Today at CES, NVIDIA announced a major addition to its three-computer solution for autonomous mobility with the introduction of NVIDIA Cosmos. This new platform comprises state of the art world foundation models (WFMs), advanced tokenizers, guardrails, and an accelerated video processing pipeline to dramatically speed the development and validation of AI systems in vehicles and robots.
What Are World Models?
World Models ingest data of all types, including images, videos, text, and movement traces to simulate and generate virtual environments that accurately capture a scene and its spatial relationships. Cosmos WFMs are purpose-built for physical AI research and development, focusing on physics-based simulation and synthetic data generation.
Building on NVIDIA’s existing three-computer approach, DGX™ systems in the data center for AI training, Omniverse™ on OVX™ for simulation and synthetic data, and the AGX™ in-vehicle computer for real-time sensor processing, Cosmos provides developers with an unprecedented “data factory” for generating near-infinite virtual driving or robotics scenarios.
“The AV data factory flywheel consists of fleet data collection, accurate 4D reconstruction and AI to generate scenes and traffic variations for training and closed-loop evaluation,” said Sanja Fidler, vice president of AI research at NVIDIA. “Using the NVIDIA Omniverse platform, as well as Cosmos and supporting AI models, developers can generate synthetic driving scenarios to amplify training data by orders of magnitude.”
Accelerating Physical AI for the Real World
Physical AI development is notoriously expensive and time-intensive, often requiring massive real-world datasets to fine-tune AI models. By introducing generative AI into the data pipeline, Cosmos helps developers automate much of the data collection and curation process. enhancing both the quantity and quality of training data.
“Developing physical AI models has traditionally been resource-intensive and costly,” said Norm Marks, vice president of automotive at NVIDIA. “Cosmos accelerates this process with generative AI, enabling smarter, faster, and more precise AI model development for autonomous vehicles and robotics.”
Among the first adopters of Cosmos for autonomous mobility are Waabi, Wayve, Foretellix, and Uber—each leveraging NVIDIA’s generative platform for various AV development goals. According to Dara Khosrowshahi, CEO of Uber, “Generative AI will power the future of mobility, requiring both rich data and very powerful compute. By working with NVIDIA, we are confident that we can help supercharge the timeline for safe and scalable autonomous driving solutions for the industry.”
Of this grouping, I am most intrigued by Wayve’s experimentation with Cosmos to search for long tail scenarios. Wayve famously utilizes multiple forms of radiance fields, in both the most recent model, PRISM-1 and Ghost Gym. I spoke to Wayve’s Chief Scientist, Jamie Shotton, in 2024 about their approach to self driving and use of Radiance Fields. It’s exciting to think about the outputs of Cosmos being leveraged in the creation of simulation ready Radiance Fields.
At the Intersection of AI, Simulation, and Real-World Testing
Waabi, an AI pioneer for self-driving technologies, will use Cosmos to search and curate real-world video data for simulation, accelerating AV software development.
Wayve is evaluating Cosmos to identify “edge” and “corner-case” scenarios critical for safe autonomous driving and model validation.
Foretellix, a provider of AV testing and validation tools, is integrating Cosmos and NVIDIA Omniverse Sensor RTX APIs to create high-fidelity testing scenarios at scale.
Uber’s massive driving datasets, combined with Cosmos and NVIDIA DGX Cloud, aim to help AV developers reinforce their AI models with even more robust, diverse, and efficient training data.
Open Access, Responsible AI
NVIDIA is opening Cosmos WFMs under an open model license on both Hugging Face and the NVIDIA NGC™ catalog, encouraging broader experimentation and fine-tuning by developers. Safety and accountability remain top of mind: Cosmos uses guardrails, watermarking, and other measures aligned with global AI safety guidelines.
Availability
Cosmos WFMs are now accessible under NVIDIA’s open model license on Hugging Face and NGC.
NVIDIA NeMo™ Curator offers an accelerated data processing pipeline for building world models more efficiently.
NVIDIA DGX™ Cloud provides enterprises with a quick, streamlined way to train and deploy these new models.
NVIDIA Cosmos was developed by NVIDIA Research. Read the research paper, “Cosmos World Foundation Model Platform for Physical AI,” for more details on model development and benchmarks. Model cards providing additional information are available on Hugging Face.
Learn more about world foundation models in an AI Podcast episode, airing Jan. 7, that features Ming-Yu Liu, Vice President of research at NVIDIA.
New to Gaussian Splatting? Start here
