Today’s cars don’t just help us park or warn us about lane departures. They can negotiate city traffic, make automated lane changes on freeways, and even take control in certain conditions. With these advancements, we need the ability to run thorough testing. Real world driving data only goes so far when your goal is to deliver safe and reliable autonomy at scale.
This is something we have repeatedly seen and is also a driving force behind NVIDIA’s recently announced World Foundation Model, Cosmos. Applied Intuition’s Neural Sim brings to the table a simulator powered by the Radiance Field representation, Gaussian Splatting. Their implementation further incorporates physically based rendering (PBR) for moving 3D assets, so that it blends with the captured world. This combination of real life captures with the infinite possibilities and efficiency of synthetic simulation to enable accurate testing, while skirting hallucinations.
Before diving into Neural Sim, it’s worth understanding why a new approach is needed in the first place. OEMs are developing next generation ADAS features that don’t just assist the driver, but can actually replace the driver under specific circumstances. Legally and ethically, this shifts a huge responsibility from the human behind the wheel to the manufacturer. These systems also operate in increasingly diverse operational design domains, from fast moving highways to crowded city streets.
Under the hood, these new Advanced driver-assistance systems (ADAS) and Autonomous Driving (AD) solutions rely on machine learning for core functions such as perception and decision making. This shift demands massive volumes of data to train and validate these algorithms, making it no longer practical to rely on real world driving alone. Moreover, purely synthetic test environments can be time consuming to create, especially when photorealism and detailed physics are required.
Neural Sim sits at the intersection of realism and flexibility, reconstructing large scale environments from raw drive data. This means you get authentic scenes complete with real lighting, road textures, and traffic patterns. Because Neural Sim supports cameras, lidar, radar, and more, teams can run robust, closed loop tests where the system under test continuously adapts to new situations.
The world of large scale simulation is of great interest for the Radiance Field representations, underscoring a broader shift in how we approach virtual worlds. As simulation continues to blur the line between real and digital, we might see even more synergy between Radiance Fields and automotive simulation. Some ways we might see this happening will be how larger radiance field reconstructions will scale, in both size as well as compute costs. Simultaneously a large volume of research is pouring into making better lighting estimations, creating more powerful novel view synthesis with closely posed cameras, and the removal of dynamic objects in reconstructions.
Learn more about Applied Intuition and Neural Sim on their website.
