This conversation was recorded live at NVIDIA's GTC conference.

At this year’s NVIDIA GTC conference, I had the chance to sit down with Rev Lebaredian, Vice President of Omniverse and simulation technology at NVIDIA. With more than three decades of experience in graphics and real-time systems, Rev has helped shape the tools behind everything from multi-GPU rendering to physics-based simulation platforms. Our conversation covered why simulation is key to advancing physical AI, how radiance fields and volumetric representations are changing what’s possible, and what the near future of robotics and imaging might actually look like. Not in sci-fi, but in factories, warehouses, and creative workflows today.

Michael:
Thanks so much for speaking with me today. I really appreciate it. With the rapid progress in large models and generative video systems, one of the things I’ve been thinking about is: Why is simulation so important in the context of artificial intelligence?

Rev:
There’s a core problem in modern AI: the intelligence we create is entirely dependent on data. What the last few years have shown is that when you gather data at large scale, you can extract tremendous insights and capabilities. That’s essentially what LLMs have done. With massive language datasets, we uncovered powerful patterns and rules transforming how machines understand language.

But the next era of AI is about physical intelligence, robots and systems that understand and operate in the physical world. To make that possible, we need data about the world itself. And we can’t afford to be bottlenecked by how much real-world data we can collect. Unfortunately, that kind of physical data isn’t as readily available as text was for LLMs.

Even in the realm of language, people are now saying we’ve mined most of the internet—books, websites, everything. It’s finite. So to keep AI moving forward, we need a renewable source of data. That’s where simulation comes in. If you can replicate the rules of the world inside a computer, you’ve created a renewable data engine. A way to generate physical-world data that can feed into training AI.

Michael:
Right, that makes a lot of sense. And as we bring in more information about the physical world, we can improve our simulations. I wanted to follow up on that. There’s no doubt that more diverse and higher-quality input improves the outcome. But how do you think about building ingestion engines that can handle all these different kinds of data? Language is relatively simple to process, but for the physical world, there are so many modalities to account for.

Rev:
That’s a great question, and it really comes down to modality. What we call “multimodal” systems. LLMs started with just text, but now they’re incorporating images, video, and other inputs. In the physical world, we can capture data from light waves (images and video), sound (audio), and other parts of the electromagnetic spectrum, like LiDAR and radar.

We can also represent that data in many forms: pixel arrays (images), point clouds, meshes, and volumes. All of these should be inputs into the world foundation models we’re building. The challenge is that some of these formats, like meshes, are hard to work with in neural networks. Neural nets like data in regular arrays of numbers, but meshes are irregular. They have varying numbers of vertices and complex topologies.

To work around that, we can convert meshes into more regular forms, like voxels or other grid-based data structures. In fact, we’ve developed a novel way to represent large 3D worlds as sparse volumes and voxels. It’s compact, handles empty space efficiently, and is compatible with neural networks for training and inference.

Michael

It sounds like a powerful ingestion engine. One that can centralize input from very different modalities and convert it into a common structure for training. That’s incredibly useful, especially for understanding the physical world in more detail.

Something else I wanted to ask you about is lifelike 3D reconstruction. Robotics clearly depends on a solid understanding of the world around it. How do you see these lifelike reconstruction capabilities helping robotics systems, either as a framework or a starting point?

Rev:
I’d say it’s more than just a starting point. Even after deployment, these systems will need to continually ingest the real world to operate effectively.

Take, for example, a factory that already exists, a brownfield factory. You want to bring robots into that environment to automate processes. You’re investing a lot of money and want to be confident they’ll work efficiently. But as the factory evolves, new products, new workflows, you’ll need to change how it’s set up. The only way to do that intelligently is through simulation.

But to simulate, you need a current representation of your factory. That means you need the ability to ingest the physical world and bring it into your simulation environment. This isn’t just for building the initial system, it’s an ongoing requirement. The simulator has to reflect real-world changes to remain useful.

Michael:
That makes perfect sense. Let’s say you’re in an automotive factory, where you have highly skilled technical workers and also roles that might be better handled by robots. Over the next three to five years, how do you envision humans and robots working together?

Rev

That's a really, really great question. I think to start off with over the next few years, what we're going to see is these robots are going to take over the jobs that no workers want to do anymore. Every single CEO I talk to who does manufacturing at scale, they're all talking about this looming crisis. 

But I think what's going to end up happening over time is we're going to see this abundance of the equivalent of human labor. If you could imagine if we had unlimited human labor to produce things and at essentially no cost or very low cost, what are all the things we can produce? So we're going to start figuring out what are all those things we can produce and the humans that are going to be involved here, they're going to be the ones deciding what it is we're going to produce, how we should do it, in what order, and why. And less and less will have humans doing the mundane repetitive tasks.

Michael:

Right and you could run simulations to test all those decisions before implementing them, allowing humans to focus more on planning and oversight.

Rev:
Exactly. A person could say, “I want to reconfigure my factory to produce this product at this volume, within this timeframe and budget.” Then the simulator can test hundreds, or thousands, or even millions of configurations to find the optimal one. It becomes a tool for better decision-making.

Michael:
That brings up a related point. Last year I spoke with Jamie Shotton from Wayve, and he brought up the challenge of knowing when a simulation is "good enough" when you’ve covered enough scenarios to trust it. With so many long-tail possibilities, how do you think about measuring simulation coverage?

Rev:

Well, I guess there are two ways to answer that. It’s really: simulation relative to what? Because the only way to do it without simulation is to try to gather information from the real world. And then the question becomes when are you confident that you've gathered enough data?

Take autonomous vehicles, for example. You send out your fleet of cars, collect data from all the sensors, bring it back, train your models, run your tests, but you still don’t know what long-tail scenarios you haven’t seen. Sometimes, you just have unknown unknowns. You don’t know what you don’t know.

Then something happens out in the world, a rare failure case. A black swan event. And suddenly you're thinking, “Wow, we weren’t prepared for that.” It might be extremely rare, but when it does happen, the consequences can be tragic. So once we know that kind of event exists, simulation becomes the best tool we have. We can recreate that exact scenario in a simulator and make sure our systems get the coverage they need moving forward.

Beyond that, there's a lot of actual science and math around testing high-dimensional systems like this. A good analogy is semiconductor testing. A computer chip is incredibly complex, the space of all the possible things it might do, all the different input/output combinations, is enormous. How do you test a chip when it comes off the line to make sure it’s going to work?

You have to probe it in all the right ways. And there's a whole field of testing science around that.

We work with a company called Foretellix, which actually came from that chip verification space. They’ve developed a formal mathematical language to describe testing scenarios and measure how much coverage you're getting. One of the reasons we’re working with them is because our simulator is integrated into their platform. So when they identify scenarios that aren't being covered well, we can use simulation to fill in those gaps and expand the coverage.

Michael:

I want to ask a more high-level question now. Have we already passed a technological milestone you didn’t think you’d see in your career? And if we haven’t yet, what would really blow your mind?

Rev:
Oh, I think we passed that a long time ago. For me, it was AlexNet, the deep learning model that kicked off all of this in 2012. That was the moment. It solved image classification. And I didn’t think that would happen in my lifetime. I think most people in computer science, most software engineers, felt the same way. It was one of those problems we just assumed would remain unsolved.

And then it was solved. Just like that. Pretty crazy.

Very quickly after that, all these other things that used to be hard, audio, speech recognition, voice synthesis, they started getting good too. And that was even before the LLM and transformer wave. So even pre-LLM, the trajectory was already surprising.

Once we saw that, we knew robotics would eventually be possible. That general robotic intelligence could happen. And the question became: what do we need to build to be ready for that? For me, simulation was the obvious answer. That’s where I decided to focus.

Since then, we’ve believed we’re on the path to solving this. But part of me still wondered, when exactly is this going to happen? Will it be soon enough?

And honestly, I’m surprised to see what’s happening right now at GTC. I mean, look at the trajectory we’re on for humanoid robotics. We expected this, but it’s still wild to see it unfolding so quickly. These robots are getting deployed into real factories, warehouses, and industrial environments. They’re doing actual jobs.

It’s going to move fast from here. I think we’re going to see a lot of them, and they’re going to have a real impact on the economy.

Michael:
It’s unbelievable and I think that’s what I’ve been trying to push toward my audience too. We really need to rethink what we believe is possible technologically.

Rev:
It’s hard. Even for me and I’ve been immersed in this for years. It’s still hard to internalize just how fast things are moving. I can’t imagine what it’s like for people who haven’t been following along.

Humanoid robotics has been in the zeitgeist forever. Sci-fi, movies—we’ve all seen it. But in the real world, it’s always felt like some kind of hobbyist side project. You know, interesting demos, but not serious.

But now it’s real. It’s happening. And I think we all need to take a moment and recognize that.

Michael:
That’s basically the thesis of my website. I focus a lot on radiance field representations, and I genuinely believe we’re now able to move imaging out of 2D and into highly lifelike 3D. I’m still trying to get people to see it though.

Rev:
Yeah. That quote from William Gibson comes to mind: “The future is already here, it’s just not evenly distributed.”

Michael:
Exactly. The crazy thing is, with radiance fields, all you need is a camera. A single still image can be enough to reconstruct the world around you. We already have the inputs, we just need to better communicate what’s possible.

So building off that, I wanted to ask: how do you see imaging evolving over the next 5 to 10 years? Whether it’s in industrial settings or just everyday life, what changes do you see in how we create or interact with visual content?

Rev:
It’s an interesting question, especially if you think about what an image even is. Philosophically.

Is it the light field hitting your sensor? Is it the pixel array that gets recorded? Or is it the experience happening in your mind, what you see and understand? I think it’s that last one that really matters.

When I look at you right now, I have an image of you in my mind. But the inputs to that image aren’t just light waves hitting my retina. It’s also my memory. My prior knowledge about faces, about people, about rooms and environments. All of that combines to form the image I experience.

So the “image,” in a meaningful sense, is a combination of sensory input and prior knowledge.

And I think our imaging technology is heading in the same direction. We’re moving from raw data, pixels, point clouds, meshes, into deeper, more abstract representations. Neural embeddings. Shared representations that combine all modalities and encode the actual structure of the world.

Once you have that kind of internal representation, you can regenerate whatever form you want—a rendered image, a point cloud, a mesh, a full 3D reconstruction.

That’s where imaging is heading. It’s not about capturing more pixels. It’s about building a complete, internal understanding of the world. Something you can then use to generate whatever output you need.

Michael:
Yes, and that’s why I’m so excited about models like Cosmos, and papers like Gen-3C. They’re pulling in priors from diffusion models or other sources, and they’re able to generate incredibly strong reconstructions from very limited input.

Rev:
Yeah, and it’s intuitive. It’s what humans do. You can give a good artist a rough sketch or low-res image and ask them to paint a detailed version. They can fill in what’s missing because they’ve seen it before. They have priors about the world and can extrapolate.

That’s the same thing these models are starting to do.

Michael:
Last question. If you could solve just one problem in simulation right now, snap your fingers and it’s done—what would it be?

Rev:
Hmm… that’s a good one. And a little personal.

If I could instantly solve something, it would be the automatic construction of a fully rich 3D representation of the world from raw sensor data.

We’ve come a long way, NeRFs, Gaussian Splatting, and so on, but those are still limited. They’re more like images than the kind of deep, richly detailed simulations we need. In a game engine or simulator, you need not just surfaces, but volumes, material properties, physics, interactions.

You need the whole thing.

Right now, we can’t go from an image directly to that level of representation. But if we could, if you could just take a picture and immediately have a fully usable, simulation-ready 3D scene? That would be huge.

Michael:
I’m excited for that future too. And I don’t think we’re far off.

Rev:
Yeah, I don’t either. It’s happening quickly. We’re on an exponential curve, and that makes it hard to predict, whether it’s going to take one year or five.

Michael:
What I learned from last year’s GTC is: don’t underestimate the pace of progress.

Rev:
Exactly. This is the place where you can feel it happening.

—————————————————————————————————————————————————

Learn more:

Learn more about Physical AI: https://nvda.ws/42dWo3V 

Get started with Physical AI: https://nvda.ws/4czxu23

Learn about Cosmos: https://nvda.ws/42dZC7A

Thank you to the NVIDIA team for facilitating this interview. If you enjoyed this, you may also enjoy my conversation with Head of AI Research at NVIDIA, Sanja Fidler.