Unlocking the Future of 3D, Neural Rendering: Luma CEO Amit Jain Discusses Photorealism, Emotion, and Accessibility on Building the Open Metaverse Podcast

Recently, Luma Co-Founder and CEO, Amit Jain sat down with Patrick Cozzi (Cesium) and Marc Petit (Epic Games) to discuss the future of 3D and neural rendering on the Building the Open Metaverse Podcast.

I highly recommend listening to the talk in its entirety and the listening (or viewing) links are at the bottom of the page. Jain shares some of his takeaways, limiting factors, while managing to hype up the state of the NeRF space and what his goals at Luma are.

On the Mission of Luma AI

When I glanced upon a NeRF for the first time, I felt a sense of amazement of feeling transported to another location, getting to experience a larger objective moment in time of someone else’s.

Jain views NeRF in a similar capacity:

“But for anything to become as big as phones, for anything to become even as big as game consoles, I believe you need that emotional pull. I believe you need that side where you're like, "Why would someone put it on?"

My answer is realistically being able to show a personal moment to a user. During the early days of photography, photographers were searching for similar answers. How do I show the average person this is something they can connect with and form that emotional bond? This led to the creation of Cabinet Cards, which allowed for larger user base for what would become modern photography. In this, I see a direct parallel from what Luma and the other NeRF platforms are creating.

P2016.101 - Unknown - [Marshall Clement playing chess with himself] - 1880s

Cabinet card photographs

Albumen silver print

By putting NeRFs in the hands of everyone with an iPhone, I believe it is probable that the emotional connect is made and the flywheel begins.

On NeRF’s Photorealism

Jain argues that through machine learning and NeRF, the result can easily be mistaken to blend into a normal video. I’ve experienced this in my own personal NeRF captures, with my dad not understanding why I was showing him a video of myself, when in fact what he was looking at was a NeRF.

To me, that truly was the highest complement I could have received as it meant I had achieved photorealism. This further reminds me of top tier gaffers in cinema — those that excel you cannot tell that it was filmed in a virtual environment or on set. NeRFs are well positioned to show a moment in time in a photorealistic way; something the internet saw yesterday with the introduction of Zip-NeRF.

How forgivable is a NeRF Capture?

Amit further touches on another important aspect NeRF — its forgiveness.

Amit’s background is in the LiDAR Scanner and what is possible when we sense the world around us. From 2020-2022, I spent day after day trying to scan humans with my iPhone LiDAR sensor. Coincidentally, that’s the project that Amit worked on at Apple and served as a base bonding point between us.

He's right; the slightest bump, jump, or speed variance would render the output unusable. NeRF allows those to not destroy the entirety of the scene and is much easier to “pause” the capture, in a meaningful way compared to LiDAR.

While I had surprisingly good results in low light with LiDAR, it can’t hold a candle (no pun intended) to NeRF:

“and you don't have to worry so much about lighting changing a little bit while you're capturing.” And in fact the ability of NeRF to understand and accurately reproduce view dependent lighting is a boon and differentiator to what makes NeRF photorealistic."

Amit Jain

Further, this is what differentiates NeRF from photogrammetry. Jain points out a relevant observation of objects falling on a spectrum between flat lighting and reflective.

“These are the two ends of the spectrum. You have something really shiny like a phone, the screen of a phone, or you have something really diffused like paper. In that spectrum, in the middle of it lies most of the objects in the world. As it turns out, though, most of the objects are a little bit closer to the shiny side. Most of the objects are a little bit reflective, and most of the objects have a little bit of shine, in our built world at least, and in the real world, too, actually.”

Only within the last year has NeRF training been expedited by a magnitude, going from days, to the approximate ~15 minutes that can be achieved by Luma, Instant NGP, and NeRFstudio.

Jain also touches on the age old maxim and how it applies to NeRF.

Garbage in; Garbage out.

This holds true for the most part right now, but as Instant NGP has demonstrated it is possible to clean up and remove floaters from the scene retroactively. How NeRF editing continues to evolve with papers such as NeRFshop and platforms such as Luma will not doubt help make scenes become even more forgiving, but I strongly believe that after a certain point, all garbage footage is unusable.

Amit does peel back the curtain slightly to discuss how it possible to model blur into the ML pipeline to artificially reduce it. How much of that tech is baked into Luma was unsurprisingly not given. Jain also speaks about NeRFs and Dreams.

Martin Nebelong has demonstrated a few times how well the two platforms can work with one another.

Jain believes that file format should be transparent. We have since seen this to be true with the introduction of Luma Fields, initially plugging into Unreal Engine, but it seems likely that Jain is thinking beyond one platform to see how he and the Luma team can empower users to bring upon the equivalent of the desktop revolution that came with word processors.

On Virtual Production and NeRFs

During the conversation, Jain reveals a somewhat shocking revelation:

“We got Unreal Engine rendering working with a new renderer in about two days.”

While there has been no mention of any of the other game engine plugins being developed by Luma, they have since included a poll on what should come next, with Blender leading the way. However, given the extreme speed in which the team was able to make Unreal Engine 5 NeRFs work, it would not surprise me to wake up one morning soon and have it be live.

“Where NeRFs are super interesting is that where the strength lies isn't photo realism; instead of doing all that work of capturing with photogrammetry, giving artists those meshes and having them produce the textures and fixing the meshes and months-long iterative cycles and a lot of processes, you can actually get to a place very soon, where you capture the scene and just move the camera in it virtually. And think about the possibilities of storytelling.”

Everyone seems to be in agreement that NeRFs are going to vastly shake up the Virtual Production industry and allow for a significantly cheaper and more efficient process, but to Jain, this benefit doesn’t end with large studios:

“To me, what is exciting is someone with an iPhone can now do this maybe 80% of the way. Maybe not make, of course, full production value, maybe not make an Avatar movie, but someone with an iPhone can make things and re-shoot. I think that is very, very exciting.”

Jain has built Luma to directly be used on an iPhone and that’s more than necessary. I firmly believe that anyone with access to a camera can build a NeRF. Because of the combination of access to cameras combined with the cloud based GPUs of Luma, it allows for a low barrier to entry for both creators and professionals. For those that don’t have an iPhone, you can upload video and photo captures via the desktop as well.

“We sit at this cusp now where it is possible to build tools that not 100,000 artists or a million artists can use, but I believe a billion people can use, where we can make 3D almost as easy to work with as photos and videos now. Photos and videos have really gotten to a place where anybody with a smartphone can at least make some level of edits, and that's kind of huge.”

As hot of a topic as artificial intelligence is currently, there is a certain barrier to entry in the form of intimidation. ChatGPT presented a way for an average user to dip their toes in and visualize what the future looks like, but in the world of content creation, 3D appears way over the average person’s head.

“That's what we want to make 3D, so people who are impressive artists today can make things that studios make, and people who can't make anything today, people who, "Oh, I want to make 3D," they go look at Blender tutorials, it's like, "I don’t want any part of this," they drop out.”

Getting past this barrier and creating an easy process with high fidelity results will empower people to move evolve past traditional photography and into the new era of content creation.

From the conversation, it is clear that Jain wants to build a company — more so a medium that connects people through emotions to feel as though they are really there with someone; to see how we as humans can connect to one another in a new and immersive way. Given the recent breakthroughs in artificial intelligence, the world seems primed to evolve forward past traditional photography and memory capture methods.

We have become complacent with obsessing over megapixels, while what we actually need is a fundamental evolution in our creation methods. I’m not advocating for a full on Harry Potter experience, but a step in the direction. I believe that Luma is headed in that direction and it represents a world that I want to live in.

Listen to the episode of Building the Open Metaverse it its entirety here: