RealmDreamer's Generative Scenes

Michael Rubloff

Michael Rubloff

Apr 11, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

You have to imagine that they're using some kind of radiance field method and they are. They use 3D Gaussian Splatting, but the process begins with the generation of a reference image from a text prompt, followed by the use of a monocular depth model to create an initial 3D point cloud.

More specifically, not wanting to start off on rocky beginnings, they initialize the scene using a pretrained 2D prior at a predefined pose, or in other words, a generated 2D image at a specific place within a larger (eventual) scene. In the examples shown, they are either using Stable Diffusion XL, Adobe Firefly, or DALLE-3. They then use a monocular depth estimator combination of Marigold and DepthAnything to convert it to 3D.

These two together help inform the initial point cloud. However, from this, there is a wide variety of potential outputs and scenarios that can be created. This means they also have the possibility of creating a bad initial starting point. They raise that initial floor, by outpainting the original generated image to extend the original point cloud view points. They use Stable Diffusion to continue to fill in discovered gaps. This step is critical to the success of RealmDreamer.

The final touch in RealmDreamer’s process is a finetuning phase that sharpens details and ensures the cohesiveness of the scene. This phase uses a text-to-image diffusion model personalized for the input image, ensuring that every aspect of the scene—from the textures to the lighting—matches the original textual prompt.

You might be interested to know that the inpainting and the finetuning stages are conducted within Nerfstudio. They use the original implmentation of Gaussian Splatting from Inria, but perhaps this could be officially supported within Nerfstudio in the coming months? In theory, you could generate an entire scene using RealmDreamer and then potentially build individual elements with SigNeRF, all without ever leaving the Nerfstudio ecosystem.

End to end it takes roughly 10 hours to run the entire generation, but encouragingly it can be accomplished on a single GPU with 24GB of VRAM, opening the door for the upper end of consumer GPUs. Like with all of Generative AI, it will only rapidly progress in both fidelity and speed. This seems like a good starting place for scene level generations. I also imagine that as diffusion based methods increase in fidelity and view consistency, 3D will be a direct benefactor, allowing for more and more hyper realistic outputs.

Check out the RealmDreamer's Project Page for more information and even more examples!

There's so much happening right now and while the majority of generative 3D has been on individual objects thus far, I would not discount the progress that is being made publicly through research and quietly behind closed doors.

Featured

Recents

Featured

Platforms

Tripo WebApp v2.0

Tripo is giving a refresh to their WebApp and more!

Michael Rubloff

Jan 11, 2025

Platforms

Tripo WebApp v2.0

Tripo is giving a refresh to their WebApp and more!

Michael Rubloff

Jan 11, 2025

Platforms

Tripo WebApp v2.0

Tripo is giving a refresh to their WebApp and more!

Michael Rubloff

News

Jensen Huang unveils two Radiance Fields in CES Keynote

Jensen welcomed CES attendees into NVIDIA through Radiance Fields.

Michael Rubloff

Jan 9, 2025

News

Jensen Huang unveils two Radiance Fields in CES Keynote

Jensen welcomed CES attendees into NVIDIA through Radiance Fields.

Michael Rubloff

Jan 9, 2025

News

Jensen Huang unveils two Radiance Fields in CES Keynote

Jensen welcomed CES attendees into NVIDIA through Radiance Fields.

Michael Rubloff

Platforms

StorySplat v1.3.6.1: Introducing SplatSwap and Walk Mode

A Discover page and more comes to StorySplat!

Michael Rubloff

Jan 9, 2025

Platforms

StorySplat v1.3.6.1: Introducing SplatSwap and Walk Mode

A Discover page and more comes to StorySplat!

Michael Rubloff

Jan 9, 2025

Platforms

StorySplat v1.3.6.1: Introducing SplatSwap and Walk Mode

A Discover page and more comes to StorySplat!

Michael Rubloff

News

NVIDIA Announces Foundational World Model, Cosmos

Generative “World Foundation Models” Complement NVIDIA’s Three-Computer Solution for AVs and Robots, Turning Thousands of Human-Driven Miles into Billions of Virtual Test Miles

Michael Rubloff

Jan 6, 2025

News

NVIDIA Announces Foundational World Model, Cosmos

Generative “World Foundation Models” Complement NVIDIA’s Three-Computer Solution for AVs and Robots, Turning Thousands of Human-Driven Miles into Billions of Virtual Test Miles

Michael Rubloff

Jan 6, 2025

News

NVIDIA Announces Foundational World Model, Cosmos

Generative “World Foundation Models” Complement NVIDIA’s Three-Computer Solution for AVs and Robots, Turning Thousands of Human-Driven Miles into Billions of Virtual Test Miles

Michael Rubloff