Research

UniSDF: Duel of the Dual Radiance Fields

Michael Rubloff

Michael Rubloff

Dec 22, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
UniSDF
UniSDF

UniSDF comes on the heels on a few advancements to create better geometry out of radiance field methods, specifically NeRFs. Earlier this year, NVIDIA won one of Time Magazine's Top Invention of 2023 for its work on Neuralangelo. UniSDF kicks these advancements up a notch with their method.

There's something about watching really clean radiance fields that is just aesthetically pleasing to me. I think it's knowing that photorealistic 3D outputs of the world are already here.

UniSDF creates noticeably cleaner outputs, specifically revolving around reflections and highly specular objects, but how does it work?

UniSDF begins by understanding the shape and form of objects in the scene. UniSDF does this using a Signed Distance Field (SDF), a mathematical function that describes how far a point in space is from the nearest surface. Next, UniSDF begins to take a look at fine details– things like the textures and small nuances. They employ Instant-NGP for this reason, in addition to its fast training speed. Time and time again, we have seen researchers use NVIDIA's method as a foundation for further development.

This is where things get really fun. UniSDF uses not one, but two radiance fields as part of their reconstruction method, each serving a specific purpose. One is a Camera View radiance field and one is a Reflected View radiance field. You can now see why the full name of the method is Unified SDF.

  • Camera View Radiance Field: This field captures the scene as seen from the camera's viewpoint, focusing on non-reflective surfaces and diffuse reflections.

  • Reflected View Radiance Field: This field specializes in capturing reflections, particularly the tricky specular ones seen on shiny surfaces.

Ok, so now they have two separate radiance fields, each specializing in their own domain, but obviously we're looking to have them match up. UniSDF does this through a learned composition process. It uses a Weight MLP to determine how much each of the two radiance fields should contribute to the final view. It's critical that this combination is accurate, otherwise we'd end up with the shiniest car you've ever seen or an output that more resembles photogrammetry.

UniSDF uses a coarse-to-fine training strategy like in Neuralangelo. It starts by understanding the broader structure of the scene before gradually refining its focus to capture finer details. This approach is complemented by regularization techniques that maintain geometric consistency and smoothness.

Throughout the training process, the Weight MLP continuously refines its understanding of how reflective or non-reflective different parts of the scene are. This adaptability allows UniSDF to accurately reconstruct scenes with a complex mix of reflective and non-reflective surfaces.

UniSDF was trained using 8 NVIDIA Tesla V100's, knocking out the ability for most people to try it on their own. But wait. What's this? The author of Instant-Angelo author took to Twitter to say he has already implemented the UniSDF findings and has achieved similar results on a 3090. Very exciting, especially considering that Instant-Angelo is both open sourced and MIT Licensed.

I'll be curious to see if other research methods begin employing multiple radiance fields to specialize in a domain. I find the idea of combining these radiance fields particularly interesting because in theory you might be able to create several fields to tackle individual drawbacks. Then again, I'm no engineer and perhaps the compute would be better spent elsewhere. UniSDF makes it hard to argue though.

Featured

Featured

Featured

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Jul 3, 2024

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Jul 3, 2024

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Jul 2, 2024

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Jul 2, 2024

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

Jul 1, 2024

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

Jul 1, 2024

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Jun 28, 2024

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Jun 28, 2024

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff

Nov 28, 2023

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff

Nov 28, 2023

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff