Research

UniSDF: Duel of the Dual Radiance Fields

Michael Rubloff

Michael Rubloff

Dec 22, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
UniSDF
UniSDF

UniSDF comes on the heels on a few advancements to create better geometry out of radiance field methods, specifically NeRFs. Earlier this year, NVIDIA won one of Time Magazine's Top Invention of 2023 for its work on Neuralangelo. UniSDF kicks these advancements up a notch with their method.

There's something about watching really clean radiance fields that is just aesthetically pleasing to me. I think it's knowing that photorealistic 3D outputs of the world are already here.

UniSDF creates noticeably cleaner outputs, specifically revolving around reflections and highly specular objects, but how does it work?

UniSDF begins by understanding the shape and form of objects in the scene. UniSDF does this using a Signed Distance Field (SDF), a mathematical function that describes how far a point in space is from the nearest surface. Next, UniSDF begins to take a look at fine details– things like the textures and small nuances. They employ Instant-NGP for this reason, in addition to its fast training speed. Time and time again, we have seen researchers use NVIDIA's method as a foundation for further development.

This is where things get really fun. UniSDF uses not one, but two radiance fields as part of their reconstruction method, each serving a specific purpose. One is a Camera View radiance field and one is a Reflected View radiance field. You can now see why the full name of the method is Unified SDF.

  • Camera View Radiance Field: This field captures the scene as seen from the camera's viewpoint, focusing on non-reflective surfaces and diffuse reflections.

  • Reflected View Radiance Field: This field specializes in capturing reflections, particularly the tricky specular ones seen on shiny surfaces.

Ok, so now they have two separate radiance fields, each specializing in their own domain, but obviously we're looking to have them match up. UniSDF does this through a learned composition process. It uses a Weight MLP to determine how much each of the two radiance fields should contribute to the final view. It's critical that this combination is accurate, otherwise we'd end up with the shiniest car you've ever seen or an output that more resembles photogrammetry.

UniSDF uses a coarse-to-fine training strategy like in Neuralangelo. It starts by understanding the broader structure of the scene before gradually refining its focus to capture finer details. This approach is complemented by regularization techniques that maintain geometric consistency and smoothness.

Throughout the training process, the Weight MLP continuously refines its understanding of how reflective or non-reflective different parts of the scene are. This adaptability allows UniSDF to accurately reconstruct scenes with a complex mix of reflective and non-reflective surfaces.

UniSDF was trained using 8 NVIDIA Tesla V100's, knocking out the ability for most people to try it on their own. But wait. What's this? The author of Instant-Angelo author took to Twitter to say he has already implemented the UniSDF findings and has achieved similar results on a 3090. Very exciting, especially considering that Instant-Angelo is both open sourced and MIT Licensed.

I'll be curious to see if other research methods begin employing multiple radiance fields to specialize in a domain. I find the idea of combining these radiance fields particularly interesting because in theory you might be able to create several fields to tackle individual drawbacks. Then again, I'm no engineer and perhaps the compute would be better spent elsewhere. UniSDF makes it hard to argue though.

Featured

Featured

Featured

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Jul 26, 2024

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Jul 26, 2024

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Jul 24, 2024

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Jul 24, 2024

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Jul 22, 2024

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Jul 22, 2024

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Jul 18, 2024

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Jul 18, 2024

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff