Research

Bilateral Guided Radiance Field Processing

Michael Rubloff

Michael Rubloff

Jun 5, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Bilateral Guided Radiance Field Processing
Bilateral Guided Radiance Field Processing

Have you ever captured a dataset only to realize that your camera was set to manual mode, causing exposure to fluctuate? These inconsistencies can break the multi-view consistency crucial for Neural Radiance Fields (NeRF), resulting in degraded or failed reconstructions.

In my consulting work, a common question I encounter is about the ability to recolor or retouch images before processing the radiance field. Both of these issues are solved with Bilateral Guided Radiance Field Processing.

Not only can it handle fluctuating exposure levels, but you only need to edit a single image, and the method will propagate the style to the entire reconstruction. They also can handle HDR through their own pipeline that resembles the HDR+ algorithm, first merging a burst of frames and then applying local tone mapping for finishing.  

The first stage involves disentangling image signal processing (ISP) enhancements. The researchers introduce a differentiable 3D bilateral grid that acts as a locally affine model, capable of approximating the complex, non-linear operations performed by ISPs on each input view. This grid captures and models view-dependent ISP enhancements, such as exposure adjustments and color corrections, by optimizing the NeRF along with these grids.

By separating the ISP processing effects from NeRF training, the reconstructed radiance fields maintain multi-view consistency and are largely free from artifacts like floaters. This method handles featureless spaces, such as walls, quite adeptly. The bilateral grids are optimized during the training process to closely match the ISP effects for each view, effectively learning the unique enhancements applied by the camera.

Once the NeRF training with disentangled ISP effects is complete, the finishing stage reapplying and enhancing these adjustments consistently across the entire 3D scene. Although it sounds like something from a Mortal Kombat game, the finishing stage is crucial.

The researchers propose a novel low-rank 4D bilateral grid to lift 2D view edits to 3D space. This grid allows users to perform detailed retouching on a single view and have those adjustments propagated consistently across the entire 3D scene. The low-rank approximation ensures the grid remains computationally feasible while effectively capturing the essential transformations needed for consistent 3D editing. Additionally they found low rank is a good prior for high-dimensional tensor completion. This idea is borrowed from the "low-rank matrix completion problem".

Users can apply various enhancements, such as color correction and brightness adjustment, to a selected view using familiar 2D photo editing tools, such as Lightroom. Adobe's recent efforts to build out their 3D reconstruction pipeline make it easy to imagine a future where Lightroom and other Adobe products extend to hyper-real 3D!

This method lifts these 2D edits to the 3D radiance fields, ensuring that adjustments are uniformly applied across all views, including full 360-degree captures. Additionally, users can continue to make edits in 2D, which can then be applied to the NeRF without needing to retrain. Keep in mind that you will need to rerun the finishing stage. For instance, you can adjust the lighting of a scene or simply recolor an object. This fine-grain control using common photo editors is a significant advantage.

The method is built on top of a PyTorch implementation of Zip-NeRF and can run on a single 3090 GPU. The process takes about two and a quarter hours, though this duration is directly tied to the NeRF method used. Adapting it to Instant-NGP or Nerfacto could result in significant speed boosts. The finishing stage operates much faster, currently taking about 15 minutes to complete.

With the code already released under an Apache 2.0 License, allowing for a multitude of use cases, it won't be long before you can revisit your HDR captures that never worked before. NeRFs continue to evolve with no known ceiling, making the future bright for radiance field technologies.

For more information, check out their project page or catch them at SIGGRAPH in Denver this July, where they will be presenting.

Featured

Featured

Featured

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Jun 18, 2024

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Jun 18, 2024

Platforms

Varjo Unveils Teleport

Finnish VR company Varjo has announced 3DGS platform, Teleport.

Michael Rubloff

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Jun 17, 2024

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Jun 17, 2024

Platforms

Wayve Announces PRISM-1

Wayve, has announced the launch of PRISM-1, an innovative 4D reconstruction model designed to significantly enhance the testing and training of advanced driver assistance systems and autonomous driving technology.

Michael Rubloff

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Jun 14, 2024

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Jun 14, 2024

Platforms

Nerfstudio Gradio UI Added

Nerfstudio now includes a Gradio UI, making it easy and intuitive to set commands and run scripts.

Michael Rubloff

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Jun 13, 2024

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Jun 13, 2024

Platforms

Graswald.ai: 3DGS and Ecommerce Platform

Graswald is a 3D product visualization platform using Gaussian Splatting, aimed primarily at the Ecommerce industry to offer hyper realistic detail at a fraction of current time and cost. 

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Featured

Featured

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Nerfstudio Releases gsplat 1.0

Michael Rubloff

Jun 7, 2024

Nerfstudio

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio