ReLight My NeRF (René) explores Relightable NeRFs

Michael Rubloff

Michael Rubloff

Jun 8, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
ReLight My NeRF
ReLight My NeRF

Through the first half of this year there have been several advancements on improving NeRFs. From realtime renderings, to editable NeRFs, to reducing the memory footprint— there's been quite a lot of innovation. One area that has been relatively neglected has been the relighting of NeRFs.

This is to be expected. In NeRFs, the individual particles emit rather than reflect light, making it extremely difficult to make it suitable to be able to relight. ReNe aims to begin solving this problem with the introduction of their research, building upon a multi resolution hash grid, also known as Instant-NGP.

^The original groundtruth is on the left. The right is the resulting NeRF.

A new dataset called Relight my NeRF (ReNé) has been introduced to address the problem of rendering novel views from a Neural Radiance Field (NeRF) under unobserved light conditions. This dataset features real-world objects framed under one-light-at-a-time (OLAT) conditions, with accurate ground-truth camera and light poses. ReNé aims to provide a benchmark for relighting methods and foster research in the field of inverse rendering, because to this point there has not been an accessible dataset to benchmark off of.

ReNé introduces two robots aptly named LightBot and CameraBot, to position the light and camera independently. This ensures that the tests are repeatable and able to calibrated to a high degree of accuracy. With multiple moving pieces, calibration is critical to ensuring the validity of the tests. It starts with the camera bot generating a path and amazingly they were able to only require it to do a single calibration test before recording starts. Naturally this results with the ability to revisit and compare reference frames between different datasets. To collect the ReNé dataset, a data acquisition methodology was developed using two robotic arms, referred to as LightBot and CameraBot.

These robots positioned the camera and light source independently, ensuring repeatable and accurate camera and light poses. The acquisition framework involved calibrating the robots with a common reference frame using a ChArUco board. Trajectories were generated for both robots to capture images from different viewpoints and lighting conditions. The dataset includes 20 scenes, each consisting of 2000 images captured from 50 different viewpoints under 40 OLAT lighting conditions. The dataset aims to provide a realistic quantitative benchmark for relighting methods.

The ReNé dataset enables a study on modifying the NeRF architecture to perform relighting with reasonable computational requirements.

They developed five different versions of ReNé (V1-V5), each building upon the one before it. These range from conditioning the network with light position to considering normal vectors, were explored.

The study revealed a lightweight architecture that effectively renders novel views under novel lighting conditions, including the casting of complex shadows. The results of this study serve as a non-trivial baseline for the ReNé dataset, offering insights into the relighting potential of NeRF and opening avenues for further research.

While the dataset has some limitations, such as the absence of 360-degree scans and a challenging but unrealistic OLAT setup, it offers a solid foundation for exploring the relighting capabilities of NeRF architectures. To me, being able to accurately and efficiently relight a NeRF will unlock several more use cases and make NeRFs even more ubiquitous in real life. Now that there is a benchmark dataset available, we'll have to see how researchers build on top of and improve relighting capabilities.

Featured

Recents

Featured

Platforms

Postshot V.5 Released

The newest version of Postshot is here!

Michael Rubloff

Dec 23, 2024

Platforms

Postshot V.5 Released

The newest version of Postshot is here!

Michael Rubloff

Dec 23, 2024

Platforms

Postshot V.5 Released

The newest version of Postshot is here!

Michael Rubloff

News

Create a Hyper-Real Holiday Card

Just in time for the holidays comes a way to share hyper real holiday wishes!

Michael Rubloff

Dec 23, 2024

News

Create a Hyper-Real Holiday Card

Just in time for the holidays comes a way to share hyper real holiday wishes!

Michael Rubloff

Dec 23, 2024

News

Create a Hyper-Real Holiday Card

Just in time for the holidays comes a way to share hyper real holiday wishes!

Michael Rubloff

Platforms

GSOPs 2.0: Now Commercially Viable with Houdini Commercial License

The 2.0 release for GSOPs is here with a commercial license!

Michael Rubloff

Dec 20, 2024

Platforms

GSOPs 2.0: Now Commercially Viable with Houdini Commercial License

The 2.0 release for GSOPs is here with a commercial license!

Michael Rubloff

Dec 20, 2024

Platforms

GSOPs 2.0: Now Commercially Viable with Houdini Commercial License

The 2.0 release for GSOPs is here with a commercial license!

Michael Rubloff

Platforms

Odyssey Announces Generative World Model, Explorer

Odyssey shows off their photo real world generator, powered by Radiance Fields.

Michael Rubloff

Dec 18, 2024

Platforms

Odyssey Announces Generative World Model, Explorer

Odyssey shows off their photo real world generator, powered by Radiance Fields.

Michael Rubloff

Dec 18, 2024

Platforms

Odyssey Announces Generative World Model, Explorer

Odyssey shows off their photo real world generator, powered by Radiance Fields.

Michael Rubloff