ReLight My NeRF (René) explores Relightable NeRFs

Michael Rubloff

Michael Rubloff

Jun 8, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
ReLight My NeRF
ReLight My NeRF

Through the first half of this year there have been several advancements on improving NeRFs. From realtime renderings, to editable NeRFs, to reducing the memory footprint— there's been quite a lot of innovation. One area that has been relatively neglected has been the relighting of NeRFs.

This is to be expected. In NeRFs, the individual particles emit rather than reflect light, making it extremely difficult to make it suitable to be able to relight. ReNe aims to begin solving this problem with the introduction of their research, building upon a multi resolution hash grid, also known as Instant-NGP.

^The original groundtruth is on the left. The right is the resulting NeRF.

A new dataset called Relight my NeRF (ReNé) has been introduced to address the problem of rendering novel views from a Neural Radiance Field (NeRF) under unobserved light conditions. This dataset features real-world objects framed under one-light-at-a-time (OLAT) conditions, with accurate ground-truth camera and light poses. ReNé aims to provide a benchmark for relighting methods and foster research in the field of inverse rendering, because to this point there has not been an accessible dataset to benchmark off of.

ReNé introduces two robots aptly named LightBot and CameraBot, to position the light and camera independently. This ensures that the tests are repeatable and able to calibrated to a high degree of accuracy. With multiple moving pieces, calibration is critical to ensuring the validity of the tests. It starts with the camera bot generating a path and amazingly they were able to only require it to do a single calibration test before recording starts. Naturally this results with the ability to revisit and compare reference frames between different datasets. To collect the ReNé dataset, a data acquisition methodology was developed using two robotic arms, referred to as LightBot and CameraBot.

These robots positioned the camera and light source independently, ensuring repeatable and accurate camera and light poses. The acquisition framework involved calibrating the robots with a common reference frame using a ChArUco board. Trajectories were generated for both robots to capture images from different viewpoints and lighting conditions. The dataset includes 20 scenes, each consisting of 2000 images captured from 50 different viewpoints under 40 OLAT lighting conditions. The dataset aims to provide a realistic quantitative benchmark for relighting methods.

The ReNé dataset enables a study on modifying the NeRF architecture to perform relighting with reasonable computational requirements.

They developed five different versions of ReNé (V1-V5), each building upon the one before it. These range from conditioning the network with light position to considering normal vectors, were explored.

The study revealed a lightweight architecture that effectively renders novel views under novel lighting conditions, including the casting of complex shadows. The results of this study serve as a non-trivial baseline for the ReNé dataset, offering insights into the relighting potential of NeRF and opening avenues for further research.

While the dataset has some limitations, such as the absence of 360-degree scans and a challenging but unrealistic OLAT setup, it offers a solid foundation for exploring the relighting capabilities of NeRF architectures. To me, being able to accurately and efficiently relight a NeRF will unlock several more use cases and make NeRFs even more ubiquitous in real life. Now that there is a benchmark dataset available, we'll have to see how researchers build on top of and improve relighting capabilities.

Featured

Recents

Featured

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Dec 10, 2024

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Dec 10, 2024

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff