ReLight My NeRF (René) explores Relightable NeRFs

Through the first half of this year there have been several advancements on improving NeRFs. From realtime renderings, to editable NeRFs, to reducing the memory footprint— there's been quite a lot of innovation. One area that has been relatively neglected has been the relighting of NeRFs.

This is to be expected. In NeRFs, the individual particles emit rather than reflect light, making it extremely difficult to make it suitable to be able to relight. ReNe aims to begin solving this problem with the introduction of their research, building upon a multi resolution hash grid, also known as Instant-NGP.

^The original groundtruth is on the left. The right is the resulting NeRF.

A new dataset called Relight my NeRF (ReNé) has been introduced to address the problem of rendering novel views from a Neural Radiance Field (NeRF) under unobserved light conditions. This dataset features real-world objects framed under one-light-at-a-time (OLAT) conditions, with accurate ground-truth camera and light poses. ReNé aims to provide a benchmark for relighting methods and foster research in the field of inverse rendering, because to this point there has not been an accessible dataset to benchmark off of.

These robots positioned the camera and light source independently, ensuring repeatable and accurate camera and light poses. The acquisition framework involved calibrating the robots with a common reference frame using a ChArUco board. Trajectories were generated for both robots to capture images from different viewpoints and lighting conditions. The dataset includes 20 scenes, each consisting of 2000 images captured from 50 different viewpoints under 40 OLAT lighting conditions. The dataset aims to provide a realistic quantitative benchmark for relighting methods.

The ReNé dataset enables a study on modifying the NeRF architecture to perform relighting with reasonable computational requirements.

They developed five different versions of ReNé (V1-V5), each building upon the one before it. These range from conditioning the network with light position to considering normal vectors, were explored.

The study revealed a lightweight architecture that effectively renders novel views under novel lighting conditions, including the casting of complex shadows. The results of this study serve as a non-trivial baseline for the ReNé dataset, offering insights into the relighting potential of NeRF and opening avenues for further research.

While the dataset has some limitations, such as the absence of 360-degree scans and a challenging but unrealistic OLAT setup, it offers a solid foundation for exploring the relighting capabilities of NeRF architectures. To me, being able to accurately and efficiently relight a NeRF will unlock several more use cases and make NeRFs even more ubiquitous in real life. Now that there is a benchmark dataset available, we'll have to see how researchers build on top of and improve relighting capabilities.