Relightable NeRFs with ReNeRF

NeRFs have revolutionized the realm of photorealism, but they've historically had a glaring limitation: the inability to modify lighting. This often means production use cases faced a substantial bottleneck. Existing methods typically baked lighting into the model, thereby eliminating the feasibility of subsequent lighting adjustments. Such constraints limited its application in diverse scenarios, especially those involving intricate lighting dynamics.

Enter ReNeRF, a promising solution straight out of the Disney vault.

Image-based relighting techniques, like One Light at a Time (OLAT), aren't new. They permit image lighting modifications, but their application is restricted to the available views. For 2D images, this limitation is palpable. Disney's innovative approach bridges this gap by applying OLAT techniques to the 2D texture plane of 3D meshes. However, challenges arise when dealing with intricate details, such as hair or fur. That's where NeRF comes into play, leveraging its volumetric density model to better capture 3D geometries.

Given Disney's extensive experience in film production, their interest in this space is hardly surprising. Relighting photorealistic 3D settings could revolutionize production quality and efficiency. While ReNeRF isn't the inaugural attempt at relighting NeRFs—recall our earlier discussion on "ReLight My NeRF (René)"—it is distinct in its approach.

The uniqueness of ReNeRF lies in its ability to intricately model the intricacies of global light transport in objects without depending on intricate simulations or over-simplified assumptions about the interaction of light with various scenes. It adeptly handles continuous lighting directions, encompassing even nearfield lighting—a domain where many predecessors stumbled. It's pretty crazy, but it's also able to simulate new lighting that was not captured in the training data.

Drawing from the principles of image-based relighting (IBRL), ReNeRF elevates this concept, transitioning from a 2D to a 3D paradigm. Moreover, it doesn't mandate a dense light stage but thrives in controlled studio settings, leveraging typical photogrammetry area lights.

Watch a detailed exploration of ReNeRF.

At its core, ReNeRF builds upon the foundational principles of NeRF. It innovatively is able to convert area light sources into point light sources during training, assuring exhaustive volumetric light transport.

The architecture of ReNeRF is designed around two Multi-Layer Perceptrons (MLP), one for the NeRF MLP and one OLAT MLP. There is also a Spherical Codebook that is a compact representation of learned OLAT codes on a 3D sphere. This spherical codebook allows for neighboring areas around data, where there is no OLAT code registered, to have consistent lighting information, despite not being exposed.

In application, ReNeRF has successfully exhibited its prowess, showcasing the relighting of intricate subjects like human faces from novel perspectives. Such capabilities hold immense potential, particularly for enriching virtual domains across digital platforms.

However, every innovation comes with its set of challenges. ReNeRF's primary bottleneck lies in its extensive training and evaluation durations. Yet, the tech world's swift pace gives hope that this challenge might soon be history. ReNeRF emerges as large progress in neural relighting. It masterfully amalgamates NeRF's strengths with the intuitive strategies of image-based relighting, heralding a future replete with realistic, dynamic, and adaptable virtual realms.