Relightable NeRFs with ReNeRF

Michael Rubloff

Michael Rubloff

Oct 6, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
ReNeRF
ReNeRF

NeRFs have revolutionized the realm of photorealism, but they've historically had a glaring limitation: the inability to modify lighting. This often means production use cases faced a substantial bottleneck. Existing methods typically baked lighting into the model, thereby eliminating the feasibility of subsequent lighting adjustments. Such constraints limited its application in diverse scenarios, especially those involving intricate lighting dynamics.

Enter ReNeRF, a promising solution straight out of the Disney vault.

Image-based relighting techniques, like One Light at a Time (OLAT), aren't new. They permit image lighting modifications, but their application is restricted to the available views. For 2D images, this limitation is palpable. Disney's innovative approach bridges this gap by applying OLAT techniques to the 2D texture plane of 3D meshes. However, challenges arise when dealing with intricate details, such as hair or fur. That's where NeRF comes into play, leveraging its volumetric density model to better capture 3D geometries.

ReNeRF

Given Disney's extensive experience in film production, their interest in this space is hardly surprising. Relighting photorealistic 3D settings could revolutionize production quality and efficiency. While ReNeRF isn't the inaugural attempt at relighting NeRFs—recall our earlier discussion on "ReLight My NeRF (René)"—it is distinct in its approach.

The uniqueness of ReNeRF lies in its ability to intricately model the intricacies of global light transport in objects without depending on intricate simulations or over-simplified assumptions about the interaction of light with various scenes. It adeptly handles continuous lighting directions, encompassing even nearfield lighting—a domain where many predecessors stumbled. It's pretty crazy, but it's also able to simulate new lighting that was not captured in the training data.

Drawing from the principles of image-based relighting (IBRL), ReNeRF elevates this concept, transitioning from a 2D to a 3D paradigm. Moreover, it doesn't mandate a dense light stage but thrives in controlled studio settings, leveraging typical photogrammetry area lights.

Watch a detailed exploration of ReNeRF.

At its core, ReNeRF builds upon the foundational principles of NeRF. It innovatively is able to convert area light sources into point light sources during training, assuring exhaustive volumetric light transport.

The architecture of ReNeRF is designed around two Multi-Layer Perceptrons (MLP), one for the NeRF MLP and one OLAT MLP. There is also a Spherical Codebook that is a compact representation of learned OLAT codes on a 3D sphere. This spherical codebook allows for neighboring areas around data, where there is no OLAT code registered, to have consistent lighting information, despite not being exposed.

In application, ReNeRF has successfully exhibited its prowess, showcasing the relighting of intricate subjects like human faces from novel perspectives. Such capabilities hold immense potential, particularly for enriching virtual domains across digital platforms.

However, every innovation comes with its set of challenges. ReNeRF's primary bottleneck lies in its extensive training and evaluation durations. Yet, the tech world's swift pace gives hope that this challenge might soon be history. ReNeRF emerges as large progress in neural relighting. It masterfully amalgamates NeRF's strengths with the intuitive strategies of image-based relighting, heralding a future replete with realistic, dynamic, and adaptable virtual realms.

Featured

Recents

Featured

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Platforms

Frame Integrates Gaussian Splatting

Powered by Babylon.js, Frame now supports 3DGS imports.

Michael Rubloff

Dec 5, 2024

Platforms

Frame Integrates Gaussian Splatting

Powered by Babylon.js, Frame now supports 3DGS imports.

Michael Rubloff

Dec 5, 2024

Platforms

Frame Integrates Gaussian Splatting

Powered by Babylon.js, Frame now supports 3DGS imports.

Michael Rubloff