Field meet Explicit Geometric Representations (FEGR) enables Novel View Relighting and Virtual Object Insertion

NeRFs off the bat bake in the lighting and while creators such as Jonathan Stephens have explored relighting a scene through Instant-NGP, today there is another paper that not only allows for Novel View Relighting, but also extends to Virtual Object Insertion.

NVIDIA, the University of Toronto, Vector Institute, and ETH Zurich present Fields meet Explicit Geometric Representations (FEGR). It reconstructs and decomposes scenes from captured images, enabling various applications like relighting and virtual object insertion.

FEGR combines the strengths of both NeRF and mesh-based methods to enable photorealistic relighting with specular and shadow effects.

Traditional methods face challenges in handling outdoor scenes or may have limitations in separating lighting and shadows from the scene. FEGR can effectively reconstruct scene geometry, materials, and lighting from a set of images, using a combination of neural fields and explicit mesh. This method enables photorealistic relighting and shadow effects on outdoor scenes and supports realistic virtual object insertion with accurate shadow casting. The technology is important because it can enhance the quality of various digital media and augmented reality applications, making them more immersive and visually appealing.

For those that are wondering how FEGR is pronounced, their paper states "feature".

FEGR also extends to use cases for VR and AR to create either more immersive experiences through lighting or climate changes as well as inserting 3D objects. For those looking to bring FEGR into other platforms, it is able to be exported into most modern pipelines and is exportable into gltf and USD.

FEGR, improves the realism of virtual objects placed into NeRFs. FEGR can accurately represent the location of light sources, like the sun, creating more realistic shadows and reflections on the inserted objects. A user study showed that the majority of participants found FEGR's results more realistic than other methods, highlighting its superior lighting estimation. This technology is important because it can enhance the quality of virtual object insertion in augmented reality applications and other digital media, making them more immersive and convincing.

However, the method does have limitations, such as being limited to static scenes and relying on manually designed priors. In the future, researchers hope to address these limitations by learning from more data and incorporating advances in dynamic neural fields.

We continue to see a trend of NeRFs focusing on large urban scenes, such as Grid-NeRF and Google Maps. All of these continue to show both researchers and corporations dedication to improving large scale immersive views.

I'm personally excited to see how large scale NeRFs impact the live ticketing and event industry.