Field meet Explicit Geometric Representations (FEGR) enables Novel View Relighting and Virtual Object Insertion

Michael Rubloff

Michael Rubloff

Apr 7, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
FEGR
FEGR

NeRFs off the bat bake in the lighting and while creators such as Jonathan Stephens have explored relighting a scene through Instant-NGP, today there is another paper that not only allows for Novel View Relighting, but also extends to Virtual Object Insertion.

NVIDIA, the University of Toronto, Vector Institute, and ETH Zurich present Fields meet Explicit Geometric Representations (FEGR). It reconstructs and decomposes scenes from captured images, enabling various applications like relighting and virtual object insertion.

FEGR combines the strengths of both NeRF and mesh-based methods to enable photorealistic relighting with specular and shadow effects.

Traditional methods face challenges in handling outdoor scenes or may have limitations in separating lighting and shadows from the scene. FEGR can effectively reconstruct scene geometry, materials, and lighting from a set of images, using a combination of neural fields and explicit mesh. This method enables photorealistic relighting and shadow effects on outdoor scenes and supports realistic virtual object insertion with accurate shadow casting. The technology is important because it can enhance the quality of various digital media and augmented reality applications, making them more immersive and visually appealing.

For those that are wondering how FEGR is pronounced, their paper states "feature".

FEGR also extends to use cases for VR and AR to create either more immersive experiences through lighting or climate changes as well as inserting 3D objects. For those looking to bring FEGR into other platforms, it is able to be exported into most modern pipelines and is exportable into gltf and USD.

FEGR, improves the realism of virtual objects placed into NeRFs. FEGR can accurately represent the location of light sources, like the sun, creating more realistic shadows and reflections on the inserted objects. A user study showed that the majority of participants found FEGR's results more realistic than other methods, highlighting its superior lighting estimation. This technology is important because it can enhance the quality of virtual object insertion in augmented reality applications and other digital media, making them more immersive and convincing.

While the virtual object insertion has almost infinite use cases, it is immediately obvious how much it impacts the autonomous driving industry. By allowing for a variety of scenarios and conditions, the simulations become all the more accurate and hopefully safer for humans.

However, the method does have limitations, such as being limited to static scenes and relying on manually designed priors. In the future, researchers hope to address these limitations by learning from more data and incorporating advances in dynamic neural fields.

We continue to see a trend of NeRFs focusing on large urban scenes, such as Grid-NeRF and Google Maps. All of these continue to show both researchers and corporations dedication to improving large scale immersive views.

I'm personally excited to see how large scale NeRFs impact the live ticketing and event industry.


Featured

Recents

Featured

Trending articles

Trending articles

Trending articles

Platforms

Scaniverse 4 Announced

Gaussian Splatting is front and center in the newest version of Niantic owned Scaniverse.

Michael Rubloff

Aug 26, 2024

Platforms

Scaniverse 4 Announced

Gaussian Splatting is front and center in the newest version of Niantic owned Scaniverse.

Michael Rubloff

Aug 26, 2024

Platforms

Scaniverse 4 Announced

Gaussian Splatting is front and center in the newest version of Niantic owned Scaniverse.

Michael Rubloff

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff