Research

GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields

Michael Rubloff

Michael Rubloff

Sep 26, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
GANeRF
GANeRF

Continuing the recent trend of papers looking to optimize neural radiance fields (NeRFs) is a team from the Technical University of Munich and Meta Reality Labs Zurich. Some of the names might be familiar, such as Matthias Nießner, who is not only the co-founder of Synthesia, but is also an author on NeRSemble and HumanRF. It's been a busy year for him!

NeRFs model precisely what they observe, which also proves to be a double edged sword for areas where nothing at all is seen. As its name implies, GANeRF takes advantage of generative adversarial networks (GAN) to improve quality of neural radiance fields (NeRF) in challenging real-world scenarios.

We have not often encountered GAN's here, so I believe it would be valuable for a quick primer. GAN's stand for Generative Adversarial Network and they are a subsect of Machine Learning. They feature two systems, a Generator and a Discriminator. These two systems work in tandem to solve a question. The generator learns to create synthetic data by trying to mimic the real data's distribution. It takes a random noise as input and generates samples that ideally are indistinguishable from real data. The discriminator, on the other hand, is trained to distinguish between the real and the generated samples. It takes in a sample and assigns a probability that the given sample is real.

During the training process, these two networks are in essence playing a game:

  1. The generator tries to produce fake data that looks as real as possible.

  2. The discriminator improves its ability to tell real from fake.

The generator's goal is to improve its ability to generate fake data, while the discriminator's goal is to get better at distinguishing real data from fake. This process continues until the generator produces high-quality data that the discriminator can hardly differentiate from the real ones, or until a specified number of training epochs is reached.

A good summary of what GANs are and do.

Understanding this gives insight into why GANs would be applied to NeRFs and how they can be beneficial.

So, how does GANeRF work?

GANeRF uses two Discriminators by learning various image patches in the training data. This representation is then trained through the adversarial model and update as it goes. What's also interesting is that the generator used takes a look at the NeRF renderings at various resolutions simultaneously, allowing for more feedback from a second discriminator. The first discriminator is trained with the adversarial objective to differentiate between real and NeRF-generated patches. Simultaneously, the NeRF is trained to generate patches that the discriminator perceives as real. The feedback from the first discriminator refines the NeRF representation by encouraging it to render more realistic patches. This is an iterative process, continuously improving NeRF’s representation of the scene.

One big feature is that they back-propagate gradients from a scene patch discriminator. This addresses the average artifacts that we are used to seeing in NeRFs. GANeRF particularly is able to see large improvements in rendering quality with areas that previously had sparse data.

By presenting the discriminator with synthetic data, it pushes the NeRF to render more realistic information, until the discriminator is convinced that what it's seeing is real.

On top of that they use a 2D adversarial loss to directly update the 3D scene representation, which improves realism in novel views. After achieving a refined NeRF representation, The conditional generator, inspired by StyleGAN2, acts as a stochastic denoiser, refining NeRF renderings further in 2D, without the mapping network. They do however utilize leaky ReLU activations, input/output skip connections and normalization layers.

The conditional generator takes the NeRF renderings and works on refining them further in 2D. It's trained concurrently with a second discriminator to refine its output, acting on discrepancies and artifacts in the NeRF renderings. Once the models are well-trained, they can synthesize novel, realistic views of the 3D scene. NeRF provides the foundational 3D rendering, the discriminator refines the 3D representation, and the conditional generator finalizes the novel view in 2D by resolving remaining imperfections.

Let's take a look at how this translates to real examples:

How cool is that! The outcome is a visible quality improvement compared to Nerfacto. There is a huge quality improvement. The authors choose to compare GANeRF to Nerfacto, but it is compatible with all existing NeRF methods, which is promising, given its success.

This is not the first application marrying NeRFs with GANs. Similar to this is the NeRF paper, NeRFLiX, which we took a look at back in May, but it inverts NeRF artifacts in an non adversarial setting.

They take a look at what happens when various steps are omitted such as the discriminator or the generator are omitted. In both findings, they show that each step has a marked increase in the sharpness and overall quality reconstruction of the NeRF.

Using GANs does come with the risk of hallucinated content, but because they back-propagate the NeRF representation and utilize a conditional generator, they're able to achieve consistent view dependent effects. One area that remains a challenge are letters and written text.

The code has not been released yet, but hopefully we will be able to try it out soon. Given that GANeRF is submitted for SIGGRAPH Asia, I expect that we will be seeing the code before the end of the year, which somehow is only three months away.

NeRF reconstruction quality continues to be an active area of research and the released methods push the envelope. Hold onto the data you've already captured; it seems like sooner than later reconstruction may be perfect.

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV