GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields

Continuing the recent trend of papers looking to optimize neural radiance fields (NeRFs) is a team from the Technical University of Munich and Meta Reality Labs Zurich. Some of the names might be familiar, such as Matthias Nießner, who is not only the co-founder of Synthesia, but is also an author on NeRSemble and HumanRF. It's been a busy year for him!

NeRFs model precisely what they observe, which also proves to be a double edged sword for areas where nothing at all is seen. As its name implies, GANeRF takes advantage of generative adversarial networks (GAN) to improve quality of neural radiance fields (NeRF) in challenging real-world scenarios.

We have not often encountered GAN's here, so I believe it would be valuable for a quick primer. GAN's stand for Generative Adversarial Network and they are a subsect of Machine Learning. They feature two systems, a Generator and a Discriminator. These two systems work in tandem to solve a question. The generator learns to create synthetic data by trying to mimic the real data's distribution. It takes a random noise as input and generates samples that ideally are indistinguishable from real data. The discriminator, on the other hand, is trained to distinguish between the real and the generated samples. It takes in a sample and assigns a probability that the given sample is real.

During the training process, these two networks are in essence playing a game:

1. The generator tries to produce fake data that looks as real as possible.

2. The discriminator improves its ability to tell real from fake.

The generator's goal is to improve its ability to generate fake data, while the discriminator's goal is to get better at distinguishing real data from fake. This process continues until the generator produces high-quality data that the discriminator can hardly differentiate from the real ones, or until a specified number of training epochs is reached.

A good summary of what GANs are and do.

Understanding this gives insight into why GANs would be applied to NeRFs and how they can be beneficial.

So, how does GANeRF work?

GANeRF uses two Discriminators by learning various image patches in the training data. This representation is then trained through the adversarial model and update as it goes. What's also interesting is that the generator used takes a look at the NeRF renderings at various resolutions simultaneously, allowing for more feedback from a second discriminator. The first discriminator is trained with the adversarial objective to differentiate between real and NeRF-generated patches. Simultaneously, the NeRF is trained to generate patches that the discriminator perceives as real. The feedback from the first discriminator refines the NeRF representation by encouraging it to render more realistic patches. This is an iterative process, continuously improving NeRF’s representation of the scene.

One big feature is that they back-propagate gradients from a scene patch discriminator. This addresses the average artifacts that we are used to seeing in NeRFs. GANeRF particularly is able to see large improvements in rendering quality with areas that previously had sparse data.

By presenting the discriminator with synthetic data, it pushes the NeRF to render more realistic information, until the discriminator is convinced that what it's seeing is real.

On top of that they use a 2D adversarial loss to directly update the 3D scene representation, which improves realism in novel views. After achieving a refined NeRF representation, The conditional generator, inspired by StyleGAN2, acts as a stochastic denoiser, refining NeRF renderings further in 2D, without the mapping network. They do however utilize leaky ReLU activations, input/output skip connections and normalization layers.

The conditional generator takes the NeRF renderings and works on refining them further in 2D. It's trained concurrently with a second discriminator to refine its output, acting on discrepancies and artifacts in the NeRF renderings. Once the models are well-trained, they can synthesize novel, realistic views of the 3D scene. NeRF provides the foundational 3D rendering, the discriminator refines the 3D representation, and the conditional generator finalizes the novel view in 2D by resolving remaining imperfections.

Let's take a look at how this translates to real examples:

How cool is that! The outcome is a visible quality improvement compared to Nerfacto. There is a huge quality improvement. The authors choose to compare GANeRF to Nerfacto, but it is compatible with all existing NeRF methods, which is promising, given its success.

This is not the first application marrying NeRFs with GANs. Similar to this is the NeRF paper, NeRFLiX, which we took a look at back in May, but it inverts NeRF artifacts in an non adversarial setting.

They take a look at what happens when various steps are omitted such as the discriminator or the generator are omitted. In both findings, they show that each step has a marked increase in the sharpness and overall quality reconstruction of the NeRF.

Using GANs does come with the risk of hallucinated content, but because they back-propagate the NeRF representation and utilize a conditional generator, they're able to achieve consistent view dependent effects. One area that remains a challenge are letters and written text.

The code has not been released yet, but hopefully we will be able to try it out soon. Given that GANeRF is submitted for SIGGRAPH Asia, I expect that we will be seeing the code before the end of the year, which somehow is only three months away.

NeRF reconstruction quality continues to be an active area of research and the released methods push the envelope. Hold onto the data you've already captured; it seems like sooner than later reconstruction may be perfect.