Research

GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields

Michael Rubloff

Sep 26, 2023

GANeRF
GANeRF

Continuing the recent trend of papers looking to optimize neural radiance fields (NeRFs) is a team from the Technical University of Munich and Meta Reality Labs Zurich. Some of the names might be familiar, such as Matthias Nießner, who is not only the co-founder of Synthesia, but is also an author on NeRSemble and HumanRF. It's been a busy year for him!

NeRFs model precisely what they observe, which also proves to be a double edged sword for areas where nothing at all is seen. As its name implies, GANeRF takes advantage of generative adversarial networks (GAN) to improve quality of neural radiance fields (NeRF) in challenging real-world scenarios.

We have not often encountered GAN's here, so I believe it would be valuable for a quick primer. GAN's stand for Generative Adversarial Network and they are a subsect of Machine Learning. They feature two systems, a Generator and a Discriminator. These two systems work in tandem to solve a question. The generator learns to create synthetic data by trying to mimic the real data's distribution. It takes a random noise as input and generates samples that ideally are indistinguishable from real data. The discriminator, on the other hand, is trained to distinguish between the real and the generated samples. It takes in a sample and assigns a probability that the given sample is real.

During the training process, these two networks are in essence playing a game:

  1. The generator tries to produce fake data that looks as real as possible.

  2. The discriminator improves its ability to tell real from fake.

The generator's goal is to improve its ability to generate fake data, while the discriminator's goal is to get better at distinguishing real data from fake. This process continues until the generator produces high-quality data that the discriminator can hardly differentiate from the real ones, or until a specified number of training epochs is reached.

A good summary of what GANs are and do.

Understanding this gives insight into why GANs would be applied to NeRFs and how they can be beneficial.

So, how does GANeRF work?

GANeRF uses two Discriminators by learning various image patches in the training data. This representation is then trained through the adversarial model and update as it goes. What's also interesting is that the generator used takes a look at the NeRF renderings at various resolutions simultaneously, allowing for more feedback from a second discriminator. The first discriminator is trained with the adversarial objective to differentiate between real and NeRF-generated patches. Simultaneously, the NeRF is trained to generate patches that the discriminator perceives as real. The feedback from the first discriminator refines the NeRF representation by encouraging it to render more realistic patches. This is an iterative process, continuously improving NeRF’s representation of the scene.

One big feature is that they back-propagate gradients from a scene patch discriminator. This addresses the average artifacts that we are used to seeing in NeRFs. GANeRF particularly is able to see large improvements in rendering quality with areas that previously had sparse data.

By presenting the discriminator with synthetic data, it pushes the NeRF to render more realistic information, until the discriminator is convinced that what it's seeing is real.

On top of that they use a 2D adversarial loss to directly update the 3D scene representation, which improves realism in novel views. After achieving a refined NeRF representation, The conditional generator, inspired by StyleGAN2, acts as a stochastic denoiser, refining NeRF renderings further in 2D, without the mapping network. They do however utilize leaky ReLU activations, input/output skip connections and normalization layers.

The conditional generator takes the NeRF renderings and works on refining them further in 2D. It's trained concurrently with a second discriminator to refine its output, acting on discrepancies and artifacts in the NeRF renderings. Once the models are well-trained, they can synthesize novel, realistic views of the 3D scene. NeRF provides the foundational 3D rendering, the discriminator refines the 3D representation, and the conditional generator finalizes the novel view in 2D by resolving remaining imperfections.

Let's take a look at how this translates to real examples:

How cool is that! The outcome is a visible quality improvement compared to Nerfacto. There is a huge quality improvement. The authors choose to compare GANeRF to Nerfacto, but it is compatible with all existing NeRF methods, which is promising, given its success.

This is not the first application marrying NeRFs with GANs. Similar to this is the NeRF paper, NeRFLiX, which we took a look at back in May, but it inverts NeRF artifacts in an non adversarial setting.

They take a look at what happens when various steps are omitted such as the discriminator or the generator are omitted. In both findings, they show that each step has a marked increase in the sharpness and overall quality reconstruction of the NeRF.

Using GANs does come with the risk of hallucinated content, but because they back-propagate the NeRF representation and utilize a conditional generator, they're able to achieve consistent view dependent effects. One area that remains a challenge are letters and written text.

The code has not been released yet, but hopefully we will be able to try it out soon. Given that GANeRF is submitted for SIGGRAPH Asia, I expect that we will be seeing the code before the end of the year, which somehow is only three months away.

NeRF reconstruction quality continues to be an active area of research and the released methods push the envelope. Hold onto the data you've already captured; it seems like sooner than later reconstruction may be perfect.

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Trending articles

Trending articles

Trending articles

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024