News|Research

Deceptive NeRF: Introducing Pseudo Observations to fill in the Gaps

Michael Rubloff

May 30, 2023

Deceptive NeRF
Deceptive NeRF

Most people have had this happen to them; they take a NeRF, throw it into their platform of choice, and get back a model that is loaded with floaters or perhaps looks great, except for one area of completely missing data. This missing data, otherwise known as "Floaters" or "Artifacts" can often be super annoying.

This is where Deceptive NeRF comes in. They feel the pain of the average NeRF creator and the artifacts associated when there are only a few images available or when the images are sparse, meaning there are gaps in the data.

One of the benefits of Deceptive NeRF is that it can handle sparse input data and remove floating artifacts that can occur in NeRF models. By synthesizing fake but plausible observations, Deceptive NeRF can create a more complete NeRF.

The big concept from this paper is getting used to the words pseudo observation. In laymen's terms a pseudo observation is an educated guess of what was located in that viewing space to put in a photorealistic lookalike. With these pseudo observations, the research team has been able to demonstrate a noticeable difference in reconstruction quality using pseudo-observations.

Now that that's out of the way, how does it work?

Deceptive NeRF involves three main steps. First, a "coarse" NeRF model is created from the sparse input images. Coarse means the original NeRF, without any corrections is created to get a baseline.

Then, pseudo-observations are generated based on the coarse model. These pseudo-observations are synthetic images that are meant to fill in the gaps in the data and make the model more complete. Finally, the NeRF model is refined using the pseudo-observations to create a high-quality 3D reconstruction.

To generate these pseudo-observations, the researchers developed a rectification latent diffusion model. This model generates images based on a coarse RGB image and depth map, which are derived from the coarse NeRF and latent text embedding from input images. The result is a set of photo-realistic pseudo-observations that are consistent with the sparse input images.

The researchers tested Deceptive NeRF on a variety of challenging scenarios and found that it outperformed state-of-the-art methods. In particular, Deceptive NeRF was able to generate more photo-realistic novel views of objects and environments compared to other methods.

However, there are some limitations to Deceptive NeRF. Because the rectification latent diffusion model does not constrain cross-view consistency when synthesizing images, there may be inconsistencies between the pseudo-observations and the input images. However, the researchers found that these inconsistencies were automatically corrected during the training of the fine NeRF model.

Interestingly, the researchers found that by introducing too many pseudo observations, the resulting NeRF suffers from even more artifacts than originally. Similarly, too few pseudo-observations and the original problem will remain. In order to address this, they recommend using only 1.

Overall, Deceptive NeRF is a promising new method for enhancing the quality of 3D models. By generating synthetic observations, Deceptive NeRF can handle sparse input data and create more complete 3D reconstructions. While there are still some limitations to the method, it represents an important step forward in NeRFs.

While the results are undoubtable, I am not entirely sure about how I feel about introducing pseudo observations into a NeRF. The results are still photorealistic, but for me, I love how NeRFs are authentically taken from a moment in time. I think for several applications it could be helpful, in a similar vein to photo coloring or restoring, but I'm not sure I would want it to become a standard. Additionally, Deceptive NeRF runs the risk of creating something photorealistic, but not actually something that exists. The team is clearly aware of this, literally calling it Deceptive NeRF. It does however make me excited for the inevitable fully Generative NeRFs!

Featured

Featured

Featured

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Featured

Featured

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Recent articles

Recent articles