Deceptive NeRF: Introducing Pseudo Observations to fill in the Gaps

Most people have had this happen to them; they take a NeRF, throw it into their platform of choice, and get back a model that is loaded with floaters or perhaps looks great, except for one area of completely missing data. This missing data, otherwise known as "Floaters" or "Artifacts" can often be super annoying.

This is where Deceptive NeRF comes in. They feel the pain of the average NeRF creator and the artifacts associated when there are only a few images available or when the images are sparse, meaning there are gaps in the data.

One of the benefits of Deceptive NeRF is that it can handle sparse input data and remove floating artifacts that can occur in NeRF models. By synthesizing fake but plausible observations, Deceptive NeRF can create a more complete NeRF.

The big concept from this paper is getting used to the words pseudo observation. In laymen's terms a pseudo observation is an educated guess of what was located in that viewing space to put in a photorealistic lookalike. With these pseudo observations, the research team has been able to demonstrate a noticeable difference in reconstruction quality using pseudo-observations.

Now that that's out of the way, how does it work?

Deceptive NeRF involves three main steps. First, a "coarse" NeRF model is created from the sparse input images. Coarse means the original NeRF, without any corrections is created to get a baseline.

Then, pseudo-observations are generated based on the coarse model. These pseudo-observations are synthetic images that are meant to fill in the gaps in the data and make the model more complete. Finally, the NeRF model is refined using the pseudo-observations to create a high-quality 3D reconstruction.

To generate these pseudo-observations, the researchers developed a rectification latent diffusion model. This model generates images based on a coarse RGB image and depth map, which are derived from the coarse NeRF and latent text embedding from input images. The result is a set of photo-realistic pseudo-observations that are consistent with the sparse input images.

The researchers tested Deceptive NeRF on a variety of challenging scenarios and found that it outperformed state-of-the-art methods. In particular, Deceptive NeRF was able to generate more photo-realistic novel views of objects and environments compared to other methods.

However, there are some limitations to Deceptive NeRF. Because the rectification latent diffusion model does not constrain cross-view consistency when synthesizing images, there may be inconsistencies between the pseudo-observations and the input images. However, the researchers found that these inconsistencies were automatically corrected during the training of the fine NeRF model.

Interestingly, the researchers found that by introducing too many pseudo observations, the resulting NeRF suffers from even more artifacts than originally. Similarly, too few pseudo-observations and the original problem will remain. In order to address this, they recommend using only 1.

Overall, Deceptive NeRF is a promising new method for enhancing the quality of 3D models. By generating synthetic observations, Deceptive NeRF can handle sparse input data and create more complete 3D reconstructions. While there are still some limitations to the method, it represents an important step forward in NeRFs.

While the results are undoubtable, I am not entirely sure about how I feel about introducing pseudo observations into a NeRF. The results are still photorealistic, but for me, I love how NeRFs are authentically taken from a moment in time. I think for several applications it could be helpful, in a similar vein to photo coloring or restoring, but I'm not sure I would want it to become a standard. Additionally, Deceptive NeRF runs the risk of creating something photorealistic, but not actually something that exists. The team is clearly aware of this, literally calling it Deceptive NeRF. It does however make me excited for the inevitable fully Generative NeRFs!