News|Research

Deceptive NeRF: Introducing Pseudo Observations to fill in the Gaps

Michael Rubloff

Michael Rubloff

May 30, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Deceptive NeRF
Deceptive NeRF

Most people have had this happen to them; they take a NeRF, throw it into their platform of choice, and get back a model that is loaded with floaters or perhaps looks great, except for one area of completely missing data. This missing data, otherwise known as "Floaters" or "Artifacts" can often be super annoying.

This is where Deceptive NeRF comes in. They feel the pain of the average NeRF creator and the artifacts associated when there are only a few images available or when the images are sparse, meaning there are gaps in the data.

One of the benefits of Deceptive NeRF is that it can handle sparse input data and remove floating artifacts that can occur in NeRF models. By synthesizing fake but plausible observations, Deceptive NeRF can create a more complete NeRF.

The big concept from this paper is getting used to the words pseudo observation. In laymen's terms a pseudo observation is an educated guess of what was located in that viewing space to put in a photorealistic lookalike. With these pseudo observations, the research team has been able to demonstrate a noticeable difference in reconstruction quality using pseudo-observations.

Now that that's out of the way, how does it work?

Deceptive NeRF involves three main steps. First, a "coarse" NeRF model is created from the sparse input images. Coarse means the original NeRF, without any corrections is created to get a baseline.

Then, pseudo-observations are generated based on the coarse model. These pseudo-observations are synthetic images that are meant to fill in the gaps in the data and make the model more complete. Finally, the NeRF model is refined using the pseudo-observations to create a high-quality 3D reconstruction.

To generate these pseudo-observations, the researchers developed a rectification latent diffusion model. This model generates images based on a coarse RGB image and depth map, which are derived from the coarse NeRF and latent text embedding from input images. The result is a set of photo-realistic pseudo-observations that are consistent with the sparse input images.

The researchers tested Deceptive NeRF on a variety of challenging scenarios and found that it outperformed state-of-the-art methods. In particular, Deceptive NeRF was able to generate more photo-realistic novel views of objects and environments compared to other methods.

However, there are some limitations to Deceptive NeRF. Because the rectification latent diffusion model does not constrain cross-view consistency when synthesizing images, there may be inconsistencies between the pseudo-observations and the input images. However, the researchers found that these inconsistencies were automatically corrected during the training of the fine NeRF model.

Interestingly, the researchers found that by introducing too many pseudo observations, the resulting NeRF suffers from even more artifacts than originally. Similarly, too few pseudo-observations and the original problem will remain. In order to address this, they recommend using only 1.

Overall, Deceptive NeRF is a promising new method for enhancing the quality of 3D models. By generating synthetic observations, Deceptive NeRF can handle sparse input data and create more complete 3D reconstructions. While there are still some limitations to the method, it represents an important step forward in NeRFs.

While the results are undoubtable, I am not entirely sure about how I feel about introducing pseudo observations into a NeRF. The results are still photorealistic, but for me, I love how NeRFs are authentically taken from a moment in time. I think for several applications it could be helpful, in a similar vein to photo coloring or restoring, but I'm not sure I would want it to become a standard. Additionally, Deceptive NeRF runs the risk of creating something photorealistic, but not actually something that exists. The team is clearly aware of this, literally calling it Deceptive NeRF. It does however make me excited for the inevitable fully Generative NeRFs!

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV