Research

InpaintNeRF-360 Introduces 360 Degree Editable NeRFs!

Michael Rubloff

Michael Rubloff

May 31, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
InpaintNeRF360
InpaintNeRF360

When InstructNeRF2NeRF first came out, I knew that it was start of something incredible. I was so excited to see more and more papers with similar functionality. Today we get a new example of that with InpaintNeRF-360.

In a nutshell, InpaintNeRF-360 allows for the removal or editing of objects within a unbounded 360 degree NeRF. It uses a promptable segmentation model to allow a user to specify a part of the NeRF that they would like to edit and ensures that the NeRF continues to look photorealistic after its been edited by applying depth-space warping and perceptual priors.

One large benefit of InpaintNeRF-360 is that it's able to tackle the full 360 of an object that a user wants to edit, rather than say just one side of it. It's able to do this through leveraging the Segment Anything Model (SAM) to get a sense of what the specific object is. While this sounds totally plausible, it gets a little trickier when you expand the request to the full shape of the object, so they use a pre trained image inpainter across each of the images that contain a different view of the object. The output is then fine tuned to ensure view consistency even across crazy camera angles and viewing perspectives.

InpaintNeRF360 encodes semantics into image space to allow for accurate object appearance editing.

Paper

NeRFs have proven to be excellent at showcasing photorealistic scenes; however, up to now, there have only been a small collection of papers such as NeRFShop and InstructNeRF2NeRF that allowed for removing or changing objects while preserving geometric and photometric consistency. As of publishing, this is the only method that is able to acheive 360 degree inpainting for a NeRF.

As part of their method, there are three primary steps to InpaintNeRF360. The first one is encoding the text descriptions directly into the NeRF through a model Grounded Language-Image Pre-training (GLIP). GLIP is able to provide segmentation to identifiable objects, but it's not powerful by itself to do what InpaintNeRF360 needs. This is where the second step comes in, utilizing depth information. Bounding boxes tend to be inaccurate when you have an unconstrained input cameras, so feeding the depth information is critical.

Without the depth in the first two images, the view angle can produce information that is not accurate. However, when that depth information represented by the red dots in the third image are included, we see the end result. After these first two steps have been refined, it's finally time to use the SAM model to accurately fill out the NeRF. You might be wondering how while it clearly is able to separate the object, what happens to the background information? To answer that, they apply dilation on each segmentation mask so that contextual background information is included! Finally once the masks are obtained, the inpainting begins.

Again, I think this will be very powerful when combined with LERF. As time goes on, I think the applications that fit in with LERF will only become more powerful and expand potential use cases. nerfstudio co-founder and original NeRF co-author, Matt Tancik, spoke a bit about how utilizing natural language processing in conjunction with NeRFs at his talk at MIT last month. We see a further application and doors opened through InpaintNeRF360. Interestingly, InpaintNeRF360 is built on Nerfstudio and hopefully will mean we get to see it hosted for general consumption soon.

InpaintNeRF-360 is an amazing step forwards for editable NeRFs and I'm excited to give it a try as soon as it's available!

Featured

Featured

Featured

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV