Platforms|Research

Instruct-NeRF2NeRF: Editable NeRFs from Text

Michael Rubloff

Michael Rubloff

Mar 24, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Instruct-NeRF2NeRF
Instruct-NeRF2NeRF

Just after the LERF paper was announced last week, I couldn't help but imagine how eventually the method could be paired with a in-NeRF editor. What year that would happen in, I could only imagine. As it turned out, I needed to wait less than two weeks, for with the publication of Instruct-NeRF2NeRF, several of my dreams became available.

UC Berkeley researchers have developed a groundbreaking new method for editing 3D scenes using simple text-based instructions, called Instruct-NeRF2NeRF. The technique allows users to make diverse local and global scene edits, offering a more accessible and intuitive 3D editing experience for everyone, even those without specialized training.

Traditional 3D editing tools require extensive training and expertise, but the emergence of neural 3D reconstruction techniques has made it easier to create realistic digital representations of real-world scenes. However, the tools for editing these scenes have remained underdeveloped, creating a barrier for those without specialized skills. Instruct-NeRF2NeRF aims to change that by allowing users to edit 3D scenes using simple text instructions.

In other words, Instruct-NeRF2NeRF allows for a user to edit a NeRF based solely off of a text prompt. On its own, it's an extremely powerful tool; paired with the power of LERF, it becomes a very dangerous combo.

If the model in the above video looks familiar, it's because it's Ethan Weber, one of the NeRFstudio founders and researcher at UC Berkeley.

The researchers evaluated their approach on a variety of captured NeRF scenes, demonstrating that their method can accomplish diverse contextual edits on real scenes. This includes environmental changes such as adjusting the time of day or localized changes that modify specific objects within the scene.

Instruct-NeRF2NeRF works by using an image-conditioned diffusion model, InstructPix2Pix, to iteratively edit input images while optimizing the underlying scene. This results in an optimized 3D scene that respects the edit instruction, allowing for more realistic and targeted edits than previous methods. By enabling a wide variety of edits using flexible and expressive textual instructions, the approach makes 3D scene editing more accessible and intuitive for everyday users.

The researchers evaluated their approach on a variety of captured NeRF scenes, demonstrating that their method can accomplish diverse contextual edits on real scenes. This includes environmental changes such as adjusting the time of day or localized changes that modify specific objects within the scene. This can either be applied on the global level of the scene or on an individual subject.


Using natural language instructions, even beginners can achieve high-quality results without the need for additional tools or specialized knowledge. This innovative method provides a more user-friendly interface for 3D editing and has the potential to revolutionize the way 3D scenes are created and modified.

These NeRFs were originally generated in Nerfstudio using the NeRFfacto model. Several of the Instruct-NeRF2NeRF authors are also on the NeRFstudio team including Matt Tancik and Professor Angjoo Kanazawa. The project page also states that an official NeRFstudio integration is coming. Several of the recent developments have been shown to work in NeRFstudio and with this, NeRFstudio remains the most open NeRF project for adding on new plugins and discoveries.

Instruct-NeRF2NeRF addresses some limitations of previous text-based stylization approaches, such as their inability to incorporate localized edits. The researchers were able to enable mask-free instructional edits by taking advantage of the recent instruction-based 2D image-conditioned diffusion model, resulting in a purely language-based interface that enables a wider range of intuitive and content-aware 3D editing.

This development comes as natural language is increasingly seen as the next "programming language" for specifying complex tasks, with large-language models (LLMs) like GPT and ChatGPT enabling more user-friendly interfaces through language instructions. The researchers' work is the first to demonstrate instructional editing in 3D, which is particularly significant given the difficulty of the base task.

Instruct-NeRF2NeRF could revolutionize the 3D editing field, offering a user-friendly, instruction-based interface that allows for a wide range of intuitive and content-aware edits. By enabling even novice users to create high-quality 3D scenes without the need for specialized knowledge, this groundbreaking technique has the potential to transform the way we create and interact with digital 3D content. With each passing week, I find myself needing to re calibrate my expectations for the evolution of the field. What will be announced next is anyone's guess, but I am fairly sure that the team of researchers at UC Berkeley will be involved.

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV