Editable NeRFs introduced with NeRFshop

In an area that hasn't had much innovation introduced, NeRFshop announced slew of amazing features allowing editable NeRFs. This extends to rotating, extending, moving, and deleting either individual or multiple objects within a NeRF.

As mentioned in the paper, current editing methods have been limited in their interactivity and abilities. In the demo video, NeRFshop is shown functioning directly within Instant-NGP.

What is becoming an exciting trend, this does not require tremendous GPU power, allowing for a wider base of NeRF users to access. Additionally, NeRFshop supports editing multiple objects in a single NeRF, so creators will not need to choose what object is most important to them in a scene.

The method is split up into 5 steps. While 5 steps may initially sound intimidating to remember and work through, after watching the demo, the steps are quick, intuitive, and effective for grabbing a object.

1. Scribbling

The user is prompted to "scribble" or paint the area that they wish to isolate. As a note, editing is not available while the NeRF is training.

2. Reprojection

Once that initial area is complete, the user hits the PROJECT button in one of the submenus. This will provide an estimation of the selected area in a three dimensional capacity

3. Region Growing

If the user wants to expand the affected area, there is a button called "GROW REGION" that enlarges the coverage.

4. Cage Building

After the user is satisfied with the requested area, hit "COMPUTE PROXY" in the cage deformation sub menu. This forms the three dimensional "cage" around the object. The user is also able to move around the NeRF to ensure that everything is enclosed.

5. Tetrahedralization

If the user wants to manipulate a certain section of the cage, you can click and drag your mouse across the requested area and it will highlight the editable area in green.

Once the final step has been complete, the objects are completely/ mobile, allowing the user to reposition them anywhere into the scene. Because there are artifacts that may be introduced after an object has been moved in a scene, there is a membrane-based interpolation to account for this.

One thing that was not shown yet, is whether or not selected objects can be copy and pasted across various NeRFs.

Some of the most astounding features of this method is the ability to transform and rotate the selected items in a 3D capacity. For instance, the demo showcases the ability of the bulldozer's blade to be tilted downwards as well as the position of fingers on a hand to be altered. The implications of this addition alone are massive for future editors and content creators.

If you are looking to have your mind blown on a Monday, I would highly recommend watching the below demo or reading the original paper here.

Now that there are editable NeRFs, another major barrier to mainstream adoption has been broken. How it will be utilized remains to be seen, but this represents a major step forwards into the proliferation of a new content medium.