NeRF Segmentation with Instance-NeRF

Over the past few weeks, we have seen a few papers pushing image segmentation forward, but today we get one specific to NeRFs. Instance-NeRF, no not Instant-NeRF, was announced by researchers at The Hong Kong University of Science and Technology as well as Kuaishou Technology.

Instance-NeRF, aims to detect all the objects within the underlying 3D scene and produces a bounding box, a continuous 3D mask, and a class label of each detected 3D object.

Instance-NeRF Paper

Researchers have developed one of the first learning-based NeRF (Neural Radiance Field) 3D instance segmentation pipelines called Instance-NeRF. The groundbreaking method allows for 3D instance segmentation of a given scene, represented as an instance field component of the NeRF model. The Instance-NeRF takes a pretrained NeRF from multi-view RGB images as input and can learn 3D instance segmentation without ground-truth instance information during inference, surpassing previous NeRF segmentation works and competitive 2D segmentation methods in segmentation performance on unseen views.

The pipeline incorporates NeRF-RPN, a method that bridges RPN (Region Proposal Network) and NeRF, with a mask head to predict 3D coarse segmentation. After projecting the 3D masks back to 2D, Instance-NeRF leverages Mask2Former and CascadePSP to match the same instance in 2D segmentation from different views and refine the resultant masks. The refined 2D segmentation of multi-view images is then used to train an instance field which encodes 3D instance information in a continuous manner as a neural field.

NeRF has become a mainstream approach to novel view synthesis. Given multi-view images with camera poses only, NeRF encodes the underlying scene in a multi-layer perceptron (MLP) by radiance propagation and generates impressive results. This has led to significant progress in improving the quality, efficiency, and generality of NeRF. The excellent approach of associating 2D with 3D through radiance field has led researchers to rethink the 3D instance segmentation problem.

Current methods on 3D instance segmentation usually perform on RGB-D images or point clouds, requiring explicit 3D geometry obtained by LiDARs or other devices. However, 3D instance segmentation directly from multi-view images has not been explored in the context of NeRFs. Some unsupervised methods involve 3D scene decomposition and instance segmentation, but they are difficult to apply to complex and large scenes akin to real-world cases.

The Instance-NeRF method is a significant development in the field, as it is one of the first attempts to perform 3D instance segmentation in NeRF without using ground-truth segmentation information during inference. The researchers conducted experiments and ablation studies on a synthetic indoor NeRF dataset to demonstrate the effectiveness of the method, which surpasses competitive 2D segmentation methods and previous works in NeRF segmentation.

This advancement in 3D instance segmentation using NeRF representation has the potential to greatly enhance NeRF object segmentation and manipulation. It can also produce multi-view consistent 2D segmentation as well as continuous 3D segmentation, making it a valuable tool for various applications involving complex scenes and objects. Instance-NeRF serves as a foundation to build on top of for detection and segmentation within a NeRF. It is exciting to see how Instance-NeRF can be utilized with recent papers such as LERF.