nerfstudio

Nerfstudio is a company founded by Matthew Tancik, Evonne Ng, and Ethan Weber.

Their goal is to create a user-friendly experience to explore radiance fields and be a contributor-friendly repo to build a community where users can easily build upon each other’s contributions.

Nerfstudio quickly expanded to become one of the most popular models to utilize, with both companies such as Polycam using it, in addition to researchers.

Nerfstudio is sponsored by both Luma AI and Berkeley Artificial Intelligence Research (BAIR).

Supported Radiance Field methods on Nerfstudio

While Nerfstudio originally launched for NeRFs, it was because at the time NeRFs were the only existing Radiance Field method. With the creation of 3D Gaussian Splatting, Nerfstudio has since expanded their offering to also include their homemade implementation, Splatfacto.

Over time the number of methods that Nerfstudio offers has continued to grow. This list will continue to be updated as new methods are added.

Implemented Methods

The following methods are supported in Nerfstudio:

• BioNeRF: Biologically Plausible Neural Radiance Fields for View Synthesis

• Instant NGP: Instant Neural Graphics Primitives with a Multi Resolution Hash Encoding

• Instruct-GS2GS: Editing Gaussian Splatting Scenes with Instructions

• Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

• Generfacto: Generate 3D models from text

• K-Planes: Unified 3D and 4D Radiance Fields

• LERF: Language Embedded Radiance Fields

• Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

• NeRF: The original NeRF paper.

• Nerfacto: our de facto NeRF method combines modules focused on quality with modules focused on faster rendering. nerfstudio easily lets us experiment with the best of both worlds! There is also Nerfacto-Big and Nerfacto-Huge, which are more powerful and VRAM consuming methods.

• Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs

• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields

• NeRFtoGSandBack: From NeRFs to Gaussian Splats, and Back

• OpenNeRF: OpenSet 3D Neural Scene Segmentation with Pixel-wise Features and Rendered Novel Views

• PyNeRF: Pyramidal Neural Radiance Fields for fast NeRF anti-aliasing.

• Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedra

• TensoRF: Tensorial Radiance Fields

• Splatfacto: Nerfstudio's homemade implementation of 3D Gaussian Splatting. There is another variation named Splatfacto-Big, for more detailed scenes.

• SeaThru-NeRF: A NeRF method targeting underwater, foggy, or hazy scenes.

• Zip-NeRF: Google's SoTA method.

Nerfstudio also allows you to submit your own methods, but it must be in their format. They have provided documentation on how to do so, here.

After the method is properly formatted, you need to follow these steps:

Add a markdown file describing the model to the docs/nerfology/methods folder

1. Update the above list of implement methods in this file.

2. Add the method to this list in docs/index.md.

3. Add a new ExternalMethod entry to the nerfstudio/configs/external_methods.py file.

Nerfstudio Extensions

There have been the following extensions created for Nerfstudio

• A Blender VFX add-on

• Unreal Engine Exporting (either through Volinga's .NVOL or Splatfacto's resulting .ply

• SDFStudio

Nerfstudio Viewer

Nerfstudio additionally has its own real time web viewer, Viser, which doesn't require any installation and allows you to view your training process on the device. Additionally, the Viser viewer now allows users to generate a share link to a capture, but it will still run on your local machine. Viser can also be modified into a custom GUI using the linked documentation.

The viewer is built on Viser using ThreeJS and packaged into a ReactJS application. This client viewer application will connect via a websocket to a server running on your machine. If you prefer to use the legacy viewer, Nerfstudio still supports using it.

Nerfstudio Tutorials

There are some great tutorials on YouTube on how to set up and use Nerfstudio. Check out some of the content created by Jonathan Stephens, the NeRFGuru