News|Research

Zip-NeRF upends previous NeRF Benchmark

Michael Rubloff

Apr 14, 2023

Zip-NeRF
Zip-NeRF

It had been too quiet recently for Jon Barron and Ben Mildenhall.

The team from Google Research have made significant progress in accelerating Neural Radiance Field (NeRF) training with Zip-NeRF by combining two previously incompatible techniques, Mip-NeRF 360 and Instant NGP (iNGP). The integration reduces error rates by 8% to 76% compared to prior techniques and trains 22 times faster than Mip-NeRF 360.

One of Google's previous papers, Mip NeRF-360 had long been a top standard within NeRF. However, it's incredibly computational taxing to the point of not being usable for a consumer and the training times were extremely long.

When I first viewed the demo video, I kept thinking, this is going to take forever to train. From all the posts I've seen this morning, people cannot believe that this technology exists; what I would like to focus on is the time in which this took to generate: ~55 minutes. The quality and area that were able to be achieved sub 1-hour is quite frankly nothing short of amazing. It's also important to note how well Zip-NeRF is able to process exposure. When you examine the video from approximately 1-1:40, Zip-NeRF handles highlights significantly better than both Mip-NeRF 360 and Instant-NGP.

The photorealistic hallmark of NeRFs is on full display here and my phone has been buzzing with people talking about how Zillow, Apartments.com, and other platforms can utilize NeRF. Yes, it's all coming and you can believe that NeRFs will be at the front of it.

With Google recently rolling out Immersive View, it becomes apparent how strong their foundation is for showcasing a place of business's atmosphere. NeRFs appear to be directly headed to transforming the way that humans experience a space. Additionally, with Zip-NeRF, atmosphere and mise en scène will become even more important.

NeRF involves training a neural network to model a volumetric representation of a 3D scene, allowing for novel views of the scene to be rendered via ray-tracing. The original NeRF model utilized a multilayer perceptron (MLP) to parameterize the mapping from spatial coordinates to colors and densities. While this approach is expressive, MLPs are slow to train, which has led researchers to accelerate training by replacing or augmenting MLPs with voxel-grid-like data structures.

Instant NGP (iNGP) is one such example. It uses a pyramid of coarse and fine grids to construct learned features processed by a tiny MLP, which greatly accelerates training. The original NeRF model also suffered from aliasing, as it reasons about individual points along a ray, resulting in jaggies in rendered images and limited ability to reason about scale. Mip-NeRF addressed this issue by casting cones instead of rays, and by featurizing the entire volume within a conical frustum for use as input to the MLP. Mip-NeRF and its successor, mip-NeRF 360, showed that this approach enables highly accurate rendering on challenging real-world scenes.

Unfortunately, the progress made on fast training and anti-aliasing are not easily compatible. Mip-NeRF's anti-aliasing strategy depends critically on the use of positional encoding to featurize a conical frustum into a discrete feature vector. In contrast, current grid-based approaches like iNGP do not use positional encoding and instead use learned features obtained by interpolating into a hierarchy of grids at a single 3D coordinate. This creates a challenge in adapting anti-aliasing approaches from rendering to grid-based NeRF models like iNGP.

To address this issue, the researchers leveraged ideas from multisampling, statistics, and signal processing to integrate iNGP's pyramid of grids into mip-NeRF 360's framework. They called their model "Zip-NeRF" due to its speed, its similarity with mip-NeRF, and its ability to fix zipper-like aliasing artifacts. On the mip-NeRF 360 benchmark, Zip-NeRF reduces error rates by as much as 18% and trains 22 times faster than the previous state-of-the-art. Compared to Instant-NGP, the results are even more staggering: 26%, 41%, and 36% reductions in RMSE, DSSIM, and LPIPS.

On a multiscale variant of that benchmark, which more thoroughly measures aliasing and scale, Zip-NeRF reduces error rates by as much as 76%.

Mip-NeRF 360 and iNGP differ significantly in how coordinates along a ray are parameterized. Mip-NeRF 360 subdivides a ray into a set of intervals, each representing a conical frustum whose shape is approximated with a multivariate Gaussian. The expected positional encoding with respect to that Gaussian is used as input to a large MLP. In contrast, iNGP trilinearly interpolates into a hierarchy of differently-sized 3D grids to produce feature vectors for a small MLP. Combining these two approaches introduces two forms of aliasing that the researchers had to address.

First, Instant NGP's feature grid approach is incompatible with mip-NeRF 360's frustum-based approach. To solve this issue, the researchers adopted a multisampling strategy, wherein they sample multiple points along the ray within each frustum. They then average the feature vectors obtained from the grid hierarchy for each of these points, resulting in a multiscale feature representation for the entire frustum.

The second challenge was that iNGP's trilinear interpolation does not produce feature vectors equivalent to mip-NeRF 360's positional encoding. To address this, the researchers introduced a differentiable estimator for the expected value of a trilinearly interpolated feature vector. This estimator combines the grid-based features and the expected positional encoding within each frustum. It also allows them to use a small MLP, like in iNGP, while still benefiting from mip-NeRF 360's conical frustum-based approach to anti-aliasing.

By combining these two solutions, the researchers created a novel method that integrates the best of both mip-NeRF 360 and iNGP, resulting in significantly improved training times and reduced aliasing issues. This new model, Zip-NeRF, is capable of producing high-quality, photorealistic renderings while minimizing artifacts such as jaggies and missing scene content.

The success of Zip-NeRF opens up new possibilities for the application of NeRF in various fields. Its fast training times could enable more rapid development and deployment of generative media, virtual reality environments, and robotic perception systems. Additionally, the reduction in aliasing artifacts improves the overall quality and realism of rendered scenes, potentially benefiting applications in filmmaking, video game design, and computational photography.

Moreover, the researchers' approach to combining mip-NeRF 360 and iNGP could inspire further research into the integration of disparate techniques in the field of neural rendering. The innovative solutions they employed to overcome the challenges posed by integrating these two methods demonstrate the value of exploring hybrid models and drawing from multiple domains to achieve optimal performance.

This breakthrough brings together the benefits of Mip-NeRF 360's anti-aliasing and iNGP's accelerated training. The resulting model, Zip-NeRF, offers a faster and more accurate approach to NeRF training, with applications in various fields such as robotics, computational photography, and generative media.

Featured

Featured

Featured

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Featured

Featured

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Recent articles

Recent articles