News|Research

What are the NeRF Metrics?

Michael Rubloff

Jun 12, 2023

PSNR
PSNR

I've become more accustomed to seeing repeating words and phrases within NeRF papers. Some of the most common are PSNR, SSIM, and LPIPS. All of them are used to evaluate the quality of each NeRF method within a given scene.

However, all three have been in existence well below NeRFs, so today I figured I would give a quick article on what the three terms are and measure, as well as highlight a new potential way to measure NeRF quality from a group of researchers at the University of Bristol.

While NeRF implementations can produce high-quality visual results, evaluating their performance accurately remains a challenge. Conventional evaluation methods provide approximate indicators and may not capture specific aspects of NeRFs. To address this issue, a group of researchers has proposed a new test framework that isolates the neural rendering network from the NeRF pipeline for more reliable evaluation.

What is PSNR (Peak Signal to Noise Ratio)?

Peak Signal-to-Noise Ratio (PSNR) is a metric widely used in the field of image and video processing to measure the quality of reconstructed (i.e., compressed and then decompressed) images or videos. It evaluates on a color-wise basis, which does have a bit of overlap with NeRFs view dependent colors. The higher the PSNR, the better is the quality of the compressed or reconstructed image, video, or NeRF.

What is SSIM (Structural Similarity Index Measure)?

SSIM is used for measuring the similarity between two images. The SSIM index is a full reference metric; in other words, the measurement or prediction of image quality is based on an initial uncompressed or distortion-free image as reference.

SSIM considers changes in structural information, perceived luminance, and contrast that can occur when images are subjected to various types of distortion. It aims to reflect the human visual system's perception more closely than simpler metrics like PSNR or mean squared error (MSE).

As SSIM computations are performed on image patches, they allow for some misalignment between the synthesized and reference images. This is helpful for evaluation as there may be variations between the NeRF camera model and the real camera used for capturing the training images.

What is LPIPS (Learned Perceptual Image Patch Similarity)?

Finally, Learned Perceptual Image Patch Similarity (LPIPS) has gained popularity in areas such as frame interpolation, measuring the similarity between features of two images extracted from a pretrained network.

Learned Perceptual Image Patch Similarity (LPIPS) is a perceptual metric that quantifies the human-perceived similarity between two images. Unlike traditional metrics such as PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measure), which calculate differences based on raw pixel values or simple transformations thereof, LPIPS leverages deep learning to better align with human visual perception. It uses the distance between features extracted by a convolutional neural network (CNN) pretrained on an image classification task as a perceptual metric.

What is Whole-scene Average Prediction Error (WAPE)?

With the three above metrics defined, a group of researchers at the University of Bristol advocate for a new metric that is NeRF specific: Whole-scene Average Prediction Error (WAPE).

The research paper introduces a configurable approach for generating representations specifically for evaluation purposes. This approach utilizes ray-casting to transform mesh models into explicit NeRF samples and "shade" these representations. By combining these methods, the researchers demonstrate how different scenes and types of networks can be evaluated within this framework. They also propose a novel metric called the Whole-scene Average Prediction Error (WAPE) to measure task complexity, considering visual parameters and the distribution of spatial data.

This framework first isolates the neural rendering network from the NeRF pipeline and then performs a parametric evaluation by training and evaluating the NeRF on an explicit radiance field representation.

By isolating the rendering network from the entire pipeline, it gives more tailored feedback to underlying errors within the reconstruction and gain a deeper understanding of how well each method works. For instance,

"as image-based metrics only evaluate the prediction quality via a 2-D projection, this can still result in loss of information about the accuracy of spatial samples in relation to their distribution in volumetric space."

The proposed framework addresses the limitations of image-based metrics by providing a parametric evaluation that compares the learnable outputs of INR rendering networks against ground truth data. By generating synthetic radiance fields from mesh-based representations and applying ray tracing, the researchers accurately represent ground truths and enhance the quality of visual features.

In theory, the more challenging data is to get a high quality NeRF, it should be graded differently to compensate. With that in mind, they introduce my favorite part of the paper, a new metric for evaluating task complexity.

This takes into account the number of input samples, the relative distribution of novel views and training views, and the functional complexity of the chosen ray tracing algorithm(s). The NeRFs that I have processed thus far have all been different in size and complexity and I believe this should be reflected in evaluation. It's also interesting that they interpret complexity as not only stemming from input views, but also extend it to the underlying positional distribution of ray samples.

As continues to be an emerging trend, WAPE's code works with nerfstudio, though it doesn't appear to have been published yet.

Only time will tell if the NeRF community will embrace WAPE and begin weighing quality markers with it. It has been fascinating to weigh each method's PSNR, SSIM, and LPIPS scores, but if there's a way to get more accurate metrics and identify areas of NeRF specific improvement more efficiently, I'm all for it.

Featured

Featured

Featured

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Apr 16, 2024

News

An Update to Radiancefields.com

You may have noticed a small change to the website over the weekend. Well, not so small actually.

Michael Rubloff

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Apr 15, 2024

Platforms

OpenSplat adds Mac GPU Acceleration

OpenSplat, which brought Mac training to 3DGS has received a big update, now allowing users to train with MPS backend with GPU acceleration.

Michael Rubloff

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Featured

Featured

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Research

Live NeRF Video Calls

Michael Rubloff

Oct 5, 2023

Radiance Field Video Call

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Lightfield and hologram capture started with a big theoretical idea 115 years ago and we have struggled to make them viable ever since. Neural Radiance fields aka NeRF along with gaming computers now for the first time provide a promising easy and low cost way for everybody to capture and display lightfields.

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Guest Article

A short 170 year history of Neural Radiance Fields (NeRF), Holograms, and Light Fields

Katrin Schmid

Mar 2, 2023

History of Neural Radiance Fields

Recent articles

Recent articles