Research

Off to the Races with F2-NeRF

Michael Rubloff

Michael Rubloff

Mar 30, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
F2-NeRF
F2-NeRF

4/22/23 Update: The code for F2-NeRF is now open sourced and available here!

The new NeRF papers continue to pour in. This time with the introduction of F2-NeRF (Fast Free NeRF).

Contrary to other NeRF platforms such as Instant-NGP and Luma, which are designed for bounded scenes, F2-NeRF utilizes a space-warping method, named Perspective Warping. This allows for high quality renders in a relatively short training time. The results might shock you at just how clear F2-NeRF is able to get.

They introduce a general space-warping scheme called perspective warping that is applicable to arbitrary camera trajectories and a space subdivision algorithm to adaptively allocate grids for foreground and background regions. Experimental results show that F2-NeRF outperforms existing grid-based NeRF methods on various datasets with different trajectory patterns.

The training time compared to Instant-NGP is roughly double, but given how quickly Instant-NGP runs, it does not appear to be a huge detractor. Especially when the rendered output is so clear. These papers are among some of my favorite because they increase the overall sharpness, while still being manageable on a consumer grade GPU (These were trained on a 2080ti!)

There have been efforts to try and get platforms such as Instant NGP to require less memory using strategies such as such as voxel pruning, tensor decomposition, and hash indexing, but these can only be utilized by bounded scenes when grids are built in the original Euclidean space. F2-NeRF is built on top of the framework of Instant NGP, but allow for free camera trajectories for large, unbounded
scenes.

So how does it work? The rendering process is subdivided into two parts: the preparation stage and
the actual rendering stage.

In the preparation stage, they subdivide the original space (a) according to the view frustums of the cameras and construct the local warping functions (b) based on the selected area for each sub-region.

In other words, they focus on the immediate view of what the user is looking at from the foreground and the background and group them together in order to most effectively apply local warping. The sizes of these grids also directly corresponds to the subject — coarse grids for background regions and fine grids for foreground regions.

Then, in the rendering stage, F2-NeRF renders the pixel color, by sampling points on the camera ray and conducting weighted accumulation of what those colors are. The sampled densities and colors are fetched from the multi-resolution hash grid. Inherently, there will be a problem where there is multiple objects within the foreground.

In comparison, F2-NeRF takes advantage of the perspective warping and the adaptive space subdivision to fully exploit the representation capacity, which enables F2-NeRF to produce better rendering quality. They did find that mip-NeRF-360 was able to pull highly accurate renderings, however this was only when the scene was allowed to train for hours. The example they showed below utilized mip-NeRF-360 short, for a period of 30 minutes.

Looking at the above photo, the results are getting close to approaching Ground Truth (original). You know that when a paper puts a disclaimer that its technology is good enough to generate misleading fake images, it's accurate. For me, the biggest thing is the text clarity on the sign. Clarity has been something that I have still been struggling with Instant-NGP and the text fidelity.

F2-NeRF represents another step in the right direction to high quality NeRFs running on consumer grade GPUs. How the various platforms incorporate the technology remains to be seen, but the evidence is undeniable.

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV