Research

Compact NGP: Sub Megabyte NeRFs

Michael Rubloff

Michael Rubloff

Jan 1, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Compact NGP
Compact NGP

Happy new year, everyone!

Shortly after SIGGRAPH in late August, I began having conversations about the place of NeRFs in Ecommerce. The almost universal feedback I received was that the file sizes were still too large and to come back when they approach roughly a megabyte. Well it appears that day has come with Compact NGP. The NGP stands for Neural Graphics Primitives, for anyone looking for a trivia question.

Compact NGP comes as follow up work to NVIDIA's seminal work, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, otherwise known as Instant-NGP. I've spent more time than most looking at the Instant-NGP GUI and it always makes me really excited and nostalgic to see improvements to NVIDIA's method. But enough about that.

How can you not be romantic about watching NeRFs training?

We're looking at submegabyte NeRFs, without sacrificing fidelity. I think this is actually a pretty big paper, in what it implies for several industries and the future of file formats, but this isn't necessarily the first NeRF compression paper we've seen.

To be clear, Compact NGP isn't something that can just be exported after a NeRF has been trained with Instant NGP. While Compact NGP utilizes a lot of Instant NGP, it's not quite at the point where you can go File > Save As > .CompactNGP. It's a standalone method that takes slightly longer to train than Instant NGP, but again that amounts to near negligible differences. While it takes slightly longer to train, Compact NGP does come with a faster inference time than Instant NGP.

As I mentioned just above, Instant NGP is utilized for the initial training portion and has been proven repeatedly as a state of the art baseline to build on top of. Compact NGP differentiates itself by using the same hash grid with learned probes, which leads to a favorable balance between storage size and computational speed, while generalizing Instant NGP's Index function. As part of this, they introduce the Index probing range and a Index codebook size.

Compact NGP

Compact NGP's ability to provide high-quality content without a dedicated decompression step makes it ideal for applications that require streaming high-fidelity graphics, such as in gaming and virtual environments. Its low memory requirement makes it suitable for environments where working memory is a constraint, which is a common challenge in video game texture compression and mobile applications. However, I still believe the larger application will be in Ecommerce and product visualization. This appears to be a trend from the tech behemoths across the past two weeks. We saw SMERF from Google and then FastSR-NeRF from Apple, both of which seem to squarely face mobile applications head on.

As an example, NeRFs may be broadcasted and decoded on large numbers of end-user devices, possibly in real-time to enable live streaming video NeRFs. More generally, (learnable) compression codecs will enable the next generation of immersive content of which live streaming of NeRFs are just an example and other applications, like video game texture compression and volumetric video, being right around the corner.

This continued research pouring in is exciting to see for the everyday mobile user. There have been clear indications from the largest and most influential companies in the world that mobile photorealistic 3D experience is not only coming, but is actively being seeked out. I tend to agree and further believe there is no company more primed to capitalize on radiance fields than NVIDIA. They have the opportunity to productize and open the doors to pursue multi-trillion dollar industries such as advertising, Ecommerce, and more to create compelling use cases for consumers.

NVIDIA has continued to made waves, following up from their Adaptive Shells release and we'll have to stay tuned to see if any of these papers make their way to the public. At the time of posting, Compact NGP's code has not been incorporated into the Instant-NGP repository and it is unclear if that will be coming. However, NVIDIA will be having the first in person GTC (since COVID) in just a couple of months, and it will be interesting to see what role NeRFs and radiance fields will play in their programming. I can't help but feel that we will see some things from the Santa Clara company.

Editor's Note: This is the primary author's Towaki Takikawa's final paper with NVIDIA. However, he continues to be advised by Sanja Fidler, VP of AI research at NVIDIA, while at the University of Toronto. He is the latest radiance field engineer to depart from a major tech company, following Ben Mildenhall's departure from Google.

Featured

Featured

Featured

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV