Compact NGP: Sub Megabyte NeRFs

Michael Rubloff

Michael Rubloff

Jan 1, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Compact NGP
Compact NGP

Happy new year, everyone!

Shortly after SIGGRAPH in late August, I began having conversations about the place of NeRFs in Ecommerce. The almost universal feedback I received was that the file sizes were still too large and to come back when they approach roughly a megabyte. Well it appears that day has come with Compact NGP. The NGP stands for Neural Graphics Primitives, for anyone looking for a trivia question.

Compact NGP comes as follow up work to NVIDIA's seminal work, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, otherwise known as Instant-NGP. I've spent more time than most looking at the Instant-NGP GUI and it always makes me really excited and nostalgic to see improvements to NVIDIA's method. But enough about that.

How can you not be romantic about watching NeRFs training?

We're looking at submegabyte NeRFs, without sacrificing fidelity. I think this is actually a pretty big paper, in what it implies for several industries and the future of file formats, but this isn't necessarily the first NeRF compression paper we've seen.

To be clear, Compact NGP isn't something that can just be exported after a NeRF has been trained with Instant NGP. While Compact NGP utilizes a lot of Instant NGP, it's not quite at the point where you can go File > Save As > .CompactNGP. It's a standalone method that takes slightly longer to train than Instant NGP, but again that amounts to near negligible differences. While it takes slightly longer to train, Compact NGP does come with a faster inference time than Instant NGP.

As I mentioned just above, Instant NGP is utilized for the initial training portion and has been proven repeatedly as a state of the art baseline to build on top of. Compact NGP differentiates itself by using the same hash grid with learned probes, which leads to a favorable balance between storage size and computational speed, while generalizing Instant NGP's Index function. As part of this, they introduce the Index probing range and a Index codebook size.

Compact NGP

Compact NGP's ability to provide high-quality content without a dedicated decompression step makes it ideal for applications that require streaming high-fidelity graphics, such as in gaming and virtual environments. Its low memory requirement makes it suitable for environments where working memory is a constraint, which is a common challenge in video game texture compression and mobile applications. However, I still believe the larger application will be in Ecommerce and product visualization. This appears to be a trend from the tech behemoths across the past two weeks. We saw SMERF from Google and then FastSR-NeRF from Apple, both of which seem to squarely face mobile applications head on.

As an example, NeRFs may be broadcasted and decoded on large numbers of end-user devices, possibly in real-time to enable live streaming video NeRFs. More generally, (learnable) compression codecs will enable the next generation of immersive content of which live streaming of NeRFs are just an example and other applications, like video game texture compression and volumetric video, being right around the corner.

This continued research pouring in is exciting to see for the everyday mobile user. There have been clear indications from the largest and most influential companies in the world that mobile photorealistic 3D experience is not only coming, but is actively being seeked out. I tend to agree and further believe there is no company more primed to capitalize on radiance fields than NVIDIA. They have the opportunity to productize and open the doors to pursue multi-trillion dollar industries such as advertising, Ecommerce, and more to create compelling use cases for consumers.

NVIDIA has continued to made waves, following up from their Adaptive Shells release and we'll have to stay tuned to see if any of these papers make their way to the public. At the time of posting, Compact NGP's code has not been incorporated into the Instant-NGP repository and it is unclear if that will be coming. However, NVIDIA will be having the first in person GTC (since COVID) in just a couple of months, and it will be interesting to see what role NeRFs and radiance fields will play in their programming. I can't help but feel that we will see some things from the Santa Clara company.

Editor's Note: This is the primary author's Towaki Takikawa's final paper with NVIDIA. However, he continues to be advised by Sanja Fidler, VP of AI research at NVIDIA, while at the University of Toronto. He is the latest radiance field engineer to depart from a major tech company, following Ben Mildenhall's departure from Google.

Featured

Recents

Featured

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Dec 10, 2024

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Dec 10, 2024

Platforms

StorySplat 1.3.5 Released.

StorySplat has received several updates.

Michael Rubloff

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Dec 10, 2024

Platforms

Into the Scaniverse: Scaniverse in VR

Scaniverse is now available on both the Quest 3 and 3S.

Michael Rubloff

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Dec 9, 2024

Platforms

Reflct: Simple 3DGS Viewer for Ecommerce

The clean 3DGS viewer for Ecommerce with SH support has begun its closed beta.

Michael Rubloff

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff

Dec 6, 2024

Research

3D Convex Splatting Radiance Field Rendering

3D Convex Splatting is another new way to render a Radiance Field. It is not Gaussian Splatting.

Michael Rubloff