Compact NGP: Sub Megabyte NeRFs

Happy new year, everyone!

Shortly after SIGGRAPH in late August, I began having conversations about the place of NeRFs in Ecommerce. The almost universal feedback I received was that the file sizes were still too large and to come back when they approach roughly a megabyte. Well it appears that day has come with Compact NGP. The NGP stands for Neural Graphics Primitives, for anyone looking for a trivia question.

Compact NGP comes as follow up work to NVIDIA’s seminal work, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, otherwise known as Instant-NGP. I’ve spent more time than most looking at the Instant-NGP GUI and it always makes me really excited and nostalgic to see improvements to NVIDIA’s method. But enough about that.

How can you not be romantic about watching NeRFs training?

We’re looking at submegabyte NeRFs, without sacrificing fidelity. I think this is actually a pretty big paper, in what it implies for several industries and the future of file formats, but this isn’t necessarily the first NeRF compression paper we’ve seen.

To be clear, Compact NGP isn’t something that can just be exported after a NeRF has been trained with Instant NGP. While Compact NGP utilizes a lot of Instant NGP, it’s not quite at the point where you can go File > Save As > .CompactNGP. It’s a standalone method that takes slightly longer to train than Instant NGP, but again that amounts to near negligible differences. While it takes slightly longer to train, Compact NGP does come with a faster inference time than Instant NGP.

As I mentioned just above, Instant NGP is utilized for the initial training portion and has been proven repeatedly as a state of the art baseline to build on top of. Compact NGP differentiates itself by using the same hash grid with learned probes, which leads to a favorable balance between storage size and computational speed, while generalizing Instant NGP’s Index function. As part of this, they introduce the Index probing range and a Index codebook size.

Compact NGP’s ability to provide high-quality content without a dedicated decompression step makes it ideal for applications that require streaming high-fidelity graphics, such as in gaming and virtual environments. Its low memory requirement makes it suitable for environments where working memory is a constraint, which is a common challenge in video game texture compression and mobile applications. However, I still believe the larger application will be in Ecommerce and product visualization. This appears to be a trend from the tech behemoths across the past two weeks. We saw SMERF from Google and then FastSR-NeRF from Apple, both of which seem to squarely face mobile applications head on.

As an example, NeRFs may be broadcasted and decoded on large numbers of end-user devices, possibly in real-time to enable live streaming video NeRFs. More generally, (learnable) compression codecs will enable the next generation of immersive content of which live streaming of NeRFs are just an example and other applications, like video game texture compression and volumetric video, being right around the corner.

Compact Neural Graphics Primitives with Learned Hash ProbingTweet

This continued research pouring in is exciting to see for the everyday mobile user. There have been clear indications from the largest and most influential companies in the world that mobile photorealistic 3D experience is not only coming, but is actively being seeked out. I tend to agree and further believe there is no company more primed to capitalize on radiance fields than NVIDIA. They have the opportunity to productize and open the doors to pursue multi-trillion dollar industries such as advertising, Ecommerce, and more to create compelling use cases for consumers.

NVIDIA has continued to made waves, following up from their Adaptive Shells release and we’ll have to stay tuned to see if any of these papers make their way to the public. At the time of posting, Compact NGP’s code has not been incorporated into the Instant-NGP repository and it is unclear if that will be coming. However, NVIDIA will be having the first in person GTC (since COVID) in just a couple of months, and it will be interesting to see what role NeRFs and radiance fields will play in their programming. I can’t help but feel that we will see some things from the Santa Clara company.

Editor’s Note: This is the primary author’s Towaki Takikawa’s final paper with NVIDIA. However, he continues to be advised by Sanja Fidler, VP of AI research at NVIDIA, while at the University of Toronto. He is the latest radiance field engineer to depart from a major tech company, following Ben Mildenhall’s departure from Google.