Research

Compact NGP: Sub Megabyte NeRFs

Michael Rubloff

Jan 1, 2024

Compact NGP
Compact NGP

Happy new year, everyone!

Shortly after SIGGRAPH in late August, I began having conversations about the place of NeRFs in Ecommerce. The almost universal feedback I received was that the file sizes were still too large and to come back when they approach roughly a megabyte. Well it appears that day has come with Compact NGP. The NGP stands for Neural Graphics Primitives, for anyone looking for a trivia question.

Compact NGP comes as follow up work to NVIDIA's seminal work, Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, otherwise known as Instant-NGP. I've spent more time than most looking at the Instant-NGP GUI and it always makes me really excited and nostalgic to see improvements to NVIDIA's method. But enough about that.

How can you not be romantic about watching NeRFs training?

We're looking at submegabyte NeRFs, without sacrificing fidelity. I think this is actually a pretty big paper, in what it implies for several industries and the future of file formats, but this isn't necessarily the first NeRF compression paper we've seen.

To be clear, Compact NGP isn't something that can just be exported after a NeRF has been trained with Instant NGP. While Compact NGP utilizes a lot of Instant NGP, it's not quite at the point where you can go File > Save As > .CompactNGP. It's a standalone method that takes slightly longer to train than Instant NGP, but again that amounts to near negligible differences. While it takes slightly longer to train, Compact NGP does come with a faster inference time than Instant NGP.

As I mentioned just above, Instant NGP is utilized for the initial training portion and has been proven repeatedly as a state of the art baseline to build on top of. Compact NGP differentiates itself by using the same hash grid with learned probes, which leads to a favorable balance between storage size and computational speed, while generalizing Instant NGP's Index function. As part of this, they introduce the Index probing range and a Index codebook size.

Compact NGP

Compact NGP's ability to provide high-quality content without a dedicated decompression step makes it ideal for applications that require streaming high-fidelity graphics, such as in gaming and virtual environments. Its low memory requirement makes it suitable for environments where working memory is a constraint, which is a common challenge in video game texture compression and mobile applications. However, I still believe the larger application will be in Ecommerce and product visualization. This appears to be a trend from the tech behemoths across the past two weeks. We saw SMERF from Google and then FastSR-NeRF from Apple, both of which seem to squarely face mobile applications head on.

As an example, NeRFs may be broadcasted and decoded on large numbers of end-user devices, possibly in real-time to enable live streaming video NeRFs. More generally, (learnable) compression codecs will enable the next generation of immersive content of which live streaming of NeRFs are just an example and other applications, like video game texture compression and volumetric video, being right around the corner.

This continued research pouring in is exciting to see for the everyday mobile user. There have been clear indications from the largest and most influential companies in the world that mobile photorealistic 3D experience is not only coming, but is actively being seeked out. I tend to agree and further believe there is no company more primed to capitalize on radiance fields than NVIDIA. They have the opportunity to productize and open the doors to pursue multi-trillion dollar industries such as advertising, Ecommerce, and more to create compelling use cases for consumers.

NVIDIA has continued to made waves, following up from their Adaptive Shells release and we'll have to stay tuned to see if any of these papers make their way to the public. At the time of posting, Compact NGP's code has not been incorporated into the Instant-NGP repository and it is unclear if that will be coming. However, NVIDIA will be having the first in person GTC (since COVID) in just a couple of months, and it will be interesting to see what role NeRFs and radiance fields will play in their programming. I can't help but feel that we will see some things from the Santa Clara company.

Editor's Note: This is the primary author's Towaki Takikawa's final paper with NVIDIA. However, he continues to be advised by Sanja Fidler, VP of AI research at NVIDIA, while at the University of Toronto. He is the latest radiance field engineer to depart from a major tech company, following Ben Mildenhall's departure from Google.

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Trending articles

Trending articles

Trending articles

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024