Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

The original authors are back and are addressing some of those criticisms, which they have distilled into three observations: the number of 3D Gaussian primitives used to represent a scene, the number of coefficients for the spherical harmonics, and the precision required to store Gaussian primitive attributes. This is by no means the only project that has tackled the memory issue since the release of 3DGS, but it does provide a great view into what makes up the memory footprint behind Gaussian Splatting.

The first observation was that the original 3DGS method generated an excessive number of 3D Gaussian primitives, many of which were redundant, leading to an unnecessary increase in memory usage. This redundancy was particularly prevalent in areas of the scene where detailed reconstruction was not required, yet a dense cluster of primitives existed.

To address this, the team developed a resolution-aware pruning method that dynamically adjusts the number of primitives based on their necessity for accurate image representation. They introduced a two-step algorithm that first calculated a redundancy score for each primitive based on its contribution to the rendered view and then culled primitives that fell below a certain utility threshold. This approach effectively reduced the total count of primitives by about 60%, significantly decreasing the memory footprint. This might sound a little familiar conceptually to how RadSplat evaluates their Gaussians.

Interestingly, this comes right on the heels of a paper that revisits the densification stage of Gaussian Splatting. I'd be curious to see how the two papers could be applied to one another.

The second observation related to the use of SH bands, which are employed to model directional radiance. The original method used three SH bands for all primitives, regardless of whether the scene's visual complexity required that level of detail. This one-size-fits-all approach led to an over allocation of memory resources.

The researchers tackled this by implementing an adaptive adjustment method for SH bands during the training phase. By analyzing the contribution of each SH band to the visual appearance of the primitives and assessing multi-view consistency, they were able to reduce the number of SH bands dynamically. This revised new method ensures that only essential data is retained, thereby reducing the memory required for storing these elements.

The final observation was that many of the attributes associated with the primitives, such as opacity and color, did not require high dynamic range or precision, yet they were stored with high precision which led to wasteful memory utilization.

To optimize memory usage for these attributes, they introduce a codebook-based quantization approach. This involved clustering similar values and replacing them with indices to a shared codebook, thus storing only the index rather than the full precision data. Additionally, they applied half-float quantization to further compress the data size without significantly affecting the visual output. This approach drastically reduced the memory requirements for storing primitive attributes.

These updates resulted in reducing the file size by 27 times and 1.7X speed boost for rendering.

They have created a viewer that still retains spherical harmonics to include view dependent effects. They have two different modes: a light one and a heavy. The light version runs smoothly on moderate devices like an M2 laptop, while the heavy version requires more powerful GPU support. Check out their Project Page here!