Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency

Gaussian Splatting, which is a Radiance Field representation, has quickly scaled in reconstruction capability, with several of the original shortcomings being solved over the last year. One of the pesky limitations of Gaussian Splatting has been these "popping" effects that happen as you rotate around a scene.

The popping effects of Gaussian Splatting is something I’ve written extensively on as something that should be fixed in all implementations. A few months ago, Stop the Pop began tackling this issue by changing how the tiles of the gaussians are sorted. 

3D Gaussian Splatting using Hybrid Transparency takes this up a notch by removing the popping effects and also has large performance boosts, such as up to doubling the rendering rate of 3DGS, depending on the scene.

You might recognize a few of the authors, such as Florian Hahlbohm from INPC: Implicit Neural Point Clouds for Radiance Field Rendering and Linus from Trilinear Point Splatting for Real-Time Radiance Field Rendering. Another one of the authors, Tim Weyrich, actually built a GPU that targets surface splatting applications, way back in 2006, when he was at ETH Zurich and Princeton.

The innovation in HTGS is the elimination of matrix inversions, which are required when evaluating a 3D Gaussian or transforming Gaussians from world space into screen space. Matrix inversions can introduce numerical instability, especially when the splats degenerate along one or more axes (i.e., when the Gaussian becomes flat). To avoid these issues, the researchers devised an alternative that leverages Plücker coordinates, a representation of rays and planes in projective geometry that allows for stable and efficient intersection computations between a viewing ray and the 3D Gaussian splat.

Arguably more important is that Hybrid Transparency mainly fixes the issues with the approximation of the perspective projection in 3DGS. This is probably the most relevant fix as the projection used in 3DGS can be very inaccurate near image borders, as shown in the bench dataset. 

They first define the pixel viewing rays in screen space using two perpendicular planes and then transform these planes to the normalized splat space. Rather than inverting the transformation matrix, the method transforms the planes defined by the pixel's viewing ray. This avoids the matrix inversion while maintaining stability. The evaluation then computes the point of maximum contribution along the ray by minimizing the perpendicular distance between the ray and the center of the Gaussian in splat space.

This perspective-correct approach improves the visual accuracy of the splats, particularly in scenes where large variations in depth or close camera angles cause significant distortion in affine projection.

While perspective-correct splat evaluation addresses projection artifacts, hybrid transparency blending is introduced to solve the depth-sorting problem that leads to popping artifacts. Traditionally, blending semi-transparent surfaces in the correct depth order is computationally expensive, as it requires sorting every fragment at each pixel. This becomes prohibitive for real-time applications, especially when handling complex scenes with a large number of splats.

To balance computational efficiency and visual accuracy, the authors implemented a hybrid transparency model that divides the blending process into two parts between Core and Tail Contributions.

For each pixel, only the top K contributions (the most significant splats based on depth) are sorted and blended using traditional alpha blending. These are known as core splats, which represent the foremost surface(s) that are visible through each pixel. 

The remaining splats (those further away) are blended without sorting using an order-independent transparency method. This allows for much faster blending of the less significant splats while still contributing to the overall image. While the tail's influence is often very small, the authors show that it’s necessary to ensure stable convergence of the optimization process.

The hybrid method ensures that the most visually important splats are blended correctly, preventing popping artifacts, while the rest are handled more efficiently. This division into core and tail contributions significantly reduces the computational burden, making the method well-suited for real-time rendering.

This is also highly scalable. Since this approach involves only a partial sort for the top K contributions, the computational complexity is significantly reduced, particularly when scenes involve many overlapping splats.

One of the major achievements of this method is that the perspective-correct projection and hybrid transparency are not only visually superior but also computationally efficient. By avoiding complex operations like matrix inversion and full sorting, the method achieves significant speed-ups compared to traditional 3DGS implementations, pushing rendering rates in excess of 400fps.

Another important feature of this method is its modularity. The perspective-correct splat evaluation and hybrid transparency blending can be integrated independently into existing 3DGS or Ray Tracing pipelines. This flexibility allows developers to selectively improve rendering quality without overhauling entire systems, depending on the specific needs of their applications. Excitingly, there are a couple other simultaneous approaches, including Taming 3DGS and EVER, which might be able to stack on top of one another.

Additionally, co-first author of Gaussian Splatting, George Kopanas has also commented on the work and will at some point work on implementing this into his Slang.d port of Gaussian Splatting!

A code release will be coming, but there is no time frame yet for it. The usage of Hybrid Transparency continues to showcase how there are still big gains to be had in the world of Radiance Fields. Check out their Project Page for more examples.