Triangle Splatting for Real Time Radiance Field Rendering

Radiance field representations have taken the world by storm over the last five years. There has been a present and unrelenting pace of research that has shown several different representations be published. These have ranged from NeRFs to Gaussian Splatting, to voxel based, ray tracing based, and Convex Splatting. 

A few authors of the latter have returned with another new radiance field representation, Triangle Splatting today. For decades, triangles have been the workhorse of computer graphics. GPUs are built to process them at lightning speeds, and most real-time engines are optimized for triangle based rendering. When it comes to learning-based methods, triangles have been tricky to use. They’re rigid, discrete, and notoriously difficult to optimize using gradient descent, which is essential for neural rendering pipelines.

Triangle Splatting overcomes that challenge by introducing a differentiable way to render triangles that allows them to be optimized just like Gaussians or voxels. Instead of treating triangles as hard-edged shapes that either fully contribute to a pixel or not at all, Triangle Splatting lets them softly “splat” onto the screen. This soft rendering approach not only makes gradients flow through the shape, but also gives each triangle the flexibility to represent appearance and geometry in more nuanced ways.

Again, just because this method has the word “splatting” in the name, does not mean that it is gaussian splatting. 

The process begins with a cloud of unconnected triangles. Each triangle floats freely in 3D space, defined by three points, a color, a level of opacity, and a parameter that controls how sharply or softly it fades at the edges. These triangles are projected onto the camera’s view, and instead of being rendered with hard borders, they blend into the image based on a new mathematical function that softly attenuates their influence near the edges. The closer you are to the triangle’s center, the more it contributes; the farther away, the more it fades out.

They also use a window function that tightly controls how the triangle contributes to the final image. It ensures that each triangle only affects the pixels inside its projected shape, and does so in a smooth, controllable way. Triangles are also pruned if they’re not contributing meaningfully. For instance, if they only show up in one view or explain very little of the image. Meanwhile, similar to MCMC, in areas where more detail is needed, existing triangles are split into smaller ones. This adaptive process allows the representation to become denser where necessary and stay lightweight elsewhere.

One question I get repeatedly in my consulting work is: when will radiance fields work seamlessly in game engines? While the list of compatible software continues to grow, there’s still work to be done. Having a representation that is already tuned to parameters the game engines expect would be a huge benefit, and potentially help accelerate adoption. 

For me, this is another acknowledgement of the versatility of radiance field representations and how much additional progress there is to be made over the coming years, with the minimum being that we are hurtling towards a hyper realistic 3D world. 

The code is not yet released, but you can learn more about Triangle Splatting from their Project Page.