3D Convex Splatting Radiance Field Rendering

Over the last year, Gaussian Splatting has dominated the world of Radiance Field rendering. However, the past few months have introduced additional ways to represent and render a Radiance Field, such as Gaussian Ray Tracing and EVER, but today we’re looking at another departure from Gaussian based representations with 3D Convex Splatting.

3D Convex Splatting replaces Gaussian primitives with 3D smooth convexes— flexible shapes capable of capturing both hard edges and dense volumes. These primitives can represent geometrically meaningful elements of a scene with fewer shapes, reducing memory overhead while maintaining visual fidelity. Defined by a set of points forming a 3D convex hull, these shapes are governed by parameters that control their smoothness and sharpness. This flexibility allows convexes to approximate flat surfaces and angular edges without requiring additional primitives, as Gaussian methods often do.

To render a scene, 3DCS projects the points of each convex onto a 2D image plane using a differentiable pinhole camera model. This projection is critical, as it enables the system to compute gradients and optimize during training. Once projected, the convex hull of the points is calculated, defining the 2D boundary of the shape, which is then rendered using a custom CUDA-based rasterizer for real-time performance.

Starting with a sparse set of primitives, 3DCS adaptively densifies convexes in regions requiring higher detail. By splitting shapes into multiple smaller convexes, the method captures finer scene details without bloating memory requirements. While offering higher quality, 3DCS reduced memory usage by up to 30% compared to 3DGS. The tradeoff here is that the training time is a bit longer compared to 3DGS. Rendering frame rates also take a hit, but still hover above real time rates.

One of the most compelling benefits of 3DCS is its ability to produce geometrically meaningful decompositions of scenes. For example, where Gaussian primitives might produce a blurred representation of complex objects, convex primitives can delineate distinct surfaces and edges. This capability makes 3DCS particularly well-suited for structured environments, such as indoor scenes, where sharp boundaries and flat surfaces dominate. Convex primitives can approximate surfaces like walls or angular edges without resorting to additional primitives, as Gaussian methods would require.

Despite its strengths, 3DCS isn’t without limitations. In outdoor scenes, its advantages over Gaussian-based methods become less pronounced. Future research could explore hybrid approaches, combining the strengths of convex and Gaussian primitives to handle diverse environments seamlessly.

3D Convex Splatting represents an exciting consideration in radiance field rendering. It is important to remember we have not yet arrived at the pinnacle of inverse rendering and 3D reconstruction.

For more details and implementation resources, visit the project page at convexsplatting.github.io. The code is also available here with an Apache 2.0 license.