vkSplatting adds 3DGUT Support

vkSplatting, NVIDIA’s open-source Vulkan renderer for gaussian splatting, has just rolled out one of its most significant updates yet. What began as a testbed for exploring the fundamentals of 3DGS has continued to evolve into a larger ecosystem.

Here we see the introduction of an Unscented Transform-based pipeline, VK3DGUT, which is of course based upon NVIDIA’s 3DGUT paper. As a reminder, traditional 3DGS relied on Elliptical Weighted Average (EWA) splat projections, which offered speed but were limited to pinhole cameras. By contrast, the Unscented Transform enables rasterization to handle complex camera models, including fisheye lenses and rolling shutters, while maintaining high performance for primary rays. VK3DGUT bridges the gap by letting rasterization achieve visual coherence with ray tracing, reserving the heavier computations for secondary effects like reflections, refractions, and shadows.

This leads directly into another exciting addition of a hybrid rendering pipeline. vkSplatting now allows developers to combine rasterization with ray tracing in the same workflow, effectively choosing the right tool for the right job. With primary rays handled through fast rasterization and secondary rays routed through ray tracing, scenes can be rendered with far greater efficiency while still benefiting from the visual fidelity ray tracing provides. Early benchmarks show that these hybrid approaches more than double the performance compared to running ray tracing alone, while preserving support for advanced effects like fisheye distortion and depth of field.

Speaking of cameras, the update also introduces two features that bring vkSplatting closer to production quality rendering. A new fisheye mode allows users to push fields of view into extreme ranges, bending and distorting the scene in ways that closely mimic real world wide angle lenses. Depth of field, meanwhile, adds naturalistic defocus, simulating bokeh and focus falloff. Multi sampling is automatically enabled when this option is active, and users can fine tune parameters such as focus distance and aperture to achieve the look they want.

To support experimentation, NVIDIA has bundled a brand new 3DGUT dataset along with updates to the 3DGRT dataset, making it easier to get started with these new pipelines without needing to prepare your own models. Compatibility has also been broadened through import support for Niantic Labs’ .spz file format, which means splats compressed through Niantic’s method can be dropped directly into vkSplatting for immediate use.

This is an exciting update that should help developers continue to experiment with gaussian based radiance fields. vkSplatting continues to be Apache 2.0 licensed and freely available on NVIDIA’s GitHub.