One of the most hotly requested capabilities for gaussian splatting has been the ability to relight and have captures interact with scenes. After months of late night development and experimentation, a currently unnamed tool demonstrates one of the first successful attempts at bringing true light interaction to splats, including spatial awareness, occlusion, contact shadows, bounce lighting, and even deformable splats.
In effect, it delivers the kind of illumination behavior typically expected from traditional renderers, but within a splat based framework. The workflow trains a multi layer perceptron (MLP) on a user defined lighting setup within Solaris in SideFX Houdini, leveraging David Rhodes’ GSOPs toolkit, which added Solaris support a couple of months ago.
According to the developer, generating and training a dataset currently takes around six hours per scene on an RTX 2070 (8GB), though inference runs significantly faster.
The system relies on an encoder/decoder architecture that translates Gaussians into physically based rendering (PBR) material values and then back into Gaussians. Two encoder/decoder pairs are trained, one under the lighting conditions “baked” into the original capture and another under the target lighting setup.
By decoding PBR values from the original model and re-encoding them using the target light model, the system effectively relights the splat. The approach hinges on two main factors. The number of training samples generated and how closely the synthetic training lighting matches the original scene. Interestingly, PBR values don’t have to originate from the decoder. They can also be derived from traditionally shaded polygonal geometry, allowing geometry to splat conversion without explicit rendering or retraining. While quality is lower in this mode, it demonstrates how the encoder and decoder can operate flexibly across different data sources.
While the early proof of concept is already functional, it comes with tradeoffs. Animated lights or moving geometry interacting with the splat require per frame retraining, making dynamic sequences computationally heavy. Additionally, there’s no true internal light transport, meaning reflective or transparent materials remain challenging for the model to simulate accurately.
Even so, this is a very exciting step in the right direction for relightable and usable splats within the entertainment industry. You can view the first demonstration video, featuring Bittel’s original “Splat Fly” scene, here. We’ll continue to follow along with the project as it’s developed and will keep you updated when it’s available to be used. The developer, Nico Klein, is also open to any name suggestions.
New to Gaussian Splatting? Start here
