Metal Viewer Relights Gaussian Splatting Reflections in Real Time on iPhone

Daisuke Majima has built a Metal viewer that strips baked in lighting from a captured splat and re-lights it in real time on an iPhone, swapping or rotating an HDR environment map while the object's specular reflections and the surrounding skybox track the change. The project, MetalGaussianSplatRelighting, runs the relightable model Ref-Gaussian on top of the open source MetalSplatter renderer, with split sum image based lighting carried over from Unreal Engine 4's approach.

A relightable representation stores material instead of color, per splat. For instance, an albedo, a surface normal, a roughness value, and a reflectance term, and recomputes the shaded color on the fly for whatever environment light is present. Ref-Gaussian is the decomposition that learns those per splat materials. This viewer is the runtime that consumes them on Apple silicon.

The shading path is deferred. Because splats are translucent and overlap, shading each one individually and compositing afterward lets neighboring normals interfere and produces visible noise, so the renderer first accumulates each splat's material into a screen space G-buffer held in Metal tile memory, averages there, and shades once per pixel after the normals settle. The environment itself is held as an HDR equirectangular map and reduced ahead of time into the two textures split sum needs. An irradiance map sampled along the normal for the diffuse term and a roughness mipped prefiltered map sampled along the reflection direction for the specular term, plus a BRDF lookup table. At runtime relighting is two texture lookups rather than a per-pixel integral, which is what makes it viable on a phone.

Earlier relightable splat work has appeared on radiancefields.com, including a Houdini-based proof of concept and a diffusion based approach. Majima's stated next step is to drive the lighting from ARKit's environment probe, lighting a placed object with the actual light of the room around it. The code, demo, and full implementation notes are on GitHub, building on MetalSplatter by Sean Cier (MIT) and Ref-Gaussian, with HDRIs from Poly Haven.