3DGS-DR's Deferred Reflections

Rendering specular reflections—essential for realistic outputs—remains a challenge. Reflections have been a hot topic recently in the research world, with papers such as NeRF-Casting greatly enhancing the fidelity of reflections, making view-dependent effects and hyper-realism of Radiance Fields shine through. Today, we're looking at an interesting paper titled "3D Gaussian Splatting with Deferred Reflection." What makes this method particularly intriguing is the reduced computational requirements.

3DGS-DR reduces training time to approximately 45 minutes on a single NVIDIA RTX 4090 GPU, making it accessible to high-end consumer GPUs. It also retains the real-time rendering rates of 3D Gaussian Splatting, achieving around 250 fps.

Specular reflections, which are mirror-like reflections from surfaces, require precise surface normal estimations to accurately model the environment map reflection. Traditional 3D Gaussian Splatting, while effective for general radiance fields, struggles with these reflections due to limited directional frequency and unstable normal estimations. This results in compromised geometry quality and poorly emulated specular effects.

There has been considerable work and interest in improving geometry quality from papers such as 2D Gaussian Splatting (2DGS) and Gaussian Frosting, but that's a topic for another article.

The authors of this paper propose a novel deferred shading method that significantly enhances the rendering of specular reflections. This approach is based on a two-pass rendering model:

1. Gaussian Splatting Pass: This initial pass generates screen-space maps, capturing base color, normal, and reflection strength for each Gaussian.

2. Pixel Shading Pass: In this pass, the method queries an environment map using the reflection direction to acquire the specular reflection color. The final color is rendered by blending the base and reflection colors, weighted by the reflection strength.

The deferred shading technique defers the computation of high-frequency shading effects like specular reflections to the pixel level, rather than at the Gaussian level. This ensures higher precision in shading, allowing for the accumulation of correct normal estimations over time and bridging the optimization of neighboring Gaussians.

Training the model involves a specialized algorithm that facilitates normal propagation. Initially, the method focuses on view-independent colors, gradually enabling reflection optimization. Reflective Gaussians with near-correct normals propagate their normal vectors to nearby Gaussians, ensuring that the entire reflective surface achieves accurate normal estimations.

Deferred shading in this context offers several advantages:

• Improved Normal Estimations: By deferring shading to the pixel level, the method achieves pixel-precise normal maps, crucial for accurate reflections.

• Enhanced Reflection Quality: The approach generates smoother and more detailed reflections by leveraging more reflection samples at the pixel level.

The research team will be presenting at SIGGRAPH this year, and the code has already been published, although it is currently without a license. Their project page can also be found here.

I envision incorporating this technique into more web-based design, using Three.js to achieve highly accurate reflections and silky smooth rendering rates for users. Reflection-based papers have become some of my favorites due to their visual impact and their potential to bring us closer to a world where hyper-realistic details are fully captured.