3DGS-DR's Deferred Reflections

3DGS-DR's Deferred Reflections

3DGS-DR's Deferred Reflections

Michael Rubloff

Michael Rubloff

Jun 27, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
3DGS-DR
3DGS-DR

Rendering specular reflections—essential for realistic outputs—remains a challenge. Reflections have been a hot topic recently in the research world, with papers such as NeRF-Casting greatly enhancing the fidelity of reflections, making view-dependent effects and hyper-realism of Radiance Fields shine through. Today, we're looking at an interesting paper titled "3D Gaussian Splatting with Deferred Reflection." What makes this method particularly intriguing is the reduced computational requirements.

3DGS-DR reduces training time to approximately 45 minutes on a single NVIDIA RTX 4090 GPU, making it accessible to high-end consumer GPUs. It also retains the real-time rendering rates of 3D Gaussian Splatting, achieving around 250 fps.

Specular reflections, which are mirror-like reflections from surfaces, require precise surface normal estimations to accurately model the environment map reflection. Traditional 3D Gaussian Splatting, while effective for general radiance fields, struggles with these reflections due to limited directional frequency and unstable normal estimations. This results in compromised geometry quality and poorly emulated specular effects.

There has been considerable work and interest in improving geometry quality from papers such as 2D Gaussian Splatting (2DGS) and Gaussian Frosting, but that's a topic for another article.

The authors of this paper propose a novel deferred shading method that significantly enhances the rendering of specular reflections. This approach is based on a two-pass rendering model:

  1. Gaussian Splatting Pass: This initial pass generates screen-space maps, capturing base color, normal, and reflection strength for each Gaussian.

  2. Pixel Shading Pass: In this pass, the method queries an environment map using the reflection direction to acquire the specular reflection color. The final color is rendered by blending the base and reflection colors, weighted by the reflection strength.

The deferred shading technique defers the computation of high-frequency shading effects like specular reflections to the pixel level, rather than at the Gaussian level. This ensures higher precision in shading, allowing for the accumulation of correct normal estimations over time and bridging the optimization of neighboring Gaussians.

Training the model involves a specialized algorithm that facilitates normal propagation. Initially, the method focuses on view-independent colors, gradually enabling reflection optimization. Reflective Gaussians with near-correct normals propagate their normal vectors to nearby Gaussians, ensuring that the entire reflective surface achieves accurate normal estimations.

Deferred shading in this context offers several advantages:

  • Improved Normal Estimations: By deferring shading to the pixel level, the method achieves pixel-precise normal maps, crucial for accurate reflections.

  • Enhanced Reflection Quality: The approach generates smoother and more detailed reflections by leveraging more reflection samples at the pixel level.

The research team will be presenting at SIGGRAPH this year, and the code has already been published, although it is currently without a license. Their project page can also be found here.

I envision incorporating this technique into more web-based design, using Three.js to achieve highly accurate reflections and silky smooth rendering rates for users. Reflection-based papers have become some of my favorites due to their visual impact and their potential to bring us closer to a world where hyper-realistic details are fully captured.

Featured

Recents

Featured

Platforms

NVIDIA Open Sources 3D Gaussian Ray Tracing and 3D Gaussian Unscented Transform

NVIDIA has released two Radiance Fields, with an exciting license!

Michael Rubloff

Mar 19, 2025

Platforms

NVIDIA Open Sources 3D Gaussian Ray Tracing and 3D Gaussian Unscented Transform

NVIDIA has released two Radiance Fields, with an exciting license!

Michael Rubloff

Mar 19, 2025

Platforms

NVIDIA Open Sources 3D Gaussian Ray Tracing and 3D Gaussian Unscented Transform

NVIDIA has released two Radiance Fields, with an exciting license!

Michael Rubloff

Research

Hybrid Transparency Gaussian Splatting Code Released

HTGS gets rid of popping artifacts in 3DGS and more!

Michael Rubloff

Mar 14, 2025

Research

Hybrid Transparency Gaussian Splatting Code Released

HTGS gets rid of popping artifacts in 3DGS and more!

Michael Rubloff

Mar 14, 2025

Research

Hybrid Transparency Gaussian Splatting Code Released

HTGS gets rid of popping artifacts in 3DGS and more!

Michael Rubloff

Platforms

Resonite Introduces Native Gaussian Splat Support: Import, Edit, and Render in VR

VR platform Resonite has integrated 3DGS for its users.

Michael Rubloff

Mar 14, 2025

Platforms

Resonite Introduces Native Gaussian Splat Support: Import, Edit, and Render in VR

VR platform Resonite has integrated 3DGS for its users.

Michael Rubloff

Mar 14, 2025

Platforms

Resonite Introduces Native Gaussian Splat Support: Import, Edit, and Render in VR

VR platform Resonite has integrated 3DGS for its users.

Michael Rubloff

Platforms

GSOPs 2.5.1 Released

Quality of Life Improvements are here for working with 3DGS in Houdini.

Michael Rubloff

Mar 13, 2025

Platforms

GSOPs 2.5.1 Released

Quality of Life Improvements are here for working with 3DGS in Houdini.

Michael Rubloff

Mar 13, 2025

Platforms

GSOPs 2.5.1 Released

Quality of Life Improvements are here for working with 3DGS in Houdini.

Michael Rubloff