Research

3DGS-DR's Deferred Reflections

Michael Rubloff

Michael Rubloff

Jun 27, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
3DGS-DR
3DGS-DR

Rendering specular reflections—essential for realistic outputs—remains a challenge. Reflections have been a hot topic recently in the research world, with papers such as NeRF-Casting greatly enhancing the fidelity of reflections, making view-dependent effects and hyper-realism of Radiance Fields shine through. Today, we're looking at an interesting paper titled "3D Gaussian Splatting with Deferred Reflection." What makes this method particularly intriguing is the reduced computational requirements.

3DGS-DR reduces training time to approximately 45 minutes on a single NVIDIA RTX 4090 GPU, making it accessible to high-end consumer GPUs. It also retains the real-time rendering rates of 3D Gaussian Splatting, achieving around 250 fps.

Specular reflections, which are mirror-like reflections from surfaces, require precise surface normal estimations to accurately model the environment map reflection. Traditional 3D Gaussian Splatting, while effective for general radiance fields, struggles with these reflections due to limited directional frequency and unstable normal estimations. This results in compromised geometry quality and poorly emulated specular effects.

There has been considerable work and interest in improving geometry quality from papers such as 2D Gaussian Splatting (2DGS) and Gaussian Frosting, but that's a topic for another article.

The authors of this paper propose a novel deferred shading method that significantly enhances the rendering of specular reflections. This approach is based on a two-pass rendering model:

  1. Gaussian Splatting Pass: This initial pass generates screen-space maps, capturing base color, normal, and reflection strength for each Gaussian.

  2. Pixel Shading Pass: In this pass, the method queries an environment map using the reflection direction to acquire the specular reflection color. The final color is rendered by blending the base and reflection colors, weighted by the reflection strength.

The deferred shading technique defers the computation of high-frequency shading effects like specular reflections to the pixel level, rather than at the Gaussian level. This ensures higher precision in shading, allowing for the accumulation of correct normal estimations over time and bridging the optimization of neighboring Gaussians.

Training the model involves a specialized algorithm that facilitates normal propagation. Initially, the method focuses on view-independent colors, gradually enabling reflection optimization. Reflective Gaussians with near-correct normals propagate their normal vectors to nearby Gaussians, ensuring that the entire reflective surface achieves accurate normal estimations.

Deferred shading in this context offers several advantages:

  • Improved Normal Estimations: By deferring shading to the pixel level, the method achieves pixel-precise normal maps, crucial for accurate reflections.

  • Enhanced Reflection Quality: The approach generates smoother and more detailed reflections by leveraging more reflection samples at the pixel level.

The research team will be presenting at SIGGRAPH this year, and the code has already been published, although it is currently without a license. Their project page can also be found here.

I envision incorporating this technique into more web-based design, using Three.js to achieve highly accurate reflections and silky smooth rendering rates for users. Reflection-based papers have become some of my favorites due to their visual impact and their potential to bring us closer to a world where hyper-realistic details are fully captured.

Featured

Featured

Featured

Platforms

splatviz: Interactive 3DGS Viewer and Python Editor

splatviz, an innovative interactive viewer available on GitHub offers real-time editing and visualization capabilities using a native Python GUI library.

Michael Rubloff

Jul 17, 2024

Platforms

splatviz: Interactive 3DGS Viewer and Python Editor

splatviz, an innovative interactive viewer available on GitHub offers real-time editing and visualization capabilities using a native Python GUI library.

Michael Rubloff

Jul 17, 2024

Platforms

splatviz: Interactive 3DGS Viewer and Python Editor

splatviz, an innovative interactive viewer available on GitHub offers real-time editing and visualization capabilities using a native Python GUI library.

Michael Rubloff

News

Amazon Prime Day Sales

Amazon Prime Day is here and with it are some deals that will help you train more NeRFs/3D Gaussian Splatting captures!

Michael Rubloff

Jul 16, 2024

News

Amazon Prime Day Sales

Amazon Prime Day is here and with it are some deals that will help you train more NeRFs/3D Gaussian Splatting captures!

Michael Rubloff

Jul 16, 2024

News

Amazon Prime Day Sales

Amazon Prime Day is here and with it are some deals that will help you train more NeRFs/3D Gaussian Splatting captures!

Michael Rubloff

Platforms

Three.js-based implemetation, GaussianSplats3D adds 2DGS Support

The Three.js-based implementation of Gaussian Splatting has officially added 2DGS support in v0.4.3.

Michael Rubloff

Jul 16, 2024

Platforms

Three.js-based implemetation, GaussianSplats3D adds 2DGS Support

The Three.js-based implementation of Gaussian Splatting has officially added 2DGS support in v0.4.3.

Michael Rubloff

Jul 16, 2024

Platforms

Three.js-based implemetation, GaussianSplats3D adds 2DGS Support

The Three.js-based implementation of Gaussian Splatting has officially added 2DGS support in v0.4.3.

Michael Rubloff

Research

3D Gaussian Ray Tracing

Today, things are taking an exciting step forward with the introduction of 3D Gaussian Ray Tracing (3DGRT).

Michael Rubloff

Jul 10, 2024

Research

3D Gaussian Ray Tracing

Today, things are taking an exciting step forward with the introduction of 3D Gaussian Ray Tracing (3DGRT).

Michael Rubloff

Jul 10, 2024

Research

3D Gaussian Ray Tracing

Today, things are taking an exciting step forward with the introduction of 3D Gaussian Ray Tracing (3DGRT).

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff