Research

VR-GS: Physics Based Gaussian Splatting in VR

Michael Rubloff

Michael Rubloff

Feb 5, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
VR-GS
VR-GS


There has been a surge in interest in VR based radiance field methods since the release of the Apple Vision Pro. I admittedly spent significantly more time in VR than I had alotted myself, contributing to this article being published a few days after its release.



In a nutshell, VR-GS expands the work done with PhysGaussian last year and extends it into the world of virtual reality. Yes that means that you can interact with your captures in a physically accurate way.





The authors took great care to ensure that VR-GS is immersive, real-time, has a user-friendly VR system, and has a unified framework. That last part is interesting to me, because they handle both the rendering and the physics simulation in the same pipeline. It goes back to the concept of what you see is what you simulate. That phrase has appeared consistently through their work. VR-GS also extends real world physics to produce lifelike interactions and manipulations of a scene, as opposed to using generative AI. VR-GS additionally introduces sophisticated segmentation and inpainting techniques, crucial for enhancing the system's immersive and user-friendly VR experience.



We've seen a few platforms and methods emerge for virtual reality based radiance fields, but this one has as slight distinction. Other methods, such as VR-NeRF, Magic NeRF Lens, and most recently MetalSplatter focus on the representation of the scene. What's really interesting about VR-GS is that it asks the question about how can people interact with the scene itself.



The core innovation of the VR-GS (Virtual Reality-Gaussian Splatting) system lies in its groundbreaking approach to interactive 3D content manipulation and rendering within virtual reality environments. This innovation is centered around the integration of Gaussian Splatting (GS) with physics-aware dynamics to facilitate real-time interaction with deformable virtual entities and intricately detailed environments. Let's delve into the specifics of this innovation and its implications for the future of VR technology.



As the name implies, at the heart of VR-GS is 3D Gaussian Splatting, which takes a series of 2D images to create lifelike three dimensional representations of what it sees.






The integration of eXtended Position-based Dynamics (XPBD) with GS kernels introduces a layer of real-time physics-based simulation that is adaptable and highly efficient. XPBD, a cutting-edge physical simulator, allows for the simulation of deformable bodies with realistic dynamic responses. This combination ensures that interactions within the VR environment not only look visually compelling but also behave in accordance with physical laws, enhancing the realism of the virtual experience.



Another key aspect of VR-GS's innovation is its two-level embedding strategy, designed to handle the deformation of GS kernels. This strategy involves constructing a tetrahedral cage for each segmented GS kernel group and embedding these groups into corresponding meshes. The deformation of these meshes, driven by XPBD, guides the deformation of the GS kernels, ensuring smooth and realistic movements. This approach effectively resolves potential artifacts that could arise from simpler embedding techniques, allowing for high-quality, dynamic rendering of complex scenes. It is also what empowers the real time editing and manipulation. Interestingly, splatfacto from nerfstudio flagged this same exact issue, which was taken from PhysGaussian (same author)!



Another standout feature is VR-GS's segmentation capability. Segmentation meticulously identifies and isolates various objects within a 3D scene, allowing users to interact with and manipulate individual elements with unparalleled precision. This process is vital for creating a truly interactive VR experience where every object behaves as it would in the real world, adhering to the laws of physics.



However, interaction often leads to the manipulation of objects, potentially exposing areas that were occluded or not captured in the original dataset. This is where inpainting comes into play. VR-GS leverages advanced inpainting techniques to dynamically fill in these gaps, ensuring that the visual integrity of the scene remains intact even as objects are moved or removed. Inpainting is essential for maintaining the continuity of the virtual environment, allowing for seamless edits that contribute to the overall realism of the experience.



Some cool, small details is that they go as far to include dynamic shadow casting and real time deformation embedding. Shadow maps are employed to add dynamic shadows to a 3D scene, enhancing the visual realism and spatial awareness crucial for immersive virtual reality experiences.



The process begins with the generation of a depth map from the perspective of the light source. This map records the closest distance from the light to the surfaces in the scene, effectively capturing which parts of the scene are directly illuminated and which are occluded. When rendering the scene from the user's viewpoint, the system compares the depth of each point in the scene (as seen from the light source) against the stored depth map. Points that match or are closer than the depth map are considered lit, while those further away are in shadow. As objects move or the light source changes position, the VR-GS system dynamically updates the depth map and recalculates the shadows. This ensures that shadows accurately reflect the current state of the environment, contributing to a realistic and responsive VR experience.






The unique challenge in VR-GS comes from integrating shadow maps with Gaussian Splatting. Given GS's explicit representation of objects using Gaussian kernels, the system must adeptly handle the calculation of shadows cast by and onto these kernels. This is achieved through careful adaptation of the shadow mapping technique, ensuring that the splatted objects not only render with high fidelity but also cast and receive shadows realistically. All of this, so that we can feel more immersed!



On top of that VR-GS is fast. Like really fast.



You might also be wondering what's powering this? Is it a 20 GPU workstation like in VR-NeRF? Not exactly. It's actually being powered within Unity on a 4090 and a Quest Pro. From parts of their experiments they created a scene where the objective is to pet a fox and this quote stood out to me.




"I love this a lot. I have a dog. Petting the fox is really like what I did to my dog at home. I(t) felt so real. VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual RealityTweet



Imagine a world where there are infinite lifelike animals to pet!



There's so much left to conquer in the world of radiance fields and VR, but I always get excited when I see methods and work like this which are charging head on into the fray of exploration. There's been no word yet on whether or not VR-GS will release and when, but given the length that the authors discuss design choices in the paper, I believe we will hopefully get access to it soon.


Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV