Research

Mesh Based Gaussian Splatting

Michael Rubloff

Michael Rubloff

Feb 11, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Mesh Based Gaussian Splatting
Mesh Based Gaussian Splatting

The pursuit of making Gaussian Splatting editable in real-time, without compromising its renowned high visual fidelity, represents an exciting topic in computer graphics research. This ambition has seen a surge in scholarly attention, evidenced by pioneering works such as PhysGaussian, Gaussian Splashing, and SuGaR.

With all of that in mind, we're getting another method pairing a mesh based method, with Gaussian Splatting.

A primary obstacle in editable Gaussian Splatting is ensuring topological guidance for Gaussians, enabling them to inform neighboring sections of the representation about movement dynamics without sacrificing appearance. Traditional approaches often led to a dichotomy: either achieve deformability at the expense of fidelity or vice versa.

Addressing this challenge, they pair 3D Gaussians with an explicit mesh, creating a symbiotic relationship that bolsters both the rendering process and the adaptive refinement of the mesh. This integration enables direct deformations to be applied to the Gaussians, fostering a dynamic interaction between the mesh and its Gaussian components. The goal is to bind it as tightly as possible, so that there's as little variation between the two. This binding serves a dual purpose: it guides the rendering process of 3D Gaussians, ensuring high visual fidelity, and directs the adaptive refinement of the mesh, crucial for capturing detailed geometric and topological nuances.

Envision sculpting with digital clay, where you aim for the dual goals of artistic freedom and lifelike realism. Traditional methods often forced a compromise: flexibility at the cost of realism, or vice versa. This core dilemma—maintaining a sculpture's lifelike appearance while freely reshaping it—mirrors the challenge in editing 3D scenes in real-time.

To solve this, imagine if the digital clay could tell its neighboring pieces how to move to keep the sculpture looking lifelike even when being reshaped. That's what the latest method does by combining two key ingredients: a mesh (a framework that gives the sculpture its shape) and Gaussian Splatting (a technique that adds the lifelike details).

By "pairing" these two, every adjustment you make to the shape (through the mesh) automatically updates the details (through Gaussian Splatting) to keep the figure looking realistic. It's as if the digital clay now has a built-in guide to maintain its realism, no matter how you reshape it.

The cornerstone of their approach is a large-scale Gaussian deformation technique that modifies the parameters of 3D Gaussians in accordance with the manipulation of the associated mesh. This method benefits significantly from existing mesh deformation datasets, allowing for more realistic, data-driven Gaussian deformation.

Here there are two main considerations that make this work: Face Splits and Normal Guidance.

Face Splits are integral to the method because they allow for the adaptive refinement of the mesh, which in turn, supports the detailed and dynamic representation of 3D scenes. By subdividing mesh faces, the method can increase the density of Gaussians in areas requiring higher detail, ensuring that the model's complexity and subtlety are accurately represented and capturing complex geometries and topologies accurately. The face splits are instructed to happen if certain criteria are met. This step is critical to the actual deformation that will take place in the scene.

The second major consideration is how to trade off the deformation ability with its ability to still maintain a certain fidelity level. This is governed by Normal Guidance. Each Gaussian is not strictly bound to lie on the surface of the mesh. Instead, Normal Guidance allows these Gaussians to be translated along the normal—the perpendicular direction—of the mesh surface they are associated with. This slight adjustment in positioning enables the Gaussians to better capture and represent the nuances of the scene's geometry.

The synergy between Face Splits and Normal Guidance underpins the method's ability to adaptively refine the mesh and precisely position Gaussians for optimal rendering. Face Splits increase the mesh's resolution where needed, while Normal Guidance ensures that the enhanced detail is captured by appropriately positioning the Gaussians. Together, these mechanisms enable the method to dynamically adjust to the scene's complexity, ensuring high fidelity in the visualization of 3D scenes. From this, they've developed an otherwise unnamed Interactive Tool to showcase modifications.

All of this still results in an output of 65fps on a 4090 GPU. While it's still at the upper end of consumer GPUs, this serves as a strong starting point to build on top of. This manipulation could involve moving, rotating, or scaling parts of the mesh to achieve the desired deformation. In order to have this interactivity, the original capture must be trained on their method, but the training time doesn't appear to be any longer than vanilla Gaussian Splatting, taking roughly 10 minutes to train.

As the user manipulates the mesh, the scene is rendered in real-time, providing immediate visual feedback. This allows the user to see the effects of their deformations as they happen, ensuring intuitive and effective scene manipulation.

While the supplementary materials, such as the video demonstration and code release, remain eagerly awaited, the described method shows an exciting foundation of real-time, high-fidelity 3D scene manipulation. By leveraging existing mesh deformation datasets for more realistic, data-driven deformations, this approach paves the way for innovations that could transform interactive graphics, virtual reality, and beyond.

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV