Research

Gaustudio

Michael Rubloff

Michael Rubloff

Apr 8, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Gaustudio
Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Gaustudio wants to be a flexible and powerful Gaussian Splatting toolbox, capable of adding new methods as they are released. On its current release, it already supports meshing. The baseline from which they are operating is already more complete than a large amount of platforms and potentially rivaling Nerfstudio in the variety of methods supported.

There are three main takeaways from what Gaustudio is launching with:

Modular Design: GauStudio is highly modular, allowing various components to be combined or replaced to tailor 3D scene modeling methods. This adaptability extends to different foreground models, background models, and other components.

Hybrid Representation: GauStudio introduces a hybrid Gaussian representation, integrating foreground models with skyball background models. This innovation significantly reduces artifacts in unbounded outdoor scenes and enhances novel view synthesis.

Gaussian Splatting Surface Reconstruction (GauS): A standout feature is the GauS module, a render-then-fuse approach that converts 3DGS inputs into high-fidelity mesh reconstructions without the need for fine-tuning.

Gaustudio addresses challenges in novel view synthesis by enabling tailored modeling pipelines for different tasks. Its architecture encompasses several key stages, including scene initialization, optimization with geometric and sparsity regularizers, enhancement of 3D Gaussians’ representation ability, and scene compression through learnable or geometric pruning.

The framework not only simplifies the integration of 3DGS techniques but also introduces efficient modules for surface reconstruction and customizable background modeling. Particularly, the GauS module easily extracts textured meshes, offering a versatile solution across different 3DGS based methods.

Their Gaussians to Mesh pipeline is quite fast, taking about 90 seconds to complete. Building on the foundation set by SuGaR, they employ a volumetric fusion approach, render the median depth, and fuse it into a mesh using VDBFusion. This method is compatible with most of the existing 3DGS platforms out there and <a href="https://radiancefields.com/kiri-engine-adds-mesh-to-3dgs/" data-type="post" data-id="8740" target="_blank" rel="noreferrer noopener">Kiri Engine is using Gaustudio</a> to power its mesh extraction from captures.

Gaussian Splatting and radiance fields are famous for their ability to model accurate view dependent features, like reflections, shifting light, and shadows. Whereas the original Gaussian Splatting implementation uses Spherical Harmonics to model their view dependent effects, it has been a discussion on the optimal choice.Aras gave a good overview of why Spherical Harmonics are potentially not the best representation choice, given their large memory footprint.

Gaustudio instead opts for Spherical Gaussians for modeling specular components within scenes and Spherical Gaussians for balancing image quality and rendering speed. With these changes, they're able to retain or exceed the same fidelity level, but increase the render speed and drop memory footprint.

However, they also support material and lighting decomposition and are achieving this through associating each point with additional PBR parameters, such as opacity, culling, and roughness in addition to indirect illumination. Also boosting real time rates are Neural Feature Vectors, where each point is associated with a compact Neural Feature Vector and then run through a small multi layer perceptron.

Gaustudio even extends to scenes with sparse input information, such as the one we looked at recently with InstantSplat, with generalized Gaussian Splatting initialization. Just like the name states, they initialize the Gaussians using pre-computed properties from a generalized Gaussian splatting model for further optimization. This helps with the consistency of the scene when looking at a novel viewpoint. I would be very curious to see and compare InstantSplat and given Gaustudio's modular approach, how they could even better be integrated with one another.

As Gaussian Splatting platforms have emerged, one of the big considerations and comparisons have been how each one handles large outdoor scenes, in terms of representing an ever extending skyline. It's quite a challenge to accurately model this through Gaussians, but certainly not impossible. There have not been a ton of open source methods to address this, Gaustudio specifically handles this by incorporating a spherical environment map composed of Gaussians to model the sky separately from the foreground in the rendering model.

It's clear that Gaustudio will support quite a few of the leading Gaussian Splatting research papers, including Gaussian Pro, SuGaR, and Mip-Splatting data. Enabling people to train these methods are part of their roadmap. Additionally as part of the roadmap are the inclusion of papers such as VastGaussian, Scaffold-GS, Triplane-GS, and FSGS.

Almost all of Gaustudio is MIT Licensed, with the exception of the rasterizer. However, Gaustudio is taking steps to prevent any miscommunication by having a flag or warning in the training pipeline to make users aware when they are utilizing non-commercial modules derived from the 3DGS codebase, similar to OpenCV and plans to offer all necessary commercial-free substitute modules for companies, ensuring they can utilize Gaustudio without any licensing conflicts.

Right now Gaustudio has only been tested on Linux, but their team is currently working on a Gradio instance and testing Windows. There is still a lot of work to continue to enhance and build out the platform, but the emergence of another modular and wide ranging model will hopefully allow and encourage people to build larger and larger implementations.

As the methods continue to evolve, it's important that the frameworks remain flexible and modular. With that in mind, Gaustudio is quite flexible and allows it to extend to different foreground models, background models, and other components, offering unprecedented flexibility.

Featured

Featured

Featured

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Oct 5, 2023

Research

Live NeRF Video Calls

Catching up with my sister has been an exercise in bridging distances. She recently moved to Copenhagen, trading the familiar landscapes of our shared childhood for the charming streets of the Danish capital.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV