PhysGaussian gives 3D Gaussian Splatting Physics

I knew the second I saw jam spreading itself across a piece of toast, that this was going to be a good paper. Gaussian Splatting has had a tremendous amount of functionalities added with great speed and today we're seeing an exploration into a new area surrounding real world physics.

In theory, it's always been possible to do, it just required several steps. On the NeRF side, it requires meshing or tetrahydras, similar to what we saw in NeRFshop. With it, you have to construct the geometry, prepare it for simulation, actually simulate it, and finally render it out.

Instead of having two distinct parallel pipelines, PhysGaussian aims to combine them into one unified workflow. This ethos is framed as “what you see is what you simulate” (WS²). This paradigm shift in 3D content creation negates the need for traditional geometry embedding methods, ensuring a more coherent and realistic integration of simulation and rendering. Because the same 3D Gaussian kernels are used for both physics simulation and visual rendering, it greatly simplifies the complex process of creating visual content, while also lowering the chance that something is lost in translation between the two.

The method's versatility extends across a variety of materials, ranging from elastic entities to non-Newtonian fluids, all while introducing kinematic attributes such as velocity and strain, along with mechanical properties like elastic energy, stress, and plasticity. I feel like an infomercial, but wait there's more:

By incorporating a custom Material Point Method (MPM), PhysGaussian not only introduces these kinematic deformation and mechanical stress attributes into these kernels, but also adheres to the principles of continuum mechanics.

With all of these advancements, you might be wondering if this introduces significant computing draw. Even with these added features, PhysGaussians is still able to run with real time performance levels on a 3090.

The Fox has been seen repeatedly in Radiance Field papers since the introduction of Instant-NGP. Researcher Alex Evans, recently explained on X (Twitter), the backstory of the fox for those that are curious.

The method's efficiency stems from its novel approach to kinematics, where physics is imparted directly to 3D Gaussian kernels. This approach allows for the attributes of velocity, strain, and mechanical properties like stress to evolve in line with physical laws. The custom MPM employed ensures that both the physical simulation and visual rendering are driven by these 3D Gaussians, thereby eliminating any disparities between simulation and rendering.

Its ability to unify physical simulation with visual rendering under a single framework opens up new avenues for creating realistic and dynamic 3D content. As the technology continues to evolve, its applications could extend far beyond its current capabilities, paving the way for more advanced and lifelike simulations in various fields.

For instance, adding accurate shadow evolution and handling more complex materials like liquids are areas that could see further development. Additionally, integrating data-driven approaches and neural networks could enhance realism and efficiency and as the paper points out, it is also possible to incorporate LLM's into the fold.

The code has not been released publicly yet, though the authors do seem to indicate that it will be at some point. I will update this article when the code has been released and will include licensing information. I would be extremely curious to see if now this is something that Super Splat or Aras would be interested in implementing to their respective platforms. Something tells me it will be sooner than later that we see publicly available tools that allow everyday users to generate these.

As we approach Thanksgiving here in the US, it's exciting to think about how technologies like PhysGaussian might revolutionize the entertainment we enjoy during these festive times. Articles like this will continue to provide updates on such fascinating developments throughout this week.