Gaussian Splashing Splashes Down

When physics based radiance fields landed last year, they certainly made a splash with PhysGaussian and PieNeRF. Now, a follow up paper is taking it up a notch with Gaussian Splashing.

They are able to incorporate dynamic fluids into gaussian splatting and it unlocks so many things. It definitely gets me thinking about what you can do with this paired with GARField.

With their last paper, I got excited about jam spreading itself across a piece of toast and introduced the concept of What you see, is what you simulate. With Gaussian Splashing, I have now seen another thing I never thought that I would. Here is a gaussian splatted cup, which is being filled with water, before dissolving itself.

While the endpoint is unique, the beginning of Gaussian Splashing is the same as all radiance field methods. They takes a collection of 2D images of a 3D scene, each featuring different viewpoints. From there, the preprocessing stage begins and it separates foreground objects from the background. and extracts their surfaces.

GSP uses advanced deep segmentation tools to isolate foreground objects and create a detailed reconstruction of them. Gaussian kernels are placed inside and on the surface of these models, acting as mass particles for simulation. As part of understanding where the boundaries lie, they need to have accurate representations and for this Gaussian Splashing turns to Neus. I would be very curious how this would work with a similar model, such as UniSDF or Instant Angelo.

During the actual training, they use a differentiable version of 3DGS, so that their visual attributes are taken into account. Normal Gaussian Splatting cannot be used here because it does not come with well defined appearance properties.

GSP employs Lagrangian fluids and position-based fluids (PBF) to simulate fluid dynamics. It also includes a position-based tension model to better capture the dynamics of the fluid surface. This approach is essential for creating realistic fluid movements and interactions with solid objects.

To prevent rendering artifacts, especially near the surfaces of solid objects, GSP employs anisotropy loss and regularization. This controls the shape of Gaussian kernels to avoid them becoming excessively elongated or compressed. To address the issue of displaced objects exposing unseen areas, the system uses an inpainting technique called Resolution-robust large mask inpainting with fourier convolutions (LaMA). It removes Gaussian kernels of the displaced object and uses a generative AI approach to inpaint these areas, improving the overall visual quality of the rendered image so that nothing looks out of the ordinary.

It makes sense why you would need that area to be inpainted. When an object is removed, how would it know what should be there?

Gaussian Splashing is accomplished with consumer grade hardware— the researchers are using 3090s and it doesn't take an unusually long time to train. Most of the examples shown in the paper/demo range from just over half an hour, going all the way up to two and a half, which is still faster than nerfacto-huge.

I, for one, am excited to see the recreation of Titanic, sans original boat and actual water. However, you can also expand this to look at simulations and evacuation methods.

Physics are how the world works, so it is always a pleasure to read about how the most lifelike models of our world that we have, can also be subject to the same rules. I can't wait for more methods to build on top of this, so that we can truly be transported and interact with the world as we see it, virtually.

Neither the code for Gaussian Splashing or supplementary video have been made available yet, but I am excited for both. You can join me in refreshing their project page, here.