SeaThru NeRF added to nerfstudio

Quietly over the weekend, underwater NeRF method, SeaThru NeRF has been added as an officially supported method to nerfstudio. We already know that radiance fields can model exactly what they see, but we typically imagine that the locations are your everyday interiors or exteriors. But what happens when that place is underwater?

SeaThru NeRF is one of the cooler papers to come out of last year and was submitted to CVPR. It might sound a little unbelievable that there are NeRF methods that specifically cater to underwater footage, but it really works! I have a few friends that have assembled diving footage and it's better than you might imagine.

Now SeaThru NeRF is available natively within nerfstudio. There are two variations of Sea Thru NeRF, a full version and a lite version. The full version requires ~23 GB of VRAM, so you'll need to be using at least a 3090. However, the lite version only consumes 7 GB, making it usable with a wide variety of NVIDIA GPUs. Originally, the published method was built on top of Mip-NeRF 360, whereas this implementation has been transferred onto nerfacto. This implementation is 65-90% faster than the original!

Traditional NeRFs and their variants have focused on clear air environments, neglecting the complexities introduced by scattering media like water or fog. These media significantly alter the appearance of objects, making accurate rendering a challenging task. The SeaThru NeRF model overcomes these challenges by assigning separate color and density parameters to both the object (scene) and the medium within the NeRF framework.

SeaThru NeRF integrates a sophisticated rendering model that accounts for the scattering properties of media. This model splits the scene into clean and backscatter components, enabling the rendering of scenes with or without the medium's influence. Backscatter refers to the phenomenon where particles in a medium, like water, scatter light back towards the source. Particularly in underwater scenarios, this approach effectively removes backscatter and can accurately simulate how light interacts with particles in media like water.

The key innovation in SeaThru NeRF is the separation of the scene into two distinct components:

• Clean Object Component: This represents the part of the scene unaffected by the scattering medium, capturing the true appearance of objects.

• Backscatter Component: This component represents the light that is scattered back towards the camera due to the medium, which usually obscures visibility and distorts colors.

In traditional NeRF models, only the object's color and density are considered. SeaThru NeRF extends this by also assigning color and density parameters to the medium itself. This dual parameter system is critical for accurately modeling the interaction between light, objects, and the medium. The volumetric rendering equations are adapted to consider the effects of the scattering medium. Unlike conventional NeRFs, where the focus is solely on objects, SeaThru NeRF takes into account the entire volume of the scene, including both the object and the medium, thus providing a more comprehensive rendering.

Interestingly, their method takes on structure from motion with scattering media. Traditional SfM has struggled with depth estimation in scattering media due to poor visibility and lack of distinct features. SeaThru NeRF, however, can estimate the depth and structure of the entire scene, including areas with bad visibility. It does this by reconstructing and reasoning about both the geometry of the scene and the properties of the medium.

While the name and method imply underwater scenes, SeaThru NeRF actually applies to any scenes where the air and light itself is either absorbs and / or scatters. Other similar scenes could be fog, haze, or smog. If only I could've tried SeaThru NeRF while living in Los Angeles.

If you don't happen to have a bunch of underwater footage to try out for yourself, the SeaThru NeRF dataset is available to download.

The nerfstudio team continues to quietly add on features, since the release of nerfstudio 1.0. It is exciting to witness more and more methods that can easily be tried by people. This is a perfect example of why nerfstudio is such a valuable platform.