Research

SeaThru NeRF added to nerfstudio

Michael Rubloff

Michael Rubloff

Jan 28, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
SeaThru NeRF
SeaThru NeRF

Quietly over the weekend, underwater NeRF method, SeaThru NeRF has been added as an officially supported method to nerfstudio. We already know that radiance fields can model exactly what they see, but we typically imagine that the locations are your everyday interiors or exteriors. But what happens when that place is underwater?

SeaThru NeRF is one of the cooler papers to come out of last year and was submitted to CVPR. It might sound a little unbelievable that there are NeRF methods that specifically cater to underwater footage, but it really works! I have a few friends that have assembled diving footage and it's better than you might imagine.

Now SeaThru NeRF is available natively within nerfstudio. There are two variations of Sea Thru NeRF, a full version and a lite version. The full version requires ~23 GB of VRAM, so you'll need to be using at least a 3090. However, the lite version only consumes 7 GB, making it usable with a wide variety of NVIDIA GPUs. Originally, the published method was built on top of Mip-NeRF 360, whereas this implementation has been transferred onto nerfacto. This implementation is 65-90% faster than the original!

Traditional NeRFs and their variants have focused on clear air environments, neglecting the complexities introduced by scattering media like water or fog. These media significantly alter the appearance of objects, making accurate rendering a challenging task. The SeaThru NeRF model overcomes these challenges by assigning separate color and density parameters to both the object (scene) and the medium within the NeRF framework.

SeaThru NeRF integrates a sophisticated rendering model that accounts for the scattering properties of media. This model splits the scene into clean and backscatter components, enabling the rendering of scenes with or without the medium's influence. Backscatter refers to the phenomenon where particles in a medium, like water, scatter light back towards the source. Particularly in underwater scenarios, this approach effectively removes backscatter and can accurately simulate how light interacts with particles in media like water.

The key innovation in SeaThru NeRF is the separation of the scene into two distinct components:

  • Clean Object Component: This represents the part of the scene unaffected by the scattering medium, capturing the true appearance of objects.

  • Backscatter Component: This component represents the light that is scattered back towards the camera due to the medium, which usually obscures visibility and distorts colors.

In traditional NeRF models, only the object's color and density are considered. SeaThru NeRF extends this by also assigning color and density parameters to the medium itself. This dual parameter system is critical for accurately modeling the interaction between light, objects, and the medium. The volumetric rendering equations are adapted to consider the effects of the scattering medium. Unlike conventional NeRFs, where the focus is solely on objects, SeaThru NeRF takes into account the entire volume of the scene, including both the object and the medium, thus providing a more comprehensive rendering.

Interestingly, their method takes on structure from motion with scattering media. Traditional SfM has struggled with depth estimation in scattering media due to poor visibility and lack of distinct features. SeaThru NeRF, however, can estimate the depth and structure of the entire scene, including areas with bad visibility. It does this by reconstructing and reasoning about both the geometry of the scene and the properties of the medium.

While the name and method imply underwater scenes, SeaThru NeRF actually applies to any scenes where the air and light itself is either absorbs and / or scatters. Other similar scenes could be fog, haze, or smog. If only I could've tried SeaThru NeRF while living in Los Angeles.

If you don't happen to have a bunch of underwater footage to try out for yourself, the SeaThru NeRF dataset is available to download.

The nerfstudio team continues to quietly add on features, since the release of nerfstudio 1.0. It is exciting to witness more and more methods that can easily be tried by people. This is a perfect example of why nerfstudio is such a valuable platform.

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV