News

Spectacular AI: Deblurring Gaussians

Michael Rubloff

Michael Rubloff

Mar 28, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Spectacular AI
Spectacular AI

I have said it time and time again, but nothing kills a radiance field faster than blurry images. Or so it's been so far. There have been a couple efforts popping up to make blurry images usable and disprove the often cited film maxim, garbage in, garbage out.

Unlike the high-end equipment used by professional cinematographers, most people today capture radiance fields with what's readily available—namely, their smartphones. Traditionally, creating high-quality 3D scene reconstructions and novel view syntheses required steady, high-quality photographs—a tall order for handheld devices prone to motion blur, rolling shutter effects, and pose noise.

Most cameras operate with rolling shutters, meaning their sensors capture data in a sequential roll across the frame. Although the delay between each segment is just microseconds, it can still significantly impact accuracy.

These phenomena, stemming from the camera's and objects' relative motion and the camera sensors' line-by-line scanning, can degrade the quality of reconstructed scenes. "Gaussian Splatting on the Move" presents a breakthrough by leveraging camera motion for high-quality reconstruction from handheld video data.

Their method models the physical image formation process by taking into account the camera shutter’s exposure time and velocity estimated using visual-inertial odometry (VIO) during image capture. Modelling these effects during 3DGS training can recover sharpened versions of input images, and can also be used to simulate motion blur and rolling shutter effects in the 3D Gaussian Splatting (3DGS) framework as a post-processing effect.

Essentially, they're using the camera's own movement to help make the final image much clearer. The proposed differentiable rendering pipeline efficiently incorporates these effects, demonstrating superior performance over traditional methods in both synthetic and real-world scenarios.

The paper comes from Spectacular AI, who provides an SDK for low-level embedded spatial AI, easy tools, scripts, and code for processing data captured with iPhones, Android devices, Kinect devices, and other RGB-D sensors used in robotics and industry.

The research was also conducted in part by Matias Turkulainen who is also one of the authors of Nerfstudio's Splatfacto, so it shouldn't be a surprise that they have implemented their method as an extension to the Nerfstudio and gsplat, providing a practical tool for advancing 3DGS in naturalistic settings.

Spectacular AI's compatibility with Nerfstudio simplifies adoption. Designed with the consumer in mind, the method was trained on a GPU with 16 GB of VRAM. It operates efficiently, completing tasks in about 20 minutes—marginally longer than Splatfacto under normal conditions. The code is available for immediate use with Nerfstudio's Splatfacto method. In order to capture usable data, you need to use Spectacular AI's free app on either iOS or Android.

With the amount of capture based startups that have emerged over the course of the last year, I imagine that this problem is high up on their lists. By raising the floor of consumer capture, a larger band of people will see success from what might have started as unusable footage. Further, if there are ways to restore and capture more details, I think we can all agree that it makes sense to pursue it.

Featured

Featured

Featured

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Jul 3, 2024

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Jul 3, 2024

Platforms

Nerfstudio Integrates 3DGS MCMC

Nerfstudio now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

Michael Rubloff

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Jul 2, 2024

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Jul 2, 2024

Research

Diffusion Based 3DGS Relighting

Research topics on relighting continue to increase sharply.

Michael Rubloff

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

Jul 1, 2024

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

Jul 1, 2024

Platforms

Voluma.ai adds Timelapse Mode

Voluma.ai, specializing in the creation and augmentation of 3D graphics, has introduced its latest feature: timelapse mode.

Michael Rubloff

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Jun 28, 2024

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Jun 28, 2024

News

GeoWeek to Present Free Webinar on Radiance Fields

The event promises to be an engaging and informative session with a strong lineup of panelists.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff

Nov 28, 2023

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff

Nov 28, 2023

Research

Mip-Splatting: Anti-Aliasing for Gaussian Splatting

If you've been trying out Gaussian Splatting, you may have noticed that the scene tends to degrade quickly, especially when you pull outside of the capture path or change the focal length, compared to NeRF.

Michael Rubloff