News

Spectacular AI: Deblurring Gaussians

Michael Rubloff

Michael Rubloff

Mar 28, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
Spectacular AI
Spectacular AI

I have said it time and time again, but nothing kills a radiance field faster than blurry images. Or so it's been so far. There have been a couple efforts popping up to make blurry images usable and disprove the often cited film maxim, garbage in, garbage out.

Unlike the high-end equipment used by professional cinematographers, most people today capture radiance fields with what's readily available—namely, their smartphones. Traditionally, creating high-quality 3D scene reconstructions and novel view syntheses required steady, high-quality photographs—a tall order for handheld devices prone to motion blur, rolling shutter effects, and pose noise.

Most cameras operate with rolling shutters, meaning their sensors capture data in a sequential roll across the frame. Although the delay between each segment is just microseconds, it can still significantly impact accuracy.

These phenomena, stemming from the camera's and objects' relative motion and the camera sensors' line-by-line scanning, can degrade the quality of reconstructed scenes. "Gaussian Splatting on the Move" presents a breakthrough by leveraging camera motion for high-quality reconstruction from handheld video data.

Their method models the physical image formation process by taking into account the camera shutter’s exposure time and velocity estimated using visual-inertial odometry (VIO) during image capture. Modelling these effects during 3DGS training can recover sharpened versions of input images, and can also be used to simulate motion blur and rolling shutter effects in the 3D Gaussian Splatting (3DGS) framework as a post-processing effect.

Essentially, they're using the camera's own movement to help make the final image much clearer. The proposed differentiable rendering pipeline efficiently incorporates these effects, demonstrating superior performance over traditional methods in both synthetic and real-world scenarios.

The paper comes from Spectacular AI, who provides an SDK for low-level embedded spatial AI, easy tools, scripts, and code for processing data captured with iPhones, Android devices, Kinect devices, and other RGB-D sensors used in robotics and industry.

The research was also conducted in part by Matias Turkulainen who is also one of the authors of Nerfstudio's Splatfacto, so it shouldn't be a surprise that they have implemented their method as an extension to the Nerfstudio and gsplat, providing a practical tool for advancing 3DGS in naturalistic settings.

Spectacular AI's compatibility with Nerfstudio simplifies adoption. Designed with the consumer in mind, the method was trained on a GPU with 16 GB of VRAM. It operates efficiently, completing tasks in about 20 minutes—marginally longer than Splatfacto under normal conditions. The code is available for immediate use with Nerfstudio's Splatfacto method. In order to capture usable data, you need to use Spectacular AI's free app on either iOS or Android.

With the amount of capture based startups that have emerged over the course of the last year, I imagine that this problem is high up on their lists. By raising the floor of consumer capture, a larger band of people will see success from what might have started as unusable footage. Further, if there are ways to restore and capture more details, I think we can all agree that it makes sense to pursue it.

Featured

Featured

Featured

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Jul 26, 2024

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Jul 26, 2024

Research

Frustum Volume Caching

A criticism of NeRFs is their rendering rates. Quietly a couple of papers have been published over the last two months which push NeRFs into real time rates.

Michael Rubloff

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Jul 24, 2024

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Jul 24, 2024

Research

N-Dimensional Gaussians for Fitting of High Dimensional Functions

It significantly improves the fidelity of reflections and other view-dependent effects, making scenes look more realistic.

Michael Rubloff

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Jul 22, 2024

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Jul 22, 2024

Platforms

Luma AI launches Loops for Dream Machine

Luma AI is starting the week off hot, with the release of Loops.

Michael Rubloff

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Jul 18, 2024

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Jul 18, 2024

Platforms

SuperSplat adds Histogram Editing

PlayCanvas is back with a new update to SuperSplat. It's the release of v0.22.2 and then the quick update to v0.24.0.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

Jun 7, 2024

Platforms

Nerfstudio Releases gsplat 1.0

Just in time for your weekend, Ruilong Li and the team at Nerfstudio are bringing a big gift.

Michael Rubloff

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

May 14, 2024

News

SIGGRAPH 2024 Program Announced

The upcoming SIGGRAPH conference catalog has been released and the conference will be filled of radiance fields!

Michael Rubloff

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff