Spectacular AI

Spectacular AI provides an SDK for low-level embedded spatial AI. They have tools that work on multiple different types of devices: Android and iOS phones; popular robotics and drone cameras such as OAK-D, RealSense, and Orbbec/Kinect devices; and completely custom hardware, all with the same output API. Everything except the tools for custom hardware are freely available for testing and non-commercial purposes.

When using the SDK or recording apps, you get access to the full raw data: IMU samples, video data and metadata, which you can either feed into Spectacular AI Mapping Tools for pose registration and key frame selection, or directly use the raw data for research purposes. Both approaches have been used in multiple research papers, and we also actively collaborate with different universities in this.

Benefits of using Spectacular AI tools for 3DGS or NeRFs is that they function as a more robust alternative to COLMAP that also retain information about the metric scale and orientation of the scene, based on IMU data. We also actively participate in 3DGS research and provide certain cutting-edge features that are not currently available in any other software, like the “Gaussian Splatting on the Move” deblurring and rolling shutter compensation method. We are also planning to augment our tools and their related open source parts with other brand new 3DGS extensions in the near future, for example, better handling of external (ToF) depth information.

We have also recently contributed to the gsplat and Nerfstudio Open Source 3DGS implementations by implementing the pose optimization fork that powered both Gaussian Splatting on the Move and BAD-Gaussians.