Gaussian Splatting Drums Up a Music Lesson

Amidst the surge of innovative research, we highlight a particularly creative application of radiance fields that has caught our attention. Paolo Tosolini, María Sánchez Isaza, and the Bellevue College XR Lab have created an interactive photorealistic drumset with educational components.

They’ve employed Gaussian Splatting, a radiance field method that takes 2D images and outputs a high-fidelity 3D capture, to create a digital twin of the drum set. This project takes advantage of Spline, a powerful tool for 3D design and animation, to make this intricate model accessible on the web. Spline natively supports Gaussian Splatting.

https://twitter.com/tosolini/status/1763067220094062622

Users can virtually play the drum set by tapping on the drum pads or using keyboard keys, each action producing realistic sound notes. This interactive model doesn’t just mimic the physical experience of playing drums but also serves as an educational tool. It introduces users to the various components that comprise a drum kit, while still being physically accurate.

The team ventured into the field to capture the drum set using Postshot‘s Gaussian Splatting implementation. The raw radiance field underwent a cleaning process with PlayCanvas’s SuperSplat, refining the model for its final presentation in Spline. The drum set is fully interactive with different notes and the ability to play the notes, either by pressing on the drum pads or through keys.

While it is widely speculated that the drumset is auditioning to join the band, the NeRFs, they have indicated they have no intention of replacing their drummer. However, it’s clear that the real purpose of this project extends far beyond musical ambitions.

This isn’t the first time Paolo Tosolini has harnessed the power of radiance fields for creative endeavors. Late last year, he utilized this technology to curate a virtual exhibition for a museum, showcasing the versatility and potential of radiance fields in presenting information and art in engaging, innovative ways.

Radiance fields are proving to be a fertile ground for educational and creative applications alike. By presenting information in a lifelike, interactive format, they open up new avenues for learning and engagement. What future innovations might emerge from this confluence of art and science, further enriching our understanding and appreciation of the world? The possibilities are as boundless as our imagination.

The drumset can be viewed here.