Platforms

EasyVolcap: a 4D nerfstudio?

Michael Rubloff

Jan 2, 2024

EasyVolcap
EasyVolcap

2024 has been ushered in with several announcements overnight. The one that caught my attention the most was EasyVolcap. EasyVolcap is a Pytorch library for accelerating neural volumetric video research.

It specializes in volumetric video capturing, reconstruction, and rendering, akin to nerfstudio but tailored for 4D applications. At this time, nerfstudio does not support multi view video datasets.

It's difficult for me to articulate how impactful this can prove to be. Use cases range from live sports, healthcare, but mostly providing a unified method for both researchers and creatives to begin leveraging. I personally am very curious what a fully equipped Jake Oleson could do with EasyVolcap.

There is an urgent need for a unified open-source library to streamline the process of volumetric video capturing, reconstruction, and rendering for both researchers and non-professional users to develop various algorithms and applications of this emerging technology.EasyVolcapTweet

EasyVolcap could be a catalyst for more personal, "at-home" captures, enabling everyday users to create photorealistic 4D content. In September of last year, we examined Infinite Realities and their pipeline. We discussed some of the potential applications of their work and we can see another avenue for people to create emerge today.

The GUI is very similar to Instant NGP and just like Instant NGP, you can create keyframes throughout the scene to render out a video. The significant divergence here is the integration of a 4D system, incorporating time as a critical component.

If the authors seem familiar, it might be because they were behind 4K4D, which we looked at earlier this year. As part of this release, they have additionally open sourced 4K4D and made its data available. You might have expected given that 4K4D would be a supported method. It is.

EasyVolcap VRAM Requirements

A notable feature of EasyVolcap is its compatibility with consumer-grade hardware, demonstrated by its modest VRAM requirements (5 GB) in a demo run on a 3060 GPU. While performance will vary with longer or higher resolution inputs, the initial demonstrations are promising. Interestingly, the input method is still just 2D images. I imagine that the folder and file sizes very quickly scale in size as additional cameras are added. I haven't had a chance to try it out for myself yet, but it's safe to say my first week back plans have meaningfully been altered.

EasyVolcap Installation

EasyVolcap can be installed from their Github under this section.

How EasyVolcap Works

As it name implies, the method is somewhat straightforward. As you feed it a standard captured video, it begins to take samples of space-time rays for rendering. These ray samples are use to generate 4D point coordinates, which are then encoded with space-time embedders. They are then decoded for their physical properties, such as their density and color and then finally fed into a volume renderer.

I have had a hard time reconciling the earlier statement about this only consuming a manageable amount of VRAM. So, how do they actually achieve that? They heuristically swap out the VRAM with the immediately following frames to be played on the pinned main memory with asynchronous and non-blocking copy, which in turn gives you a manageable memory footprint without pushing you into commercial grade GPUs.

EasyVolcap Licensing

The license to use EasyVolcap is not permissive, but honestly I don't blame them. Right now their license allows for use, copy, modify and distribute this software and its documentation for educational, research and non-profit purposes only. Any modification based on this work must be open-source and prohibited for commercial use. You must retain, in the source form of any derivative works that you
distribute, all copyright, patent, trademark, and attribution notices from the source form of this work. In order to utilize EasyVolcap for commercial applications, they ask you email: xwzhou@zju.edu.cn.

The inspiration from nerfstudio seems to be solidified and potentially intentional with EasyVolcap acknowledging nerfstudio in their documentation as well as the introduction of their paper. I will be curious to see whether or not future 4D applications will build ontop, but given the precedent of nerfstudio, I think it's likely.

Only time will tell if EasyVolcap will do what Instant NGP, Luma, and nerfstudio did for the first wave of NeRF based creators, but for 4D applications. Similar to those early days, I wouldn't expect the installation and interfaces to be as smooth as they are now for NeRFs/3DGS. They've come a long way over the course of the past ~1.5 years, but getting the platform up and running was part of the fun! EasyVolcap fills a hole in the market that has existed, and now it's time for me to begin thinking through my own library of captures to leverage dynamic scenes and stories that will inevitably be told.

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Trending articles

Trending articles

Trending articles

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024