EasyVolcap: a 4D nerfstudio?

2024 has been ushered in with several announcements overnight. The one that caught my attention the most was EasyVolcap. EasyVolcap is a Pytorch library for accelerating neural volumetric video research.

It specializes in volumetric video capturing, reconstruction, and rendering, akin to nerfstudio but tailored for 4D applications. At this time, nerfstudio does not support multi view video datasets.

It's difficult for me to articulate how impactful this can prove to be. Use cases range from live sports, healthcare, but mostly providing a unified method for both researchers and creatives to begin leveraging. I personally am very curious what a fully equipped Jake Oleson could do with EasyVolcap.

EasyVolcap could be a catalyst for more personal, "at-home" captures, enabling everyday users to create photorealistic 4D content. In September of last year, we examined Infinite Realities and their pipeline. We discussed some of the potential applications of their work and we can see another avenue for people to create emerge today.

The GUI is very similar to Instant NGP and just like Instant NGP, you can create keyframes throughout the scene to render out a video. The significant divergence here is the integration of a 4D system, incorporating time as a critical component.

If the authors seem familiar, it might be because they were behind 4K4D, which we looked at earlier this year. As part of this release, they have additionally open sourced 4K4D and made its data available. You might have expected given that 4K4D would be a supported method. It is.

EasyVolcap VRAM Requirements

A notable feature of EasyVolcap is its compatibility with consumer-grade hardware, demonstrated by its modest VRAM requirements (5 GB) in a demo run on a 3060 GPU. While performance will vary with longer or higher resolution inputs, the initial demonstrations are promising. Interestingly, the input method is still just 2D images. I imagine that the folder and file sizes very quickly scale in size as additional cameras are added. I haven't had a chance to try it out for myself yet, but it's safe to say my first week back plans have meaningfully been altered.

EasyVolcap Installation

EasyVolcap can be installed from their Github under this section.

How EasyVolcap Works

As it name implies, the method is somewhat straightforward. As you feed it a standard captured video, it begins to take samples of space-time rays for rendering. These ray samples are use to generate 4D point coordinates, which are then encoded with space-time embedders. They are then decoded for their physical properties, such as their density and color and then finally fed into a volume renderer.

I have had a hard time reconciling the earlier statement about this only consuming a manageable amount of VRAM. So, how do they actually achieve that? They heuristically swap out the VRAM with the immediately following frames to be played on the pinned main memory with asynchronous and non-blocking copy, which in turn gives you a manageable memory footprint without pushing you into commercial grade GPUs.

EasyVolcap Licensing

The license to use EasyVolcap is not permissive, but honestly I don't blame them. Right now their license allows for use, copy, modify and distribute this software and its documentation for educational, research and non-profit purposes only. Any modification based on this work must be open-source and prohibited for commercial use. You must retain, in the source form of any derivative works that you
distribute, all copyright, patent, trademark, and attribution notices from the source form of this work. In order to utilize EasyVolcap for commercial applications, they ask you email: xwzhou@zju.edu.cn.

The inspiration from nerfstudio seems to be solidified and potentially intentional with EasyVolcap acknowledging nerfstudio in their documentation as well as the introduction of their paper. I will be curious to see whether or not future 4D applications will build ontop, but given the precedent of nerfstudio, I think it's likely.

Only time will tell if EasyVolcap will do what Instant NGP, Luma, and nerfstudio did for the first wave of NeRF based creators, but for 4D applications. Similar to those early days, I wouldn't expect the installation and interfaces to be as smooth as they are now for NeRFs/3DGS. They've come a long way over the course of the past ~1.5 years, but getting the platform up and running was part of the fun! EasyVolcap fills a hole in the market that has existed, and now it's time for me to begin thinking through my own library of captures to leverage dynamic scenes and stories that will inevitably be told.