Platforms

EasyVolcap: a 4D nerfstudio?

Michael Rubloff

Michael Rubloff

Jan 2, 2024

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
EasyVolcap
EasyVolcap

2024 has been ushered in with several announcements overnight. The one that caught my attention the most was EasyVolcap. EasyVolcap is a Pytorch library for accelerating neural volumetric video research.

It specializes in volumetric video capturing, reconstruction, and rendering, akin to nerfstudio but tailored for 4D applications. At this time, nerfstudio does not support multi view video datasets.

It's difficult for me to articulate how impactful this can prove to be. Use cases range from live sports, healthcare, but mostly providing a unified method for both researchers and creatives to begin leveraging. I personally am very curious what a fully equipped Jake Oleson could do with EasyVolcap.

There is an urgent need for a unified open-source library to streamline the process of volumetric video capturing, reconstruction, and rendering for both researchers and non-professional users to develop various algorithms and applications of this emerging technology.EasyVolcapTweet

EasyVolcap could be a catalyst for more personal, "at-home" captures, enabling everyday users to create photorealistic 4D content. In September of last year, we examined Infinite Realities and their pipeline. We discussed some of the potential applications of their work and we can see another avenue for people to create emerge today.

The GUI is very similar to Instant NGP and just like Instant NGP, you can create keyframes throughout the scene to render out a video. The significant divergence here is the integration of a 4D system, incorporating time as a critical component.

If the authors seem familiar, it might be because they were behind 4K4D, which we looked at earlier this year. As part of this release, they have additionally open sourced 4K4D and made its data available. You might have expected given that 4K4D would be a supported method. It is.

EasyVolcap VRAM Requirements

A notable feature of EasyVolcap is its compatibility with consumer-grade hardware, demonstrated by its modest VRAM requirements (5 GB) in a demo run on a 3060 GPU. While performance will vary with longer or higher resolution inputs, the initial demonstrations are promising. Interestingly, the input method is still just 2D images. I imagine that the folder and file sizes very quickly scale in size as additional cameras are added. I haven't had a chance to try it out for myself yet, but it's safe to say my first week back plans have meaningfully been altered.

EasyVolcap Installation

EasyVolcap can be installed from their Github under this section.

How EasyVolcap Works

As it name implies, the method is somewhat straightforward. As you feed it a standard captured video, it begins to take samples of space-time rays for rendering. These ray samples are use to generate 4D point coordinates, which are then encoded with space-time embedders. They are then decoded for their physical properties, such as their density and color and then finally fed into a volume renderer.

I have had a hard time reconciling the earlier statement about this only consuming a manageable amount of VRAM. So, how do they actually achieve that? They heuristically swap out the VRAM with the immediately following frames to be played on the pinned main memory with asynchronous and non-blocking copy, which in turn gives you a manageable memory footprint without pushing you into commercial grade GPUs.

EasyVolcap Licensing

The license to use EasyVolcap is not permissive, but honestly I don't blame them. Right now their license allows for use, copy, modify and distribute this software and its documentation for educational, research and non-profit purposes only. Any modification based on this work must be open-source and prohibited for commercial use. You must retain, in the source form of any derivative works that you
distribute, all copyright, patent, trademark, and attribution notices from the source form of this work. In order to utilize EasyVolcap for commercial applications, they ask you email: xwzhou@zju.edu.cn.

The inspiration from nerfstudio seems to be solidified and potentially intentional with EasyVolcap acknowledging nerfstudio in their documentation as well as the introduction of their paper. I will be curious to see whether or not future 4D applications will build ontop, but given the precedent of nerfstudio, I think it's likely.

Only time will tell if EasyVolcap will do what Instant NGP, Luma, and nerfstudio did for the first wave of NeRF based creators, but for 4D applications. Similar to those early days, I wouldn't expect the installation and interfaces to be as smooth as they are now for NeRFs/3DGS. They've come a long way over the course of the past ~1.5 years, but getting the platform up and running was part of the fun! EasyVolcap fills a hole in the market that has existed, and now it's time for me to begin thinking through my own library of captures to leverage dynamic scenes and stories that will inevitably be told.

Featured

Featured

Featured

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

May 24, 2024

Platforms

OpenNeRF added to Nerfstudio

OpenNeRF is the latest method to be supported by Nerfstudio.

Michael Rubloff

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

May 24, 2024

Platforms

PlayCanvas's SuperSplat Updated with PWA support

PlayCanvas's Supersplat has continued to receive additional updates. This time it's coming with a big boost to performance, yet again.

Michael Rubloff

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

May 23, 2024

Research

Reflecting on NeRF-Casting

Late last year we looked at Uni-SDF which introduced dual radiance fields to better represent reflections in a scene. However, I just happened to see NeRF-Casting on Github a little while ago

Michael Rubloff

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV