Guide|News

How to create a NeRF with video through FFmpeg

Michael Rubloff

Michael Rubloff

Feb 23, 2023

Email
Copy Link
Twitter
Linkedin
Reddit
Whatsapp
FFmpeg NeRF
FFmpeg NeRF

To use a video in Instant-NGP or nerfstudio, you will need to use FFmpeg.

If you are looking to jump straight into it, Luma AI, will let you upload videos and automatically pull frames to use. However, it is not currently possible to dictate how many frames a second it will use.

What is FFMpeg

FFmpeg is a free, open-source software project that includes a library of codecs and tools for handling video, audio, and other multimedia files and streams. It can be used to record, convert, and stream digital audio and video in various formats. It is widely used in various applications such as video editing software, media players, and streaming platforms.

How to install FFMpeg

  1. Download the FFmpeg package from the official website: https://ffmpeg.org/download.html

    • Choose the version that corresponds to your operating system (Windows, Linux, Mac, etc.).

  2. Extract the files from the downloaded package.

  3. Open a terminal or command prompt, navigate to the folder where the FFmpeg files are located.

  4. If you are using Windows, open the command prompt and navigate to the folder where the FFmpeg files are located.

  5. Run the command ./configure (if you are using Linux or Mac) or configure (if you are using Windows) to configure the FFmpeg build.

  6. Run the command make to start building FFmpeg. This step can take a while to complete.

  7. Finally, run the command make install to install FFmpeg on your system.

Note: On Windows, you can also use pre-compiled binary version and also instead of step 4-7 you can use the wizard installer.exe You will need administrative rights to install FFmpeg on your system.

Once you have FFMPEG running on your computer, you're ready to begin working with videos. For a guide on why videos are better than photos for NeRFs, see this article.

Step 1: Find the video for NeRF you would like to use

Please note, not all videos are usable for NeRFs, but with trial and error, you'll get your capture method down.

Step 2: Place the video into your "Scripts" folder within Instant-NGP in file explorer

Step 3: How many frames can I use for NeRF without maxing out memory? (VRAM)

In order to determine how many frames per second (fps) to pull, you will first need to calculate how many photos you can use before your VRAM gets maxed out. Luckily, this is straightforward and there is a calculator for just that, here!

CD into the scripts file:

cd scripts

You'll know if its in the scripts folder if the Anaconda window looks like this

Step 4: Slice the video up into individual frames

Once you have determined how many photos you want to use, insert this line of code below, editing the yellow text with your specific file information. Please note, you will need to write the exact name of the video file, including the file type. So if the video is a .mp4, it would be videoname.mp4.

python colmap2nerf.py --video_in videoname.mov --video_fps 2.5 --run_colmap --aabb_scale 16

After this has been completed, your Anaconda window should look like this.

Type the letter "Y" and hit enter. It will pull out the individual frames and if you look in the "Images", it should be populated with the video frames. In the anaconda window, it will ask you again to hit Y/N and that 'colmap_sparse' and 'colmap_text' will be deleted. It sounds scary, but it literally doesn't matter. Hit Y and enter.

Step 5: Run the Instant-NeRF Transform

The transform will compile as it normally does for Instant-NGP and depending on the number of images and your GPU can take anywhere from 5 minutes to 1 hour+. Once it is finished and the transform.json file is created, you will need to move the images and the json file. It is significantly easier to open a new window of file explorer to keep track. Within the Instant-NGP/data folder, create a new folder to label your newest NeRF. Then select the entire images folder within scripts in addition to the transform.json file and place them within the new folder you created in Data. It is important to have the two separated and should look like this. Once this is complete, you're ready to launch Instant-NGP!

Featured

Featured

Featured

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

May 21, 2024

Platforms

Scaniverse arrives on Android

Gaussian Splatting platform, Scaniverse, is now available on Android.

Michael Rubloff

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

May 21, 2024

Platforms

Gracia launches on Steam and Sidequest

The VR Gaussian Splatting platform, Gracia, has now arrived on Steam and Sidequest, with more to come

Michael Rubloff

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

May 17, 2024

Research

CAT3D Pounces on 3D Scene Generation

We very recently were looking at RealmDreamer, which generates scenes from prompts. Just over a month later, CAT3D, short for "Create Anything in 3D," has emerged and takes things up a notch or two.

Michael Rubloff

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

May 15, 2024

Radiancefields.com launches Job Board

The latest feature has arrived onto the site and it's with the goal of connecting top talent to companies from newly launched start ups to the world's largest companies.

Michael Rubloff

To embed a website or widget, add it to the properties panel.

Trending articles

Trending articles

Trending articles

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Dec 13, 2023

Research

The MERF that turned into a SMERF

For the long time readers of this site, earlier this year, we looked into Google Research's Memory Efficient Radiance Fields (MERF). Now, they're back with another groundbreaking method: Streamable Memory Efficient Radiance Fields, or SMERF.

Michael Rubloff

Featured

Featured

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

It doesn't seem like a lot of people know this, but you can run CamP and Zip-NeRF in the cloud, straight through Google and it's actually super easy. It’s called CloudNeRF.

Michael Rubloff

May 8, 2024

Platforms

Google CloudNeRF: Zip-NeRF and CamP in the Cloud

Michael Rubloff

May 8, 2024

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Gaussian Splatting methods have continued to pour in over the first three months of the year. With the rate of adoption, being able to merge and compare these methods, shortly after their release would be amazing.

Michael Rubloff

Apr 8, 2024

Gaustudio

Research

Gaustudio

Michael Rubloff

Apr 8, 2024

Gaustudio

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

SplaTV

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Michael Rubloff

Mar 15, 2024

SplaTV