Guide|News

How to create a NeRF with video through FFmpeg

Michael Rubloff

Feb 23, 2023

FFmpeg NeRF
FFmpeg NeRF

What is FFMpegHow to install FFMpegStep 1: Find the video for NeRF you would like to useStep 2: Place the video into your "Scripts" folder within Instant-NGP in file explorerStep 3: How many frames can I use for NeRF without maxing out memory? (VRAM)Step 4: Slice the video up into individual framesStep 5: Run the Instant-NeRF Transform

To use a video in Instant-NGP or nerfstudio, you will need to use FFmpeg.

If you are looking to jump straight into it, Luma AI, will let you upload videos and automatically pull frames to use. However, it is not currently possible to dictate how many frames a second it will use.

What is FFMpeg

FFmpeg is a free, open-source software project that includes a library of codecs and tools for handling video, audio, and other multimedia files and streams. It can be used to record, convert, and stream digital audio and video in various formats. It is widely used in various applications such as video editing software, media players, and streaming platforms.

How to install FFMpeg

  1. Download the FFmpeg package from the official website: https://ffmpeg.org/download.html

    • Choose the version that corresponds to your operating system (Windows, Linux, Mac, etc.).

  2. Extract the files from the downloaded package.

  3. Open a terminal or command prompt, navigate to the folder where the FFmpeg files are located.

  4. If you are using Windows, open the command prompt and navigate to the folder where the FFmpeg files are located.

  5. Run the command ./configure (if you are using Linux or Mac) or configure (if you are using Windows) to configure the FFmpeg build.

  6. Run the command make to start building FFmpeg. This step can take a while to complete.

  7. Finally, run the command make install to install FFmpeg on your system.

Note: On Windows, you can also use pre-compiled binary version and also instead of step 4-7 you can use the wizard installer.exe You will need administrative rights to install FFmpeg on your system.

Once you have FFMPEG running on your computer, you're ready to begin working with videos. For a guide on why videos are better than photos for NeRFs, see this article.

Step 1: Find the video for NeRF you would like to use

Please note, not all videos are usable for NeRFs, but with trial and error, you'll get your capture method down.

Step 2: Place the video into your "Scripts" folder within Instant-NGP in file explorer

Step 3: How many frames can I use for NeRF without maxing out memory? (VRAM)

In order to determine how many frames per second (fps) to pull, you will first need to calculate how many photos you can use before your VRAM gets maxed out. Luckily, this is straightforward and there is a calculator for just that, here!

CD into the scripts file:

cd scripts

You'll know if its in the scripts folder if the Anaconda window looks like this

Step 4: Slice the video up into individual frames

Once you have determined how many photos you want to use, insert this line of code below, editing the yellow text with your specific file information. Please note, you will need to write the exact name of the video file, including the file type. So if the video is a .mp4, it would be videoname.mp4.

python colmap2nerf.py --video_in videoname.mov --video_fps 2.5 --run_colmap --aabb_scale 16

After this has been completed, your Anaconda window should look like this.

Type the letter "Y" and hit enter. It will pull out the individual frames and if you look in the "Images", it should be populated with the video frames. In the anaconda window, it will ask you again to hit Y/N and that 'colmap_sparse' and 'colmap_text' will be deleted. It sounds scary, but it literally doesn't matter. Hit Y and enter.

Step 5: Run the Instant-NeRF Transform

The transform will compile as it normally does for Instant-NGP and depending on the number of images and your GPU can take anywhere from 5 minutes to 1 hour+. Once it is finished and the transform.json file is created, you will need to move the images and the json file. It is significantly easier to open a new window of file explorer to keep track. Within the Instant-NGP/data folder, create a new folder to label your newest NeRF. Then select the entire images folder within scripts in addition to the transform.json file and place them within the new folder you created in Data. It is important to have the two separated and should look like this. Once this is complete, you're ready to launch Instant-NGP!

Featured

Featured

Featured

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

4/11/24

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

4/10/24

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

4/9/24

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

Apr 11, 2024

Research

RealmDreamer's Generative Scenes

Since the unveiling of the Sora's large-scale generative Radiance Fields, the tech world has been buzzing with anticipation about the future of 3D scene generation. There hasn't been much public work since then showcasing what could be coming, but today we're looking at RealmDreamer, which creates scene level generations based on original text prompts.

Michael Rubloff

4/11/24

Trending articles

Trending articles

Trending articles

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Research

RadSplat's Hybrid NeRFs and 3DGS

Excitingly, we're seeing the arrival of the first result of another widely hyped event. The meeting of NeRFs and Gaussian Splatting.

Michael Rubloff

Mar 21, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Tools

splaTV: Dynamic Gaussian Splatting Viewer

Kevin Kwok, perhaps better known as Antimatter15, has released something amazing: splaTV.

Michael Rubloff

Mar 15, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Interview

Gaussian Splatting Brings Art Exhibitions Online with Yulei

The advent of radiance fields represents a transformative leap in event photography, aiming to give people the feeling of attending an event asynchronously. Artist Yulei's recent demonstration serves as a compelling example of this technological progression.

Michael Rubloff

Feb 22, 2024

Recent articles

Recent articles

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Research

Shrinking 3DGS File Size

Gaussian Splatting has quickly become one of the most exciting research topics in Radiance Fields, thanks to its fast training, real time rendering rates, and easy to create pipeline. The one critique that emerged was the resulting file size from captures, often venturing into the high hundreds of megabytes and up.

Michael Rubloff

Apr 11, 2024

3dgs compress

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Platforms

Luma AI Android Released

Native Android support from Luma AI is finally here. Of all the questions about Luma features I get, Android support is routinely at the top of the list.

Michael Rubloff

Apr 10, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024

Research

PhysAvatar's Dynamic Dances

Playing as yourself in a video game has always seemed like a fun idea. Now, we're one step closer to making that a reality with PhysAvatar.

Michael Rubloff

Apr 9, 2024