How to create a NeRF with video through FFmpeg

To use a video in Instant-NGP or nerfstudio, you will need to use FFmpeg.

If you are looking to jump straight into it, Luma AI, will let you upload videos and automatically pull frames to use. However, it is not currently possible to dictate how many frames a second it will use.

What is FFMpeg

FFmpeg is a free, open-source software project that includes a library of codecs and tools for handling video, audio, and other multimedia files and streams. It can be used to record, convert, and stream digital audio and video in various formats. It is widely used in various applications such as video editing software, media players, and streaming platforms.

How to install FFMpeg

1. Download the FFmpeg package from the official website: https://ffmpeg.org/download.html

   • Choose the version that corresponds to your operating system (Windows, Linux, Mac, etc.).

2. Extract the files from the downloaded package.

3. Open a terminal or command prompt, navigate to the folder where the FFmpeg files are located.

4. If you are using Windows, open the command prompt and navigate to the folder where the FFmpeg files are located.

5. Run the command ./configure (if you are using Linux or Mac) or configure (if you are using Windows) to configure the FFmpeg build.

6. Run the command make to start building FFmpeg. This step can take a while to complete.

7. Finally, run the command make install to install FFmpeg on your system.

Note: On Windows, you can also use pre-compiled binary version and also instead of step 4-7 you can use the wizard installer.exe You will need administrative rights to install FFmpeg on your system.

Once you have FFMPEG running on your computer, you're ready to begin working with videos. For a guide on why videos are better than photos for NeRFs, see this article.

Step 1: Find the video for NeRF you would like to use

Please note, not all videos are usable for NeRFs, but with trial and error, you'll get your capture method down.

Step 2: Place the video into your "Scripts" folder within Instant-NGP in file explorer

Step 3: How many frames can I use for NeRF without maxing out memory? (VRAM)

In order to determine how many frames per second (fps) to pull, you will first need to calculate how many photos you can use before your VRAM gets maxed out. Luckily, this is straightforward and there is a calculator for just that, here!

CD into the scripts file:

cd scripts

You'll know if its in the scripts folder if the Anaconda window looks like this

Step 4: Slice the video up into individual frames

Once you have determined how many photos you want to use, insert this line of code below, editing the yellow text with your specific file information. Please note, you will need to write the exact name of the video file, including the file type. So if the video is a .mp4, it would be videoname.mp4.

python colmap2nerf.py --video_in videoname.mov --video_fps 2.5 --run_colmap --aabb_scale 16

After this has been completed, your Anaconda window should look like this.

Type the letter "Y" and hit enter. It will pull out the individual frames and if you look in the "Images", it should be populated with the video frames. In the anaconda window, it will ask you again to hit Y/N and that 'colmap_sparse' and 'colmap_text' will be deleted. It sounds scary, but it literally doesn't matter. Hit Y and enter.

Step 5: Run the Instant-NeRF Transform

The transform will compile as it normally does for Instant-NGP and depending on the number of images and your GPU can take anywhere from 5 minutes to 1 hour+. Once it is finished and the transform.json file is created, you will need to move the images and the json file. It is significantly easier to open a new window of file explorer to keep track. Within the Instant-NGP/data folder, create a new folder to label your newest NeRF. Then select the entire images folder within scripts in addition to the transform.json file and place them within the new folder you created in Data. It is important to have the two separated and should look like this. Once this is complete, you're ready to launch Instant-NGP!