Instant NGP from a Webcam(s)?

Over the weekend, I saw a page title that caught my eye: Immediate NeRF from webcams. Slightly incredulous and curious to learn more, I clicked through it.

Github user Ednaordinary demonstrated in a discussion of the NV Labs Github page a running version of Instant NGP utilizing 12 webcams. The results are really decent actually considering only 12 inputs are powering it. Even moreso for a bunch of $4.99 webcams.

While her computer is utilizing a 3090, it on appears to be consuming 3.62GB of VRAM, making this attainable for more people.

But the real question is how far away are we from having full NeRF powered video conferences. The answer is not close at all, but this represents an exciting baseline. NeRF has its work cut out for it to compete with Apple’s recently announced Digital Persona and Google’s Project Starline and honestly there might not be much of a competition, but it’s encouraging to see someone using Instant NGP in a near live capacity.

For those looking to try this out for themselves, Ednaordinary warns the code is not a straight out of the box solution, but does write the changes she made to make it happen.

Please modify the following lines:

83: Inside the list add the numbers for the cameras you want (to get this do v4l2-ctl --list-devices, then if you have /dev/video1 and want to use it, put 1)84-89 and 101-104: These lines are to declare lists for multithreading. The simplest thing to do while still having a working script is to remove 84-89 and replace 101-104 withthreads.append(threading.Thread(target=camloop, args=[cams, 0])), though note this will be slower. If you would like to multithread, make lists for each set of cams and make new threads.append(threading.Thread(target=camloop, args=[cams, 0])) lines, replacing cams with each list and adding an offset number of the length of the previous list of cams (so that they do not overwrite eachother). At any time 1 camera will be accessed from each list. All cams are not accessed at once because of bandwidth issues.91: Add the directory for the scene and include "/images"Once the script is ready calling it with zero arguments will capture a picture from each webcam and write it, then do nothing. This is useful for capturing pictures just once and then using colmap2nerf.py. Once you have transforms.json, run it with "--scene ./data/nerf/webcam --gui --train --n_steps 100000000000" replacing ./data/nerf/webcam with the directory to the scene. n_steps is the number of steps to train, but we don't want to stop training, so set it really high.

I for one will probably be paying Walmart.com a visit and then shortly thereafter see how successfully I can get it to run.