Stop me if this has happened to you before. You've gone out, shot a NeRF, processed the transform and launched the GUI only to realize, it's full of artifacts and floaters. The researchers behind Clean-NeRF get this frustration as their paper addresses this exact problem.
The researchers succinctly say in regards to floaters:
Such inaccuracies impede NeRF’s potential for accurate 3D NeRF registration, object detection, segmentation, etc., which possibly accounts for only limited significant research effort so far to
directly address these important 3D fundamental computer vision problems to date.
While the internet appears to be divided on whether or not floaters enhance or take away from the NeRF magic, Clean-NeRF gives you the option to have sharper scenes without any additional input or large computational load.
In order to accomplish this, Clean-NeRF is set up into two steps:
It works by detecting and discarding view-dependent appearances in training, followed by a geometric correction procedure performed on each traced ray.
In order to understand what that means, it's important to understand the difference between view-independent and view-dependent appearances. The view-independent aspect captures overall scene color, such as color and shading, while the view-dependent aspect accounts for color variations due to changing viewing angles, such as reflections.
Clean-NeRF separates view-independent and view-dependent color elements during NeRF training, while removing the impact of view-dependent appearances. Then a geometry correction is applied to each traced ray to further cut down floaters.
While hidden away within the paper, there is an exciting announcement:
Clean-NeRF can be applied as an easy plug-in component into any existing NeRF-like models, including MLP-based and voxel grid-based approaches, without resorting to extra knowledge such as normal maps, depthmaps, or lighting conditions.
Clean-NeRF
Will we be seeing this plugin into Instant-NGP, Nerfstudio, or TurboNeRF soon? Given the results, it's hard to imagine a world where Clean-NeRF is not incorporated. As a note, CleanNeRF only looks at fixed lighting conditions and no semi-transparent objects in a scene, but the examples given are promising results for cleaner NeRFs.
At the tail end of the paper, the authors write that the code will be released, but did not give a timeline. For those wanting to read the original paper, please click here.
New to Gaussian Splatting? Start here
