Nerfstudio Integrates 3DGS MCMC

In its latest merge, Nerfstudio has incorporated an exciting update to its Gaussian Splatting method. It now supports 3DGS MCMC, which should increase the fidelity of outputs across the board.

The Nerfstudio team are currently refactoring the codebase to have splatfacto flag with MCMC, but for now people can train with it using the command:

 simple_trainer_mcmc.py

MCMC reinterprets the placement and adjustment of 3D Gaussians as a process of sampling from a probability distribution. This perspective aligns the updates of 3D Gaussians with the principles of Markov Chain Monte Carlo (MCMC) sampling. Specifically, they adopt Stochastic Gradient Langevin Dynamics (SGLD), a variant of MCMC, to introduce a noise term into the Gaussian updates. This reformulation naturally promotes the exploration of the scene and improves the effectiveness of the Gaussians in faithfully reproducing the scene.

Instead of using heuristics for cloning and splitting Gaussians, MCMC employs a relocalization scheme. This scheme moves 'dead' Gaussians (those with low opacity) to the locations of 'live' Gaussians (those with high opacity). The parameters of these relocated Gaussians are adjusted to minimally impact the rendering quality, ensuring that the probability distribution of the Gaussians remains stable.

To further optimize the system, the method introduces L1 regularization on the opacity and scale of Gaussians. This encourages unused Gaussians to disappear, thus managing the number of Gaussians effectively and reducing computational overhead.

Some users have noticed a strange haze effect on a percentage of captures, but they are currently investigating a fix. Nerfstudio and gsplat can be installed from their Github page