Autonomous vehicle programs have long split two jobs across separate networks run over the same camera footage, understanding a road scene and rebuilding it for simulation. NVIDIA handles the second job with NuRec, the gaussian splatting reconstruction suite that reached general availability at GTC 2026. Foundation4D, a new paper from a 26 person NVIDIA team, folds both jobs into a single model.
Foundation4D takes multi-camera driving video and, in one pass per window of frames, predicts camera poses and depth, 3D objects, which parts of the surrounding space are occupied and by what, and a set of Gaussians ready to render. A shared backbone attends across cameras and across time in separate steps to keep long sequences affordable. Full recordings are stitched window by window, with camera trajectories refined globally and object tracks linked across windows.
The Gaussian head places one Gaussian per pixel along each camera ray at the predicted depth, assigns it either to static scenery or to a tracked actor, and fills the sky from a learned background. Moving vehicles are rendered by carrying their Gaussians rigidly along the predicted tracks, and the head does not predict motion itself.
Foundation4D running entirely on its own predicted poses and tracks scores 23.94 dB against 22.66 dB for the released Instant NuRec checkpoint across 99 internal test clips, even though Instant NuRec is handed ground truth ego-vehicle poses. Most of the gain is in static scenery. When future frames are rendered from earlier windows, Foundation4D falls to 15.84 dB on vehicles and other movable objects against 17.71 dB for Instant NuRec, and the authors list that case as open work. The two systems also use different camera counts and context windows, so the paper frames the test as a comparison of whole systems.
Foundation4D also reports the lowest depth error on two public driving benchmarks and the highest detection and occupancy scores among the methods compared, each after benchmark specific fine tuning. A 48 frame, six camera window takes about 20 seconds on one NVIDIA H100, with the Gaussian heads accounting for roughly a third of that time.
NVIDIA feeds Foundation4D's poses, depth and tracks into the camera only version of NuRec, which still optimizes each scene. Turnaround falls from 16.0 to 3.7 hours per clip, though NVIDIA notes that camera setups, software and hardware all differ between the two releases being compared.
Alongside the paper, NVIDIA has released the PAI-Foundation4D benchmark, 1,253 clips from its Physical AI Autonomous Vehicles dataset with dense depth and occupancy labels. The dataset is gated on Hugging Face under an NVIDIA license that limits use to internal autonomous vehicle development. The arXiv listing does not link to code or a project page.
The paper is available on arXiv.
