Foundation4D Puts AV Perception and Splats in One Model

NVIDIA Foundation4D predicts depth, 3D objects, occupancy, and Gaussian splats from multi-camera driving video in one model, with a 1,253-clip benchmark.

NVIDIA Foundation4D predicts depth, 3D objects, occupancy, and Gaussian splats from multi-camera driving video in one model, with a 1,253-clip benchmark.

Michael Rubloff

Michael Rubloff

Michael is the Founder and Managing Editor of Radiancefields.com

NVIDIA

Autonomous vehicle programs have long split two jobs across separate networks run over the same camera footage, understanding a road scene and rebuilding it for simulation. NVIDIA handles the second job with NuRec, the gaussian splatting reconstruction suite that reached general availability at GTC 2026. Foundation4D, a new paper from a 26 person NVIDIA team, folds both jobs into a single model.

Foundation4D takes multi-camera driving video and, in one pass per window of frames, predicts camera poses and depth, 3D objects, which parts of the surrounding space are occupied and by what, and a set of Gaussians ready to render. A shared backbone attends across cameras and across time in separate steps to keep long sequences affordable. Full recordings are stitched window by window, with camera trajectories refined globally and object tracks linked across windows.

The Gaussian head places one Gaussian per pixel along each camera ray at the predicted depth, assigns it either to static scenery or to a tracked actor, and fills the sky from a learned background. Moving vehicles are rendered by carrying their Gaussians rigidly along the predicted tracks, and the head does not predict motion itself.

Foundation4D running entirely on its own predicted poses and tracks scores 23.94 dB against 22.66 dB for the released Instant NuRec checkpoint across 99 internal test clips, even though Instant NuRec is handed ground truth ego-vehicle poses. Most of the gain is in static scenery. When future frames are rendered from earlier windows, Foundation4D falls to 15.84 dB on vehicles and other movable objects against 17.71 dB for Instant NuRec, and the authors list that case as open work. The two systems also use different camera counts and context windows, so the paper frames the test as a comparison of whole systems.

Foundation4D also reports the lowest depth error on two public driving benchmarks and the highest detection and occupancy scores among the methods compared, each after benchmark specific fine tuning. A 48 frame, six camera window takes about 20 seconds on one NVIDIA H100, with the Gaussian heads accounting for roughly a third of that time.

NVIDIA feeds Foundation4D's poses, depth and tracks into the camera only version of NuRec, which still optimizes each scene. Turnaround falls from 16.0 to 3.7 hours per clip, though NVIDIA notes that camera setups, software and hardware all differ between the two releases being compared.

Alongside the paper, NVIDIA has released the PAI-Foundation4D benchmark, 1,253 clips from its Physical AI Autonomous Vehicles dataset with dense depth and occupancy labels. The dataset is gated on Hugging Face under an NVIDIA license that limits use to internal autonomous vehicle development. The arXiv listing does not link to code or a project page.

The paper is available on arXiv.

Thumbnail for Gaussian Splatting News episode 12

11:38

Watch · New every Monday

Splats Come to Fortnite, Nike Shoots in 4D, Splats on a Nintendo 3DS

Episode 12 · Nov 9, 2026

Epic’s UEFN 43.00 brings Gaussian splats into the Unreal Editor for Fortnite, importing PLY and SPZ files that render on every platform UEFN supports, mobile included.

Featured

Recents

Platforms

sperid-swift-mlx Brings Idol Image-to-Splat to the Mac

sperid-swift-mlx ports Speri Labs' Idol to Swift and MLX, turning one photo into a 3D Gaussian splat on Apple Silicon Macs with PLY, USDZ, and GLB export.

Michael Rubloff

Nov 15, 2026

Platforms

sperid-swift-mlx Brings Idol Image-to-Splat to the Mac

sperid-swift-mlx ports Speri Labs' Idol to Swift and MLX, turning one photo into a 3D Gaussian splat on Apple Silicon Macs with PLY, USDZ, and GLB export.

Michael Rubloff

Platforms

LCC Studio Universal V2.7.0 Adds AMD and Intel GPU Support

XGRIDS LCC Studio Universal V2.7.0 brings single-scene 3DGS reconstruction, editing and application to AMD and Intel discrete GPUs, alongside its NVIDIA build.

Michael Rubloff

Nov 15, 2026

Platforms

LCC Studio Universal V2.7.0 Adds AMD and Intel GPU Support

XGRIDS LCC Studio Universal V2.7.0 brings single-scene 3DGS reconstruction, editing and application to AMD and Intel discrete GPUs, alongside its NVIDIA build.

Michael Rubloff

Platforms

ovrtx 0.7.0 Adds Diffuse Relighting for Gaussian Splats

NVIDIA ovrtx 0.7.0 lets Gaussian splats be relit by scene lights with a bundled MDL material and adds experimental lidar rendering of NuRec-trained splats.

Michael Rubloff

Nov 14, 2026

Platforms

ovrtx 0.7.0 Adds Diffuse Relighting for Gaussian Splats

NVIDIA ovrtx 0.7.0 lets Gaussian splats be relit by scene lights with a bundled MDL material and adds experimental lidar rendering of NuRec-trained splats.

Michael Rubloff

News

MPEG GS4 and GS5 Gaussian Splats Now Decode in the Browser

MPEG GS4 and GS5 Gaussian splat profiles now decode client-side in a browser demo, using WebAssembly reference decoders inside a SuperSplat-based player.

Michael Rubloff

Nov 14, 2026

News

MPEG GS4 and GS5 Gaussian Splats Now Decode in the Browser

MPEG GS4 and GS5 Gaussian splat profiles now decode client-side in a browser demo, using WebAssembly reference decoders inside a SuperSplat-based player.

Michael Rubloff

Trending Articles
  1. TRENDING
    Loading...
  2. TRENDING
    Loading...
  3. TRENDING
    Loading...
Michael Rubloff

Written by Michael Rubloff

Michael is the Founder and Managing Editor of Radiancefields.com

More from Michael Rubloff

More from Michael Rubloff

Platforms

sperid-swift-mlx Brings Idol Image-to-Splat to the Mac

sperid-swift-mlx ports Speri Labs' Idol to Swift and MLX, turning one photo into a 3D Gaussian splat on Apple Silicon Macs with PLY, USDZ, and GLB export.

Michael Rubloff

Nov 15, 2026

Sperid Idol

Platforms

sperid-swift-mlx Brings Idol Image-to-Splat to the Mac

Michael Rubloff

Nov 15, 2026

Sperid Idol

Platforms

sperid-swift-mlx Brings Idol Image-to-Splat to the Mac

sperid-swift-mlx ports Speri Labs' Idol to Swift and MLX, turning one photo into a 3D Gaussian splat on Apple Silicon Macs with PLY, USDZ, and GLB export.

Michael Rubloff

Nov 15, 2026

Sperid Idol

Platforms

LCC Studio Universal V2.7.0 Adds AMD and Intel GPU Support

XGRIDS LCC Studio Universal V2.7.0 brings single-scene 3DGS reconstruction, editing and application to AMD and Intel discrete GPUs, alongside its NVIDIA build.

Michael Rubloff

Nov 15, 2026

Platforms

LCC Studio Universal V2.7.0 Adds AMD and Intel GPU Support

Michael Rubloff

Nov 15, 2026

Platforms

LCC Studio Universal V2.7.0 Adds AMD and Intel GPU Support

XGRIDS LCC Studio Universal V2.7.0 brings single-scene 3DGS reconstruction, editing and application to AMD and Intel discrete GPUs, alongside its NVIDIA build.

Michael Rubloff

Nov 15, 2026

Platforms

ovrtx 0.7.0 Adds Diffuse Relighting for Gaussian Splats

NVIDIA ovrtx 0.7.0 lets Gaussian splats be relit by scene lights with a bundled MDL material and adds experimental lidar rendering of NuRec-trained splats.

Michael Rubloff

Nov 14, 2026

NVIDIA

Platforms

ovrtx 0.7.0 Adds Diffuse Relighting for Gaussian Splats

Michael Rubloff

Nov 14, 2026

NVIDIA

Platforms

ovrtx 0.7.0 Adds Diffuse Relighting for Gaussian Splats

NVIDIA ovrtx 0.7.0 lets Gaussian splats be relit by scene lights with a bundled MDL material and adds experimental lidar rendering of NuRec-trained splats.

Michael Rubloff

Nov 14, 2026

NVIDIA

New to Gaussian Splatting? Start here

Radiance fields, explained from the ground up: what a Gaussian splat is, how to capture one, and which tools and gear to start with.

Newsletter

Keep up with the field.

New research, tools and captures from the Radiance Fields newsroom, in your inbox. Free, no spam.

Gaussian Splatting News on YouTube

The week in radiance fields, every Monday.

Gaussian Splatting News on YouTube

The week in radiance fields, every Monday.

Gaussian Splatting News on YouTube

The week in radiance fields, every Monday.

Gaussian Splatting News on YouTube

The week in radiance fields, every Monday.