Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

查看原文 / View Original 编辑 / Edit

Computer Science > Computer Vision and Pattern Recognition arXiv:2610.30340 (cs) [Submitted on 21 Oct 2026] Title: Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles Authors: NVIDIA: Sérgio Agostinho, Maximilian Apodaca, Emmanuel Attia, Guillem Braso, George Dong, Ilya Fedorov, Michele Fenzi, Gang Huang, Jiahui Huang, Shengyu Huang, Bingxin Ke, Laura Leal-Taixé, Kuan-Ting Lee, Mingxuan Liu, Tian Liu, Tim Meinhardt, Aljoša Ošep, Naveen Kumar Rai, Kefei Ren, Cristiano Saltori, Keqiang Wu, Ning Xu, Fei Xue, Jin-Bey Yu, Lei Zhang, Qunjie Zhou.

计算机科学 > 计算机视觉与模式识别 arXiv:2610.30340 (cs) [提交于 2026 年 10 月 21 日] 标题：Foundation4D：迈向自动驾驶车辆的统一感知与重建模型作者：NVIDIA：Sérgio Agostinho, Maximilian Apodaca, Emmanuel Attia, Guillem Braso, George Dong, Ilya Fedorov, Michele Fenzi, Gang Huang, Jiahui Huang, Shengyu Huang, Bingxin Ke, Laura Leal-Taixé, Kuan-Ting Lee, Mingxuan Liu, Tian Liu, Tim Meinhardt, Aljoša Ošep, Naveen Kumar Rai, Kefei Ren, Cristiano Saltori, Keqiang Wu, Ning Xu, Fei Xue, Jin-Bey Yu, Lei Zhang, Qunjie Zhou。

Abstract: We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows.

摘要：我们介绍了 Foundation4D，这是一个用于多摄像头驾驶视频的几何估计、3D 检测、语义占用和高斯重建的统一前馈模型。共享主干网络利用分解的时空注意力机制，高效地结合了跨摄像头和跨时间的观测数据。我们的视频推理流水线使用位姿图优化来细化摄像头轨迹，并整合了跨输入窗口的对象轨迹。

We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods.

我们还开发了一个数据标注流水线，并发布了 PAI-Foundation4D，这是一个包含 1,253 个驾驶片段的密集深度和语义占用标签的基准测试集。通过针对基准测试的微调，Foundation4D 在 Waymo 和 nuScenes 上实现了最低的深度 AbsRel，在 nuScenes 上实现了最高的检测 mAP，并在 OpenOccupancy 上实现了对比方法中最高的占用 mIoU。

Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by 13.4% relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.

我们的任务组合研究显示了其对检测和几何任务的双重益处：将检测与几何和占用任务联合训练，比仅进行检测训练的顶级 mAP 提高了 1.2 个百分点；而将几何与检测联合训练，则使摄像头到平台的平移误差相对于仅进行几何训练降低了 13.4%。我们进一步将预测的几何结构和轨迹用作无激光雷达 NuRec 的先验信息，以支持用于仿真的驾驶场景重建。