EgoHSI3R: Online 4D Hand-Scene Interaction Reconstruction from Egocentric Video

Computer Science > Computer Vision and Pattern Recognition arXiv:2610.40832 (cs) [Submitted on 29 Oct 2026] Title: EgoHSI3R: Online 4D Hand-Scene Interaction Reconstruction from Egocentric Video Authors: Mingzhi Pei, Zhangchen Ye, Ruoqu Chen, Liu Cao, Mengdi Xu.

计算机科学 > 计算机视觉与模式识别 arXiv:2610.40832 (cs) [提交于 2026 年 10 月 29 日] 标题：EgoHSI3R：基于第一人称视频的在线 4D 手部-场景交互重建。作者：Mingzhi Pei, Zhangchen Ye, Ruoqu Chen, Liu Cao, Mengdi Xu。

Abstract: Egocentric videos offer an abundant source of hand-scene interactions for embodied learning, which requires recovering hands, camera, and scene consistently in a shared 3D world. Existing methods focus on either camera-frame hands or world-space hand trajectories, without jointly recovering the scene pointmaps or interaction information. Building a complete reconstruction from such specialized predictions therefore typically requires per-sequence optimization.

摘要：第一人称视频为具身学习提供了丰富的手部与场景交互来源，这要求在共享的 3D 世界中一致地恢复手部、摄像机和场景。现有方法要么关注摄像机坐标系下的手部，要么关注世界坐标系下的手部轨迹，而无法同时恢复场景点图或交互信息。因此，从这些专门的预测中构建完整的重建通常需要针对每个序列进行优化。

We propose EgoHSI3R, a unified feed-forward framework for online 4D hand-scene interaction reconstruction from egocentric monocular video. To jointly recover these components, it maintains a recurrent scene representation that accumulates geometric evidence over time and provides a shared world space and scale.

我们提出了 EgoHSI3R，这是一个用于从单目第一人称视频进行在线 4D 手部-场景交互重建的统一前馈框架。为了联合恢复这些组件，它维护了一种循环场景表示，该表示随时间积累几何证据，并提供共享的世界空间和尺度。

We introduce hand-centric prompts that fuse pretrained hand priors and local scene context, while MANO vertex queries attend to scene features to predict dense per-vertex contact. These designs enable a single network to jointly estimate camera motion, dense scene geometry, world-space bilateral MANO hands, and per-vertex hand contact.

我们引入了以手部为中心的提示，融合了预训练的手部先验和局部场景上下文，同时 MANO 顶点查询关注场景特征以预测密集的逐顶点接触。这些设计使单个网络能够联合估计摄像机运动、密集场景几何、世界坐标系下的双侧 MANO 手部以及逐顶点手部接触。

Despite operating causally, EgoHSI3R achieves the lowest world-space hand joint error on ARCTIC among the evaluated methods, while offering faster inference than existing world-space hand reconstruction pipelines and competitive camera and scene reconstruction accuracy.

尽管以因果方式运行，EgoHSI3R 在 ARCTIC 数据集上的世界坐标系手部关节误差在所有评估方法中最低，同时提供了比现有世界坐标系手部重建流程更快的推理速度，并具备具有竞争力的摄像机和场景重建精度。