Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Grounded-Exo2Ego：用于鲁棒性外向视角转第一人称视角视频生成的结构化语义基础模型

Abstract: Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions.

摘要： 从单一的外向视角（Exocentric）视频生成第一人称视角（Egocentric）视频是 AR/VR 和具身智能领域一个新兴且重要的课题。与传统的新视角合成相比，外向转第一人称视角（exo-to-ego）生成是一项难度大得多的任务，因为在极端视角变化和大面积不可见区域的情况下，标准的几何条件约束会变得极不可靠。

We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context.

我们提出了 Grounded-Exo2Ego，这是一个从架构和数据层面解决上述挑战的原则性框架。在架构上，Grounded-Exo2Ego 是一个双分支视频扩散模型：它结合了一个几何锚定分支（通过 3D 重建渲染来约束生成），以及一个新颖的语义基础分支（超越了主流的几何方法，通过基于对象级上下文合成具有挑战性的区域来提升质量）。

Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments.

此外，我们发现相机与重建结果之间的错位问题常被忽视，这严重削弱了 exo-to-ego 的学习效果。因此，我们引入了一种相机重定位算法来解决这一问题，并在所有指标上显著提升了质量。我们还开发了一个全自动合成数据引擎，用于在程序生成的环境中生成并渲染带有骨骼绑定的 3D 角色。

Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

在极具挑战性的 EgoExo4D 数据集上的评估表明，我们的方法在所有指标上均大幅优于近期最先进的方法。详细的消融实验验证了我们在数据和架构层面的每一项贡献所带来的性能提升。