iFLYTEK-Embodied-Omni Technical Report

Abstract: General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-language reasoning, video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors.

摘要： 通用具身智能体必须能够理解多模态指令，预测环境演变趋势，并在长时程内生成精确的控制动作。现有的方法通常专注于视觉-语言推理、基于视频的世界建模或动作生成；而先合成未来观测再推断动作的级联流水线，往往会引入接口瓶颈并累积预测误差。

We present iFLYTEK-Embodied-Omni, a unified multimodal foundation model that jointly models vision (videos and images), language, and action within a single Omni framework. Its modality-specific visual-language, video-generation, and action-generation components communicate through shared multimodal self-attention.

我们提出了 iFLYTEK-Embodied-Omni，这是一个统一的多模态基础模型，在单一的 Omni 框架内联合建模视觉（视频和图像）、语言和动作。其特定于模态的视觉-语言、视频生成和动作生成组件通过共享的多模态自注意力机制进行通信。

This design establishes brain-cerebellum collaboration: the vision-language model and video generation model form a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation model serves as a low-level cerebellum that directly converts planned subgoals and shared multimodal context into executable action chunks.

这种设计建立了“大脑-小脑”协作机制：视觉-语言模型和视频生成模型构成了用于指令理解、任务规划、进度跟踪和未来视觉状态预测的高层“大脑”，而动作生成模型则充当低层“小脑”，直接将规划好的子目标和共享的多模态上下文转换为可执行的动作块。

To develop these capabilities, we combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.

为了培养这些能力，我们结合了来自人类演示和机器人交互的带动作标注及无动作标注的具身视频，并融合了具身推理、具身感知以及通用图像-文本数据，构建了一个全面的数据集。此外，我们采用了一种四阶段策略，在对完整模型进行联合微调之前，逐步训练 VLM（视觉-语言模型）、VGM（视频生成模型）和 AGM（动作生成模型）。