JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents

查看原文 / View Original 编辑 / Edit

Computer Science > Computer Vision and Pattern Recognition arXiv:2610.22499 (cs) [Submitted on 15 Oct 2026] Title:JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents Authors:Junseok Ahn, Jaehyun Park, Jeongsoo Choi, Joon Son Chung, Ji-Hoon Kim View a PDF of the paper titled JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents, by Junseok Ahn and 4 other authors View PDF HTML (experimental)

计算机科学 > 计算机视觉与模式识别 arXiv:2610.22499 (cs) [提交于 2026 年 10 月 15 日] 标题：JAVIS：用于实时对话代理的联合视听交互与流式传输作者：Junseok Ahn, Jaehyun Park, Jeongsoo Choi, Joon Son Chung, Ji-Hoon Kim 查看题为《JAVIS：用于实时对话代理的联合视听交互与流式传输》的论文 PDF，作者为 Junseok Ahn 及其他 4 位作者查看 PDF HTML（实验性）

Abstract:Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, existing conversational systems have mainly focused on text or speech, leaving out the visual feedback that is essential for natural interaction.

摘要：人类交流本质上是多模态的，不仅通过语言和声音线索，还通过视觉信号来传达意图。然而，现有的对话系统主要集中在文本或语音上，忽略了自然交互中必不可少的视觉反馈。

In this paper, we propose JAVIS, an end-to-end streaming framework for real-time audio-visual conversational agents. Our key idea is to let the voice and the face speak from a single representation. On top of a shared LLM, a Joint Refined Head produces a unified audio-visual representation, which is fed into dedicated streaming branches for speech and face generation.

在本文中，我们提出了 JAVIS，这是一个用于实时视听对话代理的端到端流式传输框架。我们的核心思想是让声音和面部从单一的表征中产生。在共享大语言模型（LLM）的基础上，一个联合精炼头（Joint Refined Head）生成统一的视听表征，并将其输入到专门的语音和面部生成流式分支中。

This end-to-end design avoids intermediate bottlenecks, enabling real-time generation while keeping speech and face tightly synchronized. In addition, we present JAVIS-Syn, a 334-hour synthetic corpus designed to address the lack of expressive audio-visual dialogue data, covering emotionally rich multi-turn conversations and assistant-style exchanges.

这种端到端的设计避免了中间瓶颈，实现了实时生成，同时保持了语音和面部的高度同步。此外，我们还介绍了 JAVIS-Syn，这是一个 334 小时的合成语料库，旨在解决缺乏富有表现力的视听对话数据的问题，涵盖了情感丰富的多轮对话和助手式交流。

Experiments show that JAVIS achieves approximately 200$\times$ lower initial latency than a strong baseline, while outperforming baselines in lip synchronization and facial expression appropriateness.

实验表明，JAVIS 的初始延迟比强基线模型降低了约 200 倍，同时在唇形同步和面部表情准确性方面优于基线模型。