Putting sign language AI into users’ hands
Putting sign language AI into users’ hands
将手语 AI 交到用户手中
Google DeepMind Sign Language Team Google DeepMind 手语团队
Introducing sign-language-to-text (SL2T), our breakthrough model powering new sign language features for Deaf and hard of hearing users. 我们隆重推出“手语转文字”（SL2T）模型。这是一项突破性技术，旨在为失聪及听障用户提供全新的手语功能支持。
AI’s ability to process spoken languages has advanced rapidly over recent decades, enabling automatic translation, dictation, and conversational interfaces that feel effortless to hearing users. Yet this technological revolution has not reached the world’s more than 200 sign languages — and the estimated 70 million Deaf and hard of hearing people who use them. 近几十年来，人工智能处理口语的能力突飞猛进，实现了自动翻译、听写和对话界面，让听力正常的用户感到使用起来毫不费力。然而，这场技术革命尚未惠及全球 200 多种手语，以及使用这些手语的约 7000 万失聪和听障人士。
Today, we’re introducing a massively multilingual sign-language-to-text (SL2T) translation model that marks a breakthrough in quality and generality. With it, we are bringing sign language AI out of the lab and into consumer products for the first time: SL2T powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11, starting with American Sign Language (ASL) to English. More devices are coming soon, and additional languages will follow. 今天，我们推出了一款大规模多语言“手语转文字”（SL2T）翻译模型，这标志着在翻译质量和通用性方面取得了重大突破。通过该模型，我们首次将手语 AI 从实验室带入消费级产品：SL2T 现已支持 Pixel 11 上 Gboard 和“实时转写”（Live Transcribe）的手语转文字听写功能，首发支持美国手语（ASL）转英语。更多设备即将支持，更多语言也将陆续加入。
Similarly to how hearing users can use dictation to speak instead of typing, this feature enables Deaf users to sign to their phone anywhere they’d normally type. You can sign to search the web, draft messages or documents, and ask Gemini to solve queries or execute tasks. In Live Transcribe, you can sign responses in conversations instead of having to type back and forth. According to our testers, signing in ASL is faster, more natural, and more delightful than typing in English. 正如听力正常的用户可以使用语音听写代替打字一样，该功能让失聪用户可以在任何需要打字的地方通过手语与手机交互。你可以通过打手语来搜索网页、起草信息或文档，并让 Gemini 解决问题或执行任务。在“实时转写”中，你可以通过打手语来回复对话，而无需反复打字。据我们的测试人员反馈，使用 ASL 手语比输入英语更快捷、更自然，体验也更愉悦。
Why sign languages matter
为什么手语至关重要
Sign languages are the primary languages of Deaf communities around the world and the cornerstone of Deaf cultural identity. There is great diversity among deaf people in terms of their level of proficiency in signing, speaking, reading, and writing, so it is important to support access in all modalities. Deaf people can benefit from sign language processing in the same way that hearing people benefit from spoken language processing, plus the technology opens new possibilities for bridging the communication gap between Deaf and hearing communities. 手语是全球失聪群体的主要语言，也是失聪文化认同的基石。失聪人士在手语、口语、阅读和写作方面的熟练程度各不相同，因此支持各种形式的交流至关重要。失聪人士可以像听力正常的人从口语处理技术中受益一样，从手语处理技术中获益。此外，这项技术还为弥合失聪群体与听力正常群体之间的沟通鸿沟开辟了新的可能。
Despite this opportunity for positive social impact, progress in sign language AI has been slow — both because building AI for sign languages presents complex challenges and because widespread misconceptions exist about how the languages themselves work. 尽管这项技术具有积极的社会影响，但手语 AI 的进展一直缓慢。这既是因为为手语构建 AI 存在复杂的挑战，也是因为人们对手语本身的运作方式存在广泛的误解。
Compared to spoken language transcription, sign language translation presents two core challenges. First, transcribing speech is a matter of performing a sequential mapping from sound to text in the same language, whereas sign languages are independent, natural languages with their own distinct grammars and lexicons. As a result, they require true machine translation rather than a sequential process of sign-to-word transformations. Second, the model must learn to “see” and understand physical movement. Sign languages convey meaning through simultaneous movements of the hands, arms, torso, head, and face. Accurately tracking these at high frame rates is a difficult and computationally demanding computer vision task. 与口语转写相比，手语翻译面临两大核心挑战。首先，口语转写是将同一种语言的声音序列映射为文本，而手语是拥有独立语法和词汇的自然语言。因此，它们需要真正的机器翻译，而不是简单的“手势到单词”的序列转换。其次，模型必须学会“观察”并理解肢体动作。手语通过手、手臂、躯干、头部和面部的同步运动来传达意义。在高帧率下准确追踪这些动作是一项极具挑战性且计算量巨大的计算机视觉任务。
Given this background, it is easy to understand why some early attempts at sign language technology, like sign language gloves, were fundamentally limited: sign languages aren’t simply “English on the hands.” They require complex visual perception of fine-grained whole-body movements and full-fledged language translation. SL2T is designed to deliver both. 鉴于此，我们不难理解为什么早期的一些手语技术尝试（如手语手套）存在根本性的局限：手语不仅仅是“手上的英语”。它们需要对精细的全身动作进行复杂的视觉感知，并进行成熟的语言翻译。SL2T 的设计初衷正是为了实现这两点。
How SL2T works
SL2T 的工作原理
We built SL2T by combining a user-centric, culturally informed approach with massive data scaling. The model is trained on over 100,000 hours of data across more than 50 sign languages — with roughly a quarter of the data in ASL. Training jointly on diverse languages, dialects, and proficiency levels causes the model to learn shared underlying structures, outperforming single-language models in our experiments. 我们通过结合以用户为中心、具备文化意识的方法与大规模数据扩展，构建了 SL2T。该模型在超过 50 种手语的 10 万多小时数据上进行了训练，其中约四分之一的数据为 ASL。通过对多种语言、方言和熟练程度的数据进行联合训练，模型能够学习到共享的底层结构，在我们的实验中表现优于单一语言模型。
To protect user privacy, SL2T sees sign language as a sequence of pose landmark locations rather than a raw camera feed. An on-device model (MediaPipe Holistic) tracks the location of points on the signer, and only these geometric coordinates are sent to the server for translation, allowing the original video to be discarded immediately. 为了保护用户隐私，SL2T 将手语视为一系列姿态地标（pose landmark）位置，而非原始摄像机画面。设备端模型（MediaPipe Holistic）会追踪手语使用者身上的关键点位置，仅将这些几何坐标发送到服务器进行翻译，从而确保原始视频可以立即被丢弃。
SL2T translates this coordinate sequence directly into text, bypassing intermediate annotations known as “glosses” that are widely used in prior work on sign language translation. Glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions. Translating directly from landmarks removes artificial vocabulary limits and allows translation quality to scale directly with data. SL2T 将这些坐标序列直接翻译成文本，绕过了以往手语翻译研究中广泛使用的“注释”（glosses）。注释无法捕捉手语中丰富的非线性特征，例如非手动标记和空间结构。直接从地标进行翻译消除了人为的词汇限制，并使翻译质量能够随着数据量的增加而直接提升。