Teaching an Audio Model More About Barbados

让音频模型更了解巴巴多斯

Automatic speech recognition is very good until somebody mentions the name of a local school, a village, a politician, a festival, or a cricket ground. Then things get strange. In an earlier test with audio from Barbados, GPT Transcribe and GPT Audio 1.5 heard the event name “Rise Together” as “Rice Together”, while Qwen3.5-Omni Plus and Flash got it right. Those are different models from the Qwen3-Omni checkpoint used here, but the result motivated this experiment. 自动语音识别技术非常出色，但一旦涉及到当地学校、村庄、政治人物、节日或板球场的名称时，情况就会变得很奇怪。在之前的一次巴巴多斯音频测试中，GPT Transcribe 和 GPT Audio 1.5 将活动名称“Rise Together”听成了“Rice Together”，而 Qwen3.5-Omni Plus 和 Flash 则识别正确。虽然这些模型与本文使用的 Qwen3-Omni 检查点不同，但这一结果促成了本次实验。

Acoustically, the mistake is understandable. Culturally, it is wrong. A person who knows the local context has another signal available: they know that Rise Together is the plausible name. That led me to a question: can we give an audio-native model a stronger model of Barbados, using text that already contains the names, institutions, places, events and relationships it is likely to hear? 从声学角度看，这种错误是可以理解的；但从文化角度看，它是错误的。了解当地背景的人拥有额外的信号：他们知道“Rise Together”才是合理的名称。这引发了我的思考：我们能否利用包含当地名称、机构、地点、事件及关联关系的文本，为原生音频模型提供一个更强大的巴巴多斯知识模型？

So I took an archive of Barbados newspapers, turned it into 51.6 million tokens, and used it for domain-adaptive pretraining of the Thinker inside Qwen3-Omni. The result is promising, but not conclusive. The adapted model produced higher scores on our preliminary Barbados knowledge probe, particularly on people and institutions. It also got slightly worse on a small set of general-knowledge controls. And, most importantly, we have not yet shown that it transcribes audio more accurately. 因此，我整理了一份巴巴多斯报纸档案，将其转化为 5160 万个标记（tokens），并用于对 Qwen3-Omni 内部的“思考者”（Thinker）模块进行领域自适应预训练。结果令人鼓舞，但尚无定论。经过调整的模型在我们初步的巴巴多斯知识测试中得分更高，特别是在人物和机构方面。但在少量通用知识对照组中，其表现略有下降。最重要的是，我们尚未证明它在音频转录方面更加准确。

This is a very preliminary result. It came from our first training run, which we stopped at step 500 of a planned 801 steps. We were also still extracting the newspaper archive, so the 51.6 million training tokens represent the material available for that run rather than the full corpus we ultimately intend to use. This post is about what we have actually demonstrated, what broke along the way, and why I think the experiment is still worth pursuing. 这是一个非常初步的结果。它来自我们的首次训练，我们在计划的 801 步中于第 500 步停止了训练。当时我们仍在提取报纸档案，因此这 5160 万个训练标记仅代表该次运行可用的材料，而非我们最终打算使用的全部语料库。本文旨在说明我们实际验证了什么、过程中出现了什么问题，以及为什么我认为这项实验仍然值得继续。

The Problem Is Not Just Acoustic

问题不仅仅在于声学

A transcription model is doing more than converting sound into letters. When audio is clean and a word is common, the acoustic evidence can be enough. But real radio is compressed. Music leaks into speech. Presenters talk quickly. Phone-in guests have different microphones and accents. Proper nouns may be rare or absent from a model’s original training data. At that point, transcription becomes a contest between plausible sequences: weak acoustic evidence + language context -> selected transcript. 转录模型所做的不仅仅是将声音转换为字母。当音频清晰且词汇常见时，声学证据可能就足够了。但真实的广播音频经过压缩，音乐会混入语音，主持人语速很快，电话连线嘉宾的麦克风和口音各异。专有名词在模型的原始训练数据中可能很少见甚至缺失。此时，转录就变成了合理序列之间的竞争：微弱的声学证据 + 语言背景 -> 选定的转录结果。

For Barbados, the useful context includes things such as: Kensington Oval, Cave Hill, Samuel Jackman Prescod Polytechnic, Crop Over, constituency and parish names, local politicians, athletes, performers and organisations, and the ordinary relationships between all of those things. 对于巴巴多斯而言，有用的背景信息包括：肯辛顿椭圆球场（Kensington Oval）、凯夫希尔（Cave Hill）、塞缪尔·杰克曼·普雷斯科德理工学院（Samuel Jackman Prescod Polytechnic）、“Crop Over”节、选区和教区名称、当地政治人物、运动员、表演者和组织，以及所有这些事物之间的常规联系。

This is related to contextual biasing in automatic speech recognition. In Deep Context, Pundak and colleagues showed that an end-to-end recogniser could use supplied contextual phrases, including out-of-vocabulary terms, and reported relative word-error-rate improvements of up to 68% on their tasks. Later work on trie-based deep biasing and shallow fusion reported a 19.5% relative improvement over existing contextual-biasing methods, with particular attention to rare long-tail words. Those systems explicitly supply a biasing list at inference time. Our approach is different: we are trying to make local context part of the model’s learned language prior. 这与自动语音识别中的上下文偏置（contextual biasing）有关。在《Deep Context》一文中，Pundak 等人证明了端到端识别器可以使用提供的上下文短语（包括词汇表之外的术语），并在其任务中实现了高达 68% 的相对词错误率改进。后续关于基于前缀树（trie-based）的深度偏置和浅层融合的研究，在现有上下文偏置方法的基础上实现了 19.5% 的相对改进，特别关注了罕见的长尾词。这些系统在推理时显式地提供偏置列表。我们的方法则不同：我们试图将当地背景信息融入模型已学习的语言先验中。

Why Qwen3-Omni?

为什么选择 Qwen3-Omni？

I selected Qwen3-Omni-30B-A3B-Instruct because it is natively multimodal. It accepts text, audio, images and video, and it can produce text or speech. More importantly for this experiment, its audio understanding and text generation are parts of one end-to-end architecture. According to the Qwen3-Omni technical report, the model uses a Mixture-of-Experts Thinker-Talker design: The Thinker consumes multimodal representations and generates text. The Talker is responsible for generating streaming speech tokens. An Audio Transformer, or AuT, turns audio into representations consumed by the Thinker. 我选择了 Qwen3-Omni-30B-A3B-Instruct，因为它原生支持多模态。它能接收文本、音频、图像和视频，并能生成文本或语音。对于本次实验更重要的是，它的音频理解和文本生成属于同一个端到端架构。根据 Qwen3-Omni 技术报告，该模型采用了“专家混合”（MoE）的“思考者-说话者”（Thinker-Talker）设计：思考者处理多模态表示并生成文本；说话者负责生成流式语音标记。音频转换器（AuT）将音频转化为思考者可处理的表示。

The Thinker has roughly 30 billion total parameters with about 3 billion active for each token. The Talker is a separate 3-billion-parameter MoE component. For text-only use, Qwen exposes model.disable_talker(), allowing the speech-generation component to be omitted. This separation matters. I am not trying to teach the model a Barbadian voice, and I do not have a labelled corpus of Barbadian audio and transcripts large enough to retrain its acoustic front end. I am trying to adapt the component that interprets context and chooses text. 思考者总参数量约为 300 亿，每个标记激活约 30 亿。说话者是一个独立的 30 亿参数 MoE 组件。对于纯文本使用，Qwen 提供了 model.disable_talker() 接口，允许省略语音生成组件。这种分离至关重要。我并不是要教模型学会巴巴多斯口音，我也没有足够大的巴巴多斯音频和转录标注语料库来重新训练其声学前端。我试图调整的是负责解释上下文并选择文本的组件。

Training The Thinker, Not The Talker

训练“思考者”，而非“说话者”

We used domain-adaptive pretraining, usually shortened to DAPT. The idea is straightforward: take a model that has already been pretrained on broad data, then continue its language-modelling objective on unlabelled text from the target domain. Gururangan and colleagues’ ACL 2020 paper Don’t Stop Pretraining found that this second in-domain phase consistently improved downstream performance across several domains and resource levels. 我们使用了领域自适应预训练（DAPT）。其思路很简单：选取一个已经在广泛数据上预训练过的模型，然后在目标领域的无标注文本上继续其语言建模目标。Gururangan 等人在 ACL 2020 发表的论文《Don’t Stop Pretraining》发现，这种第二阶段的领域内训练在多个领域和资源水平上都能持续提升下游性能。

This was not instruction tuning. The newspaper chunks were not converted into question-and-answer pairs. We presented them as assistant-only continuation records and continued next-token training. The objective was to change what language and relationships the model finds probable, not to teach it a new chat format. We used LoRA rather than updating every weight. LoRA freezes the original model and injects trainable low-rank matrices into selected transformer layers. That gives us a portable adapter rather than another complete copy of a model. 这不是指令微调。报纸片段没有被转换为问答对。我们将其作为仅限助手的续写记录，并继续进行下一个标记的训练。目标是改变模型认为哪些语言和关系是可能的，而不是教它一种新的聊天格式。我们使用了 LoRA 而不是更新所有权重。LoRA 冻结了原始模型，并将可训练的低秩矩阵注入到选定的 Transformer 层中。这为我们提供了一个可移植的适配器，而不是模型的另一个完整副本。