Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

基于工作负载驱动的设备端实时字幕翻译优化

Abstract: This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions limit the value of optimizations designed for long-context or high-throughput language-model serving.

摘要： 本报告研究了在短输入、短输出、单批次推理、低延迟及隐私限制条件下，面向台湾地区的设备端英中（繁体）字幕翻译。这些限制条件使得针对长上下文或高吞吐量语言模型服务所设计的优化方案价值有限。

Starting from LMT-60-0.6B, preliminary profiling suggests that vocabulary projection becomes a more important decode-time cost after GGUF quantization reduces the relative cost of Transformer blocks. We replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrate the embedding space, and adapt the model through embedding calibration followed by full supervised fine-tuning.

以 LMT-60-0.6B 模型为起点，初步分析表明，在 GGUF 量化降低了 Transformer 模块的相对计算成本后，词汇表映射（vocabulary projection）成为了解码阶段更主要的成本来源。我们将原始的 151k 词汇表替换为 64k 的字幕领域分词器，迁移了嵌入空间，并通过嵌入校准及后续的全监督微调对模型进行了适配。

On a fixed 500-example subset of the OpenSubtitles2024 test set, the LocalSubs achieves a 59.2% tie-excluded win rate against Google Translate under GPT-4o pairwise judging. Performance is strongest on short cues and declines as cue length increases. Preliminary Apple M2 Metal measurements on a 64k-vocabulary model show a 1.63$\times$ speedup over a 151k-vocabulary profiling baseline. The raw benchmark configuration is incomplete, so the latency result is treated as preliminary.

在 OpenSubtitles2024 测试集的 500 个固定样本子集上，经 GPT-4o 成对评估，LocalSubs 在排除平局后的胜率为 59.2%（对比 Google 翻译）。该模型在短提示词上的表现最为出色，并随提示词长度增加而下降。在 Apple M2 Metal 上的初步测量显示，64k 词汇表模型相比 151k 词汇表的基准模型实现了 1.63 倍的加速。由于原始基准配置尚不完整，该延迟结果仅作为初步参考。