Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation

想要更好的合成数据？试试“激活引导”：针对低资源语言生成的激活引导技术

Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. 大型语言模型（LLMs）已成为合成数据生成的有效工具，尤其是在低资源语言领域，生成的合成数据能够显著提升下游任务的性能。

Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. 目前表现最好的方法通常依赖于使用目标语言示例的少样本提示（few-shot prompting），但这会增加推理成本，并可能因词汇锚定（lexical anchoring）效应而降低生成数据的多样性。

In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. 在这项工作中，我们研究了“激活引导”（activation steering）作为低资源合成数据生成的一种替代方案。我们研究了两种引导策略：一是“语言引导”（Language Steering），旨在锁定语言的语言学特征；二是“质量引导”（Quality Steering），通过对比人类撰写的文本与回译文本的表征，来捕捉文本的规范性。

We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. 我们通过生成情感和主题分类数据并微调小型分类器，在四个开源大模型、多个模型层级以及 11 种类型各异的语言上对这些方法进行了评估。

Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages. 我们将引导技术应用于零样本（zero-shot）和少样本（few-shot）提示设置中，并与未进行引导的对照组进行了比较。结果表明，在模型早期层级进行引导能够持续提升生成数据的多样性，并往往能带来更强的下游模型性能，尤其是在低资源语言任务中表现尤为突出。