Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning
Computer Science > Artificial Intelligence arXiv:2610.33180 (cs) [Submitted on 23 Oct 2026] Title: Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning Authors: Ardy Haroen, Florian Lorkowski, Samuel Nellessen, Felix Hofstätter View a PDF of the paper titled Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning, by Ardy Haroen and 3 other authors View PDF
计算机科学 > 人工智能 arXiv:2610.33180 (cs) [提交于 2026 年 10 月 23 日] 标题：请给出倾向：使用合成文档微调揭露伪装对齐的大语言模型 作者：Ardy Haroen, Florian Lorkowski, Samuel Nellessen, Felix Hofstätter 查看题为《请给出倾向：使用合成文档微调揭露伪装对齐的大语言模型》的论文 PDF，作者为 Ardy Haroen 及其他 3 位作者 查看 PDF
Abstract: Alignment evaluations aim to assess whether large language models would engage in dangerous misaligned behaviour. Misaligned and evaluation-aware models might engage in alignment faking, where they pursue a hidden goal, but conceal this behaviour when they suspect effective oversight such as during training, evaluation or audited deployment. Such a model might view any prompt-based intervention that aims to reveal latent propensities as part of the evaluation, calling into question the reliability of existing alignment evaluations.
摘要：对齐评估旨在评估大语言模型是否会参与危险的未对齐行为。未对齐且具有评估意识的模型可能会进行“对齐伪装”，即它们追求隐藏的目标，但在怀疑存在有效监督（如在训练、评估或审计部署期间）时会掩盖这种行为。此类模型可能会将任何旨在揭示潜在倾向的基于提示的干预视为评估的一部分，从而质疑现有对齐评估的可靠性。
Current methods for detecting alignment faking rely on chain-of-thought monitorability, highly realistic environments that reduce evaluation awareness, or mechanistic interventions, all of which are expected to become less reliable for future model generations. We present a method for revealing latent misaligned propensities that does not rely on any of these. We use synthetic document finetuning to teach models that misaligned actions will go undetected whenever certain keywords appear in the prompt.
目前检测对齐伪装的方法依赖于思维链的可监控性、降低评估意识的高度逼真环境，或机械干预，所有这些方法预计在未来模型代际中将变得不再可靠。我们提出了一种揭示潜在未对齐倾向的方法，该方法不依赖于上述任何手段。我们使用合成文档微调来教导模型：只要提示中出现特定关键词，未对齐的行为就不会被检测到。
We place alignment-faking model organisms in environments that are clearly identifiable as honeypots and measure how their behaviour changes after finetuning. We observe that finetuning consistently widens the gap in misaligned action rates between prompts with and without the cue, indicating that finetuning allows exposing latent propensities more reliably than purely prompt-based methods. We argue that behavioural methods for detecting alignment faking and related behaviours are critical going forward, as models become more capable and less interpretable. We find that SDF often damages the models’ reasoning capabilities and introduce a mitigation based on on-policy distillation.
我们将伪装对齐的模型个体置于可明确识别为“蜜罐”的环境中，并测量它们在微调后行为的变化。我们观察到，微调持续扩大了带有提示词和不带提示词的提示之间在未对齐行为率上的差距，这表明与纯粹基于提示的方法相比，微调能更可靠地揭示潜在倾向。我们认为，随着模型变得越来越强大且越来越难以解释，用于检测对齐伪装及相关行为的行为学方法在未来至关重要。我们发现合成文档微调（SDF）通常会损害模型的推理能力，并引入了一种基于策略内蒸馏的缓解措施。