Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

任务能力不等于指令遵循：评估小型语言模型中指令冲突的行为

Abstract: Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. 摘要： 指令微调旨在使语言模型能够遵循用户的请求，然而目前尚不清楚当指令与其常规任务行为发生冲突时，小型模型是否会予以配合。

We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). 我们通过三项任务——多项选择问答（MCQA）、情感分类和数学问答——研究了这一问题。研究方法是将标准指令与冲突的非标准指令（例如：选择错误选项、输出相反情感，或返回答案的两倍值）进行配对。

This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. 这种跨任务设计使我们能够测试对冲突指令的抵触是与特定任务特征相关，还是反映了一种更广泛的行为倾向。

As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. 由于所有预测结果均根据原始标准答案进行评分，因此忽略非标准指令的模型在表面上看起来仍然是准确的。

Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. 通过使用标准准确率、非标准准确率以及指令遵循失败率（IFFR），我们评估了不同规模的指令微调 Qwen 模型。

Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. 尽管在所有任务和数据集上的表现模式并不完全一致，但总体而言，标准准确率和指令遵循能力都会随着模型规模的扩大而提高。

Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. 小型模型虽然保持了任务能力，但通常会忽略非标准指令；而大型模型在两种设置之间表现出了明显的差异。

These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. 这些发现表明，任务能力的提升并不能自动转化为对模型行为的可靠控制。

Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures. 因此，任务能力和指令遵循是两种截然不同的能力，仅报告标准准确率会掩盖指令遵循方面的失败。