Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation

提示词鲁棒性取决于任务：在大模型评估中比较客观题与信念类问题

Abstract: Survey-style evaluations of large language models often treat a prompted response as a measure of a model’s values or beliefs. This assumption is particularly fragile when responses are read as evidence of political values, social attitudes, or beliefs. We ask whether prompt robustness differs between objective questions with fixed answers and subjective questions that ask for opinions or values.

摘要： 对大型语言模型（LLM）的调查式评估通常将模型的提示响应视为衡量其价值观或信念的标准。当这些响应被解读为政治价值观、社会态度或信念的证据时，这一假设显得尤为脆弱。我们探讨了提示词鲁棒性在具有固定答案的客观问题与询问观点或价值观的主观问题之间是否存在差异。

We evaluate four instruction-tuned model families on three objective datasets (MMLU, ARC, and CulturalBench) and three subjective datasets (Political Compass Test, ValueBench, and World Values Survey). For each question/statement, we apply multiple types of prompt changes, such as variations in wording, framing, and format, and measure whether the model gives the same answer across variants.

我们针对三个客观数据集（MMLU、ARC 和 CulturalBench）和三个主观数据集（政治罗盘测试、ValueBench 和世界价值观调查）评估了四个指令微调模型系列。对于每个问题或陈述，我们应用了多种类型的提示词变更，例如措辞、框架和格式的变化，并测量模型在不同变体中是否给出相同的答案。

Using a binomial generalized estimating equation, we find significant effects of model, dataset, prompt category, and their interactions. The dataset type effect is also significant, and the interaction between dataset type and prompt category is large. These results show that prompt robustness depends on the question type, the prompt change, and the model.

通过使用二项式广义估计方程，我们发现模型、数据集、提示类别及其交互作用具有显著影响。数据集类型的影响同样显著，且数据集类型与提示类别之间的交互作用巨大。这些结果表明，提示词的鲁棒性取决于问题类型、提示词变更方式以及模型本身。