Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
格式敏感度指数：大语言模型基准测试中提示词包装器的鲁棒性与模式合规性
Abstract: Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. 摘要： 提示词包装器（Prompt wrappers）往往仅在格式上有所不同，但它们足以改变模型的评分，甚至足以颠覆排行榜的结论。我们在受控的 Token 协议下研究了这种差异，并引入了两个互补的指标：格式敏感度指数（FSI，由包装器选择引起的准确率波动范围）和可解析性敏感度指数（PSI，即答案可解析性的相应波动范围）。
Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. 通过对 OpenRouter 上涵盖 7 个问答任务、5 个包装器系列以及 4 个参数规模从 7B 到 72B 的指令微调模型进行的 14 万次生成测试，我们发现不同模型的平均 FSI 差异超过 30 倍，且这种差异很大程度上是由合规性失败（compliance failures）导致的。固定效应回归分析表明，即使在控制了任务、模型和包装器变量后，可解析性仍然是准确率的强预测因子。
We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments. 我们认为，在不考虑包装器差异和合规性的情况下报告准确率在统计学上是脆弱的，并针对基准测试和结构化输出部署提出了实用建议。
Paper Details:
- Authors: Deep Pankajbhai Mehta
- Date: 2 May 2026
- Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
- DOI: 10.48550/arXiv.2607.09665
论文详情：
- 作者： Deep Pankajbhai Mehta
- 日期： 2026 年 5 月 2 日
- 学科分类： 人工智能 (cs.AI)；计算与语言 (cs.CL)；机器学习 (cs.LG)
- DOI: 10.48550/arXiv.2607.09665