PhantomFill: When the Form Demands an Answer, Language Models Invent One
PhantomFill: When the Form Demands an Answer, Language Models Invent One
PhantomFill：当表单要求回答时，语言模型会无中生有
Abstract: Language models in production do not write prose. They fill forms: JSON fields, function arguments, extraction templates. We show that the form itself causes hallucination. We ask thirteen models the same question about the same input and change only the answer format. The inputs are built so the question cannot be answered: a viral post showing 12,400 likes but no visible replies, a support ticket whose call was never transcribed.
摘要： 生产环境中的语言模型并不只是撰写文章，它们更多是在填写表单：JSON 字段、函数参数、提取模板等。我们研究发现，表单本身就会导致幻觉。我们向 13 个模型询问关于同一输入的相同问题，仅改变回答格式。这些输入被设计为无法回答：例如一个显示有 12,400 个点赞但没有可见回复的病毒式帖子，或者一个从未被转录通话内容的客服工单。
In free text, GPT-5.5 answers honestly. It says there is no reply data, 98% of the time. Given a required JSON field for sentiment, the same model invents an answer 40 times out of 40. It fabricates the mood of crowds it never saw and quotes customers it never heard. The pattern holds with force. Required fields drive fabrication to 100% in ten of thirteen models.
在自由文本模式下，GPT-5.5 的回答是诚实的。在 98% 的情况下，它会表示没有回复数据。然而，当给定一个必填的 JSON 情感字段时，同一个模型在 40 次测试中全部编造了答案。它虚构了从未见过的群体情绪，并引用了从未听过的客户言论。这种模式具有很强的普遍性：在 13 个模型中的 10 个里，必填字段将编造率推高到了 100%。
An explicit “insufficient evidence” option rescues only the frontier: all nine open-weight models ignore it. A direct instruction, do not infer sentiment, is overridden by the schema in four of six models. Resistance does not come with scale: within a single model family, the smallest model refuses, the mid-sized model fabricates, the largest refuses again. Honesty under format pressure is a training outcome that no one is measuring.
明确提供“证据不足”选项仅对前沿模型有效：所有 9 个开源权重模型都忽略了该选项。在 6 个模型中的 4 个里，即使直接指令“不要推断情感”，也会被模式（Schema）所覆盖。这种抵制能力并不随模型规模增加而提升：在同一个模型家族中，最小的模型会拒绝编造，中等规模的模型会编造，而最大的模型又会拒绝。在格式压力下的诚实度，是一个目前无人评估的训练成果。
The fabrication hides exactly where hedging is impossible: in required enums and minimum-count arrays, fields where no disclaimer fits. We release PhantomFill, a benchmark with deterministic scoring and two reportable numbers: the Coerced Fabrication Rate and the Escape Utilization Rate. The fix we test is one line of schema. The failure we measure is everywhere.
这种编造行为恰好隐藏在无法进行模糊处理的地方：在必填的枚举值（enums）和最小计数数组中，这些字段无法容纳任何免责声明。我们发布了 PhantomFill，这是一个具有确定性评分的基准测试，包含两个可报告的指标：强制编造率（Coerced Fabrication Rate）和逃避利用率（Escape Utilization Rate）。我们测试的修复方案仅需一行模式代码，但我们测得的失败现象却无处不在。