Twice the data, less power: my stability rule got blinder the harder I looked

数据翻倍，效能却下降：我的稳定性规则越深究越“盲目”

Originally published at erikhill.dev. The numbers below are checked against the repository they come from. 本文最初发布于 erikhill.dev。以下数据均已与来源代码库进行核对。

This is a finding about a measurement rule, not about a model. The models did not change. 这是一个关于测量规则的发现，而非关于模型本身的发现。模型并未发生改变。

What happened

发生了什么

I was comparing two models on a frozen 159 task suite and I had a problem most eval code does not admit to having: the same model, asked the same question three times, sometimes disagrees with itself. I had written a rule for that. A task where a config does not agree with itself is discarded, not counted. Call it strict. 我正在一个固定的 159 项任务集上比较两个模型，并遇到了大多数评估代码不愿承认的问题：同一个模型，在被问及同一个问题三次时，有时会给出不一致的答案。我为此编写了一条规则：如果某个配置在任务中无法自洽，该任务将被丢弃，不计入统计。我们称之为“严格规则”（strict）。

On the first three repetitions strict gave me Haiku ahead 7 to 1, eight informative tasks, p equal to 0.070. Not significant. So I looked at why, found that strict was throwing away thirteen tasks, and wrote a second rule that tolerates a minority disagreement. Call it rate. It gave Haiku ahead 13 to 2, fifteen informative tasks, p equal to 0.0074. Significant. 在前三次重复测试中，“严格规则”显示 Haiku 以 7 比 1 领先，有效任务数为 8 项，p 值为 0.070，未达到显著性水平。于是我探究原因，发现“严格规则”丢弃了 13 项任务，因此我编写了第二条允许少数不一致的规则，称之为“比率规则”（rate）。结果显示 Haiku 以 13 比 2 领先，有效任务数为 15 项，p 值为 0.0074，达到了显著性水平。

I had run the conservative rule, got a null, and then built the rule that returned the result I wanted. That is also exactly what everyone who p-hacks believes about themselves. 我运行了保守规则，得到了无效结果，然后构建了一条能返回我想要结果的规则。这正是每一个进行“p-hacking”（数据操纵以获取显著性）的人对自己所作所为的认知。

The evidence

证据

• How it was checked: The two rules disagreed on the same data (reps 1 to 3, suite combined-159.json):
  • strict: 7 to 1, informative 8, p 0.070
  • rate: 13 to 2, informative 15, p 0.0074
• 核对方式： 两条规则在相同数据（第 1-3 次重复，套件 combined-159.json）上表现不同：
  • “严格规则”：7 比 1，有效任务 8，p 值为 0.070
  • “比率规则”：13 比 2，有效任务 15，p 值为 0.0074

The second rule was written after seeing the first result stated in runs/PREREGISTRATION.md, committed before reps 4 to 6 existed, so the git timestamp is checkable against the run files’ mtimes. 第二条规则是在看到第一个结果后编写的，记录在 runs/PREREGISTRATION.md 中，并在第 4-6 次重复测试存在之前提交，因此 git 时间戳可以与运行文件的修改时间（mtime）进行核对。

The replication was pre-registered, analysis included predictions, refutation conditions and fixed parameters in runs/PREREGISTRATION.md; the analysis in tools/replicate.py, written while the sweeps were still running. 该复现过程经过预注册，分析包含了 runs/PREREGISTRATION.md 中的预测、反驳条件和固定参数；分析脚本 tools/replicate.py 是在扫描测试仍在运行时编写的。

The prediction I got wrong, I got wrong in public: predicted strict would again fail to reach significance; on fresh data it reached p 0.0215. 我预测错了，而且是公开预测错了：我预测“严格规则”将再次无法达到显著性；但在新数据上，它达到了 p 0.0215。

More repetitions raised the discard count:

• 2 reps: about 8.7 discarded, about 10.3 informative.
• 3 reps: 13 discarded, 8 informative.
• Pooling all six repetitions lost the result.
• Reps 4 to 6 alone: informative 10, discards 13, p 0.0215, decisive.
• Reps 1 to 6 pooled: informative 8, discards 17, p 0.0703, not significant.

更多重复测试增加了丢弃计数：

• 2 次重复：约 8.7 项丢弃，约 10.3 项有效。
• 3 次重复：13 项丢弃，8 项有效。
• 汇总全部 6 次重复测试后，结果失效。
• 仅第 4-6 次重复：有效 10 项，丢弃 13 项，p 值为 0.0215，结论明确。
• 第 1-6 次汇总：有效 8 项，丢弃 17 项，p 值为 0.0703，不显著。

What the extra data did not buy

额外数据并未带来预期的收益

This is the part I did not expect and the reason the post exists. Every extra repetition is another chance to observe a config disagreeing with itself, and strict discards a task the moment it sees one. So the discards climb with the data. Thirteen became seventeen. Informative tasks fell from ten to eight. The p-value went from 0.0215 to 0.0703, which is to say a decisive result became a null by adding measurements to it. 这是我没预料到的部分，也是我写这篇文章的原因。每一次额外的重复测试，都是观察配置自相矛盾的机会，而“严格规则”一旦发现矛盾就会丢弃任务。因此，丢弃数量随数据量增加而攀升。13 变成了 17，有效任务从 10 降至 8。p 值从 0.0215 变为 0.0703，也就是说，通过增加测量次数，一个明确的结果反而变成了无效结果。

In the limit the rule throws away every genuinely stochastic task. More measurement cannot fix that. Only a different rule can. A conservative rule is not a free choice. It will never report noise as signal, and the price is that it grows blinder the harder you look. That price is measurable, and until I measured it I had been describing the conservatism as pure upside. 从长远来看，该规则会丢弃每一个真正随机的任务。增加测量次数无法解决这个问题，只有更换规则才行。保守规则并非免费的午餐。它永远不会将噪声误报为信号，但代价是当你观察得越仔细，它就变得越“盲目”。这个代价是可以衡量的，而在我测量它之前，我一直将这种保守性视为纯粹的优势。

What the pre-registration was actually for

预注册的真正意义

The rule I trust reached significance on data it had never seen. strict on reps 4 to 6 alone: Haiku 9 to 1, ten informative tasks, p 0.0215. I had predicted it would fail again. The pre-registration says in advance what to do about that: “If 1 holds and 2 fails, that is stronger than predicted, and I will say the prediction was too conservative rather than claiming I called it.” So that is what I am saying. I did not call it. 我信任的规则在它从未见过的数据上达到了显著性。“严格规则”在第 4-6 次重复测试中：Haiku 9 比 1，10 项有效任务，p 值为 0.0215。我曾预测它会再次失败。预注册中提前说明了应对方案：“如果 1 成立而 2 失败，这比预测的更强，我会说预测过于保守，而不是声称我预判准确。”所以这就是我要说的：我没预判对。

The parameter is the part worth stealing. rate_margin was fixed at 0.5 in advance, and afterwards I removed the flag from the command line entirely, along with --alpha, because a threshold a reader can dial after seeing the result is not a threshold. The registered value did not change. The knob did. 参数是值得借鉴的部分。rate_margin 事先固定为 0.5，之后我从命令行中完全移除了该标志以及 --alpha，因为读者在看到结果后可以随意调节的阈值根本不是阈值。注册的值没有变，但调节旋钮（机制）变了。

What I might have wrong

我可能出错的地方

The weakest link is that this is one suite and two models. Haiku 4.5 beating Sonnet 4.6 here is a claim about a suite made of trap questions, not a capability ranking. A suite built to catch specific slips measures susceptibility to those slips, and I would not carry that sentence anywhere else. 最薄弱的环节在于这只是一个测试集和两个模型。Haiku 4.5 在此击败 Sonnet 4.6 是针对一个由陷阱问题组成的测试集的结论，而非能力排名。一个旨在捕捉特定失误的测试集只能衡量对这些失误的敏感度，我不会将此结论推广到其他领域。

The reading I cannot rule out is that rate is simply the better rule and strict was never worth defending, in which case the whole discard analysis is an elaborate defence of a mistake. The replication is evidence against that, since strict reached significance on fresh data, but one replication at two models is not much. 我无法排除的一种解读是：“比率规则”本身就是更好的规则，“严格规则”根本不值得辩护，在这种情况下，整个丢弃分析只是对一个错误的精心辩护。复现结果反驳了这一点，因为“严格规则”在新数据上确实达到了显著性，但仅针对两个模型的单次复现说明不了太多。

And I should be honest that the pooled result has an innocent reading. Reps 1 to 3 and reps 4 to 6 were run at different times, so pooling them mixes two sampling windows. I attribute the power loss to the discard mechanism because the discard count is the thing that visibly moved, thirteen to seventeen, but I have not run the version of this that would separate those two explanations. 我必须诚实地说，汇总结果还有一种无害的解读：第 1-3 次和第 4-6 次重复测试是在不同时间运行的，因此汇总它们混合了两个采样窗口。我将效能损失归因于丢弃机制，因为丢弃计数是唯一明显变动的数据（从 13 到 17），但我尚未运行能够区分这两种解释的测试版本。

If you maintain an eval suite with a stability or flakiness filter: count what it discards, and count it again after you add data. If the discard count rises faster than your informative count, your filter is spending your sample size, and the direction of that trade is not obvious from the code. 如果你维护着一个带有稳定性或不稳定性过滤器的评估套件：请统计它丢弃了多少任务，并在增加数据后再次统计。如果丢弃数量的增长速度快于有效任务数量，那么你的过滤器正在消耗你的样本量，而这种权衡的方向在代码中并不显而易见。

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. 我为 LLM 系统构建确定性评估和验证工具，目前正在寻找 AI 评估或 QA 工程领域的首份全职工作。支持美国东部远程办公，或位于南卡罗来纳州查尔斯顿。

https://erikhill.dev Suite, rules, pre-registration and replication: https://github.com/egnaro9/pi-eval 套件、规则、预注册和复现：https://github.com/egnaro9/pi-eval

The numbers above are quoted from FINDINGS.md and runs/PREREGISTRATION.md in that repository. The judgement of whether this was worth saying is mine. 以上数据引用自该仓库中的 FINDINGS.md 和 runs/PREREGISTRATION.md。关于这些内容是否值得分享，由我本人判断。