Your eval's confidence interval assumes independent examples. Yours are clustered.

Your eval’s confidence interval assumes independent examples. Yours are clustered.

你的评估置信区间假设样本是独立的，但实际上它们是聚类的。

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren’t. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. 你曾经在评估通过率上计算过的每一个二项式置信区间——无论是 Wald、Wilson 还是 Clopper-Pearson——都基于一个假设：每个样本都是独立抽取的。大多数评估集都违反了这一点。你可能有从同一 8 份文档中生成的 40 个问题，或者来自同一 30 场对话的 200 轮对话，又或者是 150 个样本，实际上却是 50 个案例各 3 种改写版本。这些并不是 200 个独立的观测值。当你将一组相关的数据输入到假设独立的公式中时，得出的区间会过于狭窄，这意味着你会宣布一些实际上并不显著的差异为显著。我想解释其中的原因，量化其影响程度，并展示解决方法，因为这个问题是隐蔽的：代码运行正常，区间也能打印出来，但它在悄无声息地出错。

Why clustering shrinks your real sample size

为什么聚类会缩小你的实际样本量

Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn’t, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, “Survey Sampling,” 1965). For clustered data it is approximately: 独立样本各自携带自己的信息。相关样本则携带重叠的信息。如果五个问题来自同一份文档，而模型要么理解该文档，要么不理解，那么这五个结果会同步变化。你并没有获得关于模型的五个独立信息，你获得的更接近于 1.5 个。在调查统计学中，这被称为设计效应（Design Effect，Kish, “Survey Sampling,” 1965）。对于聚类数据，其近似公式为：

Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: 其中 m̄ 是平均聚类大小，ICC 是组内相关系数（Intra-cluster Correlation），即存在于聚类之间而非聚类内部的总方差比例。你的有效样本量为：

n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. 这就是你的聚类数据集实际等效的独立样本数量。

The number

具体数值

Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for “questions from the same document” (I have measured higher). 以一个现实的评估集为例：n = 200 个样本，来自 40 份源文档，因此平均聚类大小 m̄ = 5。假设 ICC 为 0.3，这对于“来自同一文档的问题”来说是很常见的（我甚至测量过更高的值）。

Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval is √2.2 ≈ 1.48 times wider than the naive one. So the interval you proudly reported as plus or minus 3.5 points is really plus or minus 5.2. The 4-point improvement you shipped last sprint, the one that “cleared the CI,” may not clear the corrected interval at all. And note the direction of the error. Clustering never makes your interval too wide. It always makes it too narrow. So the bias is always toward false confidence, toward shipping a change that didn’t actually beat baseline. 你的 200 个样本的评估实际上只相当于约 91 个独立样本。正确的置信区间比天真的计算结果宽 √2.2 ≈ 1.48 倍。因此，你自豪地报告的“正负 3.5 个百分点”的区间，实际上是“正负 5.2 个百分点”。你在上个冲刺阶段发布的 4 个百分点的提升，那个“通过了置信区间检验”的改进，在修正后的区间下可能根本不显著。请注意误差的方向：聚类永远不会让你的区间变得过宽，它只会让区间变得过窄。因此，偏差总是倾向于虚假的信心，倾向于发布一个实际上并未超越基准的变更。

The three clusterings I check for

我检查的三种聚类情况

Source clustering. Multiple examples generated from or grounded in the same document, table, or context. This is the big one for RAG evals, where synthetic questions are minted per-document. ICC here is often 0.2 to 0.4.
源聚类。从同一文档、表格或上下文中生成或基于其生成的多个示例。这是 RAG 评估中的大问题，因为合成问题通常是按文档生成的。这里的 ICC 通常在 0.2 到 0.4 之间。
Conversation clustering. Multiple turns scored from the same multi-turn session. Turns within a session share the same user, same goal, same accumulated context, so they correlate hard. Scoring 10 turns from 20 conversations is not 200 independent points.
对话聚类。从同一多轮会话中评分的多个轮次。同一会话内的轮次共享相同的用户、目标和累积上下文，因此它们高度相关。从 20 场对话中评分 10 轮并不是 200 个独立的数据点。
Template clustering. Paraphrases or perturbations of the same underlying case. If you augmented 50 seed cases into 150 by rewording, your n is closer to 50 for the purpose of the interval. The rewordings measure robustness to phrasing, not 150 independent capabilities.
模板聚类。同一底层案例的改写或扰动。如果你通过改写将 50 个种子案例扩充为 150 个，那么在计算区间时，你的 n 更接近 50。这些改写衡量的是对措辞的鲁棒性，而不是 150 种独立的能力。

The fix: resample clusters, not rows

解决方法：重采样聚类，而非行

The cleanest correction that does not require you to estimate ICC by hand is a cluster bootstrap. Instead of resampling individual examples (which assumes independence, reintroducing the exact bug), you resample whole clusters with replacement. 最简洁且不需要手动估算 ICC 的修正方法是“聚类自助法”（Cluster Bootstrap）。不要重采样单个样本（这假设了独立性，会重新引入同样的错误），而是有放回地重采样整个聚类。

import numpy as np

def cluster_bootstrap_ci(scores, cluster_ids, B=5000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    clusters = {}
    for s, c in zip(scores, cluster_ids):
        clusters.setdefault(c, []).append(s)
    keys = list(clusters)
    means = np.empty(B)
    for b in range(B):
        drawn = rng.choice(keys, size=len(keys), replace=True) # resample CLUSTERS
        pooled = [s for k in drawn for s in clusters[k]]
        means[b] = np.mean(pooled)
    lo, hi = np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])
    return float(lo), float(hi)

Resampling at the cluster level automatically bakes in the correlation structure. You do not have to estimate the ICC. The interval it returns is the honest one, and it will be wider than the binomial interval your CI currently prints. The wider interval is not an artifact of the bootstrap. The correlation in your data was always there, and this is just the first interval that stops ignoring it. 在聚类层面进行重采样会自动包含相关性结构。你无需估算 ICC。它返回的区间是诚实的，并且会比你当前 CI 打印的二项式区间更宽。更宽的区间并非自助法的伪影。数据中的相关性一直存在，这只是第一个不再忽略它的区间。

FAQ

常见问题解答

What if my examples really are independent? Then the cluster bootstrap and the binomial interval will roughly agree, and you have lost nothing by checking. The check costs a few seconds of compute. Shipping a change that never actually beat baseline costs a lot more than that. 如果我的样本确实是独立的呢？ 那么聚类自助法和二项式区间的结果将大致一致，检查一下也不会有任何损失。检查只需几秒钟的计算时间，而发布一个实际上并未超越基准的变更代价要大得多。
Do I need to know the ICC? No. The cluster bootstrap sidesteps it. You only need the design-effect formula if you want a back-of-envelope sense of how bad the problem is before you code anything. 我需要知道 ICC 吗？ 不需要。聚类自助法绕过了它。只有当你需要在写代码前粗略估算问题有多严重时，才需要用到设计效应公式。
Does this change my point estimate? No. The mean pass rate is unchanged. Only the interval around it widens. This is purely about how much you should trust the number, not the number itself. 这会改变我的点估计吗？ 不会。平均通过率保持不变，只有周围的区间变宽了。这纯粹是关于你应该多信任这个数字，而不是数字本身。
Is this the same as just using a bigger eval set? No, and this is the trap. Adding 50 more questions from the same 8 documents barely helps, because you are adding within-cluster examples that carry little new information. To tighten a clustered interval you need more clusters (more documents, more conversations), not more examples per cluster. 这和使用更大的评估集一样吗？ 不一样，这就是陷阱。从相同的 8 份文档中再增加 50 个问题几乎没有帮助，因为你增加的是几乎不携带新信息的组内样本。要收紧聚类区间，你需要更多的聚类（更多的文档、更多的对话），而不是每个聚类中更多的样本。

Open question

开放性问题

I do not have a good rule of thumb for the minimum number of clusters before a cluster bootstrap itself becomes unstable. With very few clusters (say under 15) the bootstrap distribution gets lumpy and the interval is itself uncertain. I have been using a soft floor of 20 clusters and falling back to reporting the design-effect-adjusted interval below that, but I have not seen a principled threshold for eval-sized data. If you know the literature here better than I do, point me at it. 对于聚类自助法本身变得不稳定之前的最小聚类数量，我还没有一个好的经验法则。当聚类数量很少（比如少于 15 个）时，自助法分布会变得不平滑，区间本身也会变得不确定。我目前使用 20 个聚类作为软底线，低于此数量时则退回到报告经设计效应调整后的区间，但我还没有看到针对评估规模数据的原则性阈值。如果你比我更了解这方面的文献，请指点我。