The 95% Illusion: Why Your Confidence Interval Isn’t What You Think It Is

95% 的错觉：为什么你的置信区间并非你所想的那样

A product analyst presents an A/B test dashboard. The treatment variant’s conversion rate improved by 0.4 percentage points, and the 95% confidence interval for the lift excludes zero. When a stakeholder asks: “So there’s a 95% chance the new version is better?” No one says no. That silence is the central subject of this article.

一位产品分析师展示了一个 A/B 测试仪表板。实验组的转化率提高了 0.4 个百分点，且提升幅度的 95% 置信区间不包含零。当利益相关者问道：“所以新版本有 95% 的概率更好吗？”没有人说“不”。这种沉默正是本文探讨的核心主题。

The 95% in a confidence interval is not a probability attached to the model, the parameter, or the hypothesis. It is a probability attached to a procedure, and it has frequently misled people in medicine, product analytics, and finance because of confusion between the two.

置信区间中的 95% 并不是指模型、参数或假设本身的概率。它是一个与“程序”相关的概率。由于人们经常混淆这两者，这在医学、产品分析和金融领域频繁导致误导。

What Does a 95% Confidence Interval Actually Mean?

95% 置信区间到底意味着什么？

The confidence interval as most people use it today comes from Jerzy Neyman’s 1934–37 work on statistical estimation. Neyman was trying to solve a specific issue: any statement about an unknown parameter has to be built from data that could have come out differently. Run the same experiment again and you get a different sample, a different estimate, a different interval.

今天大多数人所使用的置信区间概念，源自 Jerzy Neyman 在 1934 年至 1937 年间关于统计估计的研究。Neyman 当时试图解决一个具体问题：任何关于未知参数的陈述都必须建立在可能产生不同结果的数据之上。如果你重复同样的实验，你会得到不同的样本、不同的估计值以及不同的区间。

His resolution was to stop asking “where is the parameter?” and instead start asking “how often does my method land on the truth?” A 95% confidence interval is defined in such a way that, when the same construction rule is applied repeatedly to new samples, 95% of the resulting intervals contain the true parameter. This is a statement about the long-run behavior of the rule, not about any single interval it generates.

他的解决方法是停止追问“参数在哪里？”，转而询问“我的方法有多大概率能命中真值？”95% 置信区间的定义方式是：当同样的构建规则被反复应用于新样本时，所产生的区间中有 95% 会包含真实参数。这是关于该规则长期表现的陈述，而非针对其生成的任何单个区间的陈述。

The mechanics, for a sample mean under the usual large-sample approximation, follow directly from the Central Limit Theorem. If x̄ is the sample mean of n draws from a distribution with true mean μ and standard deviation σ, then for large n:

对于大样本近似下的样本均值，其计算机制直接遵循中心极限定理。如果 x̄ 是从均值为 μ、标准差为 σ 的分布中抽取 n 个样本的均值，那么当 n 很大时：

(Mathematical formulas omitted for brevity)

The randomness in that statement belongs to x̄, not to μ. Once we plug in our actual observed x̄, the interval is fixed. It either contains μ or it doesn’t. The 95% describes the machine that generated the interval, not the interval itself.

该陈述中的随机性属于 x̄，而不属于 μ。一旦我们代入实际观测到的 x̄，区间就固定了。它要么包含 μ，要么不包含。95% 描述的是生成该区间的“机器”，而不是区间本身。

Neyman was clear that this was an intended compromise: in return for giving up the possibility of stating “the parameter is here with such-and-such probability”, one gains a guarantee that does not rely on anyone’s prior beliefs. It is for this reason that the interpretation is so easily misunderstood. Experienced scientists and statisticians have on several occasions discovered that a majority accept statements equivalent to “there’s a 95% probability that the parameter lies within this particular interval”, which is in fact the kind of interpretation that Neyman’s approach was specifically designed to exclude.

Neyman 很清楚这是一种预期的妥协：为了放弃“参数以某种概率位于此处”的陈述可能性，人们获得了一种不依赖于任何先验信念的保证。正因如此，这种解释极易被误解。经验丰富的科学家和统计学家曾多次发现，大多数人接受等同于“参数有 95% 的概率位于该特定区间内”的陈述，而这恰恰是 Neyman 的方法专门旨在排除的解释方式。

Bayesian Credible Intervals: A Different Question

贝叶斯可信区间：一个不同的问题

The Bayesian alternative answers a different question and, not coincidentally, gives a different kind of interval. Instead of asking about the reliability of a procedure, it asks: given this specific data and what I believed beforehand, what should I believe now?

贝叶斯替代方案回答的是一个不同的问题，且并非巧合的是，它给出了另一种区间。它不再询问程序的可靠性，而是询问：基于这些特定的数据以及我之前的信念，我现在应该相信什么？

A 95% credible interval is simply the range over which this posterior integrates to 0.95. Unlike the confidence interval, this is a direct probability statement about the parameter, conditional on the data we actually observed and the prior we brought to the table.

95% 可信区间仅仅是后验概率积分为 0.95 的范围。与置信区间不同，这是一个关于参数的直接概率陈述，其条件是我们实际观测到的数据以及我们引入的先验知识。

The catch is exactly that dependency. Two analysts with different priors, looking at identical data, will report different credible intervals. Both will be correct relative to their own starting beliefs. Frequentist theory was built partly to avoid this. A procedure’s coverage guarantee doesn’t care what our prior is, because it doesn’t use one.

问题的关键恰恰在于这种依赖性。两位持有不同先验的分析师在查看相同数据时，会报告不同的可信区间。相对于他们各自的初始信念，两者都是正确的。频率派理论的建立部分是为了避免这种情况。程序的覆盖保证并不关心我们的先验是什么，因为它根本不使用先验。

A Difference, Analyzed Twice

差异的两次分析

Let’s look at a subscription business. The basic monthly churn rate is 1.50 per cent. Following a product change, in a month when there were 20,000 active users the number of churned users was 350, which gives an observed rate of 1.75 per cent, representing a difference of 0.25 percentage points.

让我们看看一家订阅制企业。其基础月流失率为 1.50%。在一次产品变更后，某个月份有 20,000 名活跃用户，其中流失用户数为 350 名，观测到的流失率为 1.75%，即差异为 0.25 个百分点。

The frequentist reads: Under the null hypothesis that the model is correctly calibrated, the standard error of the observed default rate is… The test statistic is z = 0.25 / 0.086 ≈ 2.9, corresponding to a two-sided p-value around 0.004. The 95% confidence interval around the observed gap is 0.25 ± 1.96 × 0.086.

频率派的解读是：在模型校准正确的零假设下，观测到的默认流失率的标准误为……检验统计量 z = 0.25 / 0.086 ≈ 2.9，对应的双侧 p 值为 0.004 左右。观测到的差异的 95% 置信区间为 0.25 ± 1.96 × 0.086。