Why Your Best Predictive Model Gives the Wrong Treatment Effect

为什么你最好的预测模型会给出错误的治疗效果

Why prediction-driven variable selection misses confounders and how Bayesian Adjustment for Confounding attempts to fix it. 为什么基于预测的变量选择会漏掉混杂因素，以及贝叶斯混杂调整（BAC）如何尝试解决这一问题。

A question that sounds easy but isn’t. Model selection often chases the wrong goal. Let’s say we want to estimate the effect of an exposure on an outcome, e.g., air pollution on hospital admissions or a drug versus time to recovery. We have observational data, one exposure variable, and a pile of candidate covariates. Some are confounders. Some are noise. Now the biggest question is — which ones go in the model? 这是一个听起来简单但实则不然的问题。模型选择往往追求错误的目标。假设我们想要估计某种暴露对结果的影响，例如空气污染对住院率的影响，或者某种药物对康复时间的影响。我们拥有观测数据、一个暴露变量和一堆候选协变量。其中一些是混杂因素，另一些则是噪声。现在最大的问题是——哪些变量应该放入模型中？

The usual reflex hands the choice to a selection rule. Run Stepwise routines, compare BIC values, or even cross-validate Lasso. Full Bayesian averaging proceeds over candidate specifications. However, the problem with these approaches is that every one of these tools optimizes predictive fit. A model can predict the outcome beautifully while being systematically wrong about the effect we care about. 通常的反应是将选择权交给某种选择规则。运行逐步回归（Stepwise）、比较 BIC 值，甚至对 Lasso 进行交叉验证。完全贝叶斯平均法（Full Bayesian averaging）则会对候选模型规格进行加权。然而，这些方法的问题在于，每一个工具优化的都是“预测拟合度”。一个模型可能在预测结果方面表现得非常出色，但在我们关心的因果效应估计上却存在系统性偏差。

This issue was the subject of a 2012 Biometrics paper by Chi Wang, Giovanni Parmigiani, and Francesca Dominici, who called the underlying issue adjustment uncertainty. It is the uncertainty about which variables to adjust for, and they proposed a fix called Bayesian Adjustment for Confounding (BAC) [1]. It also has a famous frequentist twin that most people don’t realise is the same insight: post-double-selection and double machine learning [3][4]. 这一问题是 Chi Wang、Giovanni Parmigiani 和 Francesca Dominici 在 2012 年发表于《Biometrics》期刊的一篇论文的主题，他们将这一根本问题称为“调整不确定性”（adjustment uncertainty）。这是指关于“应该调整哪些变量”的不确定性，他们提出了一种名为“贝叶斯混杂调整”（BAC）的解决方案 [1]。它还有一个著名的频率派“孪生兄弟”，大多数人没有意识到它们的核心洞察是一致的：即“双重选择后估计”（post-double-selection）和“双重机器学习”（double machine learning）[3][4]。

In this post, I will demonstrate three things. I will show you the failure mode in runnable Python, build the fix, and then finally show you the fix’s own sharp edge: the case where “adjust for anything that predicts exposure” is exactly the wrong advice and makes your estimate worse. If you have been reading about propensity scores, subclassification, and matching, this is the question those methods assume you have already answered: how do you decide, from data, which covariates a valid adjustment needs? 在这篇文章中，我将演示三件事。我将通过可运行的 Python 代码展示这种失效模式，构建解决方案，最后展示该方案的局限性：即在某些情况下，“调整所有预测暴露的变量”恰恰是错误的建议，反而会使你的估计结果更糟。如果你一直在阅读关于倾向评分、子分类和匹配的内容，那么这些方法其实都预设了你已经回答了这个问题：你如何从数据中决定哪些协变量是有效调整所必需的？

A good predictor and a confounder are not the same

好的预测变量与混杂因素并非一回事

Why does a specification tuned for prediction produce a flawed effect estimate? Because the two tasks reward distinct traits. Prediction favors a covariate that accounts for outcome variance. Effect estimation favors a covariate tied to both exposure and outcome. 为什么针对预测优化的模型会产生有缺陷的效应估计？因为这两项任务所奖励的特征截然不同。预测倾向于选择那些能解释结果方差的协变量；而效应估计则倾向于选择那些同时与暴露和结果相关的协变量。

Now, the problem arises when a confounder is strongly associated with the exposure but only weakly with the outcome. To a predictive criterion, that variable looks nearly useless because it explains little outcome variance. Also, much of what it explains is already captured by the (correlated) exposure. In realistic finite samples, BIC may prefer dropping it. But dropping it changes what the exposure coefficient means. The coefficient now absorbs the confounder’s path, and the “effect” is now actually contaminated. 问题在于，当一个混杂因素与暴露变量高度相关，但与结果变量仅有微弱相关时。对于预测准则而言，该变量看起来几乎毫无用处，因为它解释的结果方差很少。此外，它所解释的大部分信息已经被（相关的）暴露变量捕获了。在现实的有限样本中，BIC 可能倾向于剔除它。但剔除它会改变暴露系数的含义。该系数现在吸收了混杂因素的路径，所谓的“效应”实际上已经被污染了。

Wang and coauthors illustrate the point with five candidate covariates. U1, U2, and U3 each link to both X and Y. U4 predicts Y alone. U5 is noise. Wang 等人通过五个候选协变量说明了这一点。U1、U2 和 U3 分别与 X 和 Y 相关。U4 仅预测 Y。U5 是噪声。

Figure 1: The setup from Wang et al. (2012), redrawn. U1, U2, and U3 are true confounders. Each touches both X and Y. U4 predicts only Y; U5 is pure noise. The trap is U2: strong on exposure, weak on outcome. Image by author. 图 1：Wang 等人 (2012) 的设置重绘。U1、U2 和 U3 是真正的混杂因素，它们都与 X 和 Y 相关。U4 仅预测 Y；U5 是纯噪声。陷阱在于 U2：它与暴露强相关，但与结果弱相关。图片由作者提供。

Now consider the entire argument reduced to two t-statistics. On the simulated data below, the variable U2 shows a t-statistic of 1.2 inside the outcome model. That yields p equal to 0.23. It appears indistinguishable from noise. Inside the exposure model, the same variable shows a t-statistic of 31.5. Same variable, same data set. One model calls it irrelevant. The other calls it central. Every routine we normally invoke examines only the first column. 现在考虑将整个论点简化为两个 t 统计量。在下面的模拟数据中，变量 U2 在结果模型中的 t 统计量为 1.2，对应的 p 值为 0.23。它看起来与噪声无异。而在暴露模型中，同一个变量的 t 统计量为 31.5。同一个变量，同一数据集，一个模型称其无关紧要，另一个模型则称其至关重要。我们通常调用的每一个程序都只检查第一列。

This outcome follows from arithmetic. Omitted variable bias from excluding U2 equals: It is the product of U2’s coefficient in the outcome equation times its coefficient in the regression of X on U2 (i.e., delta). The first factor stays small. That is exactly why BIC remains unmoved. The second factor grows large because U2 tracks exposure closely. Their product contaminates the target estimate. No outcome fit rule ever inspects the second factor. 这一结果源于算术。排除 U2 导致的遗漏变量偏差等于：U2 在结果方程中的系数与它在 X 对 U2 回归中的系数（即 delta）的乘积。第一个因子保持很小，这正是 BIC 保持不变的原因。第二个因子变得很大，因为 U2 与暴露变量紧密相关。它们的乘积污染了目标估计值。没有任何结果拟合规则会去检查第二个因子。

Two quantities. Fit rules register only one. BIC remains selection consistent. With sufficient observations, it would retain U2. The present difficulty is finite sample. The asymptotic disappearance of bias offers little solace when the data set at hand contains roughly 1,000 rows. 两个量，拟合规则只记录一个。BIC 在选择上是一致的。如果有足够的观测值，它会保留 U2。目前的困难在于有限样本。当手头的数据集只有大约 1,000 行时，偏差在渐近意义上的消失并不能带来多少安慰。

The failure mode, in Python

Python 中的失效模式

Let’s simulate the setup in Figure 1 and fit two outcome models, i.e., the fully adjusted one and one that drops U2. 让我们模拟图 1 中的设置并拟合两个结果模型，即完全调整模型和剔除 U2 的模型。

import numpy as np, pandas as pd
import statsmodels.api as sm

rng = np.random.default_rng(11)
n, M = 1000, 5
U = rng.normal(size=(n, M)) # candidates U1..U5
X = U[:,0] + U[:,1] + 0.1*U[:,2] + rng.normal(size=n) # exposure
Y = 0.1*X + U[:,0] + 0.1*U[:,1] + U[:,2] + U[:,3] \
    + rng.normal(size=n) # true effect = 0.1

cols = [f"U{j+1}" for j in range(M)]
df = pd.DataFrame(U, columns=cols); df["X"] = X; df["Y"] = Y

def fit_outcome(includes):
    feats = ["X"] + [c for c, inc in zip(cols, includes) if inc]
    return sm.OLS(df["Y"], sm.add_constant(df[feats])).fit()

full = fit_outcome((1,1,1,1,0)) # adjusts for U1..U4
noU2 = fit_outcome((1,0,1,1,0)) # silently drops confounder U2

The results (true effect is 0.1): 结果（真实效应为 0.1）：

• adjusts for U1,U2,U3,U4: β̂ = 0.144, 95% CI (0.083, 0.206), BIC = 2907.6
• drops U2: β̂ = 0.171, 95% CI (0.127, 0.214), BIC = 2902.1

Here, three things happened: 在这里，发生了三件事：

1. The model missing a true confounder has the better BIC. By the usual logic, it is the model we are going to choose. 缺失了真实混杂因素的模型具有更好的 BIC。按照通常的逻辑，这将是我们选择的模型。
2. Its confidence interval excludes the true value. We would report a significant effect of ~0.17 with confidence and actually be completely wrong. 其置信区间排除了真实值。我们会自信地报告一个约 0.17 的显著效应，但实际上完全错了。
3. The fully adjusted model, penalized for carrying a “useless” predictor, is the one whose interval actually covers the truth. The two models fit the data equally well. However, they estimate different parameters. 完全调整模型虽然因携带“无用”预测变量而受到惩罚，但其区间实际上覆盖了真实值。这两个模型对数据的拟合程度相当，但它们估计的是不同的参数。