My Model Was Cheating on Its Own Test

我的模型在自己的测试中作弊了

Machine Learning My Model Was Cheating on Its Own Test A preprocessing pipeline let my car price model peek at the test set before the exam, and the twelve points of R squared it cheated its way to Abdullahi Dattijo Aug 14, 2026 13 min read Share Image generated by the author using OpenAI’s image model. 机器学习：我的模型在自己的测试中作弊了。一个预处理流水线让我的汽车价格模型在考试前偷看了测试集，它通过作弊获得了 12 个 R 平方点的提升。作者：Abdullahi Dattijo，2026 年 8 月 14 日，阅读时长 13 分钟。图片由作者使用 OpenAI 的图像模型生成。

Data leakage happens when a model gets to see information it should not have access to yet, and that hidden peek makes its score look better than it really is. Kaufman, Rosset, Perlich, and Stitelman describe this clearly in their 2012 paper in ACM Transactions on Knowledge Discovery from Data, “Leakage in Data Mining: Formulation, Detection, and Avoidance.” 数据泄露是指模型获取了本不该接触到的信息，而这种“偷看”行为会让模型的评分看起来比实际表现更好。Kaufman、Rosset、Perlich 和 Stitelman 在他们 2012 年发表于《ACM 知识发现数据汇刊》的论文《数据挖掘中的泄露：公式化、检测与规避》中对此进行了清晰的描述。

One example they give is the INFORMS 2010 Data Mining Challenge. Competitors were supposed to predict stock price movements using only a training set, then be scored on a separate test set. Several competitors figured out which real stocks were hidden in the test set by matching patterns against public finance data. That let them pull in information the test set was supposed to keep hidden, and their scores looked better than their models actually deserved. 他们给出的一个例子是 2010 年的 INFORMS 数据挖掘挑战赛。参赛者本应仅使用训练集来预测股价走势，然后在独立的测试集上进行评分。几位参赛者通过将模式与公开的金融数据进行匹配，找出了测试集中隐藏的真实股票。这让他们获取了本应保密的测试集信息，从而使他们的模型评分看起来比其实际应得的要高。

My own leak was much smaller and much easier to miss. It did not involve figuring out hidden stock identities. It came from running two ordinary lines of preprocessing code in the wrong order. I trained a small neural network, using scikit-learn’s MLPRegressor, to predict car prices from a used car dataset. The first version of this project reported a strong test score: an R squared of 0.887. After I fixed the order of two lines, the honest score was 0.767. The model did not get worse. My first measurement had been quietly reading part of the answer key. 我遇到的泄露要小得多，也更容易被忽略。它不涉及破解隐藏的股票身份，而是源于将两行普通的预处理代码顺序写反了。我使用 scikit-learn 的 MLPRegressor 训练了一个小型神经网络，用于根据二手车数据集预测汽车价格。该项目的第一个版本报告了一个强劲的测试分数：R 平方为 0.887。在我修正了两行代码的顺序后，真实的分数是 0.767。模型并没有变差，只是我最初的测量结果在悄悄地读取部分答案。

The number that looked solid R squared is a common score for how well a model predicts a number. It runs from 0 to 1. Higher is better. An R squared of 0.887 means the model explains about 89 percent of the differences in car price on data it had never trained on. The original version of this project reported a test R squared of 0.887 and a test error, measured as mean squared error, of about 6.9 million. I reran the exact same code, with the exact same settings, to check whether that number was real, and it matched. For a small dataset of under 200 cars, that is a genuinely strong result, the kind that ends a homework assignment without further questions. 这个看起来很稳的数字——R 平方，是衡量模型预测数值准确度的一种常用指标。它的取值范围从 0 到 1，数值越高越好。R 平方为 0.887 意味着该模型能够解释其从未训练过的数据中约 89% 的汽车价格差异。该项目的原始版本报告的测试 R 平方为 0.887，测试误差（以均方误差衡量）约为 690 万。我使用完全相同的代码和设置重新运行了一遍，以验证该数字是否真实，结果是一致的。对于一个不到 200 辆车的小型数据集来说，这是一个非常强劲的结果，足以让一份家庭作业在无需进一步质疑的情况下圆满完成。

The experiment

实验

This project started from a leak I found in a car price model I had built for a class assignment. That version used a dataset with no license listed anywhere on the page it came from, so instead of reproducing it here, I rebuilt the same pipeline on UCI’s Automobile dataset, donated by Jeffrey Schlimmer in 1987 and sourced from the 1985 Ward’s Automotive Yearbook. It is released under a Creative Commons Attribution 4.0 license, which permits reuse like this with credit. 这个项目始于我在为课堂作业构建汽车价格模型时发现的一个泄露问题。那个版本使用的数据集在其来源页面上没有列出任何许可协议，因此我没有在这里复现它，而是在 UCI 的汽车数据集上重建了相同的流水线。该数据集由 Jeffrey Schlimmer 于 1987 年捐赠，源自 1985 年的《Ward 汽车年鉴》。它采用知识共享署名 4.0 许可协议发布，允许在注明出处的情况下进行此类重用。

The two datasets describe the same kind of thing: one row per car, with its specifications and its price. Each row lists a car’s make, body style, engine specs, fuel type, an assigned insurance risk rating, and price. After dropping a column with a large number of missing values and removing the rows with any remaining missing values, 193 cars remain. Of the 24 columns left to predict price from, 16 are numeric measurements, such as engine size, horsepower, and curb weight, and 8 are categorical, such as fuel type, drive wheel, and engine location. The goal is regression: predict price, which is a number, rather than sorting cars into categories. 这两个数据集描述的是同类事物：每一行代表一辆车，包含其规格和价格。每一行都列出了汽车的品牌、车身风格、发动机规格、燃料类型、分配的保险风险等级和价格。在删除了一列包含大量缺失值的列并移除了剩余包含缺失值的行后，剩下 193 辆车。在用于预测价格的 24 列中，16 列是数值测量值（如发动机排量、马力和整备质量），8 列是分类变量（如燃料类型、驱动轮和发动机位置）。目标是回归分析：预测价格（一个数值），而不是将汽车分类。

I used a multilayer perceptron, or MLP. An MLP is a type of neural network: a model made of layers of small connected units, where each unit combines its inputs and passes the result forward. This one has two hidden layers of 64 units each: 我使用了多层感知机（MLP）。MLP 是一种神经网络：由多层小型连接单元组成的模型，每个单元结合其输入并将结果向前传递。这个模型有两个隐藏层，每层 64 个单元：

MLPRegressor(
    hidden_layer_sizes=(64, 64),
    max_iter=1000,
    random_state=42,
)

I split the 193 rows into three groups, matching the same proportions as the original assignment: 60 percent for training, 20 percent for validation, and 20 percent for test, which comes to 115 training rows, 39 validation rows, and 39 test rows. The training set is what the model actually learns from. The validation set is meant to check the model while it is still being developed. The test set is a final, one time check, meant to be looked at only once the model is finished. 我将 193 行数据分为三组，比例与原始作业相同：60% 用于训练，20% 用于验证，20% 用于测试，即 115 行训练数据、39 行验证数据和 39 行测试数据。训练集是模型实际学习的数据；验证集用于在模型开发过程中进行检查；测试集是最终的一次性检查，旨在模型完成后仅查看一次。

Two lines, run in the wrong order

两行代码，顺序错误

Before any model can use this data, the raw columns need to be prepared. Numeric columns like horsepower need to be put on a common scale, since some numbers use small ranges and others use large ones. Categorical columns like fuel type need to be converted into numbers a model can use, commonly by turning each category into its own 0 or 1 column, a method called one hot encoding. This preparation step is often called preprocessing, and in scikit-learn it is usually done with a small chain of steps called a pipeline. 在任何模型使用这些数据之前，原始列都需要进行准备。像马力这样的数值列需要统一缩放，因为有些数字范围很小，而另一些则很大。像燃料类型这样的分类列需要转换为模型可以使用的数字，通常是将每个类别转换为其自己的 0 或 1 列，这种方法称为独热编码（One-hot encoding）。这个准备步骤通常被称为预处理，在 scikit-learn 中，它通常通过一系列称为流水线（Pipeline）的步骤来完成。

Here is the preprocessing code, in the order it ran to produce the strong looking result above: 以下是预处理代码，按其产生上述强劲结果的顺序排列：

# Outlier handling via IQR capping
for col in numerical_cols + ["price"]:
    Q1 = df[col].quantile(0.25)
    Q3 = df[col].quantile(0.75)
    IQR = Q3 - Q1
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR
    df[col] = df[col].clip(lower=lower_bound, upper=upper_bound)

# Preprocessing pipeline
preprocessor = ColumnTransformer(transformers=[
    ("num", StandardScaler(), numerical_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

# Transform the data
X_processed = preprocessor.fit_transform(X)

# Train-validation-test split
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X_processed, y, test_size=0.2, random_state=42
)

Nothing about this code signals a problem, which is part of the point. It is the same shape of mistake I originally found in the class assignment this project is based on, just written against a differently sourced dataset here so the result can be shared freely. Read the code in order, and watch what happens before the split. The first step caps extreme outlier values in each column, using 这段代码看起来没有任何问题，这正是问题的关键所在。这与我最初在课堂作业中发现的错误形式相同，只是在这里针对不同的数据集编写，以便可以自由分享结果。按顺序阅读代码，观察在拆分之前发生了什么。第一步使用以下方法限制了每一列中的极端异常值：