What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus

99% 的准确率到底衡量了什么？对广泛使用的假新闻语料库中“捷径学习”的可复现审计

---

Text classifiers trained on the ISOT/Kaggle “Fake and Real News” corpus routinely report accuracy and F1 above 0.98, a level of performance that sits uneasily beside the difficulty of assessing veracity. Using a transparent TF-IDF and linear-classifier pipeline as a measurement instrument, we audit the corpus along three leakage channels and two distribution-shift protocols, releasing all code and derived numbers.

在 ISOT/Kaggle “假新闻与真实新闻”语料库上训练的文本分类器通常报告准确率和 F1 分数超过 0.98，这种性能水平与评估新闻真实性的实际难度显得格格不入。我们使用透明的 TF-IDF 和线性分类器流水线作为测量工具，沿着三个泄露渠道和两个分布偏移协议对该语料库进行了审计，并公开了所有代码和衍生数据。

First, the benchmark is partly degenerate: a classifier given only the subject metadata field, with the article text discarded, attains F1 = 1.000, since the two classes have disjoint subjects.

首先，该基准测试在一定程度上是退化的：如果分类器仅使用主题元数据字段而丢弃文章正文，其 F1 分数可达到 1.000，因为两类数据的主题是完全不重叠的。

Second, removing all three leakage channels, metadata, a newswire source tag present in 99.2% of real articles, and 6,251 duplicate documents contaminating 19.4% of a naive test split, lowers F1 by only 1.21 points (0.9935 to 0.9814); the residual signal is diffuse editorial style rather than a few giveaway tokens, since deleting the 1,000 highest-weight unigrams still leaves F1 = 0.926.

其次，移除所有三个泄露渠道（元数据、存在于 99.2% 真实文章中的新闻来源标签，以及污染了 19.4% 原始测试集的 6,251 份重复文档）后，F1 分数仅下降了 1.21 个百分点（从 0.9935 降至 0.9814）；剩余的信号是分散的编辑风格，而非少数几个关键特征词，因为即使删除了权重最高的 1,000 个一元词（unigrams），F1 分数仍保持在 0.926。

Third, this style signal does not transfer: under a topic-disjoint protocol, average precision falls from 0.9995 to 0.9475 and deployed F1 from 0.9905 to 0.8067, with a prior-matched analysis confirming a genuine 5.2-point loss of discrimination, while temporal transfer is nearly lossless. A fine-tuned DistilBERT is stronger in-distribution (F1 = 0.9993) but degrades far more under topic shift, losing 12.9 average-precision points against the linear model’s 5.2.

第三，这种风格信号无法迁移：在主题不重叠的协议下，平均精度从 0.9995 下降到 0.9475，部署后的 F1 分数从 0.9905 下降到 0.8067。先验匹配分析证实了辨别力确实损失了 5.2 个百分点，而时间上的迁移几乎没有损失。经过微调的 DistilBERT 在同分布测试中表现更强（F1 = 0.9993），但在主题偏移下退化得更为严重，平均精度损失了 12.9 个百分点，而线性模型仅损失 5.2 个百分点。

Transferred to the independent LIAR benchmark, all three models fall to near-chance ranking (ROC-AUC 0.54-0.57), none beating a majority-class baseline. We conclude that within-corpus scores here quantify source and topic separability rather than veracity, that added capacity exploits the shortcut rather than avoiding it, and we recommend metadata-only, small-sample, and topic-disjoint baselines as inexpensive diagnostics for future work.

当迁移到独立的 LIAR 基准测试时，所有三个模型的表现都降至接近随机猜测的水平（ROC-AUC 为 0.54-0.57），没有一个模型能超过多数类基准。我们得出结论：该语料库内的分数衡量的是来源和主题的可分性，而非真实性；增加模型容量反而是在利用“捷径”而非避免它。我们建议将仅使用元数据、小样本和主题不重叠作为未来研究中低成本的诊断基准。