How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration

Computer Science > Machine Learning arXiv:2610.33168 (cs) [Submitted on 23 Oct 2026] Title: How Much Fresh Data Should CTR Recalibration Use? A Temporal Evaluation of Post-hoc Calibration Authors: Rui Liu.

计算机科学 > 机器学习 arXiv:2610.33168 (cs) [提交于 2026 年 10 月 23 日] 标题：点击率（CTR）重校准应使用多少新鲜数据？事后校准的时间评估 作者：Rui Liu。

Abstract: Offline evaluation of click-through rate (CTR) models still defaults to random splits, although deployed models score future traffic and downstream systems consume calibrated probabilities. On Avazu (40.4M impressions) we train logistic regression (LR), DeepFM, and DCN-v2 under two rolling temporal origins and a random split, three seeds per arm, and recalibrate each test day with Platt scaling and isotonic regression fitted on the preceding hours.

摘要：点击率（CTR）模型的离线评估仍然默认使用随机划分，尽管已部署的模型会对未来的流量进行评分，且下游系统会使用校准后的概率。在 Avazu 数据集（4040 万次展示）上，我们在两个滚动时间起点和一个随机划分下训练了逻辑回归（LR）、DeepFM 和 DCN-v2 模型，每个分支使用三个随机种子，并利用前几个小时的数据通过 Platt 缩放和保序回归对每个测试日进行重校准。

At origin 1’s nearest test day, random splits report LR’s expected calibration error (ECE) at 55-62% of the temporal value in every seed (the deep models’ ratio is seed-unstable), and no model’s ECE is monotone in temporal distance. With a 6-hour fitting slice, recalibration removes up to 61% of ECE in badly miscalibrated cells but increases it in about a third of applications (14-17 of 48 across seeds).

在起点 1 最近的测试日，随机划分报告的逻辑回归（LR）预期校准误差（ECE）在每个种子中均为时间价值的 55-62%（深度模型的比例在种子间不稳定），且没有任何模型的 ECE 随时间距离呈单调变化。使用 6 小时的拟合切片时，重校准在校准极差的单元中消除了高达 61% 的 ECE，但在约三分之一的应用中（跨种子 48 次应用中的 14-17 次）反而增加了误差。

On fixed evaluation sets, a 12-hour slice cuts harm from 13-17 to 3-5 of 36 applications, and 24 hours raises it again. The 12-hour slice subsampled to the 1-hour slice’s size (5% of the impressions; 20-25x faster to fit) recovers 88% of the full slice’s gain, so impression count alone does not explain it.

在固定评估集上，12 小时的切片将负面影响从 36 次应用中的 13-17 次减少到 3-5 次，而 24 小时切片则再次增加了负面影响。将 12 小时切片下采样至 1 小时切片的大小（展示量的 5%；拟合速度快 20-25 倍）可恢复完整切片收益的 88%，因此仅凭展示次数无法解释这一现象。

A slice-ECE gate shows no benefit out-of-origin, and in a simulated auction recalibration moves profit by only about -0.21% to +0.26%. Offline protocols should measure calibration temporally, over multiple origins and seeds; in this data, recalibration fitted on a wide time span beat the freshest few hours, which other logs must confirm.

切片-ECE 门控显示在起点之外没有收益，在模拟拍卖中，重校准仅使利润变动约 -0.21% 至 +0.26%。离线协议应在多个起点和种子下按时间维度衡量校准情况；在本数据中，基于较长时间跨度拟合的重校准效果优于最近的几个小时，这一点有待其他日志证实。