CaRE: Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

CaRE：掩码扩散语言模型的计算感知重掩码评估协议

Abstract: Masked diffusion language models (MDLMs) are advancing rapidly, yet the evaluation standards needed to reliably interpret their progress have not kept pace. Despite MDLMs becoming competitive with autoregressive language models, seven recent remasking papers evaluate under incompatible settings, varying nominal step counts, metrics, and sampling temperatures without jointly controlling these factors, rendering their strategy rankings largely incomparable and leaving open whether reported gains reflect algorithmic improvements or evaluation artifacts.

摘要： 掩码扩散语言模型（MDLM）正在迅速发展，但可靠地解读其进展所需的评估标准却未能跟上步伐。尽管 MDLM 已具备与自回归语言模型竞争的能力，但最近的七篇重掩码（remasking）论文在不兼容的设置下进行评估，在未联合控制标称步数、指标和采样温度等因素的情况下进行评估，导致其策略排名在很大程度上无法比较，也使得所报告的增益究竟是反映了算法改进还是评估伪影（evaluation artifacts）仍未可知。

We present CaRE, a compute-aware evaluation framework that audits MDLM remasking strategies by standardizing actual number of function evaluations (NFE), enforcing multi-metric reporting, and explicitly controlling stochasticity. Applied to 7 remasking strategies across LLaDA-8B-Base and Dream-7B-Base at 4 stochasticity levels and 3 step budgets on OpenWebText and LM1B, CaRE reveals that: (i) temperature explains the majority of MAUVE variance, (ii) compute-matched comparisons reverse several published strategy rankings, and (iii) informed remasking and stochastic unmasking are in tension, with high-entropy remasking reducing MAUVE by 0.296 at 256 steps at unmask_temp=0.25 (p=0.020).

我们提出了 CaRE，这是一个计算感知评估框架，通过标准化实际函数评估次数（NFE）、强制执行多指标报告并显式控制随机性，来审计 MDLM 的重掩码策略。通过在 OpenWebText 和 LM1B 数据集上，针对 LLaDA-8B-Base 和 Dream-7B-Base 模型，在 4 个随机性水平和 3 个步数预算下应用 7 种重掩码策略，CaRE 揭示了：(i) 温度解释了 MAUVE 方差的大部分；(ii) 在计算匹配的比较下，此前发表的几种策略排名被逆转；(iii) 启发式重掩码与随机去掩码之间存在冲突，在高熵重掩码下，在 256 步且 unmask_temp=0.25 时，MAUVE 指标下降了 0.296 (p=0.020)。

A CaRE leaderboard covering 12 open-weight MDLMs (150M to 8B parameters) shows that this interaction direction holds across architectures and scales. These findings demonstrate that current MDLM evaluations can systematically conflate algorithmic improvements with hidden choices of compute and stochasticity. We release the evaluation protocol, implementation, and leaderboard to ensure future remasking claims are reproducible and comparable.

一个涵盖 12 个开源权重 MDLM（参数量从 1.5 亿到 80 亿）的 CaRE 排行榜显示，这种交互方向在不同架构和规模下均成立。这些发现表明，当前的 MDLM 评估可能会系统性地将算法改进与计算和随机性的隐藏选择混为一谈。我们发布了该评估协议、实现代码和排行榜，以确保未来的重掩码研究成果具有可复现性和可比性。