Null Controls Are Necessary but Not Sufficient: Evaluating Acceptance Guards for Scientific Discovery and Interpretability
Automated discovery systems propose scientific explanations and test which to accept. Does passing a test establish the intended claim? We develop four executable operations for changing physical descriptions: identifying redundancies (L1), coarse-graining dynamics (L2), transferring structure (L3), and inferring missing quantities from conservation imbalances (L4).
自动化发现系统会提出科学解释并测试应接受哪些解释。通过测试是否就确立了预期的主张？我们开发了四种用于改变物理描述的可执行操作：识别冗余（L1）、粗粒化动力学（L2）、迁移结构（L3）以及从守恒不平衡中推断缺失量（L4）。
We evaluate these operations and three interpretability settings through null acceptance (NAR), positive acceptance (PAR), and consequence agreement (CAR). A reduction mapping every state to zero predicts its own future perfectly. It passes closure on 96% of 129 public chaotic systems despite retaining no dynamical variation. An independent observable-preservation test rejects it throughout.
我们通过零接受（NAR）、正接受（PAR）和结果一致性（CAR）来评估这些操作和三种可解释性设置。一种将每个状态映射为零的归约方法可以完美预测其自身的未来。尽管它没有保留任何动力学变化，但在 129 个公共混沌系统中，它有 96% 的情况通过了闭包测试。一项独立的可观测性保持测试始终将其拒绝。
On controlled benchmarks, the combined criteria accept 24/24 genuine reductions and reject 72/72 nulls. The other operations distinguish symmetries from redundancies, validate transfers, and reject spurious conservation imbalances. In language and ESM-2 protein models, linear CKA remains unchanged when a proposed correspondence is randomly rotated, whereas localized interventions distinguish genuine mappings from tested nulls.
在受控基准测试中，组合标准接受了全部 24 个真正的归约，并拒绝了全部 72 个无效项。其他操作区分了对称性与冗余，验证了迁移，并拒绝了虚假的守恒不平衡。在语言模型和 ESM-2 蛋白质模型中，当提议的对应关系被随机旋转时，线性 CKA 保持不变，而局部干预则能将真正的映射与测试的无效项区分开来。
In sparse autoencoders, enrichment rejects all 16 tested nulls, yet independent ablation supports 45% of evaluable enriched features, compared with 0.5% of matched controls. These results reveal two acceptance gaps: descriptive tests can accept scientifically empty candidates, and accepted candidates can lack independently verified consequences. Reliable automated discovery requires evaluating both what a test accepts and the scientific consequences it establishes.
在稀疏自动编码器中，富集（enrichment）拒绝了所有 16 个测试的无效项，但独立消融实验支持 45% 可评估的富集特征，而匹配对照组仅为 0.5%。这些结果揭示了两个接受差距：描述性测试可能会接受科学上空洞的候选对象，而通过的候选对象可能缺乏经独立验证的结果。可靠的自动化发现需要同时评估测试接受的内容及其所确立的科学后果。