Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models

基于预训练 Transformer 感知模型的对抗鲁棒溯因融合

Abstract: Deploying pre-trained perception models in novel environments degrades their accuracy under distributional shift, and assembling them alone does not recover it: combiners such as majority voting trade recall for precision and are brittle to coordinated failures.

摘要： 在新环境中部署预训练感知模型时，分布偏移会导致其准确率下降，而单纯地将它们组合起来并不能恢复性能：诸如多数投票（majority voting）之类的组合方法往往以牺牲召回率为代价换取精确率，且对协同故障非常脆弱。

Prior metacognitive methods learn logical rules that flag a model’s errors, but rely on hand-authored domain-knowledge cues (object-size priors, segmentation masks) that do not transfer to genuinely novel scenes.

先前的元认知方法通过学习逻辑规则来标记模型的错误，但这些方法依赖于人工编写的领域知识线索（如物体尺寸先验、分割掩码），这些线索无法迁移到真正新颖的场景中。

We show that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry: per-model Label Vector Pools (LVP), built from each model’s own training embeddings, yield error-detection rules from the geometry of detections relative to training-determined prototypes, reaching parity with domain-knowledge rules to within $0.002$ every F1 on test set.

我们证明，通过利用向量空间几何结构，可以在没有任何领域知识的情况下学习这种元认知层：基于每个模型自身训练嵌入构建的“标签向量池”（Label Vector Pools, LVP），能够根据检测结果相对于训练确定原型的几何关系得出错误检测规则，其在测试集上的 F1 分数与领域知识规则的差距在 $0.002$ 以内。

Because the approach remains neurosymbolic, these geometric rules share a single logical framework and can still be complemented by domain knowledge when available.

由于该方法保持了神经符号（neurosymbolic）特性，这些几何规则共享一个统一的逻辑框架，并且在有领域知识可用时，仍可对其进行补充。

We frame the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a polynomial-time heuristic.

我们将多个不完美的基于 ViT 的检测器的融合建模为一个基于一致性的溯因问题，并在测试时通过精确的整数规划（IP）和多项式时间启发式算法来求解。

On an aerial imagery benchmark of 15 weather-shifted test sets and six ViT detectors, our domain-knowledge-free layer matches the strongest majority-vote variant on clean data (within $0.005$ F1) and, unlike every majority-vote baseline, retains its performance under a coordinated label-flipping attack: at a $90%$ flip rate it averages $0.42$ F1 versus $0.35$ for MV-Plurality (a $22%$ relative gain) and attains the highest F1 on \emph{every} test set once the flip rate exceeds $0.4$.

在包含 15 个天气偏移测试集和 6 个 ViT 检测器的航空影像基准测试中，我们这种无需领域知识的层在干净数据上的表现与最强的多数投票变体相当（F1 差距在 $0.005$ 以内）；与所有多数投票基准不同的是，它在协同标签翻转攻击下仍能保持性能：在 $90%$ 的翻转率下，其平均 F1 为 $0.42$，而 MV-Plurality 为 $0.35$（相对提升 $22%$），且一旦翻转率超过 $0.4$，它在每一个测试集上都能达到最高的 F1 分数。