OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

Abstract: Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric.

摘要： 开放词汇单目 3D 检测器在领域内表现出色，但各研究采用的评估协议各不相同，且部分模型依赖于部署时无法获取的“单图像类别预言机”（per-image category oracles），此外，所有模型都将几何与语义合并为一个单一的 AP 指标。

To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer.

为了解决这些问题，我们推出了 OV3D-Bench，这是一个诊断性基准测试，旨在模拟部署环境下的真实条件，对七个室内外数据集上的开放词汇单目 3D 检测器进行比较。我们的基准测试用测试时的“数据集级类别名称提示”（dataset-level class name prompts）取代了单图像类别预言机，并从定位、语义鲁棒性和跨域迁移三个维度解耦了检测准确率。

We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D’s performance collapses from 18.6 to 5.4 AP when prompted with “a detailed high-resolution photo of a car” rather than “car”); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D’s AP by 1.9 $\times$ on ScanNet).

我们评估了七种代表性检测器，发现：(i) 它们能很好地定位物体，但经常将定位正确的框误标记为语义相近的类别；(ii) 准确率对提示词的措辞高度敏感（例如，当提示词从“car”变为“a detailed high-resolution photo of a car”时，WildDet3D 的性能从 18.6 AP 骤降至 5.4 AP）；(iii) 广泛采用的“目标感知协议”（target-aware protocol）掩盖了这些错误（例如，在 ScanNet 上将 DetAny3D 的 AP 指标虚高了 1.9 倍）。

Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector’s predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.

最后，我们证明了仅通过使用对比视觉-语言编码器（如 SigLIPv2）对冻结的闭集词汇检测器的预测结果进行重映射，其性能即可与近期专门构建的开放词汇方法相媲美。这表明几何定位技术已趋于成熟，而开放词汇语义仍是当前的主要瓶颈。