The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning
The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning
“未写之书”基准测试：多模态机器学习在抽象感知推理领域的新挑战
Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content. However, their capacity for abstract perceptual reasoning, inferring unseen information from dynamic, generative processes, remains a critical and underexplored frontier. 当前的各类多模态模型在识别静态视觉和听觉内容方面已经展现出了卓越的能力。然而，它们在抽象感知推理——即从动态生成过程中推断不可见信息——方面的能力，仍然是一个关键且尚未被充分探索的前沿领域。
In this paper, we introduce The Unwritten Benchmark, a new challenge designed to probe this abstract perceptual and cognitive ability. We define the core task as acousto-kinematic word inference: models must decipher words, across 3 different writing styles, being written solely from the audio of pen scratches and the video of hand movements, without any visible ink trace. 在本文中，我们引入了“未写之书”（The Unwritten Benchmark）基准测试，这是一项旨在探测这种抽象感知与认知能力的新挑战。我们将核心任务定义为“声-动词汇推理”：模型必须在没有任何可见墨迹的情况下，仅通过笔尖摩擦的音频和手部动作的视频，破译出以三种不同书写风格写出的单词。
Our evaluation results reveal a profound gap between human and machine performance: while human participants achieve high ordered letter accuracy (over 80%), leading Multimodal Machine Learning Models, including GPT-4o and Gemini 2.5-Pro, struggle significantly, failing to surpass 10%. 我们的评估结果揭示了人类与机器表现之间巨大的鸿沟：人类参与者在字母顺序准确率上表现优异（超过 80%），而包括 GPT-4o 和 Gemini 2.5-Pro 在内的领先多模态机器学习模型却表现挣扎，准确率未能超过 10%。
Furthermore, we identify a paradoxical fusion effect in the models, where providing both modalities often degrades performance rather than improving it. This finding indicates a fundamental breakdown in their ability to synthesize complementary perceptual cues for this cognitive task. 此外，我们还在模型中发现了一种矛盾的融合效应：同时提供两种模态的信息往往会降低而非提升模型的表现。这一发现表明，模型在为这类认知任务合成互补感知线索的能力上存在根本性的缺陷。
These findings highlight significant limitations in both cross-modal causal reasoning and the understanding of the micro-kinematics essential for such cognitive and intuitive perceptual reasoning. 这些发现凸显了模型在跨模态因果推理，以及理解此类认知和直觉感知推理所必需的微观运动学方面存在的重大局限性。