GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

查看原文 / View Original 编辑 / Edit

Computer Science > Computation and Language arXiv:2611.00319 (cs) [Submitted on 30 Oct 2026] Title: GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport Authors: Rajdeep Pal, Jotiraditya Banerjee, Saptarshi Pani.

计算机科学 > 计算与语言 arXiv:2611.00319 (cs) [提交于 2026 年 10 月 30 日] 标题：GAVAGAI：通过平衡最优传输实现婴儿规模视觉语言模型的指称对齐作者：Rajdeep Pal, Jotiraditya Banerjee, Saptarshi Pani。

Abstract: A child learns which words name which things from roughly 100 million words of input. Vision-language models trained at a comparable scale bind words to referents far less reliably, and we argue that part of the gap is structural: the usual next-token captioning objective has no variable standing for “this word refers to that region,” so training applies no direct pressure to ground words.

摘要：儿童通过大约 1 亿个词的输入来学习哪些词指代哪些事物。在相当规模下训练的视觉语言模型将词语与指称对象绑定的可靠性要低得多，我们认为这种差距的部分原因是结构性的：通常的下一个标记（next-token）字幕目标没有代表“这个词指代那个区域”的变量，因此训练过程对词语的落地（grounding）没有施加直接压力。

We add such a variable by treating each scene as a small assignment problem between the content words heard and the regions seen, solved as entropic optimal transport under two constraints. First, a word may be assigned to a special NOTHING column instead of any region, which matters because most content words in caregiver speech do not name anything visible.

我们通过将每个场景视为所听到的内容词与所看到的区域之间的小型分配问题来添加这样一个变量，并将其作为在两个约束条件下的熵最优传输问题来解决。首先，一个词可以被分配到一个特殊的“无（NOTHING）”列，而不是任何区域，这一点很重要，因为看护者话语中的大多数内容词并不指代任何可见的事物。

Second, no region may receive more than a bounded share of the assignment mass, a capacity constraint that encodes mutual exclusivity. The alignment step costs about eight Sinkhorn iterations, small next to the vision encoder.

其次，没有任何区域可以接收超过分配质量的有限份额，这是一种编码了互斥性的容量约束。对齐步骤大约需要八次 Sinkhorn 迭代，相对于视觉编码器而言开销很小。

在受控模拟中，该想法的原始版本（一种强制每个词对应某个区域的逐行 softmax）在现实的非指称率下，词图准确率下降到 0.000，低于仅使用字幕生成的 0.058；而我们的两个约束在该机制下达到了 0.075，在干净输入下达到了 0.700。该模型的一个参数版本在三种歧义条件下重现了成人跨情境学习的准确率（RMSE 0.024）。

In controlled simulation, the naive version of the idea, a row-wise softmax that forces every word onto some region, collapses to 0.000 word-to-picture accuracy under realistic non-reference rates, below the 0.058 of captioning alone, while our two constraints reach 0.075 in that regime and 0.700 on clean input. A one-parameter version of the model reproduces adult cross-situational learning accuracy across three ambiguity conditions (RMSE 0.024).

在 SAYCam 婴儿视频语料库上，将对齐损失添加到下一个标记字幕模型中，使词图准确率从 32.4% 提高到 48.7%，定位 IoU 从 0.18 提高到 0.38。在同一儿童数据上训练的对比模型在其自身的基准测试中仍然更强，因此我们的主张是该损失函数在高非指称率下改进了字幕生成目标，而不是设定了新的最佳水平。

On the SAYCam infant video corpus, adding the alignment loss to a next-token captioning model raises word-to-picture accuracy from 32.4% to 48.7% and localization IoU from 0.18 to 0.38. A contrastive model trained on the same child’s data remains stronger on its own benchmark, so our claim is that the loss improves a captioning objective under high non-reference rates, not that it sets a new best.