Aggregating Visual Information with Optimal Transport for VideoLM Token Compression

利用最优传输聚合视觉信息以实现视频语言模型（VideoLM）的 Token 压缩

Abstract: Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression.

摘要： 视频语言模型将视频处理为密集的视觉 Token 序列，其中存在大量的表征冗余。因此，压缩这些序列对于减轻语言模型解码时的视觉 Token 负担至关重要。其核心挑战在于如何在压缩过程中保留分散在各帧中的视觉信息。

To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed.

为此，我们引入了“利用最优传输聚合视觉信息”（AVIOT）方法，该方法将视频 Token 压缩建模为将帧观测值的密集经验测度传输到紧凑的目标测度上。由此产生的源到目标的耦合，为每个目标支撑点（target support）诱导出一个源观测值的分布，从而直接规定了压缩后的视频表征应如何构建。

We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content.

我们进一步在任务和空间维度上对这一构建进行了适配。问题条件化（Question conditioning）调节了源帧与目标支撑点之间的传输成本，同时影响了分配给每个时间段的支撑点数量，从而将表征能力引导至与问题相关的内容上。

At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.

在多个空间粒度上，AVIOT 计算区域特定的时间传输计划，并自适应地融合它们产生的表征，使得同一紧凑表征内的不同区域能够从不同的时间点提取信息。在不同压缩比下的评估表明，AVIOT 在多个视频理解基准测试中达到了或超过了未压缩基线的水平，同时在高压缩比下仍保持了强大的性能。