Clustering and Token Denoising for Faster and More Robust VLMs
Clustering and Token Denoising for Faster and More Robust VLMs
用于更快、更稳健视觉语言模型（VLM）的聚类与 Token 去噪技术
Recent Visual-Language Models (VLMs) have enhanced the capabilities of pre-trained LLMs by adding vision tokens alongside text, with approaches like LLaVA showing impressive results. However, the computational burden of processing up to 576 or 729 visual tokens makes edge deployment challenging. While various token pruning techniques require retraining, some are training-free and thus can easily adapt to architecture changes.
近期的视觉语言模型（VLM）通过在文本之外增加视觉 Token，增强了预训练大语言模型（LLM）的能力，其中 LLaVA 等方法展现出了令人瞩目的成果。然而，处理多达 576 或 729 个视觉 Token 所带来的计算负担，使得在边缘设备上的部署极具挑战性。虽然各种 Token 剪枝技术通常需要重新训练，但也有一些无需训练的方法，能够轻松适应架构的变化。
We introduce ClustRS, a two-part, training-free algorithm for robust token pruning. Its first component is an attention-weighted, clustering algorithm that selects representative tokens from each semantic cluster. The second component, Residual Shrinkage, is a one-pass denoising step on the selected tokens. These training-free lightweight steps make LLaVA ready for real-world data, improving robustness to a wide range of image-noise types and intensities.
我们引入了 ClustRS，这是一种由两部分组成的、无需训练的稳健 Token 剪枝算法。其第一部分是一种注意力加权的聚类算法，用于从每个语义簇中选择代表性 Token。第二部分是“残差收缩”（Residual Shrinkage），这是对所选 Token 进行的一次性去噪步骤。这些无需训练的轻量级步骤使 LLaVA 能够适应真实世界的数据，并提高了其对多种图像噪声类型和强度的稳健性。
Experimental results on the ScienceQA-IMG and MM-VET benchmarks show our method outperforms attention- and diversity-based methods by up to 20% under extreme noise and token conditions (reducing tokens by 97%, down to 16 tokens) on LLaVA 1.5 7b and achieves exceptional results on LLaVA-OneVision, where we match baseline performance with fewer than one-third of their tokens under mild noise conditions. Our study demonstrates a simple yet powerful alternative to both score-only and diversity-only pruning rules, paving the way for compute-efficient and noise-resilient VLM deployment.
在 ScienceQA-IMG 和 MM-VET 基准测试上的实验结果表明，在极端噪声和 Token 限制条件下，我们的方法在 LLaVA 1.5 7b 上比基于注意力和基于多样性的方法性能提升了高达 20%（Token 减少了 97%，降至 16 个）。在 LLaVA-OneVision 上，该方法也取得了卓越的成果：在轻微噪声条件下，仅使用不到三分之一的 Token 即可达到基准模型的性能。我们的研究展示了一种简单而强大的替代方案，能够取代仅基于分数或仅基于多样性的剪枝规则，为实现计算高效且抗噪的 VLM 部署铺平了道路。