The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
混合比例并非性能关键：诊断视觉-语言模型少样本适应中的原型混合方法
Abstract: Many few-shot adaptation methods for vision-language models classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, with a single blending ratio routinely tuned on held-out labels, often on the test set itself. We ask what the family’s own bias-variance justification invites: what is the right ratio, can it be estimated without validation data, and is finding it where the performance is?
摘要： 许多针对视觉-语言模型的少样本适应方法，通过零样本（zero-shot）文本原型与 K 个标注图像特征均值的凸组合进行分类，并通常在留出标签（往往直接在测试集上）对单一混合比例进行调优。我们基于该方法族自身的偏差-方差理论提出了疑问：什么是正确的比例？能否在没有验证数据的情况下对其进行估计？以及，找到这个比例真的是性能提升的关键吗？
First, the ratio minimising prototype mean-squared error has a closed form whose support-set plug-in is exactly a positive-part James-Stein coefficient shrinking towards the text prototype. Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity: on the 950 primary-tier cells where it is defined it trails a test-set-oracle ratio by 8.5 points. It saturates near 1, discarding the text prior for a nearest-class-mean classifier, because 78% of the text-image prototype distance it treats as bias is a class-independent offset that the arg max largely cancels. We prove the mechanism and bound its share of the damage at 26% by a counterfactual.
首先，最小化原型均方误差的比例具有闭式解，其在支持集上的代入值恰好是一个向文本原型收缩的正部 James-Stein 系数。在 4,800 个实验单元（涵盖十个数据集、五个骨干网络包括 SigLIP、五种样本量、五个随机种子、四个提示词层级）中，这一理论最优比例是对“错误目标”的可靠估计：在定义该比例的 950 个主要层级单元中，其表现比测试集预言机（oracle）比例低 8.5 个百分点。该比例趋近于 1，从而舍弃了文本先验，退化为最近类均值分类器，原因在于它所处理的 78% 的文本-图像原型距离（被视为偏差）实际上是与类别无关的偏移量，而 arg max 操作在很大程度上抵消了这一偏移。我们证明了该机制，并通过反事实分析将其造成的性能损失限制在 26%。
Second, leave-one-out on the support set alone sets a ratio landing within 0.9 points of the oracle blend, so it is estimable without validation data.
其次，仅在支持集上使用留一法（leave-one-out）即可设定一个与预言机混合比例相差不到 0.9 个百分点的比例，因此该比例无需验证数据即可估计。
Third, validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average, and at K >= 4 all four validation-free baselines sit above the oracle, the linear probes by margins excluding zero. These results locate the ceiling in the model class, not the hyperparameter: the ratio can be set near-optimally for free, and it is still not where the performance is.
第三，无需验证的线性探测（linear probes）甚至超过了预言机调优的混合方法：CLAP 平均高出 1.9 个百分点，LP++ 高出 1.5 个百分点；当 K >= 4 时，所有四个无需验证的基线方法表现均优于预言机，且线性探测的优势幅度显著。这些结果表明，性能上限在于模型类别本身，而非超参数：即使比例可以免费设定为近乎最优，它依然不是决定性能的关键所在。