What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

文本到视觉迁移了什么？视觉语言模型（VLM）的能力缩放定律与迁移动态

Abstract: Choosing the right large language model (LLM) backbone is the most consequential decision when building a vision-language model (VLM), yet it remains fundamentally unprincipled: compute-based scaling laws fail to generalize across model families, and no framework exists for directly predicting VLM performance before training begins.

摘要： 在构建视觉语言模型（VLM）时，选择合适的预训练大语言模型（LLM）作为基座是最关键的决策，但目前这一过程仍缺乏根本性的原则指导：基于计算量的缩放定律（Scaling Laws）无法在不同模型家族间泛化，且目前尚无框架能在训练开始前直接预测 VLM 的性能。

We propose the Capability-Driven Multimodal Scaling Law, the first cross-family framework that predicts VLM benchmark accuracy from directly observable textual capability. Given a low-dimensional capability score $S$ extracted from LLM textual benchmarks via PCA, we model VLM performance as a function of $S$, with a per-backbone transfer rate and an absorption rate that quantifies data-scaling efficiency.

我们提出了“能力驱动的多模态缩放定律”，这是首个能够通过直接可观测的文本能力来预测 VLM 基准测试准确率的跨家族框架。通过主成分分析（PCA）从 LLM 文本基准测试中提取低维能力得分 $S$，我们将 VLM 性能建模为 $S$ 的函数，并引入了针对每个基座的“迁移率”以及量化数据缩放效率的“吸收率”。

To fit and validate the framework, we train over 150 VLMs on 34 LLMs spanning 7 model families under a strictly controlled recipe. Evaluations on more than 200 textual and 50 multimodal benchmarks show that the law accurately extrapolates transfer rate from models up to 8B parameters to 72B-scale backbones, predicts full VLM training trajectories with high fidelity, and generalizes to entirely held-out model families.

为了拟合和验证该框架，我们在严格控制的配方下，基于 7 个模型家族的 34 个 LLM 训练了超过 150 个 VLM。在超过 200 个文本基准和 50 个多模态基准上的评估表明，该定律能准确地将迁移率从 8B 参数模型外推至 72B 规模的基座，以高保真度预测完整的 VLM 训练轨迹，并能泛化到完全未见过的模型家族。

Beyond the scaling law, our analysis surfaces actionable insights: certain textual benchmarks negatively correlate with multimodal performance, exposing latent benchmark-gaming behavior; base LLMs outperform instruction-tuned counterparts as VLM backbones due to higher absorption rates and lower data-scaling decay; and different model families occupy distinct positions in the transfer—absorption space. The framework turns backbone selection from costly empirical sweeps into a principled, quantitative decision.

除了缩放定律外，我们的分析还揭示了具有实践意义的见解：某些文本基准测试与多模态性能呈负相关，暴露了潜在的“刷榜”行为；基础版 LLM（Base LLM）由于具有更高的吸收率和更低的数据缩放衰减，作为 VLM 基座的表现优于指令微调版（Instruction-tuned）模型；此外，不同的模型家族在“迁移-吸收”空间中占据不同的位置。该框架将基座选择从昂贵的经验性试错转变为一种有原则的定量决策。

Code and data are available at this [https URL]. 代码和数据可在 [此链接] 获取。