MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

MedProb：探测视觉语言模型在医学问答中的内部表征

Abstract: Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with MedProb, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation.

摘要： 人们通常认为医学视觉问答（Med-VQA）需要进行医学微调、使用大型模型或构建复杂的多智能体流水线。我们通过 MedProb 重新审视了这一假设。MedProb 是一个轻量级的探测框架，它无需进行自由文本生成，即可直接从冻结的视觉语言模型（VLM）表征中预测多选题的 Med-VQA 答案。

Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals.

在 PATH-VQA、SLAKE 和 VQA-RAD 数据集上的实验表明，与提示工程（prompting）相比，MedProb 能够恢复出更多与答案相关的信号，且表现优于现有的医学 VLM 和智能体系统。此外，与提示工程相比，探测方法缩小了小型模型与大型模型之间的明显差距，这表明小型 VLM 中蕴含的、可恢复的 Med-VQA 信号比基于生成的评估方法所揭示的要多。

Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.

在 14 对匹配的通用型与医学型 VLM 对比中，医学领域的适应性调整并未持续提升这种线性可解码性。最后，自由文本生成表现出高达 10 个百分点的答案位置偏差；虽然 MedProb 也存在位置偏差，但其受影响的方式与提示工程不同。我们的主要研究结果针对多选题/多分类的 Med-VQA 设置；此外，我们还展示了该探测器可以通过拒绝采样评分程序扩展到开放式生成任务中。