Just Keep Prompting: Evaluating Repetitive Socratic Prompting in VLMs

持续提示：评估视觉语言模型（VLM）中的重复苏格拉底式提示

Abstract: Deploying Vision-Language Models (VLMs) in real-world settings requires not only strong visual reasoning but also stability under sustained conversational pressure. We introduce Just Keep Prompting (JKP), a multi-turn evaluation framework that measures VLM epistemic stability when users repeatedly challenge, question, or contradict a model’s answer.

摘要： 在现实场景中部署视觉语言模型（VLM）不仅需要强大的视觉推理能力，还需要在持续的对话压力下保持稳定性。我们引入了“持续提示”（Just Keep Prompting, JKP），这是一个多轮评估框架，用于衡量当用户反复挑战、质疑或反驳模型答案时，VLM 的认知稳定性。

JKP probes models for up to 10 follow-up turns using three strategies: Adversarial Negation (repeated rejection), Pure Socratic Interrogation (repeated calls to reassess certainty), and Context-Aware Socratic Summarization (reflecting the model’s prior rationale back before asking for reconsideration).

JKP 通过三种策略对模型进行最多 10 轮的后续追问：对抗性否定（反复拒绝）、纯苏格拉底式质询（反复要求重新评估确定性）以及上下文感知苏格拉底式总结（在要求重新考虑之前，将模型之前的逻辑反馈给它）。

We evaluate GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B on a subset of the STAR benchmark across 720 multi-turn runs. Aggregate accuracy changes modestly from Turn 0 to Turn 10, but trajectory-level analysis reveals substantial instability: correct answers regress, wrong answers recover, and many runs exhibit repeated answer flipping.

我们在 STAR 基准测试的一个子集上，对 GPT-4o、Gemini 2.5 Pro 和 Qwen3-VL-30B 进行了 720 次多轮运行评估。从第 0 轮到第 10 轮，总体准确率变化不大，但轨迹层面的分析揭示了显著的不稳定性：正确答案会退化，错误答案会恢复，且许多运行过程表现出反复的答案翻转。

Repeated prompting has bounded upside and often acts as a destabilizer rather than a reasoning aid. The effect is strongly model-dependent: Qwen3-VL-30B achieves the highest final accuracy but becomes confidently wrong under direct contradiction; Gemini 2.5 Pro is comparatively stable but token-expensive; GPT-4o is the most brittle and oscillatory.

重复提示的收益有限，且往往起到破坏稳定性的作用，而非辅助推理。这种影响具有很强的模型依赖性：Qwen3-VL-30B 达到了最高的最终准确率，但在直接反驳下会变得“自信地错误”；Gemini 2.5 Pro 相对稳定，但 Token 消耗巨大；GPT-4o 则表现得最为脆弱且波动剧烈。

These findings reveal that multi-turn VLM evaluation captures not just additional reasoning but pressure-response profiles: how models trade off visual grounding, calibration, and conversational compliance under repeated challenge.

这些发现表明，多轮 VLM 评估捕捉到的不仅是额外的推理能力，还有压力响应特征：即模型如何在反复挑战下权衡视觉基础、校准能力和对话顺从性。