When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play

当推理限制了选择：大语言模型游戏表现中的多样性坍缩

Abstract: Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks, but its effect on behavioral diversity in sequential decision-making remains under-explored. We study this question in a controlled suite of deterministic board games based on tic-tac-toe variants, where optimal actions are exactly computable and diversity can be measured directly.

摘要： 监督微调（SFT）被广泛用于使大语言模型适应下游任务，但其对序列决策中行为多样性的影响仍未得到充分研究。我们在基于井字棋变体的一套受控确定性棋盘游戏中研究了这一问题，在这些游戏中，最优动作是可以精确计算的，且多样性可以直接衡量。

Across state-level evaluation, arena gameplay, and training trajectories, we find that reasoning-mode generation frequently suppresses action diversity without uniformly improving action accuracy. Furthermore, standard SFT improves accuracy but often induces premature diversity collapse, which exceeds what is minimally required by the accuracy-diversity tradeoff.

通过状态级评估、竞技场游戏对战以及训练轨迹分析，我们发现推理模式下的生成往往会抑制动作的多样性，且并不能统一提升动作的准确性。此外，标准的监督微调虽然提高了准确性，但往往会导致过早的多样性坍缩，其程度超过了准确性与多样性权衡所要求的最小限度。

We then show that action augmentation, which trains on all optimal actions per state rather than a single demonstrated action, would partially mitigate this effect. Our results identify narrow-support imitation as a source of policy collapse in LLM decision-making and suggest that preserving action support during SFT is important for maintaining exploratory behavior.

随后我们证明，动作增强（即针对每个状态下的所有最优动作进行训练，而非仅针对单一示范动作）可以部分缓解这一效应。我们的研究结果指出，窄支持度模仿（narrow-support imitation）是大语言模型决策中策略坍缩的根源，并表明在监督微调过程中保持动作支持度对于维持探索性行为至关重要。