Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

随机采样在认知上是浅层的：大语言模型中温度变化与模型多样性之间的维度鸿沟

Abstract: When a language model gives different answers on repeated runs, does that variation reveal what it does not know? Self-consistency turns the variation into a per-question uncertainty estimate via majority voting. But does the same variation reveal cross-question structure — related questions flipping together, the way a diverse ensemble does?

摘要： 当语言模型在重复运行时给出不同的答案时，这种变化是否揭示了它“不知道”的内容？“自洽性”（Self-consistency）通过多数投票将这种变化转化为针对单个问题的置信度估计。但同样的变化是否能揭示跨问题的结构——即相关问题是否会像多样化模型集成（Ensemble）那样同步发生改变？

We compare two regimes on the same questions: one model run $100$ times at $\tau=1$ versus an ensemble of $24$ LLMs run once each at $\tau=0$. A Marchenko—Pastur random-matrix test separates signal from sampling noise on both sides. Within any single model, at most one dimension rises above noise across five families and three benchmarks (MMLU, HellaSwag, GSM8K).

我们针对相同的问题比较了两种机制：一种是单个模型在 $\tau=1$ 下运行 100 次，另一种是由 24 个大语言模型组成的集成，每个模型在 $\tau=0$ 下运行一次。通过 Marchenko—Pastur 随机矩阵测试，我们将两者的信号与采样噪声分离开来。研究发现，在五个模型家族和三个基准测试（MMLU、HellaSwag、GSM8K）中，对于任何单一模型，最多只有一个维度能够超越噪声。

Across the ensemble, four eigenvalues clear the noise edge, while a matched-difficulty Bernoulli null produces at most one in $500$ Monte Carlo draws. Self-consistency gives accurate per-question uncertainty but no detectable cross-question structure; only a diverse ensemble surfaces what a model does not know.

在模型集成中，有四个特征值超过了噪声边界，而匹配难度的伯努利零模型在 500 次蒙特卡洛模拟中最多只能产生一个。自洽性虽然能提供准确的单问题置信度估计，却无法检测到跨问题的结构；只有多样化的模型集成才能真正揭示模型“不知道”的内容。