Do small language models know what they don’t know?

小型语言模型知道自己“不知道什么”吗？

Abstract: We explore whether entropy-based confidence signals can be leveraged to improve the accuracy of Small Language Models (SLMs) with fewer than 3 billion parameters, running entirely on consumer hardware. We evaluate seven distinct approaches, including token-level entropy early stopping, semantic entropy estimation, and uncertainty-aware routing to larger expert models, across 7 model pairs and 5 standard NLU benchmarks.

摘要： 我们探讨了是否可以利用基于熵（entropy-based）的置信度信号，来提高参数量少于 30 亿、且完全运行在消费级硬件上的小型语言模型（SLM）的准确性。我们评估了七种不同的方法，包括基于 Token 级熵的提前停止（early stopping）、语义熵估计，以及向更大的专家模型进行不确定性感知路由（uncertainty-aware routing），并涵盖了 7 组模型对和 5 个标准的自然语言理解（NLU）基准测试。

Our key finding is that token-level entropy is effectively blind in SLMs: in 91% of dataset-model combinations, mean token entropy is near zero regardless of answer correctness, rendering token-based confidence signals unusable at this scale. We demonstrate that semantic entropy, computed by generating multiple samples, clustering answers by meaning, and measuring distributional uncertainty, recovers a viable confidence signal.

我们的主要发现是，Token 级熵在小型语言模型中实际上是“盲目”的：在 91% 的数据集与模型组合中，无论答案是否正确，平均 Token 熵都接近于零，这使得基于 Token 的置信度信号在这一规模下无法使用。我们证明，通过生成多个样本、按含义对答案进行聚类并测量分布不确定性所计算出的“语义熵”，能够恢复出一种可行的置信度信号。

Using semantic entropy to selectively route uncertain queries to a larger expert model yields accuracy improvements of up to +50 percentage points. Notably, cross-family routing (e.g., SmolLM 360M to Phi-3.5-mini) averages +22.0% improvement compared to +6.8% for same-family routing, revealing that expert model quality matters more than architectural compatibility.

利用语义熵将不确定的查询选择性地路由到更大的专家模型，可带来高达 50 个百分点的准确率提升。值得注意的是，跨系列路由（例如从 SmolLM 360M 路由到 Phi-3.5-mini）平均可带来 22.0% 的提升，而同系列路由仅为 6.8%，这表明专家模型的质量比架构兼容性更为重要。

Our results suggest that the value proposition for entropy-based methods in SLMs is not computational savings but intelligent compute allocation: spending more tokens where they matter most.

我们的研究结果表明，基于熵的方法在小型语言模型中的价值主张并非节省计算资源，而是实现智能化的计算分配：将更多的计算资源（Token）投入到最关键的地方。