Technical Manual for Toolkit for Confidence-Corpus Consistency via Fine-Tuning on a Fabricated Corpus

通过在虚构语料库上进行微调以实现置信度与语料库一致性的工具包技术手册

Abstract: A language model’s confidence in an answer is often read as a proxy for how well it knows the corresponding fact. This manual documents an open toolkit built to test that reading directly: a small causal language model is fine-tuned on a corpus that consistently asserts one fabricated arithmetic answer for each of the 81 single-digit addition pairs, and its post-fine-tuning confidence in each fabricated answer is compared against its own pre-fine-tuning confidence in the corresponding true answer, using an unchanged measurement procedure throughout.

摘要： 语言模型对答案的置信度通常被解读为衡量其对相应事实掌握程度的指标。本手册记录了一个旨在直接测试该解读的开源工具包：通过在一个语料库上对小型因果语言模型进行微调，该语料库为 81 个单数字加法对中的每一对都一致地断言一个虚构的算术答案；随后，在整个过程中使用不变的测量程序，将其微调后对每个虚构答案的置信度与微调前对相应真实答案的置信度进行比较。

We describe and justify every pipeline stage, fact-space generation, token-length-aware confidence measurement, baseline validation, corpus construction, fine-tuning, and paired before/after comparison, together with the confound each is meant to rule out, among them tokenization asymmetry between single- and double-digit answers and the difference between an answer merely losing its edge and one being actively suppressed.

我们描述并论证了每一个流水线阶段，包括事实空间生成、感知标记长度的置信度测量、基线验证、语料库构建、微调以及配对的前后比较，并说明了每一项旨在排除的混杂因素，其中包括单数字答案与双数字答案之间的分词不对称性，以及答案仅仅失去优势与被主动抑制之间的区别。

This manuscript is a methodological and implementation reference: it documents the instrument and does not report or interpret the outcome of any specific run. The toolkit and its pinned dependency environment are archived separately (Section 9) under a persistent identifier, to be cited as an instrument by work that produces and interprets empirical results with it.

本手稿是一份方法论和实现参考：它记录了该工具本身，并不报告或解释任何特定运行的结果。该工具包及其固定的依赖环境已在（第 9 节）中通过持久标识符单独存档，供后续使用该工具产生并解释实证结果的研究作为仪器进行引用。