Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

衡量微任务合格差距：现成的小语言模型（SLM）何时足以胜任智能体框架？

Abstract: Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. We ask whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why, and whether quantization changes the answer.

摘要： 智能体框架（Agent harnesses）越来越倾向于在处于前沿的大语言模型（LLM）规划器周围的微任务中运行小语言模型（SLM），例如：自动批准 Shell 命令、写入内存、选择工具以及对过往对话轮次进行排序。我们探讨了现成的 SLM 是否能达到从业者定义的阈值，如果未能达到，原因何在，以及量化是否会改变这一结果。

We build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold $\tau$ anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears $\tau$). Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), we find an eligibility gap: 0 of 16 (4 tasks $\times$ 4 models) configurations pass (verified by checking the raw outputs and parser behavior).

我们构建了一个包含 4 个此类微任务的基准测试，使用固定提示词和自动评估指标。每个任务都有一个预设阈值 $\tau$，该阈值锚定于廉价的非 LLM 基准，并采用置信区间（CI）感知的合格规则（仅当配置的置信界限超过 $\tau$ 时才算通过）。在最佳状态下（FP16、贪婪搜索、单一冻结提示词、无微调）对 Qwen3 0.6/1.7/4/8B 模型进行全面测试后，我们发现存在明显的合格差距：16 个配置（4 个任务 $\times$ 4 个模型）中没有一个通过测试（通过检查原始输出和解析器行为进行验证）。

A logprob decision-threshold diagnostic (T1/T3/T4; T2 via a context-length/cascade probe) separates the failures into capability deficits and failures that can be addressed by changing the decoding threshold (4 regimes). Quantization to 4-bit (RTN/GPTQ/AWQ) does damage that depends on model size and moves no configuration into eligibility (certified on the reconstructable hard-label tasks T1/T3, diagnostic/windowed robustness on T2/T4), so the gap tracks model size more than precision; it replicates on Llama-3.x (12/12 ineligible) and is robust to the anchor choice (a $\tau$-sweep) and to prompt wording (0/112 eligible across the original plus 3 neutral paraphrases per cell).

通过对数概率决策阈值诊断（T1/T3/T4；T2 通过上下文长度/级联探测），我们将失败原因分为能力缺陷和可以通过改变解码阈值解决的失败（共 4 种机制）。量化至 4-bit（RTN/GPTQ/AWQ）造成的损害取决于模型大小，且未能使任何配置达到合格标准（在可重构的硬标签任务 T1/T3 上进行认证，在 T2/T4 上进行诊断/窗口鲁棒性测试），因此该差距更多地与模型大小相关，而非精度。这一结论在 Llama-3.x 上得到复现（12/12 不合格），并且对锚点选择（$\tau$ 扫描）和提示词措辞具有鲁棒性（在原始提示词加上每个单元 3 个中性改写版本的情况下，0/112 合格）。

The practical implication: place SLMs behind a baseline that meets the CI-backed threshold, and use the SLM only where the baseline fails to meet the threshold; e.g. a 4B re-ranker over a BM25 shortlist beats BM25 ($+0.047$ [0.020, 0.073], without itself certifying eligibility).

实际意义在于：应将 SLM 置于满足 CI 支持阈值的基准之后，仅在基准无法达到阈值时才使用 SLM；例如，在 BM25 短名单上使用 4B 重排序模型优于单纯使用 BM25（$+0.047$ [0.020, 0.073]，尽管重排序模型本身并未达到合格认证）。