Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Back to Articles Turning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox Enterprise Article Published October 22, 2026 Upvote 11 +5 Tuhin Kundu tuhink Follow microsoft This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub.

返回文章：将数据库评分转化为强化学习奖励：使用 ThinkingBox 进行代理后训练。企业文章，发布于 2026 年 10 月 22 日，点赞 11 次，+5，Tuhin Kundu，关注 Microsoft。这是我们第一篇博文《代理说它完成了，但数据库不同意》的后续文章。ThinkingBox-Training 现已在 GitHub 上发布。

We built ThinkingBox to evaluate an agent’s work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.

我们构建了 ThinkingBox，通过比较最终数据库状态与预期结果来评估代理的工作。随后，我们将同样的执行信号用作 ThinkingBox-Bench 之外训练任务的奖励。对于 Qwen3.8-27B 模型，平均成功率从 51.70% 提升至 60.78%，而在 50 个 GRPO 步骤后，在所有 20 次尝试中均正确完成的任务数量从 38 个增加到了 107 个。

Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench. This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.

图 1：ThinkingBox 将终端数据库状态的可执行检查转化为 ThinkingBox-Bench 之外任务的代理后训练奖励。这是 Microsoft 与 Hugging Face 的联合博文，特别感谢来自 Microsoft 的 Mirco Milletari 和 Xiang Jiang、来自 Hugging Face 的 Sergio Paniego，以及我们前实习生朱卓纯（匹兹堡大学）在撰写和审阅方面的贡献。

We built ThinkingBox to catch agents that say they are done when they are not. Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer’s ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row.

我们构建 ThinkingBox 是为了捕捉那些声称已完成任务但实际并未完成的代理。ThinkingBox-Bench（我们的基准数据集）中的 507 个任务，每一个都会为代理提供一个真实的后端和一套工具，让其执行工作，然后检查数据库中实际发生了什么变化。在第一篇文章中，我们发现一个模型将客户工单标记为已解决，但其背后的承运商异常仍处于开启状态。九次看起来很专业的工具调用，却导致了一行错误的数据库记录。

Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function. So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards.

随后我们意识到我们构建了什么。对于大多数任务，可执行检查可以直接根据最终数据库状态返回 1 或 0。这不仅仅是一个评分器，它还可以是一个奖励函数。因此，我们从基准测试之外的 ThinkingBox 格式任务中构建了一个训练池，并将它们的执行结果用作奖励。

The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing. Difficulty is policy-relative: a task is hard for the current model only until that model learns it. We call this offline policy-relative difficulty. We estimated each task’s success probability once using the base policy before training and kept the selected training pool fixed throughout the run.

难点在于选择哪些任务适合代理训练：大多数任务要么已经被解决，要么对于当前模型来说毫无希望，而这两种情况都无法提供学习价值。难度是相对于策略而言的：一个任务对于当前模型来说是困难的，直到该模型学会它为止。我们称之为“离线策略相关难度”。我们在训练前使用基础策略估算了每个任务的成功概率，并在整个运行过程中保持所选训练池不变。

What happened after training We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.

训练后发生了什么？我们使用离线策略相关难度构建了训练池，然后使用 GRPO 对 Qwen3.8-27B 进行了 50 个步骤的微调。回顾一下，我们对 507 个基准任务中的每一个都进行了 20 次评估。pass@1 衡量所有 10,140 次尝试的平均成功率。pass@20 衡量至少解决一次的任务比例。Observed 20/20 衡量在所有记录的尝试中均通过的任务比例。