Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?
Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?
Fable 5 对阵 GPT-5.6 Sol：在 NP-Hard 问题上，/goal 指令真的有用吗？
Blog TL;DR: I gave Claude Fable 5 and GPT-5.6 Sol the same unpublished NP-hard optimization problem, with and without their native /goal mode. Fable 5 is an absolute beast; /goal is not a game changer.
博客摘要： 我让 Claude Fable 5 和 GPT-5.6 Sol 运行了同一个未公开的 NP-hard 优化问题，并分别测试了开启和关闭原生 /goal 模式的情况。Fable 5 表现极其强悍；但 /goal 并非改变游戏规则的关键。
Context: This is an operations research problem originally submitted to students at a hackathon. I spent a week years ago writing C++ to solve it, so I have a useful human baseline. Fable 5 was an absolute beast on this benchmark. It produced the best solution overall, and its consistency is unlike anything I have seen from a model on this problem. This is pure raw intelligence. Incredible. 背景： 这是一个最初在黑客马拉松上提供给学生解决的运筹学问题。多年前我曾花了一周时间用 C++ 编写程序来解决它，因此我有一个可靠的人类基准。Fable 5 在此基准测试中表现得非常强悍。它得出了整体最优解，且其一致性是我在模型处理此类问题时从未见过的。这是纯粹的原始智能，令人难以置信。
The other result is that /goal is not a generic “try harder” switch. It changes the control loop and the search path. Sometimes that finds a better basin. Sometimes it gives a bad idea more time to mature. All code, prompts, result tables, exclusions, and trajectory notes are in CLIArena. This is a follow-up to my first article about this benchmark.
另一个结论是，/goal 并非一个通用的“更加努力”开关。它会改变控制循环和搜索路径。有时这能找到更好的解空间，有时则会让一个糟糕的思路有更多时间去“发酵”。所有代码、提示词、结果表、排除项和轨迹记录都在 CLIArena 中。这是我关于该基准测试的第一篇文章的后续报道。
The problem KIRO is a fiber-network design problem I worked on as an engineering student in 2018. Given directed distance matrices for Grenoble, Nice, and Paris, the solver has to connect distribution points and terminals using loops and short chains while respecting several structural constraints. The objective is total cable length. Lower is better. 问题：KIRO 是我 2018 年作为工程系学生时研究的一个光纤网络设计问题。 给定格勒诺布尔、尼斯和巴黎的有向距离矩阵，求解器必须在遵守若干结构约束的前提下，使用环路和短链连接分配点与终端。目标是使总电缆长度最小化，数值越低越好。
A valid network consists of redundant loops rooted at distribution hubs, with short branches hanging from towers on those loops. Every tower must appear exactly once, and reversing a cable segment can change its cost. 一个有效的网络由以分配中心为根的冗余环路组成，短分支则挂载在这些环路的塔架上。每个塔架必须且仅能出现一次，且反转电缆段可能会改变其成本。
How large is the search space? There is no single closed-form count because a solution can use any number of loops, variable loop sizes, and differently anchored and ordered branches. But Paris alone gives a useful lower bound. Even if we ignore ordering and branches and only assign each of the 532 terminals to one of 11 distribution hubs, there are 11^532 possible assignments. 搜索空间有多大？ 没有单一的闭式解计数，因为解决方案可以使用任意数量的环路、可变的环路大小以及不同锚定和排序的分支。但仅以巴黎为例，就能得出一个有用的下界。即使忽略排序和分支，仅将 532 个终端中的每一个分配给 11 个分配中心之一，就有 11^532 种可能的分配方式。
A stronger lower bound comes from one deliberately restricted family of valid solutions: exactly 19 loops of 28 terminals each, with no branches. This covers all 532 terminals because 19 x 28 = 532, while staying below the 30-terminal limit for a loop. Order the 532 terminals, split that ordering into 19 consecutive groups, divide by 19! because the set of loops is unordered, and choose one of the 11 hubs for each loop: (532! / 19!) x 11^19 ~= 10^1223 一个更强的下界来自一个特意限制的有效解族：正好 19 个环路，每个环路 28 个终端，且没有分支。这覆盖了所有 532 个终端（因为 19 x 28 = 532），同时保持在每个环路 30 个终端的限制之下。对 532 个终端进行排序，将该序列分为 19 个连续组，除以 19!（因为环路集合是无序的），并为每个环路选择 11 个中心之一，结果约为：(532! / 19!) x 11^19 ~= 10^1223。
What I tested 测试内容
The primary experiment was intentionally narrow: 主要实验特意保持了较窄的范围：
| Setting | Value |
|---|---|
| Models | Claude Fable 5, Opus 4.8, Sonnet 5; GPT-5.6 Sol, Terra, Luna |
| Modes | Plain; native /goal |
| Optimization budget | 30 minutes |
| Outer agent timeout | 1,900 seconds |
| Reasoning | Maximum available setting for every model |
| Execution | Harbor 0.1.43, Docker, subscription authentication |
| 设置 | 数值 |
|---|---|
| 模型 | Claude Fable 5, Opus 4.8, Sonnet 5; GPT-5.6 Sol, Terra, Luna |
| 模式 | 普通模式；原生 /goal 模式 |
| 优化预算 | 30 分钟 |
| 外部代理超时 | 1,900 秒 |
| 推理 | 每个模型可用的最大设置 |
| 执行环境 | Harbor 0.1.43, Docker, 订阅验证 |
Results 结果
Before concentrating repetitions on the flagship pair, I ran one matched 30-minute no-hint pair for every model in the sweep. For Fable and Sol, the chart uses Pair 1 from the replicated headline set; the other four models have one pair each. I then repeated the flagship comparison until I had three matched runs for Fable 5 and three for Sol. 在将重复测试集中在旗舰模型对之前，我为扫描中的每个模型运行了一对匹配的 30 分钟无提示测试。对于 Fable 和 Sol，图表使用了重复标题集中的第 1 对；其他四个模型各有一对。随后，我重复了旗舰对比，直到 Fable 5 和 Sol 各有三次匹配的运行结果。
(Table omitted for brevity, showing Fable 5 and GPT-5.6 Sol performance metrics) (为简洁起见省略表格，显示 Fable 5 和 GPT-5.6 Sol 的性能指标)
Negative means /goal was better. Goal won four of six trials, so win rate alone makes the feature look useful. The means tell the other half:
负数表示 /goal 表现更好。Goal 在六次试验中赢了四次，因此仅从胜率来看，该功能似乎很有用。但平均值揭示了另一面：
Both models usually got a small benefit and occasionally suffered a large regression. That is why /goal won most runs but made both means worse. Fable was also clearly stronger. Its plain mean beat Sol’s by 1,875 points, and its goal mean beat Sol’s by 1,984. More importantly, Fable plain stayed inside a tiny 319-point range while Sol plain spanned 1,958 points. Fable goal produced the best clean score, 31,934; Fable plain was the safest configuration.
两个模型通常都能获得微小的收益，但偶尔也会出现大幅倒退。这就是为什么 /goal 虽然赢得了大多数运行，却导致两者的平均值变差的原因。Fable 的表现也明显更强。其普通模式的平均值比 Sol 高出 1,875 分，其 goal 模式的平均值比 Sol 高出 1,984 分。更重要的是，Fable 普通模式保持在 319 分的极小范围内，而 Sol 普通模式的波动范围高达 1,958 分。Fable goal 产生了最佳的纯净分数 31,934；而 Fable 普通模式则是最稳妥的配置。
Deep dive into the goal command 深入剖析 goal 指令
The same command hides two different systems. Claude Code and Codex both expose /goal, but the implementations are fundamentally different.
同一个指令背后隐藏着两个不同的系统。Claude Code 和 Codex 都提供了 /goal，但它们的实现方式有着本质区别。
Claude Code: a separate evaluator
Claude Code：独立的评估器
Claude Code implements /goal as a session-scoped Stop hook. After each main-model turn, a small evaluator model, Haiku by default, reads the condition and conversation. It returns yes or no with a reason. A no starts another turn; a yes clears the goal. The evaluator cannot use tools or inspect files. It can only judge evidence that appeared in the transcript. That can catch an early exit, but it cannot know whether another ten million solver iterations are worthwhile.
Claude Code 将 /goal 实现为一个会话范围内的停止钩子（Stop hook）。在主模型的每一轮交互后，一个小型的评估模型（默认为 Haiku）会读取条件和对话，并给出“是”或“否”的理由。如果为“否”，则开启下一轮；如果为“是”，则清除目标。评估器无法使用工具或检查文件，只能根据对话记录中的证据进行判断。这可以防止过早退出，但它无法判断再进行一千万次求解器迭代是否值得。
Codex: persisted state and lifecycle tools
Codex：持久化状态与生命周期工具
I also read the source for the benchmarked release, Codex CLI 0.144.4. Codex treats a goal as persisted thread state: The TUI saves the objective for the active thread, and SQLite stores its status and budget accounting. The working model receives create_goal, get_goal, and update_goal tools. If the thread becomes idle while the goal is active, Codex injects a continuation turn with the objective and a completion audit. Claude delegates completion to another model. Codex lets the working model declare completion, then resumes it while the persisted goal remains active. Claude’s evaluator is independent but sees only the transcript; Codex sees the files and tools but effectively grades its own work.
我还阅读了基准测试版本 Codex CLI 0.144.4 的源代码。Codex 将目标视为持久化的线程状态：TUI 为活动线程保存目标，SQLite 存储其状态和预算核算。工作模型会接收 create_goal、get_goal 和 update_goal 工具。如果线程在目标激活时处于空闲状态，Codex 会注入一个包含目标和完成审计的延续轮次。Claude 将完成判断委托给另一个模型，而 Codex 允许工作模型自行声明完成，然后在目标保持激活状态的同时恢复工作。Claude 的评估器是独立的，但只能看到对话记录；Codex 可以看到文件和工具，但实际上是在给自己打分。
Why /goal can win most runs and still be a bad default 为什么 /goal 能赢得大多数运行，却仍不是一个好的默认设置
On a normal coding task, progress is often legible: another turn can fix a test or complete a migration. Optimization is different. Once an agent chooses a solver, extra time can amplify either a good decision or a bad one. That is exactly what happened here. Goal helped when it sustained Fable’s fast compiled portfolio or Sol’s successful chain repartition.
在普通的编码任务中，进展通常是清晰可见的：多一轮交互就能修复测试或完成迁移。但优化问题则不同。一旦代理选择了求解器，额外的时间可能会放大一个好的决策，也可能放大一个糟糕的决策。这正是此处发生的情况。当 /goal 维持了 Fable 快速编译的组合策略或 Sol 成功的链重分区时，它确实起到了帮助。