Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards
Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards
Search-G1：基于表征内在奖励的扎根搜索智能体
Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Existing external rewards provide either sparse outcome supervision or richer feedback from process annotations and LLM judges. Outcome rewards scale readily but cannot distinguish grounded retrieval from redundant search, whereas richer signals require costly annotation or inference during training.
搜索增强型语言智能体应仅在必要时检索外部信息，并将其回答建立在检索到的证据之上。现有的外部奖励要么提供稀疏的结果监督，要么通过过程标注和大型语言模型（LLM）评判提供更丰富的反馈。结果奖励易于扩展，但无法区分扎根检索与冗余搜索；而更丰富的信号则需要在训练过程中进行昂贵的标注或推理。
Internal rewards based on policy-side signals such as entropy, likelihood, or information gain are graded and inexpensive to evaluate, yet mainly reflect model confidence rather than evidence grounding. We propose Search-G1, a representation-based intrinsic reward framework that measures the operational grounding of an agent’s answers through two intervention-calibrated readouts.
基于策略端信号（如熵、似然或信息增益）的内部奖励虽然分级明确且评估成本低，但主要反映的是模型置信度，而非证据扎根程度。我们提出了 Search-G1，这是一个基于表征的内在奖励框架，通过两个经干预校准的读出器（readouts）来衡量智能体回答的操作扎根性。
A prompt-state readout predicts closed-book sufficiency, whose complement defines policy-relative retrieval necessity; an answer-commit readout estimates evidence reliance from answer-stage sensitivity to evidence deletion. Together, they provide additional credit to correct searched trajectories when retrieval is estimated necessary and the answer is evidence-sensitive, favor correct direct answers when closed-book knowledge suffices, and penalize repeated search.
“提示-状态读出器”预测闭卷回答的充分性，其补集定义了策略相关的检索必要性；“回答-提交读出器”通过回答阶段对证据删除的敏感度来估计对证据的依赖性。两者结合，当评估认为检索必要且回答对证据敏感时，为正确的搜索轨迹提供额外奖励；当闭卷知识足够时，倾向于正确的直接回答；并对重复搜索进行惩罚。
After calibration, reward scoring requires neither process annotations nor LLM-as-judge inference during policy optimization. Because reinforcement learning changes policy representations, Search-G1 periodically refits both readouts on trajectories from the latest checkpoint, allowing the reward to co-evolve with the policy.
校准完成后，奖励评分在策略优化过程中既不需要过程标注，也不需要使用 LLM 作为评判进行推理。由于强化学习会改变策略表征，Search-G1 会定期根据最新检查点的轨迹重新拟合这两个读出器，从而使奖励机制能够与策略共同演进。
Experiments across multiple search-based question-answering benchmarks and two model scales show that Search-G1 improves the grounding—search-cost trade-off, producing shorter response-side trajectories at competitive task accuracy. Code is available at this https URL.
在多个基于搜索的问答基准测试和两种模型规模上的实验表明，Search-G1 改善了扎根程度与搜索成本之间的权衡，在保持竞争性任务准确率的同时，生成了更短的响应轨迹。代码可在该链接获取。