SAGE: Schema-Guided LLMs for Grant Review

SAGE：用于拨款评审的模式引导大语言模型

Abstract: Grant reviewers must apply detailed criteria to application forms, budgets, and supporting documents while producing assessments that colleagues can inspect. We present SAGE, Schema-Guided Aspect-Based Grant Evaluation, a system that translates a grant rubric into structured checks and links its judgements to evidence from the application package.

摘要： 拨款评审人员必须根据详细的准则来审查申请表、预算及证明文件，并撰写可供同事查阅的评估报告。我们提出了 SAGE（Schema-Guided Aspect-Based Grant Evaluation，模式引导的基于方面的拨款评估系统），该系统能将拨款评审准则转化为结构化的检查项，并将评估结论与申请材料中的证据相关联。

We evaluate SAGE in two stages on 35 nonprofit grant applications. A post-factum comparison with 105 reviews from the original competition shows fair ordinal agreement (kappa = 0.29). The foundation then conducted a criterion-level re-review after inspecting SAGE, producing 202 assessments. In this assisted round, SAGE reached kappa = 0.58 and outperformed a one-prompt-per-criterion baseline (kappa = 0.33 on the common subset), with higher rank correlation and lower error.

我们在 35 份非营利性拨款申请上分两个阶段对 SAGE 进行了评估。通过与原始竞赛中 105 份评审结果的事后对比，显示出尚可的序数一致性（kappa = 0.29）。随后，基金会在审阅 SAGE 的评估结果后进行了准则层面的复审，共产生了 202 份评估报告。在这一辅助评审环节中，SAGE 的 kappa 值达到 0.58，优于“每个准则对应一个提示词”的基准模型（在共同子集上的 kappa 为 0.33），且表现出更高的秩相关性和更低的误差。

A claim-level audit further identifies confirmed, disputed, and unaddressed parts of the structured draft. SAGE operationalizes the review methodology by producing a detailed, evidence-linked, and auditable draft for expert correction.

基于声明层面的审计进一步识别了结构化草案中已确认、有争议以及未处理的部分。SAGE 通过生成一份详细、证据关联且可审计的草案供专家修正，从而实现了评审方法论的实际应用。