Gating a Merge on an Eval Score in Azure Pipelines

在 Azure Pipelines 中通过评估分数控制合并

If your Azure Pipelines eval gate runs on pushes to main but never on a pull request, the YAML is not the problem. Microsoft’s documentation is explicit: for an Azure Repos Git repository you cannot configure a PR trigger in the YAML file, and the functionality is implemented by a branch policy instead. 如果你的 Azure Pipelines 评估门禁（eval gate）在推送到主分支时运行，但在拉取请求（PR）时从不运行，问题并不出在 YAML 文件上。微软的文档明确指出：对于 Azure Repos Git 仓库，你无法在 YAML 文件中配置 PR 触发器，该功能必须通过分支策略（branch policy）来实现。

Why your pr trigger does nothing

为什么你的 pr 触发器不起作用

The pr: key exists in the Azure Pipelines YAML schema, and it works — for GitHub and Bitbucket Cloud repositories. For Azure Repos Git it is inert. The Azure Repos Git documentation states that pull request triggers are implemented using branch policies, and that to enable PR validation you configure the Build validation policy on the target branch. A pr: block in the file is not an error and produces no warning; it simply never causes a run. pr: 键确实存在于 Azure Pipelines YAML 架构中，并且对于 GitHub 和 Bitbucket Cloud 仓库是有效的。但对于 Azure Repos Git，它是无效的。Azure Repos Git 文档说明，拉取请求触发器是通过分支策略实现的，要启用 PR 验证，你需要配置目标分支上的“构建验证”（Build validation）策略。文件中的 pr: 代码块不会报错，也不会产生警告；它只是永远不会触发运行。

Two related things surprise people once the policy exists. Draft pull requests do not trigger a pipeline even with a branch policy configured, so a gate that seems not to run may be running against a draft. And you must be a project administrator of the project to configure validation builds at all, which is why this is often the step that a developer cannot complete themselves. This is a product behaviour rather than a version detail, but it is the kind of thing that changes. Check the Azure Repos Git page in Microsoft’s Azure Pipelines documentation before assuming it still holds. 一旦配置了策略，有两件事常让用户感到意外。首先，即使配置了分支策略，草稿状态的拉取请求也不会触发流水线，因此看起来没运行的门禁可能是在针对草稿运行。其次，你必须是项目的管理员才能配置验证构建，这就是为什么开发者往往无法自行完成这一步。这是产品行为而非版本细节，但这类事情可能会变动。在假设规则依然有效之前，请务必查阅微软 Azure Pipelines 文档中的 Azure Repos Git 页面。

The pipeline

流水线配置

A single-stage pipeline is enough. The CI trigger below covers pushes; the pull request path comes from the policy in the next section, and no pr: key appears at all because on Azure Repos it would only be misleading to a reader. 单阶段流水线就足够了。下方的 CI 触发器涵盖了推送操作；拉取请求路径来自下一节提到的策略。这里完全没有使用 pr: 键，因为在 Azure Repos 上，它只会误导读者。

trigger:
  branches:
    include:
      - main
  paths:
    exclude:
      - docs/*

pool:
  vmImage: ubuntu-latest

variables:
  - group: llm-eval-keys
  - name: EVAL_MODEL
    value: gpt-4.1-mini-2025-04-14

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.12'
  - script: pip install -r evals/requirements.txt
    displayName: Install eval dependencies
  - script: |
      python -m evals.run \
        --cases evals/cases.jsonl \
        --thresholds evals/thresholds.json \
        --junit-out $(Build.ArtifactStagingDirectory)/eval.xml \
        --json-out $(Build.ArtifactStagingDirectory)/eval.json
    displayName: Score the golden set
    timeoutInMinutes: 15
    env:
      EVAL_API_KEY: $(EVAL_API_KEY)
      EVAL_MODEL: $(EVAL_MODEL)
  - task: PublishTestResults@2
    condition: always()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(Build.ArtifactStagingDirectory)/eval.xml
      testRunTitle: Eval gate
      failTaskOnFailedTests: true

condition: always() on the publish task is the difference between seeing which cases failed and seeing only a red script step. By default a task does not run once a previous one failed, and the eval step failing is precisely when you want the report. failTaskOnFailedTests: true is belt and braces rather than the gate itself: the script step has already failed the job by exiting non-zero. It matters only in the case where the runner is misconfigured to exit 0 while writing failing JUnit cases, and it costs nothing to have both. 发布任务上的 condition: always() 是区分“查看哪些用例失败”与“只看到一个红色脚本步骤”的关键。默认情况下，如果前一个任务失败，后续任务将不会运行，而评估步骤失败时正是你需要报告的时候。failTaskOnFailedTests: true 更多是一种双重保险，而非门禁本身：脚本步骤通过非零退出码已经使作业失败了。它仅在运行器配置错误（在写入失败的 JUnit 用例时仍返回 0）的情况下才起作用，同时保留两者并无坏处。

timeoutInMinutes on the step is the more important line — a job-level timeout also exists and, if the job timeout elapses first, the running job including your step is terminated regardless of the longer step value, so set the step bound below whatever the job allows rather than above it. 步骤中的 timeoutInMinutes 是更重要的一行——作业级超时也存在，如果作业超时先到，那么包括你的步骤在内的整个运行中作业都会被终止，无论步骤设置的时间有多长。因此，请将步骤的超时限制设置在作业允许的范围内，而不是超过它。

The branch policy people miss

人们容易忽略的分支策略

Open the repository settings, choose Branches, and open the branch policies for the target branch. Under Build validation, add a policy and select the eval pipeline as the build pipeline. Set Trigger to Automatic (whenever the source branch is updated) rather than Manual. Manual means the check exists and sits unqueued until someone remembers to run it. 打开仓库设置，选择“分支”（Branches），然后打开目标分支的分支策略。在“构建验证”（Build validation）下，添加一个策略并选择评估流水线作为构建流水线。将触发器（Trigger）设置为“自动”（Automatic，即源分支更新时触发），而不是“手动”（Manual）。手动意味着检查虽然存在，但除非有人记得去运行，否则它永远不会进入队列。

Set Policy requirement to Required. This is the setting that is missed. Microsoft’s documentation describes Optional as providing a notification of the build failure while still allowing pull requests to complete — the check goes red and the merge proceeds. 将策略要求（Policy requirement）设置为“必需”（Required）。这是最容易被忽略的设置。微软文档将“可选”（Optional）描述为：即使构建失败也会提供通知，但仍允许拉取请求完成——即检查结果变红，但合并依然可以进行。

Set a build expiration. The useful middle option expires a passing build after a number of hours if the protected branch has been updated, so a gate that passed against a three-week-old baseline is re-run rather than trusted. Optionally add a path filter so the policy does not apply to documentation-only changes. Doing it here rather than in the YAML is what keeps the reported status consistent. 设置构建过期时间。一个实用的中间选项是：如果受保护的分支已更新，则在几小时后使通过的构建过期。这样，针对三周前基准通过的门禁会被重新运行，而不是被盲目信任。可以选择添加路径过滤器，使策略不适用于仅文档的更改。在这里而不是在 YAML 中进行此操作，可以保持报告状态的一致性。

Secret variables are not in your environment

密钥变量不在你的环境中

This is the second thing that costs an hour. Ordinary pipeline variables are injected into every task’s environment automatically. Secret variables — whether from a variable group backed by a key vault or marked secret in the UI — are not. They are available for macro substitution as $(EVAL_API_KEY) but absent from the process environment unless you map them explicitly, and that is exactly what the env: block in the script step above is for. 这是第二个容易浪费一小时的问题。普通的流水线变量会自动注入到每个任务的环境中。但密钥变量（无论是来自 Key Vault 支持的变量组，还是在 UI 中标记为 secret 的变量）则不会。它们可以通过 $(EVAL_API_KEY) 进行宏替换，但除非你显式映射，否则它们不会出现在进程环境中，这正是上方脚本步骤中 env: 代码块的作用。

Omit the mapping and the symptom is an authentication error from the provider on a pipeline that has a perfectly good key configured, which reads like a credential problem and is a plumbing one. A quick check: print the length of the variable rather than the variable. 如果省略映射，症状就是流水线会报身份验证错误，尽管你配置了完全正确的密钥。这看起来像是凭据问题，实际上是配置连接（plumbing）问题。快速检查方法：打印变量的长度，而不是变量本身。

Variable groups add a second layer to this. A group has to be authorised for the pipeline before its variables resolve, and the first run after adding a group can wait on a permission prompt rather than failing outright — a pipeline that appears to hang on its first eval run is sometimes waiting for that approval rather than for a model. If the group is backed by Azure Key Vault, the service connection’s identity needs get and list on the secrets, and a missing list permission is the one that produces a confusing empty result instead of a clear access error. 变量组增加了第二层复杂性。在变量解析之前，必须先为流水线授权该组。添加组后的第一次运行可能会等待权限确认，而不是直接失败——如果流水线在第一次评估运行时看起来“挂起”了，有时它是在等待该批准，而不是在等待模型。如果该组由 Azure Key Vault 支持，服务连接的身份需要拥有对密钥的 get 和 list 权限，而缺少 list 权限往往会导致令人困惑的空结果，而不是明确的访问错误。

Publishing results so the score is findable

发布结果以便查看分数

Azure’s test tab is genuinely good at the thing an eval needs — showing which named cases failed and which are new failures rather than an aggregate. That only works if your runner emits per-case JUnit entries with stable test names, one per eval case, rather than a single test called “eval” that passes. Azure 的测试选项卡非常适合评估需求——它能显示哪些命名用例失败了，哪些是新的失败，而不是仅仅显示一个汇总结果。这只有在你的运行器为每个评估用例生成带有稳定测试名称的 JUnit 条目（每个用例一个条目）时才有效，而不是生成一个名为“eval”且总是通过的单一测试。