Testing an AI Code Reviewer Against Real GitHub Pull Requests

Building an AI code reviewer is one thing. Trusting it with real pull requests is another. I’ve been developing CodeGuard AI, a self-hosted AI-powered GitHub PR reviewer built with Java 21 and Spring Boot 3. Instead of relying only on sample code and happy-path demonstrations, I’ve been testing it against deliberately created GitHub pull requests with different bugs, test scenarios, and edge cases.

构建一个 AI 代码审查工具是一回事，而信任它去处理真实的拉取请求（Pull Request）则是另一回事。我一直在开发 CodeGuard AI，这是一个基于 Java 21 和 Spring Boot 3 构建的自托管 AI GitHub PR 审查工具。我没有仅仅依赖示例代码和理想化的演示，而是通过故意创建包含各种 Bug、测试场景和边界情况的 GitHub 拉取请求来对其进行测试。

What I’ve been testing So far, the test lab has covered scenarios across several areas: Test execution and validation Passing test suites Genuine assertion failures Test execution timeouts Builds that succeed without executing tests Unreadable test-result evidence Bug detection Unsafe Optional.get() usage Null input handling Incorrect discount calculations Boundary-value errors Incorrect implementations hidden behind passing tests False-positive resistance Safe Optional.orElse() usage Correct discount calculations Valid test assertions Harmless code that should not trigger unnecessary findings Test-quality analysis Missing regression tests Missing edge-case coverage Whether tests actually validate changed behavior

我的测试内容：到目前为止，测试实验室已经涵盖了多个领域的场景：测试执行与验证（通过的测试套件、真实的断言失败、测试执行超时、未执行测试但构建成功的案例、不可读的测试结果证据）；Bug 检测（不安全的 Optional.get() 使用、空输入处理、错误的折扣计算、边界值错误、隐藏在通过测试背后的错误实现）；抗误报能力（安全的 Optional.orElse() 使用、正确的折扣计算、有效的测试断言、不应触发不必要警告的无害代码）；测试质量分析（缺失的回归测试、缺失的边界情况覆盖、测试是否真正验证了变更的行为）。

Some interesting findings A few results have been particularly useful: CodeGuard correctly identified a genuine failing assertion and showed the expected versus actual values. It detected an incorrect discount calculation even though the existing test passed. It correctly classified a timed-out test execution as TIMED_OUT. It avoided flagging valid Optional.orElse() usage. It identified missing regression coverage without claiming that the production implementation contained a confirmed bug.

一些有趣的发现：有几个结果特别有用：CodeGuard 正确识别了一个真实的断言失败，并显示了预期值与实际值。它检测到了一个错误的折扣计算，尽管现有的测试通过了。它正确地将超时的测试执行归类为 TIMED_OUT。它避免了标记有效的 Optional.orElse() 使用。它识别出了缺失的回归覆盖，且没有断言生产环境的实现一定存在 Bug。

However, testing also exposed areas that need improvement. For example, some failing test scenarios received a warning instead of a blocking merge recommendation. In another scenario, test execution was unverified while the merge decision was still marked as safe. These are exactly the kinds of inconsistencies I want to uncover before calling the product reliable.

然而，测试也暴露了一些需要改进的地方。例如，一些失败的测试场景仅收到了警告，而不是阻止合并的建议。在另一个场景中，测试执行未经验证，但合并决策仍被标记为安全。这些正是我在称该产品可靠之前想要发现的不一致之处。

What I’m learning An AI code reviewer should not simply produce more findings. It should: Distinguish real bugs from false positives. Understand whether tests actually executed. Separate verified evidence from AI inference. Avoid suggesting fixes that weaken valid tests. Make merge recommendations consistent with verification results. A green build does not automatically mean the tests passed. And a suggested fix is not necessarily a correct fix.

我的心得：AI 代码审查工具不应仅仅产生更多的发现。它应该：区分真实 Bug 和误报；了解测试是否真正执行；将验证过的证据与 AI 推理分开；避免建议削弱有效测试的修复方案；使合并建议与验证结果保持一致。构建成功（Green build）并不自动意味着测试通过，而建议的修复方案也不一定是正确的修复方案。

What’s next? I’m working through a 100-scenario validation plan. The next focus areas include repository context, regression detection, team policies, suggested-patch validation, review consistency, and merge decisions. I’m documenting the failures alongside the successful results because both matter.

接下来做什么？我正在执行一项包含 100 个场景的验证计划。接下来的重点领域包括仓库上下文、回归检测、团队策略、建议补丁验证、审查一致性和合并决策。我正在记录失败和成功的结果，因为两者都很重要。

CodeGuard AI is available as a source-code product for developers interested in exploring the implementation. I’d especially like feedback from developers who maintain CI/CD pipelines or GitHub PR review workflows: What should an AI code reviewer absolutely verify before recommending that a pull request is safe to merge? Built with Java 21, Spring Boot 3, GitHub integration, and AI provider support. Explore CodeGuard AI and get the source code here: https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=devto&utm_medium=article&utm_campaign=codeguard_testing_100

CodeGuard AI 作为源代码产品提供给有兴趣探索其实现的开发者。我特别希望收到维护 CI/CD 流水线或 GitHub PR 审查工作流的开发者的反馈：AI 代码审查工具在建议拉取请求可以安全合并之前，绝对应该验证什么？该工具基于 Java 21、Spring Boot 3 构建，支持 GitHub 集成和 AI 提供商。探索 CodeGuard AI 并获取源代码：https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=devto&utm_medium=article&utm_campaign=codeguard_testing_100