How to test your LLM app for prompt injection: promptfoo vs garak vs Giskard vs PyRIT vs sentinel-scan-cli

如何测试你的 LLM 应用是否存在提示词注入：promptfoo vs garak vs Giskard vs PyRIT vs sentinel-scan-cli

If you’re shipping anything with an LLM in the request path (a support bot, an agent that calls tools, a RAG pipeline reading untrusted documents) you eventually have to answer “did we actually test this for prompt injection, or did we just eyeball it.” There are a handful of open source tools that can help, and they are not interchangeable. Picking the wrong one wastes a day; picking none wastes a lot more than that. I went through the five tools that come up most often, ran them where I could, and read the source/docs closely where I couldn’t. Here’s what each one actually does, where they overlap, and where they don’t.

如果你在请求路径中集成了 LLM（例如支持机器人、调用工具的 Agent、或读取不可信文档的 RAG 流水线），你最终必须回答这个问题：“我们真的针对提示词注入进行了测试，还是仅仅凭肉眼观察？”目前有一些开源工具可以提供帮助，但它们并非可以互换。选错工具会浪费一天时间；而不做测试则会浪费更多。我研究了最常被提及的五款工具，在可行的情况下进行了运行，并在无法运行的情况下仔细阅读了源码和文档。以下是每款工具的实际功能、重叠之处以及差异所在。

Update, Aug 2026: Microsoft archived PyRIT on GitHub on March 27, 2026. It’s read-only now, no commits, no releases, no issue triage. I’ve kept the section below for context since a lot of existing guides and tutorials still point people to it, but if you’re picking a tool today, skip to the “which one, for what” section, or read the dedicated post on what to use instead.

更新（2026 年 8 月）：微软已于 2026 年 3 月 27 日将 PyRIT 在 GitHub 上归档。目前该项目为只读状态，不再有提交、发布或问题处理。我保留了下文以供参考，因为许多现有的指南和教程仍指向它。但如果你现在要选择工具，请直接跳到“如何选择”部分，或阅读关于替代方案的专题文章。

The two layers people conflate

人们容易混淆的两个层面

Most confusion here comes from mixing up two different things you can test: App-layer testing: does your application (prompts, guardrails, tool-calling logic, RAG retrieval) resist attacks when wired together end to end. Model-layer testing: does the underlying model itself have exploitable behavior, independent of any app wrapped around it.

这里的大多数困惑源于混淆了两个不同的测试层面：应用层测试：当你的应用程序（提示词、护栏、工具调用逻辑、RAG 检索）端到端连接时，是否能抵御攻击。模型层测试：底层模型本身是否存在可被利用的行为，而不考虑其外层的应用包装。

promptfoo, Giskard, and sentinel-scan-cli are primarily app-layer tools. garak is model-layer. PyRIT can do either depending on how you configure it. Knowing which layer you actually need to test decides most of this for you before you even look at features.

promptfoo、Giskard 和 sentinel-scan-cli 主要属于应用层工具。garak 属于模型层。PyRIT 则取决于你的配置，两者皆可。在查看功能之前，明确你需要测试哪个层面，就能帮你解决大部分选择问题。

promptfoo

App-layer, and the most broadly adopted of the bunch. Zero-install via npx promptfoo, 50+ built-in red-team plugins, and it ships report presets mapped to OWASP LLM Top 10, NIST, and MITRE ATLAS out of the box. It’s used internally at places like OpenAI and Anthropic (per their own repo), which tells you it holds up at scale, not just for toy demos.

应用层工具，也是这几款中应用最广泛的。通过 npx promptfoo 可实现零安装，内置 50 多种红队插件，并开箱即用提供映射到 OWASP LLM Top 10、NIST 和 MITRE ATLAS 的报告预设。据其仓库显示，OpenAI 和 Anthropic 等公司内部都在使用它，这说明它不仅适用于演示，也能经受住大规模生产环境的考验。

Use it when: you want the most complete free coverage of your actual application, you’re willing to spend real setup time wiring it into your prompts/agent config, and you want output that maps cleanly to a compliance framework someone above you is going to ask about. Tradeoff: the plugin surface is large enough that getting a useful first run takes more than five minutes. It rewards investment; it doesn’t do much for you in a five-minute check.

适用场景：当你想要对实际应用进行最全面的免费覆盖，愿意投入时间将其集成到提示词/Agent 配置中，并且需要输出符合合规框架（上级可能会询问）的报告时。权衡：插件覆盖面很广，因此获得有用的首次运行结果需要超过五分钟。它需要投入才能见效；五分钟的快速检查无法发挥其价值。

garak (NVIDIA)

Model-layer. Pip installable, 50+ probes, 8.1k GitHub stars, actively maintained by NVIDIA. It’s testing the model’s own susceptibility to jailbreaks, encoding tricks, data leakage, etc., not how your specific prompts and app logic hold up.

模型层工具。可通过 Pip 安装，拥有 50 多种探测器，GitHub 获 8.1k 星，由 NVIDIA 积极维护。它测试的是模型自身对越狱、编码技巧、数据泄露等的易感性，而不是你的特定提示词和应用逻辑的稳健性。

Use it when: you’re evaluating which base model to build on, or you maintain a model and need to know its raw failure modes before anyone wraps an app around it. It is not natively organized around OWASP LLM Top 10, so if a compliance mapping is the deliverable you need, you’ll be doing that translation yourself. Don’t use it as your only test if what you actually ship is an application with system prompts, tool access, and retrieval, because none of that is in scope for a model-layer scan.

适用场景：当你正在评估选择哪个基础模型进行构建，或者你维护着一个模型，需要在其被应用包装前了解其原始故障模式时。它并非原生围绕 OWASP LLM Top 10 组织，因此如果你需要合规映射报告，则需要自行转换。如果你的产品是包含系统提示词、工具访问和检索的应用，请不要仅使用它进行测试，因为这些都不在模型层扫描的范围内。

Giskard

App-layer, and it does have OWASP-mapped detectors, similar spirit to promptfoo. The catch is that the continuous-scan Hub, the part that would let you run this repeatedly against a live app over time, is a paid product. The open source scanner itself is real and usable, but plan for the free tier to be a point-in-time check, not a monitoring setup.

应用层工具，拥有类似 promptfoo 的 OWASP 映射检测器。问题在于，其允许针对在线应用进行持续扫描的 Hub 是付费产品。开源扫描器本身是真实可用的，但请将免费版视为一次性检查工具，而非监控方案。

Use it when: you want an OWASP-mapped scan and you’re open to (or already have) their commercial tooling for anything ongoing.

适用场景：当你需要 OWASP 映射扫描，并且愿意（或已经）使用他们的商业工具进行持续监控时。

PyRIT (Microsoft) - archived

PyRIT (Microsoft) - 已归档

Status as of March 27, 2026: archived by Microsoft. The repo is read-only, no commits, no releases, no issue triage. Whatever version you have installed is the last one you’ll get. Everything below describes what it did while it was maintained.

截至 2026 年 3 月 27 日的状态：已被微软归档。仓库为只读，无提交、无发布、无问题处理。你安装的版本将是最终版本。以下内容描述了它在维护期间的功能。

Multi-turn attack orchestration, built for red teamers who need to script conversational attack sequences (escalating a jailbreak over several turns, chaining techniques). It was more of a framework than a turnkey scanner, and it had noticeably lower general-developer adoption than the other four, mostly because the audience was security researchers running structured red-team engagements, not app developers doing a pre-ship check.

它专注于多轮攻击编排，专为需要编写对话式攻击序列（在多轮中升级越狱、链接攻击技术）的红队人员设计。它更像是一个框架而非开箱即用的扫描器，其在普通开发者中的采用率明显低于其他四款，主要是因为其受众是进行结构化红队演练的安全研究人员，而非进行发布前检查的应用开发者。

It doesn’t have a direct one-for-one replacement for its multi-turn orchestration model. If that’s specifically what you need, promptfoo’s red-team plugins cover a lot of the same ground for app-layer testing, or you’re looking at building a small harness yourself on top of a maintained model API.

它的多轮编排模型没有直接的替代品。如果你确实需要此功能，promptfoo 的红队插件在应用层测试中涵盖了大部分相同领域，或者你可以考虑在受维护的模型 API 之上自行构建一个小型的测试工具。

sentinel-scan-cli (ours)

sentinel-scan-cli (我们开发的)

Full disclosure: I work on this one, so weigh that accordingly, and I’ve tried to be honest about where it does and doesn’t fit rather than pretend it’s the answer for everyone. It’s a small, app-layer, dependency-free CLI (Python and Node, byte-identical output between the two) covering 15 attack patterns, each individually tagged with its OWASP LLM Top 10 category in both the console and JSON output.

免责声明：我参与了此工具的开发，请酌情参考。我尽量客观地说明了它的适用场景和局限性，而不是假装它是万能的。这是一个小型的、应用层、无依赖的 CLI 工具（Python 和 Node 版本输出完全一致），涵盖 15 种攻击模式，每种模式在控制台和 JSON 输出中都单独标记了其对应的 OWASP LLM Top 10 类别。

The whole pitch is speed to first result: —demo runs with zero config and no API keys, and you get an OWASP-tagged pass/fail readout in under a minute. Where it’s genuinely useful: a fast, zero-setup first pass before you reach for something heavier, or a CI gate check when you want a quick OWASP-mapped signal without standing up a bigger red-team framework. Where it isn’t: it covers 15 attacks, not 50+. If you need broad coverage or you’re past the “quick check” stage and building an actual ongoing red-team practice, promptfoo or Giskard cover more ground and you should use those instead.

其核心卖点是获取结果的速度：--demo 模式无需配置、无需 API Key，在一分钟内即可获得带有 OWASP 标签的通过/失败报告。它真正的用途是：在转向更复杂的工具之前进行快速、零配置的初步检查，或者在 CI 流水线中需要快速获取 OWASP 映射信号而无需部署大型红队框架时使用。它的局限性在于：它只覆盖 15 种攻击，而非 50 多种。如果你需要广泛的覆盖范围，或者已经过了“快速检查”阶段并正在构建持续的红队实践，那么 promptfoo 或 Giskard 覆盖面更广，你应该使用它们。

Which one, for what

如何选择

Testing your app’s prompts/guardrails/tool logic, want the most thorough free coverage: promptfoo.
Testing raw model behavior, not a specific app: garak.
Want OWASP mapping and might pay for continuous monitoring later: Giskard.
Running structured, multi-turn attacks: (No direct replacement, consider promptfoo or custom harness).
Fast, zero-setup, CI-friendly OWASP check: sentinel-scan-cli.
测试应用提示词/护栏/工具逻辑，且需要最全面的免费覆盖：promptfoo。
测试原始模型行为，而非特定应用：garak。
需要 OWASP 映射，且未来可能付费进行持续监控：Giskard。
进行结构化的多轮攻击：（无直接替代品，考虑 promptfoo 或自定义测试工具）。
快速、零配置、CI 友好的 OWASP 检查：sentinel-scan-cli。