I scanned Cal.com with my evidence-based CLI. The best result was what it refused to say

I scanned Cal.com with my evidence-based CLI. The best result was what it refused to say

我用基于证据的 CLI 扫描了 Cal.com,最棒的结果是它拒绝回答的内容

I pointed my scanner at Cal.com this week, fully expecting to write a post about everything it got wrong. The most interesting result was the thing it refused to say. 本周,我将扫描器指向了 Cal.com,本以为会写一篇关于它所有错误之处的文章。但最有趣的结果,却是它拒绝回答的内容。

Quick context: I build DevTime, a local-first CLI that scans a repository and helps it explain itself from evidence. It detects a closed set of concepts (authentication, billing webhooks, background jobs, data export, admin permissions, file uploads), links every claim to evidence files, and reports uncertainty when it cannot prove something. No cloud, no telemetry, no code execution during scan, no AI. 简单背景:我开发了 DevTime,这是一个本地优先的 CLI 工具,用于扫描代码仓库并根据证据进行自我解释。它能检测一组特定的概念(身份验证、计费 Webhook、后台任务、数据导出、管理权限、文件上传),将每个结论链接到证据文件,并在无法证明某事时报告不确定性。它不依赖云端、不收集遥测数据、扫描时不执行代码,也不使用 AI。

The Setup Cal.com is a well-known open-source scheduling platform. Big TypeScript monorepo, Next.js frontend, NestJS API, Stripe integration. If you wanted to stress a heuristic scanner, this is a good place to do it. 准备工作 Cal.com 是一个知名的开源调度平台。它是一个庞大的 TypeScript 单体仓库,包含 Next.js 前端、NestJS API 和 Stripe 集成。如果你想测试一个启发式扫描器,这里是一个绝佳的场所。

The run: pipx install devtime-ei git clone —depth 1 https://github.com/calcom/cal.com cd cal.com dtc init dtc scan 运行过程: pipx install devtime-ei git clone —depth 1 https://github.com/calcom/cal.com cd cal.com dtc init dtc scan

The scan: Scan complete. Scanned 5736 files, 24516 signals, 3 concepts in 4.2s. Local, 4.2 seconds, nothing left the machine. 扫描结果:扫描完成。在 4.2 秒内扫描了 5736 个文件、24516 个信号和 3 个概念。完全本地化,耗时 4.2 秒,没有任何数据离开机器。

What It Found

  1. Authentication confidence: high debt: medium
  2. Billing Webhooks confidence: low debt: high
  3. Background Jobs confidence: low debt: high 它发现了什么
  4. 身份验证:置信度:高,技术债:中
  5. 计费 Webhook:置信度:低,技术债:高
  6. 后台任务:置信度:低,技术债:高

Authentication was solid. Real evidence, real routes:

  • Authentication is present and supported by behavior evidence. confidence: 0.82 evidence: apps/web/app/api/auth/forgot-password/route.ts, apps/web/app/api/auth/oauth/me/route.ts 身份验证非常稳固。有真实的证据,真实的路由:
  • 存在身份验证,并由行为证据支持。置信度:0.82,证据:apps/web/app/api/auth/forgot-password/route.ts, apps/web/app/api/auth/oauth/me/route.ts

But Billing Webhooks at LOW confidence? Cal.com is a scheduling SaaS with Stripe integration. Of course it has billing webhooks. I assumed I had found my first miss. 但计费 Webhook 的置信度为何是“低”?Cal.com 是一个集成了 Stripe 的调度 SaaS,它当然应该有计费 Webhook。我以为我发现了第一个漏报。

The Refusal Here is what DevTime actually said:

  • Possible Billing Webhooks signals detected. Billing Webhooks behavior is not established from current evidence. confidence: 0.45 Uncertainty:
  • Only dependency or manifest evidence was found for Billing Webhooks; presence is not confirmed behavior. 拒绝回答 DevTime 实际上是这样说的:
  • 检测到可能的计费 Webhook 信号。根据现有证据,无法确定计费 Webhook 的行为。置信度:0.45 不确定性:
  • 仅发现计费 Webhook 的依赖项或清单证据;无法确认其行为存在。

So I went to check manually. The repo has exactly the file you would expect: apps/web/pages/api/stripe/webhook.ts Case closed, right? A file named stripe/webhook.ts in a Stripe-integrated SaaS. Any developer grepping the repo would say “here is their webhook handler”. Any AI agent reading file paths would state it as fact. 于是我手动检查了一下。仓库里确实有你预期的那个文件:apps/web/pages/api/stripe/webhook.ts 案子结了,对吧?在一个集成了 Stripe 的 SaaS 中有一个名为 stripe/webhook.ts 的文件。任何在仓库中搜索的开发者都会说“这就是他们的 Webhook 处理程序”。任何读取文件路径的 AI 代理都会将其视为事实。

Here is the entire handler: export default function handler(_req: NextApiRequest, res: NextApiResponse) { res.status(404).json({ message: “Billing webhooks are not available in community edition” }); } 这是该处理程序的全部内容: export default function handler(_req: NextApiRequest, res: NextApiResponse) { res.status(404).json({ message: “Billing webhooks are not available in community edition” }); }

It is a stub. The open-source snapshot of Cal.com does not process billing webhooks. That behavior lives in their enterprise edition. The second webhook route in the repo is the same story. DevTime could not find behavior evidence because there is no behavior to find. The refusal was the correct answer. 这只是一个存根(stub)。Cal.com 的开源快照并不处理计费 Webhook。该行为存在于他们的企业版中。仓库中的第二个 Webhook 路由也是同样的情况。DevTime 找不到行为证据,因为根本没有行为可找。拒绝回答才是正确的答案。

That is the whole thesis of the tool in one example. A confident story built from file names would have been wrong. The repository could not prove billing webhooks, so the scanner said so. 这一个例子就概括了该工具的核心理念。如果仅凭文件名编造一个自信的故事,那将是错误的。仓库无法证明存在计费 Webhook,所以扫描器如实报告了这一点。

Where It Was Actually Wrong I promised misses, and there are real ones. Background Jobs was under-called. Cal.com has a genuine job system: a Tasker package with an internal tasker, a Redis-backed tasker, and a task processor, plus cron routes. That is real worker and queue behavior, and DevTime only surfaced dependency and test evidence at low confidence. The guard that suppresses fake background-job signals (calendar cleanup crons pretending to be job queues) is suppressing a true positive here. 它真正出错的地方 我承诺过会指出漏报,确实存在。后台任务(Background Jobs)被低估了。Cal.com 有一个真正的任务系统:一个包含内部任务调度器、基于 Redis 的任务调度器和任务处理器的 Tasker 包,外加 cron 路由。这是真实的 Worker 和队列行为,但 DevTime 仅以低置信度显示了依赖项和测试证据。用于抑制虚假后台任务信号(如伪装成任务队列的日历清理 cron)的防护机制,在这里误删了一个真正的正例。

Admin Permissions was missed entirely. The repo has real admin surfaces, including admin API routes and admin settings layouts. DevTime detected nothing. 管理权限(Admin Permissions)则完全被漏掉了。仓库中有真正的管理界面,包括管理 API 路由和管理设置布局。DevTime 什么也没检测到。

Both misses become regression fixtures now: small repo patterns encoding the expected output, so this exact failure can never return silently. 这两个漏报现在都变成了回归测试用例:通过小型仓库模式编码预期的输出,确保这种特定的失败永远不会再悄无声息地发生。

Funny detail: two of the oldest fixtures in the suite came from Cal.com-shaped false positives (calendar “subscriptions” being mistaken for billing). Those fixtures fixed overclaiming. Today Cal.com exposed the opposite failure, overcorrection. Same factory, opposite direction. 有趣的是:该套件中最古老的两个测试用例来自 Cal.com 形状的误报(日历“订阅”被误认为是计费)。那些测试用例修正了过度声明的问题。今天,Cal.com 暴露了相反的失败:过度修正。同一个工厂,相反的方向。

One more honesty note: during the scan, DevTime printed this on its own: Framework coverage warning: NestJS controller parsing is not supported in V0; backend route coverage may be incomplete. Cal.com’s API v2 is NestJS. The tool told me where its own blind spot was before I found it. 再补充一点诚实的情况:在扫描过程中,DevTime 自己打印了以下内容: 框架覆盖警告:V0 版本不支持 NestJS 控制器解析;后端路由覆盖可能不完整。 Cal.com 的 API v2 是 NestJS 编写的。该工具在我发现之前,就告诉了我它自己的盲点在哪里。

Why I Care About This Pattern Every tool in this space is optimized to sound sure. Summaries, autocomplete, agents: confidence is the product. But the Cal.com webhook stub is exactly the kind of thing confident tools get wrong. The story “this repo handles Stripe webhooks” is plausible, well-supported by file names, and false. If your AI agent believed it, it might modify payment logic that does not exist in this edition, or trust dedupe behavior that is not there. 我为什么关注这种模式 这个领域里的每一个工具都被优化得听起来很“确定”。摘要、自动补全、代理:置信度就是产品。但 Cal.com 的 Webhook 存根正是那些“自信”的工具会出错的地方。关于“此仓库处理 Stripe Webhook”的故事听起来很合理,文件名证据也很充分,但它是假的。如果你的 AI 代理相信了它,它可能会修改此版本中不存在的支付逻辑,或者信任并不存在的去重行为。

The rule I keep coming back to: No claim without evidence. Weak evidence produces uncertainty, not confidence. Uncertainty is not the tool failing. Uncertainty is the tool working. 我一直坚持的原则是:没有证据,就没有结论。薄弱的证据产生的是不确定性,而不是置信度。不确定性不是工具的失败,不确定性正是工具在正常工作。

Try It on Your Repo pipx install devtime-ei cd your-repo dtc init dtc scan dtc concepts 在你的仓库上试试 pipx install devtime-ei cd your-repo dtc init dtc scan dtc concepts

There is also a read-only MCP server (dtc mcp start) so coding agents can query the same evidence instead of guessing. If DevTime gets something wrong on your repository, that is not an embarrassment, it is a contribution. Wrong outputs become fixtures, and fixtures are how this tool earns trust. 此外还有一个只读的 MCP 服务器(dtc mcp start),这样编码代理就可以查询相同的证据,而不是靠猜。如果 DevTime 在你的仓库上出错了,那不是尴尬,而是一种贡献。错误的输出会变成测试用例,而测试用例正是该工具赢得信任的方式。