My AI agent writes my marketing. Today I made it prove every post is true.
Title: My AI agent writes my marketing. Today I made it prove every post is true. 标题:我的 AI 代理负责撰写营销内容。今天,我要求它证明每一篇帖子都是真实的。
I build an app called Scribo by myself. To save time, I let an AI agent write my marketing posts. It’s fast. But today I noticed something. It never checks if what it says is true. I spent the day at Kane CLI Hack Day by TestMu AI, so I tested my own posts with Kane CLI. I found two problems, and both were on my own app. 我独自开发了一款名为 Scribo 的应用程序。为了节省时间,我让一个 AI 代理来撰写我的营销帖子。这确实很快,但我今天注意到了一件事:它从不核实所写内容是否属实。我在 TestMu AI 举办的 Kane CLI 黑客马拉松上度过了一天,于是我用 Kane CLI 测试了我自己的帖子。我发现了两个问题,而且都出在我自己的应用上。
The link that told the wrong story. My agent wrote a Threads post and linked it to my website. Looks fine. Kane opened the post in a real Chrome browser and clicked the link. The App Store button on my website changes the campaign tag. The post sends people with ct=threads-oct24, and the button replaces it with ct=site-en-hero. So every install from Threads would have looked like it came from my website. There was no error. Nothing crashed. The number was just wrong. And that’s the scary part. My agent learns from these numbers. With this bug, it would have decided Threads doesn’t work and stopped posting there. 链接讲述了错误的故事。我的代理写了一篇 Threads 帖子并链接到我的网站,看起来没问题。Kane 在真实的 Chrome 浏览器中打开了这篇帖子并点击了链接。我网站上的 App Store 按钮会更改营销标签。帖子将用户引导至 ct=threads-oct24,但按钮将其替换为 ct=site-en-hero。因此,来自 Threads 的每一次安装看起来都像是来自我的网站。没有报错,也没有崩溃,只是数据错了。这才是最可怕的地方。我的代理会从这些数据中学习,如果存在这个漏洞,它会误以为 Threads 渠道无效,从而停止在那里发帖。
The posts that got old. The second problem was time. I rebuilt Scribo this month. It used to be a handwriting vocabulary app. Now it’s an app that rewrites videos and articles at your level. My old posts are still online. They still say “a vocabulary notebook you write by hand.” People still find them, open the App Store, and see a different app. 过时的帖子。第二个问题是时间。我这个月重构了 Scribo。它曾经是一款手写词汇应用,现在则是一款根据你的水平重写视频和文章的应用。我以前的帖子还在网上,上面仍然写着“一本手写的词汇笔记本”。人们仍然能找到它们,打开 App Store,却看到了一个完全不同的应用。
What I built. So I built Omni Verify, a gate between my agent and the publish button. The agent (OpenAI) drafts a post and turns every promise in it into a check. The link’s tag, each sentence, each feature. Kane CLI runs those checks in a real browser. The agent reads Kane’s result. If something fails, it fixes the post and Kane checks again. Publish stays locked until Kane passes that exact version. 我的解决方案。于是我构建了 Omni Verify,作为我的代理和发布按钮之间的一道关卡。代理(OpenAI)起草帖子,并将其中的每一个承诺转化为检查项,包括链接标签、每一个句子和每一项功能。Kane CLI 在真实浏览器中运行这些检查,代理读取 Kane 的结果。如果检查失败,它会修改帖子并让 Kane 再次检查。在 Kane 通过该版本之前,发布按钮将保持锁定状态。
For the link bug, Kane failed the first draft (KR-63). The agent linked straight to the App Store with the right tag, and Kane passed it (KR-68). Then I clicked Re-verify, Kane replayed the test, and it passed again (KR-85). Only then did the publish button unlock. This post went out through that gate today. 针对链接漏洞,Kane 驳回了初稿(KR-63)。代理随后直接链接到带有正确标签的 App Store,Kane 通过了测试(KR-68)。接着我点击了“重新验证”,Kane 重复了测试,再次通过(KR-85)。只有在那时,发布按钮才解锁。今天这篇帖子就是通过这道关卡发布的。
For the old posts, Kane checked six promises from three of my old posts against my website today. Three passed. Three failed, all about handwriting. The agent suggests a new version of each post, or taking it down. It doesn’t change anything on Threads by itself. The agent also remembers. The three failed promises go into a memory file as “retired.” When it writes the next post, it can’t use them. I tested this by giving it two drafts with the old promise. It refused both. 对于旧帖子,Kane 今天根据我的网站检查了我三篇旧帖子中的六个承诺。三个通过,三个失败,失败的都与“手写”有关。代理建议为每篇帖子提供新版本或将其删除,它不会自行更改 Threads 上的内容。代理还会进行记忆,那三个失败的承诺被存入一个名为“已退役”的记忆文件中。当它撰写下一篇帖子时,就不能再使用这些承诺了。我通过给它两个包含旧承诺的草稿进行了测试,它拒绝了这两个草稿。
Where Kane surprised me. Kane didn’t only run my checks. I gave it the campaign, and it designed its own tests, with me approving each step. It also mapped my site with Kane Explore before the agent wrote anything. It wasn’t perfect. Once, Kane said a sentence wasn’t on the page when it was. Asking it a simple true or false for each phrase fixed it. I wrote that down in the repo too. Kane 令我惊喜的地方。Kane 不仅仅运行我设定的检查。我给它提供了营销活动目标,它设计了自己的测试,并由我批准每一步。在代理撰写任何内容之前,它还使用 Kane Explore 对我的网站进行了映射。虽然并不完美,比如有一次 Kane 说页面上没有某句话,但实际上有。通过要求它对每个短语进行简单的真伪判断,问题就解决了。我也把这一点记录在了代码库中。
What a pass doesn’t mean. A pass doesn’t mean more installs. I haven’t checked App Store Connect yet. It means the post tells the truth when it goes out. That’s the point for me. Before my agent learns from a post, I want to know the post was honest and the numbers will be right. AI can write the promise. Kane checks if the product keeps it. 通过测试并不意味着安装量会增加。我还没查看 App Store Connect 的数据。这意味着帖子在发布时是真实的,这对我来说才是重点。在我的代理从帖子中学习之前,我想确保帖子内容诚实且数据准确。AI 可以写下承诺,而 Kane 则负责检查产品是否兑现了这些承诺。