Agentic AI: Who’s responsible? The AI? Or the developer?

代理式 AI：谁该负责？是 AI 本身？还是开发者？

This week, while I was having lunch, I overheard someone talking about the incident that happened in July 2025 where a Florida mother was coerced into paying $15,000 after an AI cloned her daughter’s voice in a way that made it sound like she was in danger. 本周，我在吃午饭时无意中听到有人谈论 2025 年 7 月发生的一起事件：佛罗里达州一位母亲被 AI 克隆的女儿声音所欺骗，对方伪造了她女儿处于危险中的假象，迫使她支付了 1.5 万美元。

Immediately, I remembered the article I wrote the previous week about how agentic AI has impacted our decision-making processes. However, this person who brought up the topic asked a very good question: Who’s truly responsible for the decisions AI makes? The AI itself? Or the developer who made it? 我立刻想起了上周写的一篇文章，内容是关于代理式 AI 如何影响我们的决策过程。然而，这位谈论此事的人提出了一个非常好的问题：AI 所做出的决策，到底该由谁负责？是 AI 本身？还是开发它的开发者？

The truth is that it entirely depends on the circumstances that led to the development of the AI, what happened throughout the development process (i.e. if any guardrails were added), and the impact it has had on the individuals using it. But overall, the responsibility falls entirely on the developers as they are responsible for distributing their product to their clients and/or customers. 事实是，这完全取决于 AI 开发背后的具体情况、开发过程中发生了什么（例如是否添加了护栏机制），以及它对使用者造成的影响。但总的来说，责任完全在于开发者，因为他们负责将产品分发给客户和/或用户。

AI is solely the mouthpiece of the goals and ambitions of the developers. While there is unintended behavior that they may not have anticipated, it doesn’t absolve anyone who developed it from being held accountable. AI is a helpful tool, but it’s also a serious one that can go sideways for businesses and individuals who use it. AI 仅仅是开发者目标和野心的传声筒。虽然可能会出现他们未曾预料到的意外行为，但这并不能免除开发者的责任。AI 是一个有用的工具，但它也是一个严肃的工具，如果使用不当，可能会给企业和个人带来严重后果。

There are 5 notable instances in which AI or its agentic variant have caused serious consequences as a result of unintended behavior, negligence, or malicious intent stemming from the developers’ own decisions: 以下是 5 个值得注意的案例，由于开发者的决策导致了意外行为、疏忽或恶意意图，AI 或其代理变体造成了严重后果：

Air Canada’s False Chatbot Response: In November 2022, Jake Moffatt asked a question to Air Canada’s chatbot regarding their bereavement refund policy after his grandmother passed away. This led to a claim being denied, resulting in a tribunal case that eventually led to Air Canada being held liable in February 2024, approximately 15 months after the incident. The latter subsequently claimed the AI agent was, in their words, “a separate legal entity” responsible for its own actions. 加拿大航空的虚假聊天机器人回复： 2022 年 11 月，Jake Moffatt 在祖母去世后向加拿大航空的聊天机器人询问有关丧亲退款政策的问题。这导致他的索赔被拒绝，引发了一场法庭诉讼，最终在事件发生约 15 个月后的 2024 年 2 月，加拿大航空被判承担责任。加航随后辩称，该 AI 代理是“一个独立的法律实体”，应对其自身行为负责。

Waymo Mishaps in Texas: Instances include the deadly March 2026 West Sixth Street shooting in Austin, Texas where a responding Austin PD officer was forced to move a Waymo vehicle after it got stuck to make way for an ambulance responding to the scene, an apartment explosion that occurred in Dallas, Texas in early June 2026 where first responders’ path was blocked, and similar incidents occurring from March to May 2026 across North, Central and Southeast Texas, resulting in Waymo suspending their services in the affected cities. Waymo 在德克萨斯州的事故： 案例包括 2026 年 3 月德克萨斯州奥斯汀市西第六街发生的致命枪击案，当时一名响应的奥斯汀警官被迫移动一辆卡住的 Waymo 车辆，以便为救护车让路；2026 年 6 月初德克萨斯州达拉斯市发生的公寓爆炸案中，急救人员的路径被阻挡；以及 2026 年 3 月至 5 月期间在德克萨斯州北部、中部和东南部发生的类似事件，导致 Waymo 在受影响城市暂停了服务。

March 2026 Amazon Outages: Amazon experienced multiple outages following a change to its retail platform within the span of a week. This resulted in multiple operations going down worldwide, ranging from checkout to delivery services. While Amazon officials originally cited generative AI as a contributing factor, it subsequently reversed its position, putting the blame on one of their software engineers following “inaccurate advice” from outdated documentation that an AI agent referred to them. 2026 年 3 月亚马逊宕机事件： 亚马逊零售平台在一周内进行更改后经历了多次宕机。这导致全球范围内的多项业务瘫痪，从结账到配送服务均受影响。虽然亚马逊官员最初称生成式 AI 是促成因素，但随后改变了立场，将责任归咎于其一名软件工程师，称其遵循了 AI 代理提供的过时文档中的“不准确建议”。

False advertising utilizing well-known South Florida lawyer’s image: In June 2026, AI has been used to create false advertisements using South Florida lawyer Ángel Leal’s image with the goal of promising deportees a chance of returning to the United States if they paid $1,500 to use his legal services. While not necessarily the result of a developer’s actions, the lack of guardrails allowed the scammers to promote their campaign, resulting in multiple federal investigations and Leal’s reputation being tarnished through no fault of his own. 利用南佛罗里达知名律师形象进行虚假广告宣传： 2026 年 6 月，有人利用 AI 制作虚假广告，使用南佛罗里达律师 Ángel Leal 的形象，承诺被驱逐出境者如果支付 1,500 美元使用其法律服务，就有机会返回美国。虽然这不一定是开发者的直接行为，但缺乏护栏机制使得诈骗者能够推广其活动，导致了多项联邦调查，Leal 的声誉也因此无辜受损。

Character.AI and Meta AI Studio August 2025 investigations: The Texas Attorney General’s office released a statement in August 2025, announcing investigations into Character.AI and Meta AI Studio regarding their chatbot platforms and data privacy/collection practices. The Attorney General’s office alleges that both companies have potentially engaged in false advertising and deceptive trade practices by distributing its AI agents to its users without the proper guardrails in place. They allege that as a result, it has led to vulnerable individuals and minors alike to use it for emotional support, despite the products lacking a medical license to practice it. Character.AI 和 Meta AI Studio 2025 年 8 月的调查： 德克萨斯州总检察长办公室于 2025 年 8 月发表声明，宣布对 Character.AI 和 Meta AI Studio 的聊天机器人平台及数据隐私/收集行为展开调查。总检察长办公室指控这两家公司在未设置适当护栏的情况下向用户分发 AI 代理，可能涉及虚假广告和欺骗性贸易行为。他们声称，这导致弱势群体和未成年人将其用于情感支持，尽管这些产品并不具备医疗执业许可。

Based on these five instances, they all point to a similar pattern: a lack of accountability mixed with a lack of guardrails and safety measures in place, resulting from a lack of oversight and responsibility on the developers’ part to ensure that these systems were set up in the first place before deploying it to production. 基于这五个案例，它们都指向一个相似的模式：缺乏问责制，加上缺乏护栏和安全措施，这是由于开发者在将系统部署到生产环境之前，缺乏监督和责任感来确保这些系统设置到位。

This is the standard that I hold myself to when developing products such as Benny. While no development process is perfect, we can reduce the risk of human error by ensuring that we do our due diligence on what we want our AI agents or any other software product to do. We have to also realize that as soon as an error is found, we are ultimately responsible for the impact it has on our customers, clients, and end users utilizing the product. 这就是我在开发 Benny 等产品时对自己设定的标准。虽然没有完美的开发过程，但我们可以通过确保对 AI 代理或其他软件产品的功能进行尽职调查，来降低人为错误的风险。我们还必须意识到，一旦发现错误，我们最终要对它给客户、委托人和终端用户造成的影响负责。

I heavily emphasize this as a software developer because as someone who has worked with customers and clients in the past, every action we take will lead to a chain reaction of small events that lead to severe consequences if they are not addressed properly. If you are still doubting the severity, you can refer to the incidents that have plagued Amazon’s retail services for a week or any other horror story involving a form of AI or its agentic variant. 作为一名软件开发者，我非常强调这一点，因为作为曾经与客户和委托人共事过的人，我们采取的每一个行动都会引发一系列连锁反应，如果处理不当，就会导致严重的后果。如果你仍然怀疑其严重性，可以参考困扰亚马逊零售服务一周的事件，或任何其他涉及 AI 或其代理变体的恐怖故事。

Currently, I am setting up Benny to detect false leads through the use of different actions, such as detecting user inactivity, context, and past references using a combination of retrieval augmented generation and software engineering principles/ethics with human-in-the-loop approaches (i.e. manual reviews, additional references, escalation, etc.). Even with all the guardrails in place, Benny will eventually flag an actual lead as a false positive, and vice versa. It often results from malicious actors evolving their tactics. 目前，我正在设置 Benny，通过不同的操作来检测虚假线索，例如结合检索增强生成（RAG）、软件工程原则/伦理以及“人在回路”（human-in-the-loop）方法（即人工审核、额外参考、升级处理等）来检测用户不活跃状态、上下文和过往引用。即使设置了所有的护栏，Benny 最终还是会将真实的线索标记为误报，反之亦然。这通常是因为恶意行为者在不断演变他们的策略。