AI chatbots have failed people in crisis. Can that be fixed?
AI chatbots have failed people in crisis. Can that be fixed?
AI 聊天机器人辜负了处于危机中的人们，这能被修复吗？
This year alone, there have been numerous known instances—often via lawsuits—of AI chatbots (most often, OpenAI’s ChatGPT) that have gone horrifically wrong. A January lawsuit described the story of a man who took his own life after being allegedly “coached” into suicide. A college student in Georgia sued OpenAI, claiming that ChatGPT “pushed him into psychosis.” In June, a Canadian family also sued OpenAI and argued that ChatGPT agreed with the young woman’s dismissiveness when it first gave her the option to seek professional mental health advice. ChatGPT allegedly “encouraged” her to end her life, too, and she did so.
仅今年一年，就已经发生了多起广为人知的案例（通常通过诉讼曝光），涉及 AI 聊天机器人（最常见的是 OpenAI 的 ChatGPT）造成了可怕的后果。一月份的一起诉讼描述了一名男子在被指控受到“诱导”自杀后结束了自己生命的故事。佐治亚州的一名大学生起诉 OpenAI，声称 ChatGPT “将他推向了精神错乱”。六月，一个加拿大美国家庭也起诉了 OpenAI，并指出当 ChatGPT 最初向该年轻女子提供寻求专业心理健康建议的选项时，它却附和了她对这些建议的排斥态度。据称，ChatGPT 还“鼓励”她结束自己的生命，最终她真的这样做了。
So what should OpenAI—and other AI companies generally—do differently to reduce harm among people who use their products? Silicon Valley is certainly aware of the legal liability it now faces as these products are being used in ways that they were not intended for, and it seems to be trying to improve. On Thursday, OpenAI announced that it had partnered with the American Psychological Association to “bring psychological science into how we think about responsible AI development and use among young people.”
那么，OpenAI 以及其他 AI 公司通常应该做出哪些改变，以减少其产品使用者受到的伤害？硅谷显然已经意识到，随着这些产品被用于非预期用途，他们正面临法律责任，并且似乎正在努力改进。周四，OpenAI 宣布与美国心理学会合作，旨在“将心理科学引入我们对年轻人负责任的 AI 开发和使用的思考中”。
Experts told Ars that, while large language model safety has seemingly improved, there are some broad suggestions—more transparency into the models and a de-anthropomorphization of chatbots being chief among them—that would likely further reduce harm. “Third-party evaluation suggests newer LLMs generally recognize distress and can respond with seeming empathy, and actively damaging responses are infrequent,” Shaddy Saba, a professor of social work at New York University, emailed Ars. “Where they fall short is actually probing for risk, guiding people to human care, and holding appropriate boundaries around what an AI should and shouldn’t do in these situations.”
专家告诉 Ars，虽然大语言模型的安全性似乎有所提高，但仍有一些广泛的建议——其中最主要的是提高模型的透明度以及降低聊天机器人的拟人化——可能会进一步减少伤害。“第三方评估表明，较新的大语言模型通常能识别出痛苦并以看似同理心的方式回应，主动造成伤害的回复并不常见，”纽约大学社会工作教授 Shaddy Saba 在给 Ars 的邮件中写道。“它们的不足之处在于未能真正探测风险、引导人们寻求人工护理，以及在这些情况下未能就 AI 该做什么和不该做什么保持适当的界限。”
AI is not a mental health professional
AI 不是心理健康专业人士
It’s no secret that many people are using chatbots to make emotional or interpersonal decisions, even when companies tell them not to. While the cases that make the news may have resulted in some of the worst-known outcomes, according to the results of a published November 2025 medical survey, many more people are using chatbots in this way, mostly with innocuous results. In that paper, over 13 percent of respondents said they had done so. If extrapolated nationwide, that would mean millions of Americans have used a chatbot “for advice or help” when faced with a difficult emotional situation.
许多人正在使用聊天机器人来做出情感或人际关系决策，这早已不是什么秘密，尽管公司明确告知不要这样做。虽然登上新闻的案例可能导致了已知最糟糕的结果，但根据 2025 年 11 月发表的一项医学调查结果显示，有更多的人以这种方式使用聊天机器人，且大多没有产生严重后果。在那篇论文中，超过 13% 的受访者表示他们曾这样做过。如果推及全国，这意味着数百万美国人在面对困难的情绪状况时，曾使用聊天机器人“寻求建议或帮助”。
A panel of mental health professionals convened earlier this year by the National Academy of Medicine found that “chatbots are likely harming people, but we can’t measure how much.” It appears those deleterious effects may be diminishing, but they haven’t been eliminated. An April 2026 preprint paper by a team from the City University of New York and King’s College London found that “unsafe” models, including Chat GPT-4o, Grok 4.1 Fast, and Gemini 3 Pro, “did more than validate delusional claims; they elaborated on them, absorbed the user’s interpretive frame as their own, and progressively lost the capacity to distinguish a user in crisis from a narrative to be extended.” However, since that paper came out, all of these models have been deprecated by their respective makers.
今年早些时候由美国国家医学院召集的一个心理健康专家小组发现，“聊天机器人很可能正在伤害人们，但我们无法衡量伤害程度。”这些有害影响似乎正在减弱，但并未消除。2026 年 4 月，纽约市立大学和伦敦国王学院的一个团队发表的一篇预印本论文发现，包括 ChatGPT-4o、Grok 4.1 Fast 和 Gemini 3 Pro 在内的“不安全”模型，“不仅验证了妄想性主张，还对其进行了阐述，将用户的解释框架吸收为自己的框架，并逐渐失去了区分处于危机中的用户与需要延伸的叙事的能力。”然而，自该论文发表以来，所有这些模型都已被其各自的制造商弃用。
Of the major chatbot makers, only Anthropic responded to Ars’ request for comment. OpenAI did not respond, while Google sent a link to “An update on our mental health work” after publication. “Claude is not designed or intended to act as a mental health professional, and it makes that clear in conversations where these topics arise,” said Michael Aciman, a spokesperson for Anthropic. “When users raise mental health concerns, Claude is designed to respond with care while encouraging users to seek guidance from licensed professionals.” He noted that Anthropic says it has worked to reduce sycophancy in its models.
在主要的聊天机器人制造商中，只有 Anthropic 回应了 Ars 的置评请求。OpenAI 没有回应，而谷歌在文章发布后发送了一个关于“我们心理健康工作的更新”的链接。“Claude 的设计初衷并非作为心理健康专业人士，当这些话题出现时，它会在对话中明确这一点，”Anthropic 发言人 Michael Aciman 表示。“当用户提出心理健康问题时，Claude 的设计旨在以关怀的态度回应，同时鼓励用户寻求持证专业人士的指导。”他指出，Anthropic 表示已致力于减少其模型中的“阿谀奉承”（sycophancy）现象。
Thursday’s announcement marks a number of public steps that OpenAI has taken recently in an effort to mitigate dangerous outcomes. These range from creating an “expert council” of mental health experts (October 2025) to inviting users to create an optional “Trusted Contact” (April 2026) that ChatGPT can contact if it detects serious emotional distress. OpenAI has previously said it has “deep responsibility to help those who need it most.”
周四的公告标志着 OpenAI 最近为减轻危险后果而采取的一系列公开举措。这些举措包括成立心理健康专家“专家委员会”（2025 年 10 月），以及邀请用户创建可选的“受信任联系人”（2026 年 4 月），以便在 ChatGPT 检测到严重情绪困扰时可以联系该联系人。OpenAI 此前曾表示，它有“帮助最需要帮助的人的深重责任”。
“Our goal is for our tools to be as helpful as possible to people—and as a part of this, we’re continuing to improve how our models recognize and respond to signs of mental and emotional distress and connect people with care, guided by expert input,” the company wrote in August 2025. In October 2025, OpenAI also wrote that it had “expanded access to crisis hotlines, re-routed sensitive conversations originating from other models to safer models, and added gentle reminders to take breaks during long sessions.”
“我们的目标是让我们的工具尽可能地为人们提供帮助——作为其中的一部分，在专家意见的指导下，我们正在不断改进模型识别和响应心理及情绪困扰迹象的方式，并将人们与护理服务连接起来，”该公司在 2025 年 8 月写道。2025 年 10 月，OpenAI 还写道，它已经“扩大了对危机热线的访问权限，将源自其他模型的敏感对话重新路由到更安全的模型，并增加了在长时间会话中休息的温和提醒。”
Black boxes
黑箱
It’s not always easy, though, to know precisely what changes to reduce dangerous mental health outcomes have been effective. “It does become tricky without knowing how many conversations went on,” John Torous, a professor of psychiatry at Harvard Medical School, told Ars. “Do the safeguards work for most people? Where do they fail? It’s a black box of how it’s happening or how it’s responding.”
然而，要确切知道哪些旨在减少危险心理健康后果的改变是有效的，并不总是那么容易。“在不知道进行了多少次对话的情况下，这确实变得很棘手，”哈佛医学院精神病学教授 John Torous 告诉 Ars。“这些保障措施对大多数人有效吗？它们在哪里失效了？它是如何发生或如何响应的，这完全是一个黑箱。”
Similarly, Saba, the NYU professor, noted that most of the professional medical and mental health world has a very opaque view into what is happening inside these AI companies. Altering that, he said, would go a long way. “Models also update far faster than traditional research and publication timelines,” he wrote. “Companies should publish their safety evaluation methods and results, submit to open benchmarks, and build with clinicians, researchers, lawmakers, and people with lived experience at the table.” Absent a closer look from the inside, some researchers are trying to poke and prod from the outside. Ragy Girgis, a pr…
同样，纽约大学教授 Saba 指出，大多数专业医疗和心理健康界对这些 AI 公司内部发生的事情了解甚少。他说，改变这一点将大有裨益。“模型的更新速度也远快于传统的研究和出版周期，”他写道。“公司应该公布其安全评估方法和结果，提交给开放基准测试，并与临床医生、研究人员、立法者以及有亲身经历的人共同构建。”在缺乏从内部进行更深入观察的情况下，一些研究人员正试图从外部进行探索和推动。Ragy Girgis，一位…