Your AI Product Needs a Plan for Model Retirement

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Your AI Product Needs a Plan for Model Retirement

An appointment agent still answers the phone after a model change. Its replies sound natural. Yet it now asks one extra question before checking availability, summarizes an unconfirmed time as booked, or takes long enough to make callers repeat themselves. The endpoint works while the product’s behavior has changed.

在模型更换后，预约代理依然能够接听电话。它的回复听起来很自然。然而，它现在会在检查可用性之前多问一个问题，将未确认的时间总结为已预订，或者因为反应迟钝导致来电者不得不重复自己的话。虽然端点仍在工作，但产品的行为已经发生了改变。

Disclosure: An AI agent researched and drafted this article from my public project documentation and the linked primary sources. The model-change scenario and migration plan are illustrative recommendations, not reports of a SmartVoiceX incident or a completed model migration. My SmartVoiceX case study describes a working phone-operations product in ongoing development. It connects calling flows with calendars, routing, human transfer, follow-up, transcripts, summaries, account administration, and billing. That scope makes model replacement a product-maintenance question: the surrounding workflow still has to produce the right next step.

披露：一个人工智能代理根据我的公开项目文档和链接的原始资料研究并起草了本文。文中的模型变更场景和迁移计划仅为说明性建议，并非 SmartVoiceX 的事故报告或已完成的模型迁移。我的 SmartVoiceX 案例研究描述了一个正在开发中的电话运营产品。它将通话流程与日历、路由、人工转接、后续跟进、转录、摘要、账户管理和计费连接起来。这种范围使模型替换成为一个产品维护问题：周围的工作流仍然必须产生正确的下一步操作。

A retirement notice is a deadline for a decision. On October 15, GitHub announced selected Copilot model deprecations for November 16, 2026. The notice covers Copilot experiences and identifies suggested alternatives, with an exception for one model for individual annual subscribers. It is not a retirement announcement for the underlying providers’ APIs or a claim about SmartVoiceX’s dependencies. The operational lesson is broader: a model choice has a support lifecycle. A product can outlive the particular model used to build its first version. Owning the application therefore includes knowing how to evaluate and replace that dependency.

退役通知是决策的最后期限。10月15日，GitHub宣布了部分Copilot模型将于2026年11月16日弃用。该通知涵盖了Copilot体验并指出了建议的替代方案，但针对个人年度订阅者使用的一款模型除外。这并非针对底层提供商API的退役公告，也不是关于SmartVoiceX依赖项的声明。运营方面的经验教训更为广泛：模型选择具有支持生命周期。产品的寿命可能超过构建其第一个版本时所使用的特定模型。因此，拥有该应用程序意味着必须知道如何评估和替换该依赖项。

For a deployed product, I would keep a short inventory of the model identifiers and providers used by each workflow, the configuration that selects them, the owner who monitors lifecycle notices, and the behavior that must survive a change. Record version information where the provider exposes it. A friendly model name alone may not describe the exact configuration that was tested. This is especially useful at handoff. The next operator should be able to locate a dependency and its test procedure without reconstructing the original builder’s chat history.

对于已部署的产品，我会保留一份简短的清单，记录每个工作流所使用的模型标识符和提供商、选择它们的配置、负责监控生命周期通知的人员，以及必须在变更中保持不变的行为。在提供商公开版本信息的地方记录版本信息。仅凭友好的模型名称可能无法描述所测试的确切配置。这在交接时特别有用。下一位操作员应该能够在无需重构原始构建者聊天记录的情况下，找到依赖项及其测试程序。

Make the replacement earn its place in the workflow. A suggested alternative is a candidate for evaluation. Compatibility at the request interface does not establish equivalent behavior in a multi-step task. For a hypothetical appointment workflow, I would use controlled calendar data and a set of scenarios with explicit outcomes: a successful booking, no available slot, an ambiguous date, a request to change an earlier detail, a tool timeout, and a request for a person. Keep real callers and live calendar writes out of this evaluation environment.

让替换方案在工作流中证明其价值。建议的替代方案只是评估的候选对象。请求接口的兼容性并不能确保多步骤任务中行为的一致性。对于一个假设的预约工作流，我会使用受控的日历数据和一组具有明确结果的场景：成功预订、无可用时段、日期模糊、更改先前细节的请求、工具超时以及人工转接请求。请将真实的来电者和实时的日历写入操作排除在此评估环境之外。

Check what exists afterward. Did the tool create the intended appointment? Was an unavailable time left unbooked? Does the summary distinguish a confirmed result from a request awaiting follow-up? A polished sentence should not be the only evidence that an action completed. Anthropic’s January 9, 2026 evaluation guidance separates a trial’s transcript from its final outcome and describes evaluating the model together with its harness. That is useful here: a migration must exercise the instructions, tool schemas, application logic, and model as a working combination. Keep the business rules stable for the initial comparison. If the new model needs different prompts or tool descriptions, record those changes and rerun the scenarios. Otherwise the team cannot tell which change caused a gain or regression.

检查之后的结果。工具是否创建了预期的预约？不可用的时间是否未被预订？摘要是否区分了已确认的结果和等待跟进的请求？优美的句子不应是操作完成的唯一证据。Anthropic 2026年1月9日的评估指南将试验的转录与最终结果分开，并描述了将模型与其测试框架一起评估的方法。这在这里很有用：迁移必须将指令、工具模式、应用程序逻辑和模型作为一个工作组合进行测试。在初步比较时保持业务规则稳定。如果新模型需要不同的提示词或工具描述，请记录这些更改并重新运行场景。否则，团队将无法判断是哪种更改导致了提升或倒退。

Preserve useful variation while checking concrete outcomes. The replacement does not need to repeat the old model’s wording. Several different responses may guide a caller through the same successful task. Tests that demand a particular sentence can reject a useful improvement while missing an incorrect booking. I would separate outcome checks from a rubric for the conversation. The outcome checks can inspect tool arguments and stored test records. The rubric can assess whether the caller receives a clear explanation, whether unnecessary questions accumulate, and whether a handoff preserves the context a person needs. Run multiple trials where outputs vary, and include scenarios held back from prompt tuning. Retain failed examples so the next iteration has a specific problem to solve. An evaluation set is evidence about the cases it covers, not a guarantee about every future call.

在检查具体结果的同时保留有用的变化。替换模型不需要重复旧模型的措辞。几种不同的回复都可以引导来电者完成相同的成功任务。要求特定句子的测试可能会拒绝有用的改进，同时漏掉错误的预订。我会将结果检查与对话评分标准分开。结果检查可以审查工具参数和存储的测试记录。评分标准可以评估来电者是否收到了清晰的解释、是否积累了不必要的问题，以及转接是否保留了人工所需的环境信息。运行多次输出不同的试验，并包含未用于提示词调优的场景。保留失败的案例，以便下一次迭代有具体的问题需要解决。评估集只是其所涵盖案例的证据，并不能保证未来的每一次通话都完美无缺。

For voice, transcript-based checks are only part of acceptance. Exercise the candidate through the actual audio path as well. Measure the time callers wait for a useful response and test recovery when a service fails. Speech recognition, synthesis, networking, and application tools can affect the experience even when the language model is unchanged. A fallback must still exist after the deadline. Before changing the default, decide what happens if the replacement fails an essential requirement. Routing a subset of supported requests to a person or disabling one automated action may be preferable to allowing an untested workflow to keep making commitments. Rolling back to the old model can be useful while it remains available. It is not a durable fallback if that model is about to disappear. The recovery plan needs an option the team can actually use after retirement.

对于语音而言，基于转录的检查只是验收的一部分。还要通过实际的音频路径来测试候选模型。测量来电者等待有效回复的时间，并测试服务失败时的恢复情况。即使语言模型未变，语音识别、合成、网络和应用程序工具也可能影响体验。截止日期后必须存在回退方案。在更改默认设置之前，请决定如果替换方案未能满足基本要求该怎么办。将部分受支持的请求路由给人工，或禁用某项自动化操作，可能比允许未经测试的工作流继续做出承诺要好。在旧模型仍然可用时，回滚到旧模型可能很有用。但如果该模型即将消失，它就不是一个持久的回退方案。恢复计划需要一个团队在退役后真正能够使用的选项。

A staged rollout also has a cost. Keeping a candidate configuration, evaluation fixtures, and monitoring takes effort. An adapter that centralizes provider-specific behavior can make changes easier, but a universal interface can conceal capabilities or assumptions that do not transfer cleanly. Document those differences instead of assuming every model is interchangeable. I would leave the operator a compact release record: the selected configuration, known limitations, evaluation results, monitoring signals, and the steps for entering the fallback mode. Keep secrets in the appropriate secret store rather than embedding them in that document. For small teams, the practical benefit.

分阶段推广也有成本。维护候选配置、评估装置和监控需要付出努力。集中处理特定提供商行为的适配器可以使变更更容易，但通用接口可能会掩盖无法完全迁移的功能或假设。记录这些差异，而不是假设每个模型都是可互换的。我会给操作员留下一份简洁的发布记录：所选配置、已知限制、评估结果、监控信号以及进入回退模式的步骤。将密钥保存在适当的密钥存储中，而不是嵌入到该文档中。对于小团队来说，其实际利益。