A Faster Model Will Not Fix Your Slow Voice Agent

更快的模型无法解决语音代理响应迟缓的问题

The first thing I check when a client tells me their AI voice agent “feels slow” is not the model. It is almost never the model. I have swapped a faster model into a sluggish agent and had callers notice nothing at all, because the seconds they were feeling were never being spent where everyone assumed. 当客户告诉我他们的 AI 语音代理“感觉很慢”时，我首先检查的不是模型。几乎从来都不是模型的问题。我曾将更快的模型替换进迟缓的代理中，但通话者完全没有察觉到任何变化，因为他们感受到的延迟时间根本没有花在大家以为的地方。

If you have ever profiled a slow page and found the real cost was a render blocking script rather than the function you spent all week optimizing, you already understand this post. Voice just makes the mistake more expensive, because a phone call has no spinner. No loading state, no skeleton UI, no way to signal “still working” other than talking. 如果你曾经分析过一个加载缓慢的网页，却发现真正的开销是一个阻塞渲染的脚本，而不是你花了一周时间优化的函数，那么你已经理解这篇文章了。语音交互让这种错误代价更高，因为电话没有加载转圈动画。没有加载状态，没有骨架屏，除了说话之外，没有任何方式可以示意“系统正在处理”。

Silence is the only progress indicator you get, and to a human on a phone, silence means the line dropped, the other person is confused, or nobody is listening. All three make callers talk over the agent, repeat themselves, or hang up. 沉默是你唯一的进度指示器。对于电话那头的人来说，沉默意味着掉线、对方困惑，或者根本没人听。这三种情况都会导致通话者打断代理、重复说话或直接挂断。

I build production voice agents on Retell wired into n8n, GoHighLevel and Twilio. Before that I spent years in Unity and AR, where a late frame was a safety problem rather than a conversion problem. That is where I learned to treat latency as a property of the whole chain rather than a number attached to one component. 我使用 Retell 构建生产级语音代理，并将其连接到 n8n、GoHighLevel 和 Twilio。在此之前，我在 Unity 和 AR 领域工作多年，在那里，帧率延迟是安全问题而非转化问题。正是在那里，我学会了将延迟视为整个链路的属性，而不是某个单一组件的指标。

The metric is turn latency, not time to first token

核心指标是“轮次延迟”，而非“首字延迟”

The number everybody quotes is time to first token, or whatever response time the vendor publishes. The number callers experience is turn latency: the gap between the moment they stop speaking and the moment they hear the agent start. That gap is a sum, and the model is one term in it: 大家都在引用的指标是“首字延迟”（Time to First Token），或者是供应商发布的任何响应时间。而通话者实际体验到的指标是“轮次延迟”（Turn Latency）：即他们停止说话到听到代理开始回应之间的间隔。这个间隔是一个总和，模型只是其中的一项：

Endpointing. The system waits until it is confident the caller is done. A deliberate wait, and frequently the largest single item in the budget.
端点检测（Endpointing）。 系统等待直到确信通话者已说完。这是一种刻意的等待，通常也是延迟预算中占比最大的一项。
Transcription. Speech becomes text. Usually streaming, so most of it overlaps with the caller still speaking.
语音转文字（Transcription）。 语音变为文本。通常是流式的，因此大部分时间与通话者说话的过程重叠。
Reasoning. The model reads state and decides what to say, and whether to call a tool.
推理（Reasoning）。 模型读取状态并决定说什么，以及是否调用工具。
Tool calls. Anything the agent must ask an external system before it can answer.
工具调用（Tool calls）。 代理在回答前必须向外部系统查询的任何内容。
Speech generation. The reply becomes audio, and audio takes real time to play no matter how fast it was generated.
语音生成（Speech generation）。 回复变为音频，无论生成速度多快，音频播放都需要真实时间。
Network and telephony. Delay you do not control.
网络与电话线路（Network and telephony）。 你无法控制的延迟。

Two things fall out of writing the chain down. Shaving 200ms off step 3 while step 1 sits at two seconds is the classic optimizing-the-wrong-thing move. And steps 4 and 5 are where badly behaved agents actually lose people. 写下这个链路后，你会发现两件事。在步骤 1 耗时两秒的情况下，将步骤 3 缩短 200 毫秒是典型的“优化方向错误”。而步骤 4 和 5 往往是表现不佳的代理导致用户流失的关键。

Endpointing is the config you set once and never looked at again

端点检测是那种“设置一次就再也不管”的配置

Endpointing is the agent deciding you finished your sentence. Too short and it interrupts people mid thought, the rudest failure mode a phone system has. Too long and every turn carries a dead pause, so the agent feels slow even when everything downstream is fast. 端点检测是代理判断你是否说完话的机制。设置太短，它会在人思考时打断对方，这是电话系统中最粗鲁的故障模式。设置太长，每一轮对话都会带有死寂的停顿，导致即使下游处理极快，代理依然显得迟缓。

The mistake is treating it as one global constant. Different moments deserve different patience: 错误的做法是将其视为一个全局常量。不同的时刻需要不同的耐心：

Short answers get short waits. “Yes.” A confirmation. A one word choice. Lingering here is what makes an agent feel sluggish across an entire call.
简短回答需要短等待。 “是的。”确认信息。单字选择。在这里拖延会让代理在整个通话过程中显得迟钝。
Open questions get long waits. Ask someone why they are calling and they think out loud, pause, resume. Cut them off and you lose the actual reason for the call.
开放性问题需要长等待。 询问对方来电目的时，他们会边想边说、停顿、继续。如果此时打断，你就会错过通话的真正原因。
Anything recited digit by digit gets the most patience. People say phone numbers in chunks with real gaps. An agent that jumps in after the first chunk gets the number wrong, and a wrong callback number is a lost lead no matter how snappy the call felt.
逐位朗读的内容需要最大的耐心。 人们说电话号码时会分段并留有真实间隙。如果代理在第一段后就插话，会导致号码错误，而错误的号码意味着线索丢失，无论通话过程看起来多么敏捷。

Same shape as the rest of agent design: loose where the caller is exploring, tight where the caller is confirming. 这与代理设计的其他部分逻辑一致：在通话者探索时放宽限制，在通话者确认时收紧限制。

Rule one: nothing on the critical path the next sentence does not need

规则一：不要在关键路径上放置下一句对话不需要的东西

This one rule fixes most “slow agent” complaints I get handed. A voice agent wired into a real business touches a CRM, a calendar, a messaging platform, an automation layer. The instinct is to do all of it inline, mid conversation, while the caller listens to nothing. That is how you get four seconds of silence after someone says “yes, book me in”. 这一条规则解决了大部分我接手的“代理太慢”的投诉。连接到真实业务的语音代理会触及 CRM、日历、消息平台和自动化层。人的直觉是把所有事情都在对话中同步完成，让通话者在沉默中等待。这就是为什么当有人说“好的，帮我预约”后，会出现四秒钟的死寂。

Split the work by one question: does the caller’s next sentence depend on it? 通过一个问题来拆分工作：通话者的下一句话是否依赖于此？

Blocking work is anything the agent must know before it can speak truthfully. Real availability before it offers a slot. Whether an account exists before it claims one does. Keep this list short and defend its shortness.
阻塞性工作是指代理在诚实回答前必须知道的事情。比如在提供预约时段前必须确认真实空档，或者在声称账户存在前必须先核实。保持这个列表尽可能短，并捍卫它的简洁。
Non-blocking work is everything else. Creating the contact, tagging the lead, firing the confirmation SMS, writing the summary, kicking off follow up. None of it belongs in the caller’s waiting time.
非阻塞性工作是其他所有事情。创建联系人、标记线索、发送确认短信、撰写摘要、启动后续跟进。这些都不应该占用通话者的等待时间。

In practice the agent fires a webhook at the automation layer and keeps talking. This is the job queue pattern every backend engineer already knows, applied where latency is measured in human patience. 在实践中，代理应向自动化层触发一个 Webhook，然后继续对话。这是每个后端工程师都熟悉的“任务队列模式”，只不过应用在了以人类耐心为衡量标准的场景中。

The one wait worth taking, and how to cover it

唯一值得等待的情况，以及如何掩盖它

Calendar availability is the honest exception. If the agent offers a slot it has not verified, you traded a two second pause for a double booking, which is far more expensive to explain to a client. So take the wait, but never take it in silence. 日历空档查询是一个合理的例外。如果代理在未核实的情况下提供时段，你用两秒的停顿换取了重复预约，这向客户解释起来代价要大得多。所以，该等就等，但绝不要在沉默中等待。

Speak first, then look up. “Let me check what we have this week” costs about a second and a half of speech, roughly what the lookup needs, and the caller experiences zero dead air because a voice was talking the whole time. That is not a trick. It is what a receptionist does while their booking screen loads, and it is the same instinct as an optimistic UI update. 先说话，再查询。“让我看看这周有哪些空档”大约需要一秒半的语音时间，这正好是查询所需的时间。通话者不会感受到任何死寂，因为语音一直在播放。这不是技巧，这是接待员在加载预约界面时会做的事，这与“乐观 UI 更新”的直觉是一样的。

The rule generalizes: if you cannot remove a wait, cover it with speech that was going to happen anyway. What you must not do is fill the gap with filler promising progress the system is not making. Callers notice an agent that says “one moment” three times in a row. 这个规则可以推广：如果你无法消除等待，就用本该发生的对话来掩盖它。你绝对不能做的是用虚假的“正在处理”填充空白。通话者会注意到一个连续说了三次“请稍等”的代理。

Rule two: the agent’s own sentences are part of the budget

规则二：代理自己的句子也是延迟预算的一部分

Generated speech plays at human speed. A four sentence answer takes four sentences worth of seconds no matter how fast the model produced it. If your agent opens with a paragraph, you spent more of the caller’s patience on your greeting than on any technical delay in the stack. Cap reply length in the prompt and mean it. 生成的语音以人类语速播放。无论模型生成得有多快，四句长的回答就需要四句的时间。如果你的代理开场就是一大段话，那么你在问候语上消耗的通话者耐心，远比技术栈中的任何延迟都要多。在提示词中限制回复长度，并严格执行。