A caller does not experience a pipeline. They experience the gap between the end of their sentence and the first sound of the answer. That gap is the product. Everything else is instrumentation.
In a natural conversation, the pause between turns is often under half a second. A voice agent that takes two seconds feels like a system reading a file. It may be accurate. It is not a conversation. The 480-millisecond turn is our working budget for a conversational response on a Singapore edge, and the discipline is less about hitting a number than about knowing where every millisecond went.
The number you feel
There are two tempting ways to measure a voice agent. The first is to time the model from a text prompt. The second is to quote a vendor's "sub-second" number without saying where it starts and where it ends. Both flatter the system and mislead the buyer.
We measure from the end of the caller's speech to the first audible frame of the agent's response. That includes endpointing, the final ASR hypothesis, the agent's first token, the TTS first audio and playout. It excludes the time the caller is still talking, because that is not latency; that is conversation.
The budget
The budget is written down before a deployment and measured per stage. It is not an average of the happy path. It is a set of allowances that add up to a turn a person will tolerate.
| Stage | Budget | What the time buys |
|---|---|---|
| SIP edge | 20 ms | Termination, jitter buffer, SRTP |
| Endpointing | 140 ms | Deciding the caller has finished, not just paused |
| Final ASR | 60 ms | The final hypothesis after streaming recognition |
| Agent first token | 110 ms | Policy, dialogue state and tool dispatch |
| TTS first audio | 100 ms | Speech begins while the sentence is still being generated |
| Playout | 50 ms | Frame pacing and concealment |
| Turn | 480 ms | Speech end to first audio |
The sum is deliberately tight. A tool call that blocks the response can add 250 ms or more. We do not hide that inside the turn budget; we design the conversation so the tool runs in parallel with a holding phrase, or we accept a longer turn and say why.
Where time hides
Endpointing is the largest variable. A fixed silence timer is simple and wrong. It cuts off a slow speaker, misses a code-switch and turns a thoughtful pause into an interruption. We combine acoustic evidence with semantic context, and we let the caller resume. The extra 40 ms of patience is usually worth more than the 40 ms it costs.
Tools are the second variable. A CRM lookup, a payment authorisation or a dispatch query can take 300–900 ms. A voice agent that waits in silence sounds broken. We stream a holding phrase, cap the call at 800 ms and design a fallback that keeps the conversation alive.
TTS is the third. Generating the whole sentence before speaking is a waste. Streaming the first clause gets audio out early; the rest of the sentence catches up. The hard part is barge-in: when the caller interrupts, the queued audio has to stop inside a frame, not after the sentence.
The line itself is the fourth. A resold API adds a hop you did not choose. A codec renegotiation can add a round trip. A jitter buffer that is too small drops packets; one that is too large adds delay. These are not model problems, which is why a model-only benchmark cannot predict them.
A conversation is a deadline. The average is a story; the tail is the experience.
p50 is not enough
A median turn tells you what a typical call feels like. It does not tell you what happens on the call that matters. We report p50 and p95, and we look at the distribution by stage, language and intent. A 480 ms median with a 1.4-second p95 is not a 480 ms system. It is a system that fails some callers badly and hides them in the average.
Our reference p95 is 610 ms. That is still a budget, not a boast. It is measured on the same path as the p50, with the same edge, the same models and the same playout. If the p95 grows, we treat it as a regression even when the median is flat.
What we measure
Every call gets a call ID and a trace. We timestamp the edge, the endpoint decision, the ASR final, the first agent token, the first TTS frame and the first playout frame. The trace is per call, not a dashboard average. The numbers are joined to the outcome so we can see whether a slower turn actually changed the resolution.
We test with consented recordings and a synthetic regression suite, and we replay the same calls after a model, prompt or carrier change. A change that improves the demo and worsens the p95 is a regression. The test is not whether the agent can answer; it is whether it can answer in time, on the line, for the person who is actually calling.
What owning changes
Owning the stack does not automatically make a system faster. It makes the trade-offs visible. We can choose a smaller model for a short turn, stream the first clause, cache a prompt, move a tool call off the critical path, or add an edge in a region. We can also see when a carrier is adding 80 ms and decide whether to route around it.
What we refuse to do is measure from a text box, hide tool latency behind a progress animation, or quote a single number without its percentile and its path. Those are not optimisations. They are ways of not knowing.
The deadline
Voice is unforgiving because the caller is present. They hear the pause. They hear the breath. They decide whether the system is worth talking to before the second sentence.
The reference turn budget is how we keep that decision on our side. It is not a marketing number. It is a set of promises we make to the line, one stage at a time.
Reference numbers describe the current Singapore edge and are agreed per deployment. They are not an SLA or a guarantee.