The latency budget of a natural voice agent

Human conversational turn-taking has a gap of roughly 200 milliseconds. Voice agents do not hit that today, but there is a threshold where an interaction stops feeling mechanical, and in our testing it sits somewhere under a second.

The budget breaks into four parts: endpointing (deciding the caller has finished speaking), transcription, model reasoning, and speech synthesis. Endpointing is the most commonly mishandled. Aggressive settings interrupt the caller; conservative settings add dead air to every single turn.

Streaming everything is non-negotiable. Transcribe while the caller speaks, begin generating on partial transcripts where it is safe to do so, and start synthesising the first sentence before the full response exists.

Design for interruption. A caller who starts speaking mid-answer must be able to cut the agent off cleanly, and the agent must discard the rest of its planned response rather than resuming it seconds later.

Keep a filler strategy. A short, natural acknowledgement while a slow tool call runs is better than silence, which callers reliably interpret as a dropped line.