Voice AI latency, explained — and how we measure it honestly

If you've seen a voice AI demo that felt natural and one that felt awkward, latency is probably the main reason. A fast response feels like a conversation. A slow one feels like a telephone tree.

Let's break down where the delay comes from, what we've done to reduce it, and — importantly — how we measure and report it honestly.

The four segments of a voice turn

In a typical voice AI system like Televox, every time you say something and the agent responds, the response time is the sum of four sequential steps:

1. End-of-utterance detection (EOU)strong: The system has to decide you've finished speaking. This isn't as simple as detecting silence — a natural pause mid-sentence looks similar to the end of a sentence. Televox uses a combination of voice activity detection (Silero VAD) and a semantic turn-detector model that evaluates whether the utterance is likely complete.

2. Speech-to-text (STT)strong: Once the system detects the end of your utterance, it needs to transcribe what you said. We use Deepgram with streaming, which means transcription happens in parallel with speech — the STT cost is largely hidden in the EOU detection window and is not a significant sequential delay.

3. LLM response (TTFT)strong: The transcript is sent to the language model, which generates a response. The key metric here is TTFT (time to first token) — how long before the first word of the response arrives. This is often the longest segment.

4. Text-to-speech (TTS)strong: The LLM output is synthesized into audio. We use Deepgram Aura, which is optimized for low latency — first audio byte typically arrives quickly.

Total response latency is roughly: EOU detection + TTFT + TTS first audio byte.

What we measured (a real call, June 2026)

We flagged an inbound turn with a notably slow response time and broke it down using our per-turn latency inspector:

EOU (endpointing wait): 1.02sstrong — the system waited over a second to confirm the caller had finished speaking.

STT: ~0msstrong — streaming, no sequential cost.

LLM TTFT: 2.07sstrong — time to first token. This was the bottleneck.

TTS (first audio byte): 0.27sstrong

Total: 3.37s for that turn. That's too slow for a natural conversation — it's in the range where callers start to wonder if the connection dropped.

What we changed

We identified two main levers:

Model upgrade: gpt-4.1-ministrong. We switched from gpt-4o-mini to gpt-4.1-mini. OpenAI's newer model has comparable or better intelligence for customer-service and booking tasks, with meaningfully lower TTFT in our testing. It also has better reliability on tool-calling, which reduces the rate of booking errors.

Prompt caching reorderstrong. OpenAI automatically caches any stable prompt prefix longer than 1,024 tokens (reducing TTFT by up to ~80%, at no cost). However, caching only activates if the static content — system instructions, knowledge base — comes first, and dynamic content — per-call context — comes last. Our original prompts had the knowledge base appended after dynamic content, defeating caching. After reordering, warm-cache turns are significantly faster.

Important caveat: prompt caching only helps on warm cache hits. The first turn of the first call in an idle period will be cold. We measure and report latency at the p50 across all turns, not just ideal conditions.

How we instrument this

Every turn in every call is instrumented with four timing fields: EOU wait, STT, LLM TTFT, and TTS first audio byte. This data is visible in our observability dashboard — operators can inspect any call, any turn, and see exactly where the time went.

We report the p50 (median) latency across recent turns. The p50 reflects the typical caller experience. We also track p95 to catch tail latency — the turns that feel slow even if the median is good.

What "sub-second" means and doesn't mean

You'll often see AI voice products advertise "sub-second latency." This is worth examining:

A sub-second response time is achievable for warm turns with short prompts and simple responsesem. It is not guaranteed for every turn in every call.

Longer prompts, cold model cache, complex LLM reasoning, and network conditions between the worker and the AI providers all affect latency. The honest framing is: median response time, for typical turns, under normal conditions.

We aim for sub-second median latency on warm turns. We publish our methodology so you can hold us to it.