Why Is My Voice AI Agent Slow? 8 Sources of Latency (2026)

Voice agent delay comes from eight places: the endpointing silence threshold, a deliberate response wait, prompt prefill, model choice, tool calls, transcription mode, TTS buffering, and the network or telephony leg. Retell sets 1.5 seconds as the estimated total latency to stay under.

Measure one thing before you change anything: the time from the end of the customer's speech to the first byte of audio they hear back. Every other number is a decomposition of that one. An operator in a September 2026 r/AI_Agents thread on where voice latency hides after you have already turned streaming on put it plainly: if you are not logging that single timestamp pair per turn, you will keep chasing the wrong component. Retell publishes 1.5 seconds as the estimated total latency to stay under, which is a reasonable target to hold yourself to.

Where does voice agent latency come from?

1. Endpointing: the silence threshold you are paying for on every turn

The agent does not start thinking when you stop talking. It starts thinking when the system decides you have stopped talking, and that decision waits for a configured amount of silence. The best summary of this came from an operator in the r/AI_Agents thread: if your silence threshold is 700ms, then 700ms of your latency is a config value rather than compute. People spend a week on token throughput while the largest single chunk is a constant they could change in one line.

Every platform exposes it differently. Vapi's stop-speaking plan defaults voiceSeconds to 0.2 seconds and backoffSeconds to 1 second, and its smart endpointing plan can be handed to Krisp, Deepgram Flux, Assembly, LiveKit or Vapi's own text-based detector, with the Krisp confidence threshold defaulting to 0.5. LiveKit offers five strategies: its turn detector model, a realtime model's built-in detection, VAD alone for minimum latency, STT endpointing, or manual control. Its own telephony example for snappier turn closing on noisy phone audio uses 300ms of prefix padding and 400ms of silence.

The tradeoff is real and you should make it deliberately. A shorter threshold makes the agent quicker and makes it interrupt people who pause mid-sentence. Longer is politer and slower. Transcription-based endpointing splits the difference by using punctuation, which is why Vapi exposes separate onPunctuationSeconds and onNoPunctuationSeconds values.

2. The deliberate delay somebody configured and forgot

Several platforms ship a wait before the agent speaks, and it is easy to set generously during testing and never revisit. Vapi's start-speaking plan defaults waitSeconds to 0.4 seconds. Retell exposes a Response Wait time with a maximum of 5.5 seconds, and its own troubleshooting page tells you to adjust it downward when latency is high.

Email agents have the same setting, but on voice it is more visible: a team running a five-minute reply delay is making a workflow choice, while a team running a two-second response wait is usually making an accident.

The check. Open the agent's speech or start-speaking settings and write down every configured delay before you look at a single model metric. Add them up. That sum is your floor.

3. Prefill: the prompt that gets slower every turn

Time to first token is not a constant. It scales with how many tokens you send, and a voice conversation sends the whole history back on every turn. Turn 1 is fast. Turn 12 is not, and the operator who described this in the r/AI_Agents thread named it exactly: prefill on a growing history.

The system prompt is the half you control. A developer on r/AI_Agents in May 2026 reported 3 to 7 second pauses from a Twilio plus ElevenLabs plus Railway stack, and the top diagnosis in the thread was his own prompt: the model has to process all of it before generating the first token, so static reference data belonging in a lookup should not be sitting in the system prompt. Retell puts a number on the same advice, warning that prompts beyond 8,000 tokens become noticeably slower and telling you to move detail into the knowledge base or tool descriptions.

The fix. Move catalogs, price lists, staff rosters and opening-hours tables out of the prompt and into tools or a knowledge source. Then cap or summarize history rather than resending it. Context window size for AI support agents covers the same tradeoff on the text side.

4. Model choice, and where the model is running

Once the prompt is lean, the model itself is the next honest lever, and it is usually a bigger one than people expect. An operator in the September thread reported moving an agent loop off one hosted provider onto another after comparing decode benchmarks, and going from roughly 8 seconds end to end to about 1.8. Treat that as one team's result on one stack rather than as a benchmark, and verify with your own trace before switching anything.

Both Retell and ElevenLabs give the same first advice here: pick a smaller or faster model. It is also the change most likely to cost you accuracy, which is why the developer with the 3-to-7-second stack rejected a faster model as not capable enough for his logic. The honest sequence is prompt first, model second.

5. Tool calls, sitting in the critical path

The moment your agent looks something up mid-conversation, your own API's p95 becomes part of the caller's experience. Two things make this worse than teams expect. The orchestration around the call often costs more than the call: one operator noted the function itself being fast while the orchestration layer added far more than expected. And a slow tool call is indistinguishable, to the listener, from a slow model.

The fix. Give every tool a hard timeout shorter than the silence you are willing to leave, and design what happens when it expires. Fire lookups in parallel with the agent's acknowledgment rather than after it. And keep the tool descriptions short, because they are part of the prompt from cause 3.

6. Transcription: speed mode against accuracy mode

Speech recognition has a dial with a published cost. Retell documents that its accuracy-optimized transcription mode adds about 200 milliseconds over the speed-optimized one, and that boosted keywords add more. That is a real 200ms you are choosing to spend, and it is worth spending on a medical intake line and not on a booking confirmation.

7. Text to speech, and the buffer after it

TTS inference is rarely the problem on its own. ElevenLabs publishes about 75 milliseconds of model inference for typical short inputs on its Flash model, with the caveat that this is a benchmark under representative conditions rather than a guarantee, and that it rises for longer inputs and under load. Two things around it cost more than the inference does.

The first is clause buffering. An operator in the September thread described the failure precisely: the TTS sitting on its hands for 400 milliseconds waiting for a sentence end. The second is the player. ElevenLabs notes that a 500-millisecond audio buffer is common, which means half a second can be spent after the audio is generated and before the caller hears it.

The fix. Stream at clause boundaries rather than sentence boundaries, and consider splitting the reply into independently cancellable units so a short preamble can start playing while the rest is still generating. That pattern only works if every unit carries a turn ID and late audio from a superseded turn is discarded rather than played, which is the discipline one operator in that thread spelled out in full.

8. Network, telephony and the hour of the day

This is the cause that makes the dashboard disagree with the phone. A Vapi user on r/vapiai reported a dashboard response time around 600 milliseconds against a felt delay of 2 to 3 seconds on an actual call. The gap is the carrier leg, the jitter buffer and the routing, none of which the platform's own timer sees.

ElevenLabs gives the baseline: network round trip is typically 20 to 200 milliseconds depending on geographic proximity, with server processing in single-digit milliseconds. Retell tells you to keep round-trip ping under 300 milliseconds and treats consistent times above that as a cause of latency spikes, and it recommends using phone numbers in the same region as your callers.

Then there is load, which is the part no configuration fixes. A March 2026 r/ElevenLabs thread from a team on the $99 Pro plan reports latency that is fine in the European morning and bad in the afternoon, with several other European users in the same thread confirming it, one naming 2pm UTC. Two of them moved to other providers. Retell's answer to the same class of problem is a paid fast tier that routes onto dedicated infrastructure, which is worth reading as what it is: capacity you can buy back.

Keep p50 and p95 apart while you do this. A clean demo hides a bad tail, and the tail is what your callers get once concurrency rises.

Why does cutting latency make the agent interrupt people?

Every millisecond you cut from endpointing makes the agent more likely to cut people off, so the two problems are one problem. A February 2026 r/speechtech thread from a team running voice agents on real phone calls lays out what they tried and what each approach cost:

• A simple VAD threshold. Too sensitive and the agent stops every time someone breathes; too insensitive and it talks over people. They describe tuning it endlessly with no perfect setting.
• Energy-based filtering. Fine for background noise, bad for soft-spoken callers and quiet acknowledgments.
• A semantic interrupt classifier on the partial transcript, separating "wait, actually" from "mm-hmm". They call this the best approach, and report that it adds latency and still ran around a 15% error rate in their testing.
• Platform-level handling. Decent on ElevenLabs in their experience, and not configurable enough to keep talking through a backchannel in one case and stop immediately in another.

Vapi's equivalent dial is numWords in the stop-speaking plan: 0 reacts immediately, and 2 or 3 makes the agent ignore brief acknowledgments. That single setting is the cheapest fix for an agent that stops every time somebody says "okay".

The same thread raises the mirror problem, which has no good answer yet: when a caller goes quiet for five seconds, the agent cannot tell thinking from confused from gone, and a prompt that fires too early reads as pushy.

How do you find your own latency source in an afternoon?

1. Instrument the turn. Timestamp last detected speech, final transcript, first model token, last model token, tool round trip, first TTS byte and first audio heard. One operator recommends running a single canned turn 20 times and reading the distribution rather than a single call.
2. Add up the configured constants first. Endpointing silence, response wait, player buffer. Anything left over is compute.
3. Attribute the remainder. If end-of-speech to final transcript dominates, tune endpointing and transcription mode. If first-byte-out to first-audio-heard dominates, it is buffering or the carrier leg.
4. Re-run at concurrency. Then compare p95 against p50, and compare the same test at your busiest hour.

Without step 1 you are guessing, and every thread in this cluster contains at least one person who guessed the model and was wrong.

Who pays for a slow voice agent?

Worth naming the incentive: most voice platforms bill per minute, so a slow turn is revenue. Nobody is going to send you an alert saying your endpointing threshold is 200 milliseconds longer than it needs to be. The two platforms above that publish latency targets and troubleshooting pages deserve credit for it, and they are still selling you the minutes that the delay fills.

For support teams specifically, the decision that matters more than any of these eight is what the voice agent is allowed to do. An agent that only reads a knowledge base will be fast and will not resolve much; an agent that looks up an order is slower by exactly the length of your API call and is worth it. Macha fits teams whose support tickets already live in Zendesk, Freshdesk, Gorgias, Front, HubSpot or Intercom and who want the same agent and the same tools reachable by voice, through the ElevenLabs integration; billing is per ticket, from $299 a month for 750 tickets, with setup and monitoring by the Macha team included. It is the wrong choice if what you are building is an outbound calling operation, where a dedicated voice platform is the right tool. Retell and Vapi are the two we have written up in full, and Gorgias Voice covers the help-desk-native option.

How we researched this

We checked every figure attributed to a vendor against that vendor's own current documentation on 2026-09-21 (rechecking Retell's and Vapi's figures on 2026-09-24), and we used a headless browser for the pages that refuse scripted requests: Retell's high-latency troubleshooting page, ElevenLabs' latency concepts page, Vapi's speech configuration reference and LiveKit's turn detection guide. The lived numbers come from six community threads we read in full the same day. We have not run our own benchmark across these platforms for this post, so no number here is ours, and the ordering of the eight causes reflects how often each is named across those threads rather than a measured distribution.

Frequently asked questions

What is a good target for voice agent latency? Retell publishes 1.5 seconds as the estimated total to stay under, measured from the end of the caller's speech to the agent's response. Below about one second a conversation feels normal; above two seconds callers start talking over the agent because they assume it did not hear them.

Why does my dashboard show 600ms when the call feels like three seconds? The platform's timer usually starts after audio has reached it and stops when it emits audio, which leaves out the carrier leg, the jitter buffer and the player's own buffer. One r/vapiai user reported exactly that gap. Measure from the caller's side, or accept that the dashboard number is a component and not the experience.

Does a faster model fix it? Sometimes, and it is rarely the first thing to try. Both Retell and ElevenLabs recommend a smaller model as a high-impact change, and both also tell you to shorten the prompt first, because time to first token scales with the tokens you send. Prompt, then history, then model.

Why is my agent slower in the afternoon? Shared capacity. European users on r/ElevenLabs report consistently good morning latency and bad afternoon latency on the same configuration, which points at load rather than settings. Check the provider's status page, test the same canned turn at two times of day, and look at whether a dedicated or fast tier exists.

How do I stop the agent interrupting people? Lengthen the silence threshold, and separately raise the number of words required to count as an interruption. On Vapi that is numWords in the stop-speaking plan; 2 or 3 makes it ignore brief acknowledgments. Expect to trade some responsiveness for it, because these are the same dial.

Should I use VAD alone or a turn detector model? VAD alone is the lowest-latency option and the least accurate on noisy phone audio. LiveKit's own recommendation is its turn detector model, which uses the meaning of the speech as well as its acoustic properties on top of VAD. On a quiet web widget VAD alone is often fine; on telephony it usually is not.

Do tool calls really matter that much? Yes, because they sit in the critical path and because their descriptions inflate the prompt. Give every tool a timeout shorter than the silence you are willing to leave, fire it in parallel with an acknowledgment, and log the tool round trip as its own number so it does not get blamed on the model.

Can I fix this by moving to a different platform? Only if the cause is capacity or routing. Endpointing, prompt size, tool latency and buffering travel with you, and teams who switch without instrumenting first usually reproduce the same delay on the new stack.

Sources:

• Retell AI, Troubleshoot high latency
• Retell AI, Check actual latency
• ElevenLabs, Understanding latency
• Vapi, Speech configuration
• LiveKit, Turn detection and interruptions
• r/AI_Agents, Where does voice-agent latency still hide after you fix streaming?
• r/AI_Agents, My AI receptionist has 3–7s latency… how do I fix this?
• r/vapiai, Anyone else getting 2 to 3 second delays with voice agents?
• r/ElevenLabs, ElevenLabs latency is terrible during afternoon
• r/ElevenLabs, ElevenLabs Voice Agent Too Slow for Cold Calls
• r/speechtech, Handling interruptions in voice AI is an unsolved problem