AI voice agents are software that answers or places phone calls and holds the conversation itself. Most of them run a loop that turns the caller's speech into text and hands it to a large language model (LLM), the kind of model behind ChatGPT. The reply is then converted back into speech.
Callers judge that loop on timing before content. People reply to each other within a few hundred milliseconds, so an agent that takes two seconds sounds broken however good its answer is. Our list of agents every business should build in 2026 puts the support voice agent first.
What an AI voice agent does on a single call
An AI voice agent is a chain of models passing the conversation along. The classic design, called a cascaded pipeline, has three stages:
- Speech-to-text (STT), also called automatic speech recognition, transcribes the caller's audio as it arrives.
- The LLM reads the transcript and the conversation so far, then writes a reply or calls a tool, such as an order lookup.
- Text-to-speech (TTS) turns the reply into audio and streams it back to the caller.
Each stage is typically a call to a hosted model, so every turn stacks three network round trips, each following the pattern in our guide to how an AI API works.
The cascade is popular because each piece is swappable, so you can change the voice without touching transcription.
The price is that information leaks at every handoff. The authors of the Moshi paper point out that once speech becomes text, cues such as emotion are lost, and the pipeline can only work in tidy speaker turns. It is a game of telephone, played by very fast machines, over an actual telephone.
The alternative is a speech-to-speech model, which goes from audio to audio with no text in the middle. Moshi, a research model, listens while it talks and reports a practical latency of 200 milliseconds. These systems sound more natural but are harder to inspect stage by stage.
Principles of building AI agents that hold a conversation
The principles of building AI agents carry over to voice, from clear instructions to well-designed tools. Voice adds a clock. A text agent can think for five seconds and nobody minds, while a voice agent that does the same sounds like it has hung up.
Treat latency as a budget
The Moshi paper cites an average gap of 230 milliseconds between speakers in natural conversation, measured across ten languages. It puts a typical cascaded pipeline at several seconds.
Every stage spends part of that budget, and streaming is how you fit. Transcription emits partial text while the caller is still talking. The LLM starts generating the moment the turn ends, and TTS starts speaking from the first sentence of the reply while the rest is still being written.
A caller will forgive a robotic voice long before they forgive silence. Two seconds of dead air and people start saying "hello?" to their phone, checking whether it has quietly died in their hand.
Tool calls are where budgets go to die, so have the agent say "let me check that for you" before a slow lookup. It buys time for a machine the same way it does for a person.
Decide when the caller has finished
Endpointing means deciding when the caller has finished. Most systems use voice activity detection (VAD), a small model that separates speech from silence, and treat a long enough pause as the end of the turn.
A fixed silence threshold fails in both directions. Set it short and the agent cuts in while someone pauses to find their card. Set it long and every reply feels sluggish. Better systems weigh the pause against what was said, so "my account number is" followed by silence reads as unfinished.
Stop talking the moment you are interrupted
Barge-in means stopping the moment the caller starts talking, and it comes up constantly: overlapping speech fills 10 to 20% of spoken time, according to research the Moshi paper cites. The agent has to cut its audio and discard the rest of its queued reply.
It also has to remember only what the caller heard. Store the full planned reply in the conversation history and the model will later assume the caller knows a refund policy they talked over in the first sentence.
Read back anything that matters
Transcription is weakest on names and numbers, and "fifteen" and "fifty" sound alike on a bad line. Have the agent read back any value it will act on before it books or charges anything.
AI agent frameworks built for real-time voice
Most general AI agent frameworks assume text arrives as a complete message. Voice needs one built around continuous audio, and open source options such as Pipecat and LiveKit Agents exist for this job.
The Pipecat pipeline model is worth learning even if you never use the library. A pipeline is a list of processors connected in sequence, passing small units of data called frames. A typical voice agent chains them like this:
- Transport input (the caller's audio)
- Speech-to-text
- User context aggregator (what the caller said)
- The LLM
- Text-to-speech
- Transport output (audio back to the caller)
- Assistant context aggregator (what the agent said)
Two details show why a dedicated framework earns its place. Interruptions travel as high-priority system frames, so a barge-in jumps the queue ahead of audio still waiting to play. The assistant context aggregator sits after the transport output, which is the natural place to record what the caller actually heard.
The framework also owns the transport, the connection carrying the audio, whether that is WebRTC for an app or a phone line through a telephony provider. Nobody has ever demoed packet buffering to a standing ovation, yet it decides whether the call sounds clean.
Where an AI call center voice agent earns its keep
An AI call center voice agent works best on frequent, predictable calls. Good AI agent examples in a contact center share a shape: the caller wants one specific thing, and the answer already lives in a system the agent can query.
| Call type | What the agent does | Hand off to a human when |
|---|---|---|
| Order status | Looks up the order and reads out the delivery date | The order is lost or arrived damaged |
| Appointment changes | Checks the calendar and moves the booking | The caller needs a slot that does not exist |
| After-hours triage | Takes details and opens a ticket for the morning | The issue sounds urgent or safety-related |
The handoff to a human decides how callers judge the whole system, so pass the transcript and transfer reason along with the call. Making a caller repeat their account number to a person after reading it to a machine is the fastest way to turn an automation win into a story they tell at dinner.
Track containment, the share of calls resolved without a human, next to transfer reasons. If containment and complaints rise together, the agent is trapping people.
If you are weighing build against buy, AI voice agent services for businesses vary widely. Our guide to what a good agent development engagement delivers covers what to ask, and the best answer to any of it is a recording of a real call that went wrong.
Can AI agents make outbound calls legally?
They can, and in the US the rules for AI-voiced calls apply in full. This is a summary for engineers, so confirm your own situation with counsel before any outbound campaign.
In February 2024 the Federal Communications Commission issued a declaratory ruling confirming that current AI voice technology counts as an "artificial or prerecorded voice" under the Telephone Consumer Protection Act (TCPA). The consequences are concrete:
- A call placed with an AI voice needs the prior express consent of the person called, unless it serves an emergency purpose or an exemption applies.
- A call that advertises or telemarkets needs prior express written consent.
- An artificial voice message must state clearly, at the start, the identity of the business responsible for the call.
- No exception exists for systems that behave like a live agent. The ruling says the TCPA "does not allow for any carve out" for such technology.
That last point closes the loophole many teams hope for: a natural-sounding agent is still an artificial voice. Inbound calls, where the customer dials you, are a different situation. Outside the US, consent rules differ by country.
A checklist for your first AI voice agent
Start with one high-volume inbound call type and record every call from day one. Then:
- Stream every stage, and measure the time from the caller's last word to the agent's first sound.
- Tune endpointing on real recordings, including callers who pause mid-sentence.
- Support barge-in, and store only what the caller heard.
- Read back every value the agent will act on.
- Design the human handoff before the happy path, and send the transcript with it.
- Get legal sign-off before the agent places a single outbound call.
Underneath the audio, a voice agent is an agent like any other, with a reasoning loop and a set of tools. Agentic AI Engineering covers designing system prompts and tools for that loop, and building agents in Python for real deployments.
Get the timing right and callers forget they are talking to a pipeline. Get it wrong and they spend the whole call saying "hello?" to it.