The AI voice agent guide: what happens in that half second.

The pipeline, end to end

A voice agent is a loop that runs continuously while the call is connected. Each pass through the loop has to complete faster than a human would notice.

1. Media streaming

   The call arrives over telephony infrastructure — Twilio, for example — and the audio is streamed to the agent as it happens rather than recorded and processed afterwards. Streaming is what makes interruption possible at all.

2. Speech-to-text

   A streaming recognition model transcribes the audio incrementally, producing partial results that firm up as more audio arrives. Accent, line quality and background noise all bite here.

3. Endpointing

   The system decides when the caller has finished speaking rather than merely paused for breath. Cut in too early and you interrupt; wait too long and the call feels sluggish. This is the least glamorous and most important component.

4. Retrieval

   The question is used to search your business knowledge, and the most relevant passages are selected as the factual basis for the reply.

5. Generation

   A language model writes the reply, constrained to the retrieved facts and to the behaviour rules you configured — tone, length, and what to do when unsure.

6. Text-to-speech

   The reply is synthesised as audio and streamed back, usually starting playback before the whole sentence has been generated so the caller hears a response sooner.

The latency budget

Human conversation tolerates roughly a couple of hundred milliseconds of silence between turns before it starts to feel wrong. A voice agent has to fit transcription, retrieval, generation and speech synthesis into something close to that.

This is why architecture matters more than model choice. Systems that wait for a complete transcript, then a complete answer, then a complete audio file, feel like walkie-talkies. Systems that stream at every stage — partial transcripts, token-by-token generation, audio that begins playing mid-sentence — feel like a conversation. When a vendor demos, listen for the gap after you stop talking. It tells you more than any spec sheet.

Turn-taking, interruptions and the barge-in problem

Real callers interrupt. They say “sorry, actually—” halfway through your agent’s sentence, and a good agent stops talking immediately, discards what it was about to say, and listens.

This is called barge-in, and handling it well requires the agent to keep listening while it speaks, distinguish the caller’s voice from its own audio, and abandon in-flight speech cleanly. Poor handling produces the two classic failure modes: an agent that talks over the caller for three more sentences, or one that stops dead every time a door closes in the background.

• Test it by interrupting mid-answer. Does it stop, and does it understand what you said instead?
• Test it with background noise — a television, a busy street, a second person talking.
• Test long pauses. Does it wait while someone finds their diary, or does it fill the silence?

Where the answers come from

A voice agent with no knowledge base is a polite way of taking messages. The value appears when it can answer the questions your team answers all day, which means the retrieval layer has to be fed properly.

In practice that means uploading the documents you already have — price lists, service descriptions, policies, menus — and importing your website. Good systems then let you see which source produced an answer, so when something comes out wrong you fix the document rather than argue with a model.

The rule to insist on is simple: no answer without a source, and an honest fallback when there is no source. On a phone call there is no page to double-check, so an invented answer travels further than it would in chat.

Telephony: what actually changes on your line

The most common setup is call forwarding. Your existing business number stays with your existing provider, and you forward it — always, or only when unanswered, or only outside hours — to a number that reaches the agent. Nothing about your contract or your handsets changes, and you can undo it in a minute.

The alternative is a new dedicated number, which is useful for a specific campaign, a second location or a department. Both routes support transferring a live call to a human, provided the transfer targets are configured in advance.

Designing the handoff

The handoff design is where a voice deployment succeeds or annoys people. Three decisions cover most of it.

• Triggers: which topics, phrases or caller requests always reach a person — and whether the agent offers a transfer or waits to be asked.
• Targets and hours: who receives transfers, in what order, and what happens outside working hours when nobody is there to take one.
• Fallback: what the agent does when a transfer goes unanswered — take a structured message, offer a callback slot, or send the details to a shared inbox.

Whatever you choose, the person picking up should receive the context: who called, what they asked, and what the agent already told them. A transfer that starts with “can you explain that again?” wastes the goodwill the fast answer earned.

A short evaluation script

When you trial a voice agent, run the same five calls against every vendor. Consistency reveals more than a guided demo.

• The easy one: opening hours. Check the answer is right and the reply is fast.
• The multi-part one: “what do you charge for X, and how soon could I come in?” — does it hold both threads?
• The unknown one: something plausible that is deliberately not in the knowledge base. Does it admit it?
• The interruption: cut across it mid-sentence and change the subject.
• The escalation: ask for a human and see what the person on the other end receives.