My Voice Agent Stack, Torn Down Component by Component

What's actually inside a production AI voice agent: telephony, streaming STT, the LLM core, tool calling, TTS, barge-in handling, and observability — with failure modes and cost drivers for each.

Key takeaways

• A production voice agent is seven layers — telephony, streaming STT, an LLM core, tool calling, TTS, orchestration, and observability — and the caller experiences the sum of every layer's latency and failure modes at once.
• The sub-second response budget shapes every choice in the stack: fast models beat smart ones for conversational turns, and streaming at every layer is non-negotiable.
• Barge-in (interruption) handling and the post-launch tuning loop are where demos and production diverge — this is a system you operate, not a weekend project.

Layer 1: Telephony — the part everyone forgets until it bites

Layer 2: Streaming speech-to-text — the ears

Layer 3: The LLM reasoning core — where the latency budget bites hardest

Layer 4: Tool calling — where talk becomes action

Layer 5: Text-to-speech — the voice itself

Layer 6: Orchestration and interruption — where demos go to die

Layer 7: Observability — the layer that makes the other six improvable

The tuning loop, and the honest close

I've written about what AI voice agents do for a business and where they sit in my 2026 tools stack. This post is the other view: the machine with the case off. What's actually inside a production voice agent, component by component — what each layer does, what I use or have evaluated, how it fails, and what it costs.

Why bother, if you're a buyer rather than a builder? Because every vendor demo you'll ever see is designed to hide exactly the seams this post describes, and knowing where the seams are is how you ask questions that separate real systems from duct tape. Standard disclosure: my agency builds these for a living, so I profit when this looks hard. I'll try to earn the bias by being specific.

One framing number before the tour: a natural human conversational gap is under a second. Everything below — hearing the caller, understanding, deciding, acting, and speaking — has to fit inside that budget, simultaneously, on every conversational turn. That single constraint explains almost every architectural choice in the stack.

And the budget is tighter than it sounds, because it's spent in series. Detecting that the caller has finished speaking costs a slice. The model's first token costs a slice. The first audible syllable of synthesized speech costs another, with network hops between every layer. Stack ordinary, individually reasonable latencies end to end and you've burned the whole second before anyone did anything wasteful — which is why every layer below streams, overlaps, and starts before its predecessor finishes. A voice agent isn't a pipeline; it's a relay team where everyone leaves early.

What it does: gets a phone call into and out of your software. Phone numbers, SIP trunks, call routing, transfers, and the audio stream itself. Owning a phone company has made me unreasonably opinionated about this layer, because it's where "AI project" collides with a century of telecom reality: carrier registration, number provisioning, caller-ID reputation, E911 obligations.

What I use: programmable telephony infrastructure — Twilio is the reference example I've cited in my vendor comparisons, and most assembled platforms ride on it or something shaped like it. If you buy a bundled voice-agent product, this layer comes included and you'll never think about it, which is precisely the value.

How it fails: ugly and non-obviously. Choppy audio that the AI layers get blamed for. Transfers that drop context or drop the call. Outbound numbers flagged as spam because nobody registered them properly. My rule from the phone business: when a voice agent "sounds bad," check telephony first — it's the layer least likely to be diagnosed and most likely to be guilty.

Cost driver: per-minute carrier charges plus number rental. Cheap individually, but it's the floor under every other per-minute cost in the stack.

What it does: converts caller audio to text as it's being spoken. The streaming part is the whole game — batch transcription that waits for the caller to finish would blow the entire latency budget on its own. Good streaming STT emits partial hypotheses that firm up as the sentence completes, so the reasoning layer can start thinking before the caller stops talking.

What I've evaluated: the major streaming STT providers plus the transcription bundled into assembled platforms. I'll skip vendor endorsements here — the category moves fast and the honest answer is that several are good enough. What separates them in my testing is the messy edges: accents, industry jargon, callers on speakerphone in a moving truck.

How it fails: silently, which makes it dangerous. The agent doesn't know it misheard "Kaufman" as "Coffman" — it confidently proceeds on wrong data, and you find out when the CRM lookup misses or the callback number is wrong. This is why I weight transcript review so heavily in the observability layer, and why my vendor checklist tells buyers to read raw demo transcripts rather than trust the highlight reel.

Cost driver: per-minute transcription fees — typically the second-cheapest layer, but always on.

What it does: takes the transcript-so-far plus the conversation state and decides what to do next: answer, ask a clarifying question, call a tool, or escalate to a human.

The counterintuitive lesson: voice is the one place where I don't reach for the smartest model available. A frontier model that takes two-plus seconds to respond loses to a fast model that answers well in a few hundred milliseconds, because in voice, slow is indistinguishable from broken. Sub-500ms turn gaps feel natural; approach a second and callers start talking over the agent, which then stresses the interruption layer below. The pattern I've settled on is tiered: a fast model handles the conversational loop, and harder decisions get routed to a stronger model or a human — asynchronously where possible, so the caller never waits on the expensive thinking. If the phrase "the model decides and acts" is new territory, my plain-English agentic AI explainer covers the concepts underneath this layer.

How it fails: confidently. Hallucinated policy answers, invented appointment availability, or an agent that agrees to things it can't do. The mitigations are unglamorous: tight prompts, strict grounding in retrieved business data, and hard rules about what the agent may never promise. None of that comes from the model; all of it comes from engineering around the model.

Cost driver: tokens per turn — usually the most volatile line item, and the one that balloons when someone "improves" the prompt to three times the length.

What it does: lets the agent actually do things mid-call: check calendar availability, book the appointment, look up an order, create the CRM record, send the confirmation text. This layer is the difference between an expensive FAQ machine and a receptionist. It's also where the voice stack stops being a voice problem and becomes an integration problem — the same one that dominates the rest of my automation work.

What I use: the same integration discipline as any workflow build — well-defined tools with narrow permissions, connected to systems of record through their APIs. Everything I wrote about API-auditing your applications in my tech stack guide applies double here: a voice agent in front of software with no API is a very fluent dead end.

How it fails: at the seams, mid-conversation. The calendar API takes four seconds to respond and the agent has to vamp naturally instead of going dead silent. A booking succeeds but the confirmation write fails, leaving systems disagreeing about reality. Good stacks treat every tool call as something that can be slow, wrong, or half-completed — and script the caller experience for each case.

Cost driver: mostly engineering time, not runtime. Integrations are where build budgets actually go.

What it does: turns the response text into audio, streamed so the first syllable plays while the rest is still generating. Modern TTS is good enough that voice quality is no longer the differentiator vendors pretend it is — callers forgive a slightly synthetic voice; they don't forgive a two-second pause or being cut off.

What I've evaluated: the leading TTS providers and the voices bundled into platforms. Same posture as STT: several are excellent, pick for streaming latency first and voice aesthetics second, and spend the saved deliberation time on the orchestration layer below, which matters more.

How it fails: mispronounced names (your business's, your caller's — both bad), robotic handling of numbers and addresses, and latency spikes under load. Test with your actual street names and your actual price list, not the vendor's demo script.

Cost driver: per-character or per-minute synthesis fees. Modest, but it stacks onto the same per-minute pile as everything else.

What it does: conducts all of the above in real time. Turn-taking, endpointing (deciding the caller has actually finished a thought, not merely paused), managing conversation state — and above all, barge-in handling: what happens when a human interrupts the agent mid-sentence. Which humans do constantly, because that's how phone calls work.

Barge-in sounds trivial and is anything but. The system has to detect real speech fast (while ignoring a cough, a "mm-hmm," or a TV in the background), kill the audio that's already streaming, throw away the rest of what it planned to say, and fold the interruption into the conversation state without losing the thread. Get it wrong in one direction and the agent bulldozes over callers; wrong in the other direction and it stops dead every time a truck drives past. This layer is the single sharpest divider between a demo and a production system — scripted demos never contain interruptions, and real calls contain little else. It's the reason my standing advice to buyers is to test any vendor with a live, messy, unscripted call.

What I use: this is the layer where assembled platforms — the Retell/Bland tier from my earlier vendor comparison — genuinely earn their per-minute margin, because they've burned engineering years here that a self-builder has to repeat. If I'm honest about where DIY stacks I've reviewed fall down, it's here and in observability, almost never in model choice.

Cost driver: platform margin if you buy; a punishing engineering tail if you build.

What it does: records what actually happened. Full transcripts with audio, per-layer latency traces, resolution tagging (booked, answered, escalated, abandoned), and escalation analytics — which call types keep ending up with humans, and why.

This is the least glamorous layer and my strongest opinion in the whole stack: a voice agent without observability isn't a system, it's a liability with a pleasant voice. You cannot tune what you can't see, and — per the STT section — several of this stack's worst failure modes are silent. The operational cadence that makes the whole thing work is mundane: humans reading a sample of transcripts every week, tagging failures, and feeding fixes back into prompts, tools, and routing rules.

How it fails: by not existing. It's the first thing cut from a quick build and the first thing you need on day two of production.

Here's the part that reframes everything above: launch is the midpoint, not the finish. Every production agent I've been near follows the same arc — it handles maybe half of its target calls acceptably in week one, and the difference between the ones that climb from there and the ones that quietly get unplugged is entirely the loop: review transcripts, find the top failure pattern, fix the layer responsible, re-measure, repeat. The stack is seven components; the product is the loop.

So, build versus buy, with the bias disclosed again: this is a real-time distributed system with seven layers that each fail differently, wrapped in an operational practice that never ends. If voice is your product, build — it's a genuinely fun system to engineer. If voice is your front door and your actual business is dentistry or HVAC or law, buy the assembled platform or hire someone accountable for the whole loop. That second path is what my team sells, so discount accordingly — and if you want the buyer's-side version of this post before talking to anyone (me included), the buyer's-side guide my team wrote covers the ten questions I'd ask any vendor.

This teardown lives on the workshop side of my writing — the Builder Notes hub has the rest of the benches.

I write these from the operator's seat — I run a phone company and an automation agency, which means I've been paged for failures at layer one and layer six of this stack, sometimes on the same call. More teardowns in Builder Notes, or get in touch.