Technical Implementation of a Real-Time Voice Agent.

In the previous post I mapped the abstract anatomy of a real-time voice agent — STT, LLM, TTS, VAD, orchestrator, RAG, and the latency budget that ties them together. That post was the blueprint. This one is the construction site.

Every code snippet, every latency number, every architectural choice here is pulled directly out of a working voice-agent platform I built — the same one I shipped as Velox AI. So instead of whiteboard sketches, you get the production answers: how the pieces are actually wired, what surprised me, what I’d reverse if I started over, and where the milliseconds really go.

The conversation lifecycle

Before we go anywhere, here’s what we’re optimising for. One full conversational turn — from the moment a user finishes speaking to the moment they hear the agent reply — passes through nine stages and crosses three networks. Every millisecond on this rail is a millisecond the user can hear:

Roughly 850 ms end-to-end. Sounds slow on paper. Feels almost-but-not-quite-human in practice — and most of the engineering that follows is about chipping away at that number without losing audio quality or context.

Before any code: the single most important architectural decision you’ll make is whether your pipeline is vendor-locked or vendor-agnostic. Every voice-agent tutorial on the internet hard-codes one stack — “OpenAI Whisper + GPT-4 + ElevenLabs” — and ships it. That works for a demo. It does not work in production.

The reason: the optimal stack depends on language, latency budget, voice taste, and price. An English customer-support bot in the US wants Deepgram + Groq + Deepgram Aura. A Hindi receptionist wants Sarvam everywhere. A premium character-voice game NPC wants ElevenLabs. A privacy-paranoid customer wants self-hosted Piper. Lock yourself into one stack and you’ve locked yourself out of those use cases.

The fix is an adapter pattern — one interface per layer, many implementations behind it, chosen per agent at config time:

# Every STT provider implements the same protocol.
class STTAdapter(Protocol):
    async def send_audio(self, pcm: bytes) -> None: ...
    async def events(self) -> AsyncIterator[STTEvent]: ...
    async def close(self) -> None: ...

# Picking is a one-liner at call setup:
stt = build_stt(agent.stt_provider, language=agent.stt_language)
#       └─► returns DeepgramAdapter | SarvamAdapter | AssemblyAIAdapter | …

# The orchestrator never imports a specific vendor.
async for event in stt.events():
    handle(event)

Three layers, three adapters. STT, LLM, TTS. Swap any one without touching the other two, the orchestrator, the audio plumbing, or the UI. Here’s what I shipped — and what you can plug in instead:

Zoom out. Here’s the whole system on one map — browser, dashboard backend, voice runtime, persistence, and every external provider it talks to. The split that matters is the colour: control plane (where agents are defined) versus data plane (where they run).

Control plane vs data plane

The clearest mental model for a voice-agent platform is the same one cloud-infra teams use: control plane vs data plane.

• Control plane = the dashboard. Where agents are defined. A user signs in, edits an agent (system prompt, provider choices, voice, knowledge base), and the result lands in MongoDB. Pure CRUD, no real-time anything, no exotic infrastructure.
• Data plane = the voice runtime. Where agents run. A WebSocket opens, the backend reads the agent config from Mongo (cached in Redis), spins up an asyncio task graph (STT loop, LLM stream, TTS worker, audio sender), and streams audio both directions until the user hangs up.

The two share exactly one thing: the agent document. They can deploy together (one box, two containers — what I actually run today on AWS) or scale independently the moment one gets hot. Because the data plane holds no cross-call in-process state — everything per-call lives on a single asyncio task graph — you can scale it horizontally by adding boxes behind a load balancer the day you need to. That decoupling is worth more than any specific tech choice.

Concrete choices, with the “why” in one line and the alternatives you can swap in. None of these is the One True Choice — they’re the trade-offs that fit a real-time, B2B, multi-tenant SaaS.

Audio transport: raw WebSocket, not WebRTC

Most tutorials reach for WebRTC because it’s “the real-time standard.” For a browser-to-server voice agent you almost certainly don’t need it. WebRTC buys you three things — NAT traversal, jitter buffering, and AEC — and none are valuable here:

• NAT traversal — you have a public server. No peer-to-peer hole-punching.
• Jitter buffering— you’re going to build your own client-side ~150 ms buffer anyway (TTS chunks arrive bursty).
• AEC — getUserMedia({ echoCancellation: true }) already does it on the browser side.

Skipping WebRTC means no SDP exchange, no STUN/TURN servers, no SFU. Just one TCP+TLS connection carrying binary WebSocket frames in both directions — lower setup latency, far less infra. The audio format on the wire:

The downsample 48 kHz Float32 → 16 kHz Int16 is the single most-important client-side detail. Doing it in the AudioWorklet (not the main thread) keeps the audio path off React’s render loop — a React re-render blocking the audio thread is what causes mysterious glitches in toy implementations.

The orchestrator loop

Everything above is plumbing. The brain is one async function that owns the call’s state machine: it consumes STT events, fires the LLM, slices its token stream into sentences, queues those for TTS, and pushes audio bytes back over the WebSocket — while listening for interruptions the whole time.

# Per-call task graph (simplified from services/llm_orchestrator.py)
task_manager = TaskManager(websocket)          # audio queue, interrupt signal, llm task handle
history = [{"role": "system", "content": agent.system_prompt}]
stt = build_stt(agent.stt_provider, language=agent.stt_language)

# 1) STT loop — listens for transcripts & barge-in
async def stt_loop():
    async for event in stt.events():
        if event.type == "interim_transcript" and task_manager.is_busy:
            await task_manager.handle_interruption()       # ← see Barge-In section
        elif event.type == "utterance_end":
            asyncio.create_task(run_llm_and_tts(event.final_text))

# 2) The pipeline: LLM → sentence buffer → TTS queue
async def run_llm_and_tts(user_text):
    history.append({"role": "user", "content": user_text})
    sentence_buffer = ""
    async for delta in stream_llm(history[-25:], tools=TOOL_SCHEMAS):
        if delta.tool_call:                                # tool calls handled separately
            await dispatch_tool(delta.tool_call); return
        sentence_buffer += delta.content
        for sentence in extract_sentences(sentence_buffer):  # punctuation-boundary slice
            if task_manager.interrupt_signal.is_set():       # bail on barge-in
                return
            await task_manager.tts_queue.put(sentence)
        sentence_buffer = leftover_after_extraction
    if sentence_buffer.strip():
        await task_manager.tts_queue.put(sentence_buffer)    # flush tail (don't drop it!)

# 3) TTS worker — drains the sentence queue
async def tts_worker():
    while True:
        sentence = await task_manager.tts_queue.get()
        async for chunk in tts.synthesize(sentence):
            if task_manager.interrupt_signal.is_set():
                break
            await task_manager.audio_queue.put(chunk)

# 4) Audio sender — drains the audio queue to the client
async def audio_sender():
    while True:
        chunk = await task_manager.audio_queue.get()
        await websocket.send_bytes(chunk)

await asyncio.gather(stt_loop(), tts_worker(), audio_sender())

Four concurrent tasks. Three queues between them. One shared TaskManagerholding the interrupt signal that every task checks before doing anything. That’s the entire architecture — everything else in this post is detail about how each of those four tasks behaves.

Why does Velox feel snappy when most voice bots feel laggy? Sentence buffering.

A naive pipeline waits for the LLM to finish, sends the whole response to TTS, waits for TTS to synthesise all of it, then plays it. Three serial round-trips — users sit through ~2–3 seconds of dead air every turn. The fix is to chunk the token stream by punctuation boundaries and pipe each finished sentence to TTS the instant it lands:

# The whole magic, in one regex.
SENTENCE_BOUNDARY = re.compile(r"([.!?:;])\s+|\n")

# Stream tokens in; whenever a terminator is seen, slice off the completed
# sentence and ship it to TTS. Everything after the cut stays in the buffer.

Three sub-decisions matter here:

1. 01Boundary set. . ! ? : ; \n. Anything finer (token count, mid-clause) produces glitchy half-words from TTS. Anything coarser (whole paragraphs) starves the user of audio.
2. 02Hold the tail. If the LLM ends mid-sentence (no terminator), the last buffered chunk has to be flushed at end-of-stream — otherwise the final clause silently disappears. (Ask me how I know.)
3. 03Drop too-short fragments.Sentences shorter than ~3 chars are held back — otherwise an “OK.” costs a full TTS round-trip to say one syllable.

Net effect: the user hears the first sentence ~600 ms after they stop speaking, not ~2.5 s. TTS for sentence two happens while sentence one is playing. The whole pipeline is naturally pipelined.

Barge-in — the user talking over the agent mid-sentence — was the bug class that bit the most before it was solved. The hard part isn’t detecting that the user spoke; it’s making the agent shut up within ~100 ms, across three concurrent tasks, two queues, and one TCP stream’s worth of buffered audio.

Two engineering choices made it tractable:

1. Detect barge-in semantically, not acoustically.A naive implementation watches mic energy — anything above a threshold counts as speech. In the real world that fires on keyboard clicks, doors, AC fans, the user’s own breathing. Velox uses the STT engine’s interim transcriptsinstead: if Deepgram returns a partial that contains a recognisable word while the agent is speaking, that’s a real interrupt. Otherwise it’s noise — ignore it.

2. Treat interruption as a cascade, not a flag. When the interrupt fires, six things have to happen in order. Get any of them wrong and you hear overlapping voices, ghost audio, or the agent resuming where it left off after a two-second pause:

The known imperfection: when the user interrupts mid-sentence, the conversation history records the full intended response, not what the user actually heard. If they reference something the agent never finished saying, the model can get confused. The fix is to truncate history to the last word the user heard — which requires knowing which audio chunks actually made it past the jitter buffer. Not solved yet.

A voice agent without business knowledge can’t answer “what’s your return policy?” It needs RAG — retrieval-augmented generation. The pipeline is dead simple, but every parameter is a trade-off:

Choices worth defending:

• 500-char / 50-overlap chunks, character-based, not token-based. Small enough to fit 3–5 in an LLM call without bloating cost; large enough to carry real context. The recursive splitter respects paragraph → sentence → word boundaries before falling back to mid-word.
• Local embeddings (all-MiniLM-L6-v2) over OpenAI’s text-embedding-3-small. 384-dim is plenty for SMB-sized KBs (~50–500 chunks per agent), inference is sub-50 ms on CPU, and there’s no per-token bill on the hot path. The recall gap vs OpenAI isn’t noticeable below a few thousand chunks. For technical / legal / multilingual KBs, swap in something stronger.
• One Qdrant collection, payload-filter isolation. Every point has an agent_idfield with a payload index; every query filters by it. Cheap, simple, scales to thousands of agents. If a customer ever demands physical separation, the migration is “one collection per tenant” — a refactor, not a rewrite.
• Top-3 vector-only retrieval. No hybrid keyword search, no re-ranker. Top-3 with 500-char chunks puts ~1500 chars of context in the LLM — concise factual answers that read naturally when spoken aloud.

# Tenant isolation = one line of Qdrant filter
query_filter = models.Filter(must=[
    models.FieldCondition(
        key="agent_id",
        match=models.MatchValue(value=agent_id),
    )
])

results = qdrant.search(
    collection_name="agent_knowledge",
    query_vector=embed(user_question),     # 384-dim
    query_filter=query_filter,
    limit=3,
)

The retrieved chunks aren’t stuffed into the system prompt — they’re injected as the response of a search_knowledge_basetool call. The LLM decides when to ask for them. That’s crucial: the model can choose notto search when the user is just exchanging pleasantries, saving a Qdrant round-trip every turn. Which leads to…

Tool calling on a non-streaming HTTP chat endpoint is easy. Tool calling inside a streaming voice pipeline is genuinely tricky, because tool calls arrive as partial JSON interleaved with regular content tokens, and you have to decide on the fly:

• Suppress any “narration” tokens the model emits alongside the tool call (it shouldn’t say “I’m going to look that up…” out loud while it’s already doing it).
• Mask the tool latency with a filler phrase so the user hears something.
• Execute the tool, append the result to history, then re-call the LLM — which streams the actual spoken answer back through the sentence buffer.

async def run_llm_and_tts(user_text):
    history.append({"role": "user", "content": user_text})
    sentence_buffer = ""
    tool_call_buffer = None

    async for delta in stream_llm(history[-25:], tools=TOOL_SCHEMAS):
        # 1) Accumulate partial tool-call JSON across stream chunks
        if delta.tool_call:
            tool_call_buffer = accumulate(tool_call_buffer, delta.tool_call)
            continue                       # don't TTS this delta

        # 2) Regular content tokens flow into the sentence buffer
        sentence_buffer += delta.content
        for sentence in extract_sentences(sentence_buffer):
            if task_manager.interrupt_signal.is_set(): return
            await task_manager.tts_queue.put(sentence)
        sentence_buffer = leftover_after_extraction

    # 3) End of stream — if a tool was called, execute and recurse
    if tool_call_buffer:
        # Latency mask: queue a filler while the tool runs
        await task_manager.tts_queue.put(random.choice(FILLER_PHRASES))
        result = await execute_tool(tool_call_buffer)        # search KB, end call, etc.
        history.append({"role": "tool", "content": result,
                        "tool_call_id": tool_call_buffer.id})
        await run_llm_and_tts(_continuation=True)            # second LLM call answers
    elif sentence_buffer.strip():
        await task_manager.tts_queue.put(sentence_buffer)    # flush tail

The filler phrase (“Let me check the knowledge base for that…”) masks ~400–700 ms of dead air on knowledge searches. Small thing, but it’s the difference between an agent that feels thoughtful and one that feels stuck. The same trick applies to slow LLMs in general — queue a generic “thinking” phrase before the call, and the user hears engagement instead of silence.

Real numbers from a production pipeline on a good connection (English path: Deepgram STT + NVIDIA NIM LLM + Deepgram Aura TTS, client in ap-south-1). These are estimatesfrom log-stamped TTS first-byte events — honest disclaimer: there’s no P50/P95 dashboard yet, which is genuinely the biggest operational gap left in the system.

Two observations worth taking away:

1. 01The LLM is rarely the bottleneck anymore.Going in, I assumed “AI is slow” and built around hiding model latency. With Groq / Cerebras / NVIDIA NIM returning first tokens in 150–250 ms, the LLM is faster than the network round-trip to TTS. The real bottlenecks are STT endpointing waiting for silence, and TTS providers that don’t stream.
2. 02Users notice silence after they speak, not delay before they’re answered.200 ms of silence between “Hi” and the agent reacting feels broken. 500 ms of audio delay once the agent is talking feels totally fine. That asymmetry should shape where you spend your optimisation budget.

What’s left to attack:Sarvam’s TTS doesn’t stream — it returns full utterances in one HTTP response. For a one-sentence reply that’s 300–600 ms of dead air on Indian-language agents. The day Sarvam ships a streaming endpoint, that’s a one-line config change for a 30% latency win. STT endpointing (utterance_end_ms=1000) is the runner-up — dropping it to 500 ms shaves half a second off perceived latency at the cost of more occasional early triggers. A tunable knob, not a code change.

Three things I didn’t expect going in — worth sharing because every voice-agent post on the internet was wrong about at least one of them:

1. 01The LLM is the fastest part.See above. Most posts treat the LLM as the slow, expensive thing you wrap with caching and prompt compression. With modern inference providers it’s faster than your network round-trip. Spend your budget on STT endpointing and TTS streaming — that’s where the seconds live.
2. 02Semantic caching isn’t worth it for voice.Theory: 40% of support queries are repeats; cache them. Reality: voice queries are wildly variable in phrasing (“how do I…” / “can you tell me…” / “what’s the way to…”), hit rates are low, and the operational complexity isn’t worth it when the uncached path is already ~700 ms.
3. 03You need fewer queues than you think. Early prototypes had a queue between every stage. All you actually need is a sentence queue (LLM → TTS) and an audio queue (TTS → client). Everything else is in-process async/await — queues add latency and a place for state to drift out of sync with the interrupt signal.