Skip to main content
A voice agent hears a question, answers it, and speaks the answer back. The hard part is not the audio; it is what the answer is grounded in. Vector search over transcripts returns whichever chunk sounds similar. A Wazoo world returns facts with subjects, predicates, and SPARQL on top, so the agent can answer “who manages Ethan” by querying a relation instead of hoping a chunk contains it. This guide builds a voice agent with three parts:
  1. Streaming speech-to-text (AssemblyAI) turns speech into question text.
  2. An LLM turn (Vercel AI SDK) answers using @wazoo/tools recall tools: searchWorld and executeSparql against a world.
  3. Streaming text-to-speech (AssemblyAI) speaks the reply over one WebSocket session for the whole conversation.
The audio legs come from the AssemblyAI JavaScript SDK (text-to-speech is in the SDK from 4.42.0). Everything is runnable without a microphone: the guide drives the speech-to-text leg from an audio file, the same path a mic pipeline uses.

Prerequisites

  • A Wazoo private beta account. Sign up at wazoo.dev/beta.
  • A platform token (wzp_) and a world token (wzw_) with data-plane access. Generate them on the Console tokens page. The two tokens are not interchangeable; see Your first query for what each grants.
  • An AssemblyAI API key.
  • Node.js 18+.

Architecture

Two details matter more than the rest of the design, and both come from the AssemblyAI APIs: the TTS side wants one session per conversation (each connection burns a per-account new-sessions-per-minute slot, so opening a connection per sentence hits the limit fast), and the STT side only finalizes a turn when you send a terminate message, which the SDK’s close() does.

Steps

1

Install the SDKs

Any AI SDK model provider works in place of DeepSeek; the agent turn is provider-agnostic.
2

Hear the TTS leg: one request, one WAV

Before wiring anything, verify the voice leg with one HTTP request. The Text-to-Speech API takes text and returns a complete WAV:
The response headers tell you what you got: x-tts-sample-rate: 24000, x-tts-encoding: pcm_s16le, x-tts-billed-characters for what you were charged, and x-tts-audio-duration-ms. Read the sample rate from the response rather than hardcoding it.Synthesis is not deterministic: the same sentence can come back a few hundred milliseconds longer or shorter. Billing counts characters, not seconds.
3

Speak a whole reply over one session

A voice agent speaks many short texts, one per turn. The TTS WebSocket API keeps one session open for the whole conversation, each sentence is a request on it, and audio streams back while the sentence is still being synthesized. In the JavaScript SDK:
First audio arrives about 100 ms after each sentence is sent, so splitting the reply into sentences and speaking them in order keeps playback continuous. The chunks are headerless PCM; to audition a file outside the agent:
4

Transcribe speech with streaming STT

The streaming speech-to-text API transcribes a WebSocket audio stream and emits turn events: partial transcripts while you speak, a final when the turn ends. For a runnable version with no microphone, stream a WAV file in real-time-sized frames. Any clip converts to the expected format with ffmpeg -i question.wav -ar 16000 -ac 1 question16.wav:
The terminate step is not optional polish. The turn stays open until you terminate, so waiting for a final before closing deadlocks: send the audio, let the trailing silence play out, then close. With a real mic pipeline the same rule applies on every turn: finalize before the agent starts answering.Valid encodings are pcm_s16le, pcm_mulaw, opus, ogg_opus, and aac. For opus, ogg_opus, and aac the stream describes its own sample rate; for the raw PCM encodings set sampleRate yourself.Streaming STT bills on connection time, not audio volume. Close sessions when the conversation ends.
5

Give the agent a world

@wazoo/tools exposes recall tools that wrap a Worlds client. Point them at the hosted data plane by adapting @worlds/client:
The hosted beta currently runs keyword full-text search. Vector ranking and the hybrid flow are coming; see Search for the current status. SPARQL is the structured recall path either way, so prompt the agent to prefer it for questions about relations (who manages what, which entities have which property) and to use search for fuzzy text lookups.
One behavior to know when seeding a new world: a Turtle import can report chunks: 0 even though the quads parsed and landed. Keyword search reads chunks, not quads, so search comes back empty until the index is built:
The response reports the chunk count once it completes. For relation questions the agent does not need the search index at all; SPARQL queries the quads directly.For a zero-dependency local agent (CI, tests, demos with no network), swap the same recall tools onto an embedded world:
The embedded search index matches text inside literals only, so seed worlds for it with literal-bearing triples. The hosted world is the default path for anything real.
6

Run the loop

The whole agent is one turn function and one event loop: the transcriber’s final turn triggers an LLM call with the recall tools, the reply is split into sentences, and each sentence is spoken into the TTS session:
Two prompt details earn their keep with spoken replies. “Never announce a lookup” keeps the model from speaking filler like “let me check that for you” while it works through tool rounds, which callers hear as dead air followed by a second answer. And limiting the tool rounds keeps latency bounded: the loop can spend several seconds before the first sentence exists, and every second is audible.
7

Cut latency with retrieve-then-speak

The tool loop is flexible but slow: every round is a full model call, and the model may spend several rounds exploring before it speaks. Production voice agents flip the shape: retrieve first, speak second.
  • Recall runs while the user is still talking. The streaming STT API emits partial transcripts as words arrive; four words in, fire the retrieval pre-pass, and reuse its result if the final transcript only differs in spelling.
  • Keyword-only search matches literal substrings, so query per meaningful token (stopwords stripped, two or three tokens) in parallel and merge the hit subjects. Filter results to http(s) subjects: search can return blank-node chunk ids, and a relative IRI inside a SPARQL VALUES clause silently breaks the whole query.
  • Fetch every hit subject’s triples in one batched query:
  • Answer with no tools at all: the facts go into the system prompt, the model streams, and each completed sentence is spoken immediately. The tool loop remains as the fallback when retrieval comes up empty, plus a final no-tools pass that summarizes the tool results if the loop ran out of rounds without speaking.
Measured against a hosted world with keyword-only recall and deepseek-v4-flash: first spoken word 10 to 20 seconds after the question ends, dominated by backend search-call variance (0.7 to 8 seconds per call observed), and the first TTS audio arrives about 100 ms after each sentence is sent. Enabling hybrid recall (embeddings) shortens the retrieval leg further.
8

Deploy the browser side safely

A browser cannot set an Authorization header on a WebSocket, and it must never hold the API key. The TTS API accepts a short-lived token in the token query parameter, minted server-side with product=tts, one per connection:
The same pattern applies to the streaming STT WebSocket. Never put the API key in a URL that runs on a user’s device.

Choosing the voice providers

The guide builds on AssemblyAI’s streaming APIs, but the audio legs are swappable: both sides are WebSockets in and out, and the agent turn is decoupled from either. When picking an STT or TTS provider for a voice agent, measure against your own use case instead of vendor benchmarks:
  • Voice Arena publishes human-eval leaderboards for both legs: streaming and offline STT ranked by semantic WER on public and private test sets, and TTS ranked by Bradley-Terry Elo from blind pairwise votes with time-to-first-audio listed per model. The TTS board splits real-time-only models, which is the filter that matters for a conversational agent; an offline-quality model with 600 ms TTFA is the wrong choice even if it tops the general ranking.
  • For STT, check the streaming-specific rows: a provider’s batch model can lead the general board while its realtime model trails (this ordering exists on the current board too). The demo needs end_of_turn turn events and partial transcripts, so confirm those are first-class in whichever API you pick.
  • For TTS, the number to compare is time to first audio, not Elo alone: the demo speaks each sentence as the answer stream completes it, so TTFA dominates perceived latency while the graph recall dominates the wait before speech starts.
  • Both boards are human-eval based; run your own domain audio (your callers, your accent mix, your vocabulary) through the finalists before committing. Public test sets underrepresent domain terms, and the guide’s whole point is that vocabulary (entity names, relation labels) is what your graph holds.
The demo pins AssemblyAI for the hackathon’s featured-stack constraint and because the streaming APIs expose the pieces this guide relies on (turn events, sentence-level sessions, temporary browser tokens). The recall layer does not change if you swap either audio provider.

Where to take it

  • Swap the question source from file to mic capture; the transcriber loop is identical.
  • Add barge-in: stream partial transcripts during TTS playback and cancel the session when the speaker starts talking.
  • Give the agent write access with the ingest tools so the conversation itself updates the world.