- Streaming speech-to-text (AssemblyAI) turns speech into question text.
- An LLM turn (Vercel AI SDK) answers using
@wazoo/toolsrecall tools:searchWorldandexecuteSparqlagainst a world. - Streaming text-to-speech (AssemblyAI) speaks the reply over one WebSocket session for the whole conversation.
Prerequisites
- A Wazoo private beta account. Sign up at wazoo.dev/beta.
- A platform token (
wzp_) and a world token (wzw_) with data-plane access. Generate them on the Console tokens page. The two tokens are not interchangeable; see Your first query for what each grants. - An AssemblyAI API key.
- Node.js 18+.
Architecture
close() does.
Steps
1
Install the SDKs
2
Hear the TTS leg: one request, one WAV
Before wiring anything, verify the voice leg with one HTTP request. The
Text-to-Speech API takes text and returns a complete WAV:The response headers tell you what you got:
x-tts-sample-rate: 24000,
x-tts-encoding: pcm_s16le, x-tts-billed-characters for what you were
charged, and x-tts-audio-duration-ms. Read the sample rate from the
response rather than hardcoding it.Synthesis is not deterministic: the same sentence can come back a few
hundred milliseconds longer or shorter. Billing counts characters, not
seconds.3
Speak a whole reply over one session
A voice agent speaks many short texts, one per turn. The TTS WebSocket
API keeps one session open for the whole conversation, each sentence is a
request on it, and audio streams back while the sentence is still being
synthesized. In the JavaScript SDK:First audio arrives about 100 ms after each sentence is sent, so
splitting the reply into sentences and speaking them in order keeps
playback continuous. The chunks are headerless PCM; to audition a file
outside the agent:
4
Transcribe speech with streaming STT
The streaming speech-to-text API transcribes a WebSocket audio stream and
emits turn events: The terminate step is not optional polish. The turn stays open until you
terminate, so waiting for a final before closing deadlocks: send the
audio, let the trailing silence play out, then close. With a real mic
pipeline the same rule applies on every turn: finalize before the agent
starts answering.Valid encodings are
partial transcripts while you speak, a final when
the turn ends. For a runnable version with no microphone, stream a WAV
file in real-time-sized frames. Any clip converts to the expected format
with ffmpeg -i question.wav -ar 16000 -ac 1 question16.wav:pcm_s16le, pcm_mulaw, opus, ogg_opus, and
aac. For opus, ogg_opus, and aac the stream describes its own
sample rate; for the raw PCM encodings set sampleRate yourself.Streaming STT bills on connection time, not audio volume. Close sessions
when the conversation ends.5
Give the agent a world
@wazoo/tools exposes recall tools that wrap a Worlds client. Point them
at the hosted data plane by adapting @worlds/client:The hosted beta currently runs keyword full-text search. Vector
ranking and the hybrid flow are coming; see Search
for the current status. SPARQL is the structured recall path either
way, so prompt the agent to prefer it for questions about relations
(who manages what, which entities have which property) and to use
search for fuzzy text lookups.
chunks: 0 even though the quads parsed and landed. Keyword
search reads chunks, not quads, so search comes back empty until the
index is built:6
Run the loop
The whole agent is one turn function and one event loop: the
transcriber’s final turn triggers an LLM call with the recall tools, the
reply is split into sentences, and each sentence is spoken into the TTS
session:Two prompt details earn their keep with spoken replies. “Never announce a
lookup” keeps the model from speaking filler like “let me check that for
you” while it works through tool rounds, which callers hear as dead air
followed by a second answer. And limiting the tool rounds keeps latency
bounded: the loop can spend several seconds before the first sentence
exists, and every second is audible.
7
Cut latency with retrieve-then-speak
The tool loop is flexible but slow: every round is a full model call, and
the model may spend several rounds exploring before it speaks. Production
voice agents flip the shape: retrieve first, speak second.
-
Recall runs while the user is still talking. The streaming STT API
emits
partialtranscripts as words arrive; four words in, fire the retrieval pre-pass, and reuse its result if the final transcript only differs in spelling. -
Keyword-only search matches literal substrings, so query per meaningful
token (stopwords stripped, two or three tokens) in parallel and merge
the hit subjects. Filter results to
http(s)subjects: search can return blank-node chunk ids, and a relative IRI inside a SPARQLVALUESclause silently breaks the whole query. -
Fetch every hit subject’s triples in one batched query:
- Answer with no tools at all: the facts go into the system prompt, the model streams, and each completed sentence is spoken immediately. The tool loop remains as the fallback when retrieval comes up empty, plus a final no-tools pass that summarizes the tool results if the loop ran out of rounds without speaking.
deepseek-v4-flash: first spoken word 10 to 20 seconds after the
question ends, dominated by backend search-call variance (0.7 to 8
seconds per call observed), and the first TTS audio arrives about 100
ms after each sentence is sent. Enabling hybrid recall (embeddings)
shortens the retrieval leg further.8
Deploy the browser side safely
A browser cannot set an The same pattern applies to the streaming STT WebSocket. Never put the
API key in a URL that runs on a user’s device.
Authorization header on a WebSocket, and it
must never hold the API key. The TTS API accepts a short-lived token in
the token query parameter, minted server-side with product=tts, one
per connection:Choosing the voice providers
The guide builds on AssemblyAI’s streaming APIs, but the audio legs are swappable: both sides are WebSockets in and out, and the agent turn is decoupled from either. When picking an STT or TTS provider for a voice agent, measure against your own use case instead of vendor benchmarks:- Voice Arena publishes human-eval leaderboards for both legs: streaming and offline STT ranked by semantic WER on public and private test sets, and TTS ranked by Bradley-Terry Elo from blind pairwise votes with time-to-first-audio listed per model. The TTS board splits real-time-only models, which is the filter that matters for a conversational agent; an offline-quality model with 600 ms TTFA is the wrong choice even if it tops the general ranking.
- For STT, check the streaming-specific rows: a provider’s batch model can lead
the general board while its realtime model trails (this ordering exists on the
current board too). The demo needs
end_of_turnturn events and partial transcripts, so confirm those are first-class in whichever API you pick. - For TTS, the number to compare is time to first audio, not Elo alone: the demo speaks each sentence as the answer stream completes it, so TTFA dominates perceived latency while the graph recall dominates the wait before speech starts.
- Both boards are human-eval based; run your own domain audio (your callers, your accent mix, your vocabulary) through the finalists before committing. Public test sets underrepresent domain terms, and the guide’s whole point is that vocabulary (entity names, relation labels) is what your graph holds.
Where to take it
- Swap the question source from file to mic capture; the transcriber loop is identical.
- Add barge-in: stream partial transcripts during TTS playback and cancel the session when the speaker starts talking.
- Give the agent write access with the ingest tools so the conversation itself updates the world.