> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wazoo.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a voice agent

> A voice agent that answers from a Wazoo world: streaming speech-to-text, grounded recall over search and SPARQL, and streaming text-to-speech.

A voice agent hears a question, answers it, and speaks the answer back. The hard
part is not the audio; it is what the answer is grounded in. Vector search over
transcripts returns whichever chunk sounds similar. A Wazoo world returns facts
with subjects, predicates, and SPARQL on top, so the agent can answer "who
manages Ethan" by querying a relation instead of hoping a chunk contains it.

This guide builds a voice agent with three parts:

1. Streaming speech-to-text (AssemblyAI) turns speech into question text.
2. An LLM turn (Vercel AI SDK) answers using `@wazoo/tools` recall tools:
   `searchWorld` and `executeSparql` against a world.
3. Streaming text-to-speech (AssemblyAI) speaks the reply over one WebSocket
   session for the whole conversation.

The audio legs come from the AssemblyAI JavaScript SDK (text-to-speech is in the
SDK from 4.42.0). Everything is runnable without a microphone: the guide drives
the speech-to-text leg from an audio file, the same path a mic pipeline uses.

## Prerequisites

* A Wazoo private beta account. Sign up at
  [wazoo.dev/beta](https://wazoo.dev/beta).
* A platform token (`wzp_`) and a world token (`wzw_`) with data-plane access.
  Generate them on the [Console tokens page](https://docs.wazoo.dev/console/tokens.md). The two tokens
  are not interchangeable; see [Your first query](https://docs.wazoo.dev/worlds/first-query.md) for what
  each grants.
* An AssemblyAI API key.
* Node.js 18+.

## Architecture

```
mic (or file) -> streaming STT (ws)
              -> LLM turn with @wazoo/tools recall tools
                   searchWorld + executeSparql against the world
              -> reply text, split into sentences
              -> streaming TTS session (one ws for the conversation)
              -> speaker (24 kHz 16-bit mono PCM)
```

Two details matter more than the rest of the design, and both come from the
AssemblyAI APIs: the TTS side wants one session per conversation (each
connection burns a per-account new-sessions-per-minute slot, so opening a
connection per sentence hits the limit fast), and the STT side only finalizes a
turn when you send a terminate message, which the SDK's `close()` does.

## Steps

**1. Install the SDKs**

```bash theme={null}
npm install assemblyai@4.42.0 ai @ai-sdk/deepseek @wazoo/tools @worlds/client
```

Any AI SDK model provider works in place of DeepSeek; the agent turn is
provider-agnostic.

**2. Hear the TTS leg: one request, one WAV**

Before wiring anything, verify the voice leg with one HTTP request. The
Text-to-Speech API takes text and returns a complete WAV:

```bash theme={null}
curl -sS --fail-with-body https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Hi, thanks for calling. How can I help you today?",
       "voice_id": "aai-jane", "container": "wav", "stream": false}' \
  -o hello.wav
```

The response headers tell you what you got: `x-tts-sample-rate: 24000`,
`x-tts-encoding: pcm_s16le`, `x-tts-billed-characters` for what you were
charged, and `x-tts-audio-duration-ms`. Read the sample rate from the
response rather than hardcoding it.

Synthesis is not deterministic: the same sentence can come back a few
hundred milliseconds longer or shorter. Billing counts characters, not
seconds.

**3. Speak a whole reply over one session**

A voice agent speaks many short texts, one per turn. The TTS WebSocket
API keeps one session open for the whole conversation, each sentence is a
request on it, and audio streams back while the sentence is still being
synthesized. In the JavaScript SDK:

```ts theme={null}
import { createWriteStream } from "node:fs";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

const session = client.tts.session({
  voiceId: "aai-jane",
  language: "english",
});
await session.connect();

for (const sentence of sentences) {
  for await (const chunk of session.speak(sentence)) {
    out.write(chunk); // raw 24 kHz s16le mono PCM
  }
}
await session.close();
```

First audio arrives about 100 ms after each sentence is sent, so
splitting the reply into sentences and speaking them in order keeps
playback continuous. The chunks are headerless PCM; to audition a file
outside the agent:

```bash theme={null}
ffplay -f s16le -ar 24000 -ac 1 reply.pcm
```

**4. Transcribe speech with streaming STT**

The streaming speech-to-text API transcribes a WebSocket audio stream and
emits turn events: `partial` transcripts while you speak, a `final` when
the turn ends. For a runnable version with no microphone, stream a WAV
file in real-time-sized frames. Any clip converts to the expected format
with `ffmpeg -i question.wav -ar 16000 -ac 1 question16.wav`:

```ts theme={null}
import { readFile } from "node:fs/promises";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const finals: string[] = [];
const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const transcriber = client.streaming.transcriber({
  speechModel: "universal-3-6-pro",
  encoding: "pcm_s16le", // the wire name; raw 16-bit little-endian PCM
  sampleRate: 16000,
});
transcriber.on("turn", (turn) => {
  if (turn.end_of_turn && turn.transcript) finals.push(turn.transcript);
});

await transcriber.connect();
const pcm = await readFile("question16.wav");
const frame = 3200; // ~100 ms of 16 kHz s16le mono
for (let off = 0; off < pcm.length; off += frame) {
  transcriber.sendAudio(pcm.subarray(off, off + frame));
  await new Promise((r) => setTimeout(r, 100));
}
await transcriber.close(); // terminate: finalizes the open turn
```

The terminate step is not optional polish. The turn stays open until you
terminate, so waiting for a final before closing deadlocks: send the
audio, let the trailing silence play out, then close. With a real mic
pipeline the same rule applies on every turn: finalize before the agent
starts answering.

Valid encodings are `pcm_s16le`, `pcm_mulaw`, `opus`, `ogg_opus`, and
`aac`. For `opus`, `ogg_opus`, and `aac` the stream describes its own
sample rate; for the raw PCM encodings set `sampleRate` yourself.

Streaming STT bills on connection time, not audio volume. Close sessions
when the conversation ends.

**5. Give the agent a world**

`@wazoo/tools` exposes recall tools that wrap a Worlds client. Point them
at the hosted data plane by adapting `@worlds/client`:

```ts theme={null}
import { createClient, searchWorld, sparqlWorld } from "@worlds/client";
import { createRecallTools } from "@wazoo/tools";

const client = createClient({
  baseUrl: "https://data.wazoo.dev",
  auth: process.env.WORLDS_TOKEN, // a wzw_ world token
});

const hosted = {
  search: async ({ query, topK, minScore }) => {
    const response = await searchWorld({
      client,
      path: { worldId: process.env.WORLD_ID! },
      body: { query, limit: topK, minScore },
    });
    return response.data; // { results: [...] }
  },
  sparql: async ({ query }: { query: string }) => {
    const response = await sparqlWorld({
      client,
      path: { worldId: process.env.WORLD_ID! },
      body: { query },
    });
    // The tool surfaces `data`; `kind` must not be "void".
    return { kind: "select", data: response.data };
  },
};

const recall = createRecallTools({ client: hosted });
```

> [!NOTE]
>
> The hosted beta currently runs keyword full-text search. Vector
> ranking and the hybrid flow are coming; see [Search](https://docs.wazoo.dev/worlds/search.md)
> for the current status. SPARQL is the structured recall path either
> way, so prompt the agent to prefer it for questions about relations
> (who manages what, which entities have which property) and to use
> search for fuzzy text lookups.

One behavior to know when seeding a new world: a Turtle import can
report `chunks: 0` even though the quads parsed and landed. Keyword
search reads chunks, not quads, so search comes back empty until the
index is built:

```bash theme={null}
curl -s -X POST "https://data.wazoo.dev/worlds/$WORLD_ID/reindex" \
  -H "Authorization: Bearer $WORLDS_TOKEN"
```

The response reports the chunk count once it completes. For relation
questions the agent does not need the search index at all; SPARQL
queries the quads directly.

For a zero-dependency local agent (CI, tests, demos with no network),
swap the same recall tools onto an embedded world:

```ts theme={null}
import { MemoryStore, WazooSparqlEngine } from "@wazoo/sparql-engine";
import { WorldsSdk } from "@worlds/sdk";
import { RdfjsQuadStore, RdfjsSearchIndex } from "@worlds/sdk/rdfjs";

const worlds = new WorldsSdk({
  quadStore: new RdfjsQuadStore({ store: new MemoryStore() }),
  searchIndex: new RdfjsSearchIndex(store),
  sparqlEngine: new WazooSparqlEngine({ store }),
});
await worlds.import({
  source: { kind: "serialized", data: turtle, contentType: "text/turtle" },
});

const recall = createRecallTools({ client: worlds });
```

The embedded search index matches text inside literals only, so seed
worlds for it with literal-bearing triples. The hosted world is the
default path for anything real.

**6. Run the loop**

The whole agent is one turn function and one event loop: the
transcriber's final turn triggers an LLM call with the recall tools, the
reply is split into sentences, and each sentence is spoken into the TTS
session:

```ts theme={null}
import { generateText, stepCountIs } from "ai";
import { deepseek } from "@ai-sdk/deepseek";

async function agentTurn(question: string) {
  const { text } = await generateText({
    model: deepseek("deepseek-chat"),
    system:
      "You are a voice agent answering spoken questions from a Wazoo " +
      "knowledge world. Ground answers in the tools: prefer " +
      "executeSparql for structured questions (relations, lists) and " +
      "searchWorld for fuzzy lookups. Never announce a lookup; answer " +
      "in two or three short spoken sentences. No markdown.",
    tools: recall,
    stopWhen: stepCountIs(6),
    prompt: question,
  });
  return text;
}
```

Two prompt details earn their keep with spoken replies. "Never announce a
lookup" keeps the model from speaking filler like "let me check that for
you" while it works through tool rounds, which callers hear as dead air
followed by a second answer. And limiting the tool rounds keeps latency
bounded: the loop can spend several seconds before the first sentence
exists, and every second is audible.

**7. Cut latency with retrieve-then-speak**

The tool loop is flexible but slow: every round is a full model call, and
the model may spend several rounds exploring before it speaks. Production
voice agents flip the shape: retrieve first, speak second.

* Recall runs while the user is still talking. The streaming STT API
  emits `partial` transcripts as words arrive; four words in, fire the
  retrieval pre-pass, and reuse its result if the final transcript only
  differs in spelling.

* Keyword-only search matches literal substrings, so query per meaningful
  token (stopwords stripped, two or three tokens) in parallel and merge
  the hit subjects. Filter results to `http(s)` subjects: search can
  return blank-node chunk ids, and a relative IRI inside a SPARQL
  `VALUES` clause silently breaks the whole query.

* Fetch every hit subject's triples in one batched query:

  ```sparql theme={null}
  SELECT ?s ?p ?o WHERE {
    VALUES ?s { <https://demo.wazoo.dev/ethan> <https://demo.wazoo.dev/wazoo> }
    ?s ?p ?o
  }
  ```

* Answer with no tools at all: the facts go into the system prompt, the
  model streams, and each completed sentence is spoken immediately. The
  tool loop remains as the fallback when retrieval comes up empty, plus
  a final no-tools pass that summarizes the tool results if the loop
  ran out of rounds without speaking.

Measured against a hosted world with keyword-only recall and
`deepseek-v4-flash`: first spoken word 10 to 20 seconds after the
question ends, dominated by backend search-call variance (0.7 to 8
seconds per call observed), and the first TTS audio arrives about 100
ms after each sentence is sent. Enabling hybrid recall (embeddings)
shortens the retrieval leg further.

**8. Deploy the browser side safely**

A browser cannot set an `Authorization` header on a WebSocket, and it
must never hold the API key. The TTS API accepts a short-lived token in
the `token` query parameter, minted server-side with `product=tts`, one
per connection:

```ts theme={null}
// server side, one per connection
const token = await client.tts.createTemporaryToken({
  expires_in_seconds: 60,
});
// browser side
const session = client.tts.session({ token, voiceId: "aai-jane" });
```

The same pattern applies to the streaming STT WebSocket. Never put the
API key in a URL that runs on a user's device.

## Choosing the voice providers

The guide builds on AssemblyAI's streaming APIs, but the audio legs are
swappable: both sides are WebSockets in and out, and the agent turn is decoupled
from either. When picking an STT or TTS provider for a voice agent, measure
against your own use case instead of vendor benchmarks:

* [Voice Arena](https://voicearena.com) publishes human-eval leaderboards for
  both legs: streaming and offline STT ranked by semantic WER on public and
  private test sets, and TTS ranked by Bradley-Terry Elo from blind pairwise
  votes with time-to-first-audio listed per model. The TTS board splits
  real-time-only models, which is the filter that matters for a conversational
  agent; an offline-quality model with 600 ms TTFA is the wrong choice even if
  it tops the general ranking.
* For STT, check the streaming-specific rows: a provider's batch model can lead
  the general board while its realtime model trails (this ordering exists on the
  current board too). The demo needs `end_of_turn` turn events and partial
  transcripts, so confirm those are first-class in whichever API you pick.
* For TTS, the number to compare is time to first audio, not Elo alone: the demo
  speaks each sentence as the answer stream completes it, so TTFA dominates
  perceived latency while the graph recall dominates the wait before speech
  starts.
* Both boards are human-eval based; run your own domain audio (your callers,
  your accent mix, your vocabulary) through the finalists before committing.
  Public test sets underrepresent domain terms, and the guide's whole point is
  that vocabulary (entity names, relation labels) is what your graph holds.

The demo pins AssemblyAI for the hackathon's featured-stack constraint and
because the streaming APIs expose the pieces this guide relies on (turn events,
sentence-level sessions, temporary browser tokens). The recall layer does not
change if you swap either audio provider.

## Where to take it

* Swap the question source from file to mic capture; the transcriber loop is
  identical.
* Add barge-in: stream partial transcripts during TTS playback and cancel the
  session when the speaker starts talking.
* Give the agent write access with
  [the ingest tools](https://jsr.io/@wazoo/tools) so the conversation itself
  updates the world.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.