Build a voice agent (Next.js)

Build a minimal voice-agent loop in Next.js while a server-side route protects your Rime API key.

Build a minimal voice-agent loop in Next.js: the browser captures speech, a server route decides what to say, and Rime speaks the reply. This starter uses browser speech recognition and a stub response function; replace both with production components.

Keep RIME_API_KEY on the server. Call Rime over HTTPS with fetch; no Rime-specific npm package is required. The optional WebSocket bridge adds the only new dependency, ws.

Architecture: the browser never talks to Rime directly. Your API key stays server-side, and the browser calls your own /api/tts route:

Browser (mic + playback)  →  POST /api/tts (Next.js route)  →  POST https://users.rime.ai/v1/rime-tts

Prerequisites

• A Rime API key from the API Tokens page. Put it in .env.local as RIME_API_KEY=...
• A Next.js 14+ project using the App Router (npx create-next-app@latest defaults work)

1. Server: the TTS proxy route

Create app/api/tts/route.ts (or src/app/api/tts/route.ts if your project uses src/):

app/api/tts/route.ts

export async function POST(req: Request) {
  const { text, speaker = "astra", modelId = "coda" } = await req.json();

  if (!text) {
    return Response.json({ error: "text is required" }, { status: 400 });
  }

  const rimeRes = await fetch("https://users.rime.ai/v1/rime-tts", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.RIME_API_KEY}`,
      "Content-Type": "application/json",
      Accept: "audio/mpeg", // MP3; see the API reference for Opus, WAV, PCM, μ-law
    },
    body: JSON.stringify({ text, speaker, modelId }),
  });

  if (!rimeRes.ok) {
    const detail = await rimeRes.text();
    return Response.json(
      { error: `Rime API error ${rimeRes.status}`, detail },
      { status: rimeRes.status },
    );
  }

  // Stream the audio straight through to the browser
  return new Response(rimeRes.body, {
    headers: { "Content-Type": "audio/mpeg" },
  });
}

Verify it works before touching the frontend:

npm run dev
curl -X POST http://localhost:3000/api/tts \
  -H 'Content-Type: application/json' \
  -d '{"text": "Hello from my Next.js voice agent."}' \
  --output test.mp3

test.mp3 should be playable audio. If you get a 401, check that RIME_API_KEY is set in .env.local and the dev server was restarted after adding it.

2. Client: mic in, Rime audio out

Create app/voice-agent/page.tsx. It uses the browser’s built-in SpeechRecognition for input (Chrome/Edge/Safari), a stub respond() function as the agent brain, and your /api/tts route for the voice:

app/voice-agent/page.tsx

"use client";

import { useRef, useState } from "react";

// Replace this stub with a call to your LLM of choice.
function respond(userText: string): string {
  return `You said: ${userText}. This voice is Rime's Coda model.`;
}

export default function VoiceAgent() {
  const [messages, setMessages] = useState<string[]>([]);
  const [listening, setListening] = useState(false);
  const audioRef = useRef<HTMLAudioElement>(null);

  async function speak(text: string) {
    const res = await fetch("/api/tts", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text }),
    });
    if (!res.ok) throw new Error(`TTS failed: ${res.status}`);
    const url = URL.createObjectURL(await res.blob());
    audioRef.current!.src = url;
    await audioRef.current!.play();
  }

  async function handleUserText(userText: string) {
    const reply = respond(userText);
    setMessages((m) => [...m, `You: ${userText}`, `Agent: ${reply}`]);
    await speak(reply);
  }

  function listen() {
    const SR =
      (window as any).SpeechRecognition ??
      (window as any).webkitSpeechRecognition;
    if (!SR) return alert("SpeechRecognition not supported in this browser.");
    const rec = new SR();
    rec.onresult = (e: any) => handleUserText(e.results[0][0].transcript);
    rec.onend = () => setListening(false);
    setListening(true);
    rec.start();
  }

  return (
    <main style={{ maxWidth: 480, margin: "4rem auto", fontFamily: "sans-serif" }}>
      <h1>Rime voice agent</h1>
      <button onClick={listen} disabled={listening}>
        {listening ? "Listening…" : "🎤 Speak"}
      </button>
      <ul>{messages.map((m, i) => <li key={i}>{m}</li>)}</ul>
      <audio ref={audioRef} hidden />
    </main>
  );
}

Open http://localhost:3000/voice-agent, select Speak, say something, and the agent answers in Rime’s astra voice.

3. Stream over WebSockets when latency matters

The HTTP route above is the right default for most apps. Switch to WebSockets for incremental synthesis, where audio starts while the rest of the sentence is still being generated, or for barge-in handling with word-level timestamps.

Two things make the WebSocket setup different:

1. Rime’s WebSocket endpoints authenticate with an Authorization header, which browser WebSockets cannot send. The bridge must live on your server, where it also keeps the API key off the client.
2. Next.js route handlers can’t hold WebSocket connections, so you need a small custom server.

Browser WS (/ws/tts)  →  server.mjs bridge  →  wss://users-ws.rime.ai/ws3  (Authorization header)

Install ws, then create server.mjs in the project root:

npm install ws

server.mjs

import { createServer } from "node:http";
import next from "next";
import { WebSocketServer, WebSocket } from "ws";

const app = next({ dev: process.env.NODE_ENV !== "production" });
await app.prepare();
const handle = app.getRequestHandler();

const server = createServer(handle);
const wss = new WebSocketServer({ server, path: "/ws/tts" });

wss.on("connection", (client) => {
  // One upstream Rime connection per browser client.
  // speaker/modelId/audioFormat are query params; auth is a header.
  const rime = new WebSocket(
    "wss://users-ws.rime.ai/ws3?speaker=astra&modelId=coda&audioFormat=mp3",
    { headers: { Authorization: `Bearer ${process.env.RIME_API_KEY}` } },
  );

  // Browser → Rime: {text: "..."} to speak, {operation: "clear"} to interrupt.
  // Queue anything that arrives before the upstream connection finishes opening;
  // the browser side is usually connected first, and ws drops sends on a
  // CONNECTING socket.
  const pending = [];
  client.on("message", (data) => {
    const msg = data.toString();
    if (rime.readyState === WebSocket.OPEN) rime.send(msg);
    else pending.push(msg);
  });
  rime.on("open", () => {
    for (const msg of pending) rime.send(msg);
    pending.length = 0;
  });

  // Rime → browser: {type: "chunk", data: <base64 mp3>}, {type: "timestamps", ...},
  // {type: "done"}. Relay verbatim and let the client decode.
  rime.on("message", (data) => {
    if (client.readyState === WebSocket.OPEN) client.send(data.toString());
  });

  const closeBoth = () => { rime.close(); client.close(); };
  client.on("close", closeBoth);
  rime.on("close", closeBoth);
  rime.on("error", (err) => {
    client.send(JSON.stringify({ type: "error", message: String(err) }));
  });
});

server.listen(3000, () => {
  console.log("Next.js + Rime WS bridge on http://localhost:3000");
});

Point your dev/start scripts at it:

package.json

{
  "scripts": {
    "dev": "node server.mjs",
    "build": "next build",
    "start": "NODE_ENV=production node server.mjs"
  }
}

On the client, connect to your bridge and play chunks as they arrive:

const ws = new WebSocket(`ws://${location.host}/ws/tts`);

// Collect base64 MP3 chunks per utterance, play on "done".
// (For true chunk-by-chunk playback, feed them to a MediaSource instead.)
let chunks: Uint8Array[] = [];

ws.onmessage = async (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === "chunk") {
    chunks.push(Uint8Array.from(atob(msg.data), (c) => c.charCodeAt(0)));
  }
  if (msg.type === "done") {
    const url = URL.createObjectURL(new Blob(chunks, { type: "audio/mpeg" }));
    chunks = [];
    await new Audio(url).play();
  }
};

// Speak: sentences synthesize as they complete (segment=bySentence default)
ws.onopen = () => ws.send(JSON.stringify({ text: "Hello from Rime over WebSockets." }));

// Interrupt (barge-in): drop whatever is still buffered server-side
function interrupt() {
  ws.send(JSON.stringify({ operation: "clear" }));
  chunks = [];
}

The Coda WebSocket reference defines the full /ws3 schema, including chunk, timestamps, done, and error events plus the flush, clear, and eos operations. The WebSocket overview covers segmentation and interruption patterns.

Production building blocks

WebSocket API overview

Endpoints, word-level timestamps, context IDs, and interruption handling.

Voices

Swap astra for any Coda voice. Coda covers eight languages, and each voice serves one of them.

Streaming formats

Choose between Opus, MP3, WAV, PCM, and μ-law for your latency budget.

LiveKit & Pipecat

Use the ready-made Rime plugins when a framework should own transport and orchestration.