Build a voice agent (FastAPI)

Build a minimal voice-agent loop in FastAPI while a server-side endpoint protects your Rime API key.

Build a minimal voice-agent loop in FastAPI: the browser captures speech, your backend decides what to say, and Rime speaks the reply. This starter uses browser speech recognition and a stub response function; replace both with production components.

Keep RIME_API_KEY on the server and call Rime over HTTPS with httpx. No Rime-specific PyPI package is required.

Browser (mic + playback)  →  POST /api/tts (FastAPI)  →  POST https://coda.api.rime.ai/

Prerequisites

• A Rime API key from the API Tokens page, exported as RIME_API_KEY
• Python 3.10+ with pip install fastapi uvicorn httpx

1. Server: FastAPI app with a TTS endpoint

Create main.py:

main.py

import os

import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel

app = FastAPI()

RIME_API_KEY = os.environ["RIME_API_KEY"]
# Each model has its own host.
RIME_TTS_URLS = {"coda": "https://coda.api.rime.ai/", "mistv3": "https://mist.api.rime.ai/"}


class TTSRequest(BaseModel):
    text: str
    speaker: str = "astra"
    modelId: str = "coda"


@app.post("/api/tts")
async def tts(req: TTSRequest) -> Response:
    async with httpx.AsyncClient(timeout=30.0) as client:
        rime_res = await client.post(
            RIME_TTS_URLS.get(req.modelId, RIME_TTS_URLS["coda"]),
            headers={
                "Authorization": f"Bearer {RIME_API_KEY}",
                "Content-Type": "application/json",
                # MP3; see the API reference for Opus, WAV, PCM, μ-law
                "Accept": "audio/mpeg",
            },
            json={"text": req.text, "speaker": req.speaker},
        )

    if rime_res.status_code != 200:
        raise HTTPException(
            status_code=rime_res.status_code,
            detail=f"Rime API error: {rime_res.text}",
        )

    return Response(content=rime_res.content, media_type="audio/mpeg")


@app.get("/")
async def index() -> FileResponse:
    return FileResponse("index.html")

Verify it works before touching the frontend:

uvicorn main:app --reload --port 8000
curl -X POST http://localhost:8000/api/tts \
  -H 'Content-Type: application/json' \
  -d '{"text": "Hello from my FastAPI voice agent."}' \
  --output test.mp3

test.mp3 should be playable audio. A KeyError: 'RIME_API_KEY' at startup means the environment variable isn’t set.

2. Client: mic in, Rime audio out

Create index.html next to main.py. It uses the browser’s built-in SpeechRecognition for input (Chrome/Edge/Safari), a stub respond() function as the agent brain, and your /api/tts endpoint for the voice:

index.html

<!doctype html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Rime voice agent</title>
    <style>
      main { max-width: 480px; margin: 4rem auto; font-family: sans-serif; }
    </style>
  </head>
  <body>
    <main>
      <h1>Rime voice agent</h1>
      <button id="speak">🎤 Speak</button>
      <ul id="messages"></ul>
      <audio id="player" hidden></audio>
    </main>
    <script>
      // Replace this stub with a call to your LLM of choice.
      function respond(userText) {
        return `You said: ${userText}. This voice is Rime's Coda model.`;
      }

      async function speak(text) {
        const res = await fetch("/api/tts", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ text }),
        });
        if (!res.ok) throw new Error(`TTS failed: ${res.status}`);
        const player = document.getElementById("player");
        player.src = URL.createObjectURL(await res.blob());
        await player.play();
      }

      function log(line) {
        const li = document.createElement("li");
        li.textContent = line;
        document.getElementById("messages").appendChild(li);
      }

      document.getElementById("speak").addEventListener("click", () => {
        const SR = window.SpeechRecognition ?? window.webkitSpeechRecognition;
        if (!SR) return alert("SpeechRecognition not supported in this browser.");
        const rec = new SR();
        rec.onresult = async (e) => {
          const userText = e.results[0][0].transcript;
          const reply = respond(userText);
          log(`You: ${userText}`);
          log(`Agent: ${reply}`);
          await speak(reply);
        };
        rec.start();
      });
    </script>
  </body>
</html>

Open http://localhost:8000, select Speak, say something, and the agent answers in Rime’s astra voice.

Add streaming when latency matters

Legacy Coda /ws3. Use this section only for an existing /ws3 integration. For new clients, use the Coda WebSocket v1 JSON quickstart. Coda WebSocket v1 uses a different message format and does not return word timestamps.

For an existing /ws3 integration, connect your server to wss://users-ws.rime.ai/ws3 with the Python websockets library and relay events through FastAPI’s WebSocket support. Audio can then start while later sentences are still generating. Browser WebSockets cannot send the required Authorization header, so the Rime connection stays server-side.

Production building blocks

WebSocket API overview

Compare Coda WebSocket v1 with the legacy endpoints, their message formats, and their synthesis controls.

Voices

Swap astra for any Coda voice. Coda covers ten languages, and each voice serves one of them.

Streaming formats

Choose between Opus, MP3, WAV, PCM, and μ-law for your latency budget.

Pipecat

Building a full Python voice pipeline? Use the ready-made Rime plugin.