Build a voice agent (FastAPI)

Build a minimal voice-agent loop in FastAPI: the browser captures speech, your backend decides what to say, and Rime speaks the reply. This starter uses browser speech recognition and a stub response function; replace both with production components. Keep RIME_API_KEY on the server and call Rime over HTTPS with httpx. No Rime-specific PyPI package is required.

Browser (mic + playback)  →  POST /api/tts (FastAPI)  →  POST https://users.rime.ai/v1/rime-tts

Prerequisites

A Rime API key from the API Tokens page, exported as RIME_API_KEY
Python 3.10+ with pip install fastapi uvicorn httpx

1. Server: FastAPI app with a TTS endpoint

Create main.py:

main.py

import os

import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel

app = FastAPI()

RIME_API_KEY = os.environ["RIME_API_KEY"]
RIME_TTS_URL = "https://users.rime.ai/v1/rime-tts"


class TTSRequest(BaseModel):
    text: str
    speaker: str = "astra"
    modelId: str = "coda"


@app.post("/api/tts")
async def tts(req: TTSRequest) -> Response:
    async with httpx.AsyncClient(timeout=30.0) as client:
        rime_res = await client.post(
            RIME_TTS_URL,
            headers={
                "Authorization": f"Bearer {RIME_API_KEY}",
                "Content-Type": "application/json",
                # MP3; see the API reference for Opus, WAV, PCM, μ-law
                "Accept": "audio/mpeg",
            },
            json=req.model_dump(),
        )

    if rime_res.status_code != 200:
        raise HTTPException(
            status_code=rime_res.status_code,
            detail=f"Rime API error: {rime_res.text}",
        )

    return Response(content=rime_res.content, media_type="audio/mpeg")


@app.get("/")
async def index() -> FileResponse:
    return FileResponse("index.html")

Verify it works before touching the frontend:

uvicorn main:app --reload --port 8000
curl -X POST http://localhost:8000/api/tts \
  -H 'Content-Type: application/json' \
  -d '{"text": "Hello from my FastAPI voice agent."}' \
  --output test.mp3

test.mp3 should be playable audio. A KeyError: 'RIME_API_KEY' at startup means the environment variable isn’t set.

2. Client: mic in, Rime audio out

Create index.html next to main.py. It uses the browser’s built-in SpeechRecognition for input (Chrome/Edge/Safari), a stub respond() function as the agent brain, and your /api/tts endpoint for the voice:

index.html

<!doctype html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Rime voice agent</title>
    <style>
      main { max-width: 480px; margin: 4rem auto; font-family: sans-serif; }
    </style>
  </head>
  <body>
    <main>
      <h1>Rime voice agent</h1>
      <button id="speak">🎤 Speak</button>
      <ul id="messages"></ul>
      <audio id="player" hidden></audio>
    </main>
    <script>
      // Replace this stub with a call to your LLM of choice.
      function respond(userText) {
        return `You said: ${userText}. This voice is Rime's Coda model.`;
      }

      async function speak(text) {
        const res = await fetch("/api/tts", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ text }),
        });
        if (!res.ok) throw new Error(`TTS failed: ${res.status}`);
        const player = document.getElementById("player");
        player.src = URL.createObjectURL(await res.blob());
        await player.play();
      }

      function log(line) {
        const li = document.createElement("li");
        li.textContent = line;
        document.getElementById("messages").appendChild(li);
      }

      document.getElementById("speak").addEventListener("click", () => {
        const SR = window.SpeechRecognition ?? window.webkitSpeechRecognition;
        if (!SR) return alert("SpeechRecognition not supported in this browser.");
        const rec = new SR();
        rec.onresult = async (e) => {
          const userText = e.results[0][0].transcript;
          const reply = respond(userText);
          log(`You: ${userText}`);
          log(`Agent: ${reply}`);
          await speak(reply);
        };
        rec.start();
      });
    </script>
  </body>
</html>

Open http://localhost:8000, select Speak, say something, and the agent answers in Rime’s astra voice.

Add streaming when latency matters

For incremental synthesis, connect your server to wss://users-ws.rime.ai/ws3 with the Python websockets library and relay events through FastAPI’s WebSocket support. Audio can then start while later sentences are still generating. Browser WebSockets cannot send the required Authorization header, so the Rime connection stays server-side. The Coda WebSocket reference includes runnable Python code and the full /ws3 schema.

Production building blocks

WebSocket API overview

Endpoints, word-level timestamps, context IDs, and interruption handling.

Voices

Swap astra for any Coda voice. Coda covers eight languages, and each voice serves one of them.

Streaming formats

Choose between Opus, MP3, WAV, PCM, and μ-law for your latency budget.

Pipecat

Building a full Python voice pipeline? Use the ready-made Rime plugin.

Overview

Quickstart

Voice agents

Voices & models

Customizing speech

Streaming & performance

Integrations

On-prem

Account & updates

Build a voice agent (FastAPI)

Prerequisites

1. Server: FastAPI app with a TTS endpoint

2. Client: mic in, Rime audio out

Add streaming when latency matters

Production building blocks

WebSocket API overview

Voices

Streaming formats

Pipecat

​Prerequisites

​1. Server: FastAPI app with a TTS endpoint

​2. Client: mic in, Rime audio out

​Add streaming when latency matters

​Production building blocks

WebSocket API overview

Voices

Streaming formats

Pipecat

Prerequisites

1. Server: FastAPI app with a TTS endpoint

2. Client: mic in, Rime audio out

Add streaming when latency matters

Production building blocks