Websockets JSON

Mist v3 JSON WebSocket (/ws3): structured events with base64 audio chunks and word-level timestamps.

Python

import asyncio
import json
import websockets
import base64

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws3?speaker={speaker}&modelId=mistv3&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_messages(self, websocket, messages):
        for message in messages:
            await websocket.send(json.dumps(message))

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            message = json.loads(audio)

            if message['type'] == 'chunk':
              self.audio_data += base64.b64decode(message['data'])

            if message['type'] == 'timestamps':
                print("Rime model pronounced the words...\n")
                for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
                    print(f"'{w}' at time {t}")

    async def run(self, messages):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_messages(websocket, messages),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)
        print(f"\n Audio saved at {file_path}")


message = [
    {"text": "This "},
    {"text": "is "},
    {"text": "an "},
    {"text": "incomplete "},
    {"text": "sentence "},
    {"operation": "eos"},
]

client = RimeClient("cove", api_key="YOUR_API_KEY")
asyncio.run(client.run(message))

client.save_audio("output.mp3")

The Rime API authenticates every request with a bearer token in the Authorization header: Authorization: Bearer YOUR_API_KEY. See API authentication for how to create a key.

Overview

This WebSocket sends and receives JSON messages. Compared with the plain-text WebSocket, it adds word-level timestamps, a done event after each synthesis, and contextId tagging. You pass every synthesis parameter as a query parameter when you open the connection, and those settings apply for the life of the connection.

The API buffers input until it detects a sentence boundary, typically at ., ?, or !, so synthesis starts once it has enough text for natural prosody. The wait is most noticeable on the first sentence. After that, later text usually arrives while earlier audio is still playing, and the buffering becomes largely invisible. To change when synthesis starts, set segment.

Messages

Send

Text

Sends text for synthesis. This is the message your client sends most often.

schema:

type TextMessage = {
  text: string,
  contextId?: string,
}

examples:

{
    "text": "this is the minimum text message."
}

{
    "text": "this is a text message with a context id.",
    "contextId": "159495B1-5C81-4C73-A51A-9CE10A08239E"
}

If you set contextId, the server attaches it to the events it sends back for that text. The server tracks one context ID at a time.

Clear

Discards buffered text that hasn’t been synthesized yet. Send it when the user interrupts. It doesn’t cancel a synthesis already in progress, so stop playback in your client and drop any audio that still arrives.

{ "operation": "clear" }

Flush

Synthesizes whatever text is buffered, without waiting for a sentence boundary, and sends the audio.

{ "operation": "flush" }

EOS

Synthesizes whatever text is buffered, sends the audio, and then the server closes the connection.

{ "operation" : "eos" }

Receive

Chunk

Carries a piece of the synthesized audio. This is the event your client receives most often.

type Base64String = string

type AudioChunkEvent = {
  type: "chunk",
  data: Base64String,
  contextId: string | null,
}

data holds base64-encoded audio in the audioFormat set at connection time. Decode each chunk and append the bytes in order to rebuild the audio. If the text that produced this audio had a contextId, the event includes it.

Timestamps

The server sends word-level timestamps alongside the audio chunks, so your client can tell which words have been spoken at any point. The server estimates them from the text rather than measuring the audio. Treat them as approximate. When the user talks over the output, map the playback position back to the last word they actually heard.

Timestamps are emitted only when lang is en/eng or es/spa, or when lang is omitted. Requests in any other language receive chunk and done events with no timestamps event and no error (this includes fr and de). Don’t block playback while waiting for a timestamps event.

type TimestampsEvent = {
  type: "timestamps",
  word_timestamps: {
    words: string[],
    start: number[],
    end: number[],
  },
  contextId: string | null,
}

The three arrays inside word_timestamps are the same length and index-aligned: for a given index i, words[i] is spoken from start[i] to end[i]. Times are in seconds, measured from the beginning of the audio for the current synthesis. If a context id was attached to the text that produced this audio, it is included on the event.

Example payload:

{
  "type": "timestamps",
  "word_timestamps": {
    "words": ["Testing", "mistv3", "timestamps."],
    "start": [0, 0.35396, 1.41584],
    "end":   [0.35396, 1.41584, 3.18564]
  },
  "contextId": null
}

Done

After the last audio chunk for a synthesis batch has been sent, the server emits a done event. This signals that the current synthesis is fully complete. If the client sends more text and triggers further synthesis, another done will follow.

type DoneEvent = {
  type: "done",
  contextId: string | null,
}

When exactly done fires depends on the segment setting. See Segmentation and behavior settings for full details.

Error

If a message is malformed or unexpected, the server responds immediately with an error event. The server does not close the connection and keeps accepting well-formed messages, so your client decides whether to close it.

type ErrorEvent = {
  type: "error",
  message: string,
}

Variable parameters

speaker · string · required

Must be a voice from the Rime voice catalog.

modelId · string · default: "mistv3"

Set to mistv3.

audioFormat · string

One of wav, mp3 (or mpeg), ogg (Opus in OGG), webm (Opus in WebM), pcm (or l16), or mulaw (or pcmu). Unrecognized values fall back to wav.

lang · string · default: "eng"

If provided, the language must match the language spoken by the selected speaker. Verify the pairing in the Rime voice catalog.

pauseBetweenBrackets · bool · default: "false"

When set to true, adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds. Example: Hi. <200> I'd love to have a conversation with you. adds a 200ms pause. Learn more about custom pauses.

samplingRate · int

The sampling rate in Hz. Default: 24000. Values above 24000 are upsampled.

inlineSpeedAlpha · string

Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: “This is [slow] and [fast]”, use “3, 0.5” to make “slow” slower and “fast” faster.

speedAlpha · float · default: "1.0"

Adjusts the speed of speech. Higher than 1.0 is faster and lower than 1.0 is slower.

segment · string · default: "bySentence"

Controls how text is segmented for synthesis. Available options:

• “immediate” - Synthesizes text immediately without waiting for complete sentences
• “never” - Never segments the text, waits for explicit flush or EOS
• “bySentence” (default) - Waits for complete sentences before synthesis

Python

import asyncio
import json
import websockets
import base64

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws3?speaker={speaker}&modelId=mistv3&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_messages(self, websocket, messages):
        for message in messages:
            await websocket.send(json.dumps(message))

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            message = json.loads(audio)

            if message['type'] == 'chunk':
              self.audio_data += base64.b64decode(message['data'])

            if message['type'] == 'timestamps':
                print("Rime model pronounced the words...\n")
                for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
                    print(f"'{w}' at time {t}")

    async def run(self, messages):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_messages(websocket, messages),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)
        print(f"\n Audio saved at {file_path}")


message = [
    {"text": "This "},
    {"text": "is "},
    {"text": "an "},
    {"text": "incomplete "},
    {"text": "sentence "},
    {"operation": "eos"},
]

client = RimeClient("cove", api_key="YOUR_API_KEY")
asyncio.run(client.run(message))

client.save_audio("output.mp3")