Websockets JSON

Mist v2 JSON WebSocket (/ws2): structured events with base64 audio chunks and word-level timestamps.

Python

import asyncio
import json
import websockets
import base64

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws2?speaker={speaker}&modelId=mistv2&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_messages(self, websocket, messages):
        for message in messages:
            await websocket.send(json.dumps(message))

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            message = json.loads(audio)

            if message['type'] == 'chunk':
              self.audio_data += base64.b64decode(message['data'])

            if message['type'] == 'timestamps':
                print("Rime model pronounced the words...\n")
                for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
                    print(f"'{w}' at time {t}")

    async def run(self, messages):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_messages(websocket, messages),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)
        print(f"\n Audio saved at {file_path}")


message = [
    {"text": "This "},
    {"text": "is "},
    {"text": "a "},
    {"text": "test "},
    {"operation":"clear"},
    {"text": "This "},
    {"text": "is "},
    {"text": "an "},
    {"text": "incomplete "},
    {"text": "sentence "},
    {"operation": "eos"},
]

client = RimeClient("cove", api_key="xxx")
asyncio.run(client.run(message))

client.save_audio("output.mp3")

The Rime API authenticates every request with a bearer token in the Authorization header: Authorization: Bearer YOUR_API_KEY. See API authentication for how to create a key.

Overview

This WebSocket sends and receives JSON messages. Compared with the plain-text WebSocket, it adds word-level timestamps, a done event after each synthesis, and contextId tagging. You pass every synthesis parameter as a query parameter when you open the connection, and those settings apply for the life of the connection.

The API buffers input until it detects a sentence boundary, typically at ., ?, or !, so synthesis starts once it has enough text for natural prosody. The wait is most noticeable on the first sentence. After that, later text usually arrives while earlier audio is still playing, and the buffering becomes largely invisible. To change when synthesis starts, set segment.

Messages

Send

Text

Sends text for synthesis. This is the message your client sends most often.

schema:

type TextMessage = {
  text: string,
  contextId?: string,
}

examples:

{
    "text": "this is the minimum text message."
}

{
    "text": "this is a text message with a context id.",
    "contextId": "159495B1-5C81-4C73-A51A-9CE10A08239E"
}

If you set contextId, the server attaches it to the events it sends back for that text. The server tracks one context ID at a time, and each event carries the most recent context ID at the time its audio was requested. In the examples above, even if the server receives both messages before it sends any audio, the first sentence’s audio is tagged contextId: null and the second sentence’s audio is tagged with its UUID.

Clear

Discards buffered text that hasn’t been synthesized yet. Send it when the user interrupts. It doesn’t cancel a synthesis already in progress, so stop playback in your client and drop any audio that still arrives.

{ "operation": "clear" }

Flush

Synthesizes whatever text is buffered, without waiting for a sentence boundary, and sends the audio.

{ "operation": "flush" }

EOS

Synthesizes whatever text is buffered, sends the audio, and then the server closes the connection.

{ "operation" : "eos" }

Receive

Chunk

Carries a piece of the synthesized audio. This is the event your client receives most often.

type Base64String = string

type AudioChunkEvent = {
  type: "chunk",
  data: Base64String,
  contextId: string | null,
}

data holds base64-encoded audio in the audioFormat set at connection time. Decode each chunk and append the bytes in order to rebuild the audio. If the text that produced this audio had a contextId, the event includes it.

Timestamps

The server sends word-level timestamps alongside the audio chunks, so your client can tell which words have been spoken at any point. When the user talks over the output, map the playback position back to the last word they actually heard.

Timestamps are emitted only when lang is en/eng or es/spa, or when lang is omitted. Requests in any other language receive chunk and done events with no timestamps event and no error. Don’t block playback while waiting for a timestamps event.

type TimestampsEvent = {
  type: "timestamps",
  word_timestamps: {
    words: string[],
    start: number[],
    end: number[],
  },
  contextId: string | null,
}

The three arrays inside word_timestamps are the same length and index-aligned: for a given index i, words[i] is spoken from start[i] to end[i]. Times are in seconds, measured from the beginning of the audio for the current synthesis. If a context id was attached to the text that produced this audio, it is included on the event.

Example payload:

{
  "type": "timestamps",
  "word_timestamps": {
    "words": ["Hello", "from", "a", "timestamps", "probe", "."],
    "start": [0, 0.35991, 0.51084, 0.59211, 1.10295, 1.41642],
    "end":   [0.35991, 0.51084, 0.59211, 1.10295, 1.41642, 1.5093]
  },
  "contextId": null
}

Done

After the last audio chunk for a synthesis batch has been sent, the server emits a done event. This signals that the current synthesis is fully complete. If the client sends more text and triggers further synthesis, another done will follow. The eos operation emits a final done for any remaining buffered content before the server closes the connection. If the buffer is already empty, the server closes without sending a done.

type DoneEvent = {
  type: "done",
  done: true,
  contextId: string | null,
}

On /ws2, the done event carries an extra boolean done: true field in addition to type: "done". On /ws3 the event contains only type and contextId. Match on type === "done" to handle both.

When exactly done fires depends on the segment setting. See Segmentation and behavior settings for full details.

Error

If a message is malformed or unexpected, the server responds immediately with an error event. The server does not close the connection and keeps accepting well-formed messages, so your client decides whether to close it.

type ErrorEvent = {
  type: "error",
  message: string,
}

Variable parameters

speaker · string · required

Must be a voice from the Rime voice catalog.

modelId · string

Set to mistv2.

audioFormat · string

One of mp3, mulaw, or pcm

lang · string · default: "eng"

If provided, the language must match the language spoken by the selected speaker. Verify the pairing in the Rime voice catalog.

pauseBetweenBrackets · bool · default: "false"

When set to true, adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds.
Example: “Hi. <200> I’d love to have a conversation with you.” adds a 200ms pause between the first and second sentences.

phonemizeBetweenBrackets · bool · default: "false"

When set to true, you can specify the phonemes for a word enclosed in curly brackets.
Example: “{h’El.o} World” pronounces “Hello” with the phonemes you supplied. Learn more about custom pronunciation.

inlineSpeedAlpha · string

Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: “This is [slow] and [fast]”, use “3, 0.5” to make “slow” slower and “fast” faster.

samplingRate · int

The value, if provided, must be between 4000 and 44100. Default: 22050

speedAlpha · float · default: "1.0"

Adjusts the speed of speech. Lower than 1.0 is faster and higher than 1.0 is slower.

Note: this is the legacy Mist v2 convention. Coda and Mist v3 invert it, so for those models higher than 1.0 is faster.

noTextNormalization · bool · default: "false"

mist/mistv2 only. Skips text normalization before synthesis. This reduces latency, but digits and abbreviations reach the model unexpanded and may be mispronounced.

segment · string · default: "bySentence"

Controls how text is segmented for synthesis. Available options:

• “immediate” - Synthesizes text immediately without waiting for complete sentences
• “never” - Never segments the text, waits for explicit flush or EOS
• “bySentence” (default) - Waits for complete sentences before synthesis

Note: For backward compatibility, setting immediate=true in query params is equivalent to segment=immediate. If a null value is provided, it will default to “bySentence”.

Python

import asyncio
import json
import websockets
import base64

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws2?speaker={speaker}&modelId=mistv2&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_messages(self, websocket, messages):
        for message in messages:
            await websocket.send(json.dumps(message))

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            message = json.loads(audio)

            if message['type'] == 'chunk':
              self.audio_data += base64.b64decode(message['data'])

            if message['type'] == 'timestamps':
                print("Rime model pronounced the words...\n")
                for w, t in zip(message['word_timestamps']['words'], message['word_timestamps']['start']):
                    print(f"'{w}' at time {t}")

    async def run(self, messages):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_messages(websocket, messages),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)
        print(f"\n Audio saved at {file_path}")


message = [
    {"text": "This "},
    {"text": "is "},
    {"text": "a "},
    {"text": "test "},
    {"operation":"clear"},
    {"text": "This "},
    {"text": "is "},
    {"text": "an "},
    {"text": "incomplete "},
    {"text": "sentence "},
    {"operation": "eos"},
]

client = RimeClient("cove", api_key="xxx")
asyncio.run(client.run(message))

client.save_audio("output.mp3")