Websockets

Mist v2 plain-text WebSocket (/ws): send text, receive raw audio bytes.

Python

import asyncio
import websockets

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws?speaker={speaker}&modelId=mistv2&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_tokens(self, websocket, message):
        for token in message:
            await websocket.send(token)

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            self.audio_data += audio

    async def run(self, message):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_tokens(websocket, message),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)

message = [
    "This ",
    "is ",
    "a ",
    "test ",
    "<CLEAR>",
    "This ",
    "is ",
    "a ",
    "sentence, ",
    "that ",
    "will ",
    "produce ",
    "audio ",
    "across ",
    "two ",
    "messages.",
    "<EOS>",
]

client = RimeClient("cove", api_key="xxx")
asyncio.run(client.run(message))

client.save_audio("output.mp3")

All requests require authentication with a bearer token in the Authorization header: Authorization: Bearer YOUR_API_KEY. See API authentication for how to create a key.

Overview

Rime’s websocket implementation accepts bare text, and responds with audio bytes of the selected format. All synthesis arguments are provided as query parameters when establishing the connection.

The WebSocket API buffers inputs up to one of the following punctuation characters: ., ,, ?, !. This is most pertinent for the initial messages sent to the API, as synthesis won’t begin until there are sufficient tokens to generate audio with natural prosody. After the first synthesis of any given utterance, typically enough time has elapsed that subsequent audio contains multiple clauses, and the buffering becomes largely invisible.

Messages

Send

The messages your client will send to the websocket API will be bare (non-serialized) text.

This will be converted to audio via websockets

Receive

The messages your client will receive will be raw audio bytes in the audio format specified at connection time.

<FF>^@^@^@9LAME3.100^AP^@^@^@^@^@^@^@^@^T<A0>$^D>"^@^@<A0>^@^@<A8><C0><BA><9D>G^N^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@

Commands

Use these commands to manipulate the stored text buffer.

<CLEAR>

This clears the current buffer. Used in the event of interruptions.

<FLUSH>

This forces whatever buffer exists, if any, to be synthesized, and the generated audio to be sent over.

<EOS>

This forces whatever buffer exists, if any, to be synthesized, and for the server to close the connection after sending the generated audio.

Variable parameters

speaker · string · required

Must be a voice from the Rime voice catalog.

text · string · required

The text you’d like spoken. Character limit per request is 1,000 via the API and in the dashboard UI.

modelId · string

Set to mistv2.

audioFormat · string

One of mp3, mulaw, or pcm

lang · string · default: "eng"

If provided, the language must match the language spoken by the selected speaker. Verify the pairing in the Rime voice catalog.

pauseBetweenBrackets · bool · default: "false"

When set to true, adds pauses between words enclosed in angle brackets. The number inside the brackets specifies the pause duration in milliseconds.
Example: “Hi. <200> I’d love to have a conversation with you.” adds a 200ms pause between the first and second sentences.

phonemizeBetweenBrackets · bool · default: "false"

When set to true, you can specify the phonemes for a word enclosed in curly brackets.
Example: “{h’El.o} World” will pronounce “Hello” as expected. Learn more about custom pronunciation.

inlineSpeedAlpha · string

Comma-separated list of speed values applied to words in square brackets. Values < 1.0 speed up speech, > 1.0 slow it down. Example: “This is [slow] and [fast]”, use “3, 0.5” to make “slow” slower and “fast” faster..

samplingRate · int

The value, if provided, must be between 4000 and 44100. Default: 22050

speedAlpha · float · default: "1.0"

Adjusts the speed of speech. Lower than 1.0 is faster and higher than 1.0 is slower.

Note: this is the legacy Mist v2 convention. Coda and Mist v3 invert it; for those, higher than 1.0 is faster.

noTextNormalization · bool · default: "false"

mist/mistv2 only. Skips text normalization of the input text prior to synthesizing audio. This will reduce latency at the cost of some possible mispronunciation of digits and abbreviations.

segment · string · default: "bySentence"

Controls how text is segmented for synthesis. Available options:

• “immediate” - Synthesizes text immediately without waiting for complete sentences
• “never” - Never segments the text, waits for explicit flush or EOS
• “bySentence” (default) - Waits for complete sentences before synthesis

Note: For backward compatibility, setting immediate=true in query params is equivalent to segment=immediate. If a null value is provided, it will default to “bySentence”.

Python

import asyncio
import websockets

class RimeClient:
    def __init__(self, speaker, api_key):
        self.url = f"wss://users-ws.rime.ai/ws?speaker={speaker}&modelId=mistv2&audioFormat=mp3"
        self.auth_headers = {
            "Authorization": f"Bearer {api_key}"
        }
        self.audio_data = b''

    async def send_tokens(self, websocket, message):
        for token in message:
            await websocket.send(token)

    async def handle_audio(self, websocket):
        while True:
            try:
                audio = await websocket.recv()
            except websockets.exceptions.ConnectionClosedOK:
                break
            self.audio_data += audio

    async def run(self, message):
        async with websockets.connect(self.url, additional_headers=self.auth_headers) as websocket:
            await asyncio.gather(
                self.send_tokens(websocket, message),
                self.handle_audio(websocket),
            )

    def save_audio(self, file_path):
        with open(file_path, 'wb') as f:
            f.write(self.audio_data)

message = [
    "This ",
    "is ",
    "a ",
    "test ",
    "<CLEAR>",
    "This ",
    "is ",
    "a ",
    "sentence, ",
    "that ",
    "will ",
    "produce ",
    "audio ",
    "across ",
    "two ",
    "messages.",
    "<EOS>",
]

client = RimeClient("cove", api_key="xxx")
asyncio.run(client.run(message))

client.save_audio("output.mp3")