LiveKit Quickstart

Build a real-time voice agent in your browser using LiveKit Agents + Rime TTS.

Build a working browser voice agent with LiveKit transport, OpenAI transcription and responses, and Rime speech. The completed agent handles turn-taking, speech recognition, agent logic, and streaming TTS.

Step 1: Prerequisites

1.1 Rime API key

Sign up for a Rime account and copy your API key from the API Tokens page. This enables access to the Rime API for text-to-speech (TTS).

1.2 OpenAI API key

Create an OpenAI account and generate an API key from the API keys page. This key enables speech-to-text (STT) and LLM responses.

1.3 LiveKit Cloud

Create a LiveKit Cloud account for real-time audio transport:

1. Create a new project called rime-agent
2. Go to Settings → API keys → Create key
3. Copy your WebSocket URL, API key, and API secret. You need all three.

1.4 Python

Install Python 3.10 or later. Verify your installation by running python --version in your terminal.

LiveKit terms and architecture

• A Room is a virtual space where participants connect and share media in real time.
• An Agent is a server-side participant that processes media streams and interacts with users.
• LiveKit Cloud transports real-time audio between participants.

The agent sends user audio through LiveKit to OpenAI for transcription and response generation, then returns Rime speech through the same room:

Step 2: Project setup

2.1 Create the project folder

Create a new folder for your project and navigate into it:

mkdir rime-voice-agent
cd rime-voice-agent

2.2 Set up environment variables

In the new directory, create a file called .env and add the keys that you created in Step 1:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
OPENAI_API_KEY=your_openai_api_key
RIME_API_KEY=your_rime_api_key

Replace the placeholder values with your actual API keys and credentials.

2.3 Configure dependencies

Create a virtual environment and activate it:

macOS/Linux

python -m venv venv
source venv/bin/activate

Next, install the uv package manager:

pip install uv

This helps with LiveKit’s complex dependencies.

Create a file called pyproject.toml and add the following dependencies:

[project]
name = "rime-voice-agent"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "livekit-agents[openai,rime,silero,turn-detector]~=1.6",
    "livekit-plugins-noise-cancellation~=0.2",
    "python-dotenv>=1.1.1",
]

These packages add the following:

• openai plugin: Connects to OpenAI for STT and LLM
• rime plugin: Connects to Rime for text-to-speech
• silero plugin: Voice Activity Detection (knows when you start/stop speaking)
• turn-detector plugin: Detects when you’ve finished your turn in the conversation
• noise-cancellation plugin: Filters out background noise
• python-dotenv: Loads your API keys from the .env file

Then, install the dependencies by running:

uv sync

Step 3: Create the agent

Create a file called agent.py. To run the agent without building it up piece by piece, skip to Step 3.6: Full agent code.

3.1 Load environment variables

Add the following imports and initialization code to agent.py:

from dotenv import load_dotenv

load_dotenv()

This loads the API keys from your .env file so they’re available throughout the application. Don’t skip this step: the LiveKit agents CLI doesn’t load .env files automatically, so without load_dotenv() the worker won’t find your LiveKit and API credentials.

3.2 Define an Agent class

Add the following class definition to agent.py:

class Assistant(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are a helpful voice assistant. 
            Keep your responses short and conversational - no more than 2-3 sentences.
            Be friendly and natural."""
        )

Assistant extends the Agent base class and sets the agent’s personality through a system prompt. Step 5.2 shows a longer prompt kept in a separate file.

Import the Agent class at the top of agent.py, along with the rest of the required imports:

from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentSession,
    AutoSubscribe,
    JobContext,
    JobProcess,
    RoomInputOptions,
    WorkerOptions,
    cli,
)
from livekit.plugins import openai, noise_cancellation, rime, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

These provide access to the LiveKit Agents SDK and the plugins for STT, LLM, TTS, and VAD (voice activity detection).

3.3 Code the conversation pipeline

Add the following entrypoint function to agent.py:

async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
    await ctx.wait_for_participant()

    session = AgentSession(
        stt=openai.STT(model="gpt-4o-transcribe"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=rime.TTS(model="coda", speaker="lyra"),
        vad=ctx.proc.userdata["vad"],
        turn_detection=MultilingualModel(),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_input_options=RoomInputOptions(
            noise_cancellation=noise_cancellation.BVC()
        ),
    )

    await session.say("Hey there! How can I help you today?")

Each time a user connects, this function:

• Connects to the LiveKit room and waits for a participant to join
• Creates an AgentSession that wires together the voice pipeline components: OpenAI for STT and LLM, Rime for TTS, and LiveKit for VAD and turn detection
• Starts the session with noise cancellation enabled to filter out background noise from the user’s microphone
• Greets the user with an initial message

3.4 Initialize the VAD plugin

Add the following prewarm function below the entrypoint function:

def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()

This function loads the VAD plugin once when the worker starts, rather than for each new conversation.

3.5 Create the main entrypoint

Add the following __main__ block to agent.py:

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        ),
    )

This starts the agent server that listens for incoming connections from LiveKit. WorkerOptions is configured with the two functions you created above: prewarm_fnc runs once per worker process to preload models, and entrypoint_fnc runs each time a user connects to a room.

3.6 Full agent code

At this point, your agent.py should look like the complete example below:

Full agent code

from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentSession,
    AutoSubscribe,
    JobContext,
    JobProcess,
    RoomInputOptions,
    WorkerOptions,
    cli,
)
from livekit.plugins import openai, noise_cancellation, rime, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

load_dotenv()

class Assistant(Agent):
    def __init__(self):
        super().__init__(
            instructions="""You are a helpful voice assistant.
            Keep your responses short and conversational - no more than 2-3 sentences.
            Be friendly and natural."""
        )

def prewarm(proc: JobProcess):
    proc.userdata["vad"] = silero.VAD.load()

async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
    await ctx.wait_for_participant()

    session = AgentSession(
        stt=openai.STT(model="gpt-4o-transcribe"),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=rime.TTS(model="coda", speaker="lyra"),
        vad=ctx.proc.userdata["vad"],
        turn_detection=MultilingualModel(),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_input_options=RoomInputOptions(
            noise_cancellation=noise_cancellation.BVC()
        ),
    )

    await session.say("Hey there! How can I help you today?")

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
        ),
    )

Step 4: Test your agent

4.1 Start the agent

Start your agent by running:

uv run agent.py dev

The dev argument starts the agent in development mode. You’ll see output like:

INFO   livekit.agents     starting worker {"version": "1.6.5", ...}
INFO   livekit.agents     initializing process {"pid": 12345, ...}
INFO   livekit.agents     process initialized
INFO   livekit.agents     HTTP server listening on :12332

4.2 Connect to your agent

Open the LiveKit Agents Playground in your browser.

• Select your project, then select Use [project_name].
• At the top right of the Playground, select Connect.
• Allow microphone access when prompted.

If everything is set up correctly, you should hear your agent say the greeting that you configured above:

await session.say("Hey there! How can I help you today?")

You can now talk to your agent using your microphone or by typing in the chat section.

Step 5: Customize your agent

5.1 Change the voice

Update the tts line in your AgentSession to try a different voice:

tts=rime.TTS(model="coda", speaker="astra"),

Rime has many voices. See the full list on the Voices page.

5.2 Fine-tune agent personalities

Create a new file called personality.py with the following content:

Example personality file

SYSTEM_PROMPT = """
CHARACTER:
You are Gary, an overly enthusiastic tech support agent who is convinced 
that every problem can be solved by turning it off and on again. You work 
for a fictional company called "Reboot Solutions" and take your job very seriously.

PERSONALITY:
- Extremely optimistic, even when things are clearly going wrong
- Obsessed with the power of rebooting
- Uses tech jargon incorrectly but confidently
- Gets genuinely excited when users describe their problems
- Occasionally references "the great outage of 2019" as a formative experience

SPEECH STYLE:
- Keep responses to 2-3 sentences maximum
- Add natural fillers like "um", "well", "you know"
- Express enthusiasm with phrases like "Oh, fantastic!" and "Love it!"
- Always end with a question to keep the conversation going

RESTRICTIONS:
- Never break character
- Don't use emojis or special characters
- Avoid technical accuracy - confidence matters more than correctness
"""

INTRO_MESSAGE = "Oh hey there! Welcome to Reboot Solutions, I'm Gary! What can I help you restart today?"

Update your agent.py to import and use this prompt:

from personality import SYSTEM_PROMPT, INTRO_MESSAGE

class Assistant(Agent):
    def __init__(self):
        super().__init__(instructions=SYSTEM_PROMPT)

Update the greeting at the end of entrypoint:

await session.say(INTRO_MESSAGE)

Keeping the system prompt in a separate file lets you change the character without editing the agent logic.

5.3 Adjust TTS chunking

By default, rime.TTS(...) uses Rime’s HTTP synthesis path. LiveKit wraps non-streaming TTS providers so streamed LLM output can be sent to TTS in sentence-aligned chunks.

If you want to tune those chunks, wrap Rime explicitly:

from livekit.agents import tokenize, tts
from livekit.plugins import rime

tts = tts.StreamAdapter(
    tts=rime.TTS(model="coda", speaker="wawona"),
    sentence_tokenizer=tokenize.blingfire.SentenceTokenizer(
        retain_format=True,
        min_token_len=250,
        max_token_len=500,
    ),
)

Chunking options:

• min_token_len: Batches sentences until the chunk reaches this size. Increase it for smoother multi-sentence prosody.
• max_token_len: Caps chunk size. Use it to avoid very large TTS requests.
• retain_format: Keeps original whitespace and newlines. True matches LiveKit’s default wrapper behavior.

Tuning chunk size is a prosody/latency tradeoff: larger chunks usually sound better, while smaller chunks usually start faster.

Troubleshooting

Agent doesn’t respond to speech

• Check microphone permissions: Ensure your browser has microphone access enabled.
• Verify VAD is working: Look for speech detected logs in the terminal. If missing, check your Silero plugin installation.
• Test with text input: Use the chat input in the Playground to confirm the agent logic works.

”Connection refused” or agent won’t start

• Check environment variables: Ensure all keys in .env are set correctly with no extra spaces.
• Verify LiveKit credentials: Confirm your LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET match your LiveKit Cloud project.

Incorrect voice detection

• Enable noise cancellation: Verify noise_cancellation.BVC() is included in your RoomInputOptions.
• Check your microphone: Test with a different input device or headset
• Reduce background noise: The VAD may struggle to detect speech in noisy environments.