Skip to main content
Tune concurrency against an explicit service-level target. Rime does not provide rate limiting or request queues because those controls depend on the deployment, so enforce the concurrency limit in your own infrastructure.

​
What to measure

  • Initial latency, or time to first frame or byte (TTFF/TTFB), measures the time from sending a request until the first frame arrives. Lower is better.
  • Real-time factor (RTF) is processing time divided by stream duration. RTF must remain at or below 1 for real-time delivery. Lower is better.
  • Concurrency is the number of simultaneous requests the service can handle while meeting the target. Higher is better.
Latency and RTF usually rise with concurrency. Queue or reject requests above the measured limit instead of allowing overload to degrade every active stream.

​
Coda

For Coda, GENERATOR_MAX_BATCH and INFERENCE_CONCURRENCY_CAPACITY are the tuning controls, and their defaults work for most deployments. INFERENCE_CONCURRENCY_CAPACITY (default 128) sets the generator’s admission limit and sizes its KV cache, so raising it costs GPU memory. Increase GENERATOR_MAX_BATCH to 128 when you serve higher traffic or run on more modern GPUs. A larger batch supports higher concurrency at the cost of longer container startup time. Leave the other controls at their defaults.