Load balancing

Load-balance Rime on-prem deployments using the ORCA cost-header signal.

Real-time streaming often requires limiting how many inference requests a model processes concurrently.

Rime’s Coda and Mist model images return an Open Request Cost Aggregation (ORCA) header on every engine HTTP response, reporting concurrent request load to your load balancer. Set ENDPOINT_LOAD_METRICS_ENABLE=false on the model container to suppress it.

HTTP ORCA header

The ORCA header in HTTP responses looks like:

endpoint-load-metrics: TEXT application_utilization=0.5, cpu_utilization=0.3128, mem_utilization=0.2453, rps_fractional=0.0000, eps=0.0000

Max concurrency

The application_utilization metric is calculated by dividing the number of concurrent inference requests by a preconfigured max concurrency.

You can override the max concurrency after parameter tuning by setting the INFERENCE_CONCURRENCY_CAPACITY to the desired max concurrency.

On Mist images, INFERENCE_CONCURRENCY_CAPACITY only scales the utilization figure reported to the load balancer; it does not reject or queue requests. On Coda images it additionally sets the generator’s admission limit and sizes its KV cache, so lowering it queues excess requests and raising it costs GPU memory.