Load balancing

Load-balance Rime on-prem deployments using the ORCA cost-header signal.

Real-time streaming often requires limiting how many inference requests a model processes concurrently.

Rime’s Arcana model images return an Open Request Cost Aggregation (ORCA) header that reports concurrent request load to your load balancer.

The full set of ORCA headers is returned by Arcana model containers since the 20260115 release.

HTTP ORCA header

The ORCA header in HTTP responses looks like:

endpoint-load-metrics: TEXT application_utilization=0.5, cpu_utilization=0.3128, mem_utilization=0.2453, rps_fractional=0.0000, eps=0.0000

Max concurrency

The application_utilization metric is calculated by dividing the number of concurrent inference requests by a preconfigured max concurrency.

You can override the max concurrency after parameter tuning by setting the INFERENCE_CONCURRENCY_CAPACITY to the desired max concurrency.

The INFERENCE_CONCURRENCY_CAPACITY variable is only used to calculate the utilization to inform the load balancer. Setting it does not reject or queue overflowing requests.