> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Load balancing

> Load-balance Rime on-prem deployments using the ORCA cost-header signal.

Rime's Coda and Mist model images report their concurrent request load to your
load balancer in an [Open Request Cost Aggregation (ORCA)
header](https://github.com/envoyproxy/envoy/issues/6614) on every engine HTTP
response. A balancer that reads it can send new requests to less-loaded engines,
because latency and real-time factor usually rise with an engine's
concurrency. Set `ENDPOINT_LOAD_METRICS_ENABLE=false` on the model
container to suppress the header.

## HTTP ORCA header

HTTP responses carry the report as text:

```
endpoint-load-metrics: TEXT application_utilization=0.5, cpu_utilization=0.3128, mem_utilization=0.2453, rps_fractional=0.0000, eps=0.0000
```

For gRPC, the engine sends the same load report as protobuf ORCA data in
`endpoint-load-metrics-bin` response metadata. Your gateway must understand
that encoding to use the report. WebSocket context events do not each carry
a fresh load report.

## Max concurrency

`application_utilization` is the number of concurrent inference requests
divided by the engine's max concurrency, `INFERENCE_CONCURRENCY_CAPACITY`. The
image default is `128` for Coda and `32` for Mist v3. Override it with the value
you measure during [parameter tuning](/docs/on-prem/performance).

At capacity, the [standalone engine](/docs/on-prem/quickstart) rejects new
synthesis. It returns gRPC `UNAVAILABLE`, HTTP 503, or WebSocket error kind
`unavailable`. `INFERENCE_CONCURRENCY_CAPACITY` sets this limit for both Coda
and Mist. The engine does not queue excess requests. For Coda, this setting
also sizes the generator's KV cache, so a higher limit costs GPU memory.

## Shutdown and long-lived connections

Your gateway must support HTTP/2 gRPC on engine port `7463` and HTTP/WebSocket
on engine port `8080`. Use `/readyz` or gRPC health to decide where to send
traffic. The HTTP ORCA header cannot move active gRPC streams or WebSocket
contexts to another engine. An open WebSocket stays on its selected engine,
even when the client starts another context on that connection.

Before you stop an engine, remove it from routing and wait for the change to
propagate. `SIGUSR2` marks the engine as not ready but lets it accept requests
that still arrive. `SIGTERM` starts request drain. At that point, the engine
rejects new synthesis, including new contexts on existing WebSocket connections.
Accepted work can finish until `--shutdown-drain-timeout`, which defaults to
ten minutes.

Give the engine time to finish. Set the container stop timeout or Kubernetes
`terminationGracePeriodSeconds` to cover request drain and backend cleanup.
The quickstart allows eleven minutes for the default configuration. Your
gateway's stream and idle timeouts must also allow the request duration your
application needs.

After shutdown, reconnect clients to a ready engine. Check what audio they
already received before you retry. An automatic retry can play the same audio
twice.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.