Skip to main content
Rime’s Coda and Mist model images report their concurrent request load to your load balancer in an Open Request Cost Aggregation (ORCA) header on every engine HTTP response. A balancer that reads it can send new requests to less-loaded engines, because latency and real-time factor usually rise with an engine’s concurrency. Set ENDPOINT_LOAD_METRICS_ENABLE=false on the model container to suppress the header.

​
HTTP ORCA header

HTTP responses carry the report as text:
For gRPC, the engine sends the same load report as protobuf ORCA data in endpoint-load-metrics-bin response metadata. Your gateway must understand that encoding to use the report. WebSocket context events do not each carry a fresh load report.

​
Max concurrency

application_utilization is the number of concurrent inference requests divided by the engine’s max concurrency, INFERENCE_CONCURRENCY_CAPACITY. The image default is 128 for Coda and 32 for Mist v3. Override it with the value you measure during parameter tuning. At capacity, the standalone engine rejects new synthesis. It returns gRPC UNAVAILABLE, HTTP 503, or WebSocket error kind unavailable. INFERENCE_CONCURRENCY_CAPACITY sets this limit for both Coda and Mist. The engine does not queue excess requests. For Coda, this setting also sizes the generator’s KV cache, so a higher limit costs GPU memory.

​
Shutdown and long-lived connections

Your gateway must support HTTP/2 gRPC on engine port 7463 and HTTP/WebSocket on engine port 8080. Use /readyz or gRPC health to decide where to send traffic. The HTTP ORCA header cannot move active gRPC streams or WebSocket contexts to another engine. An open WebSocket stays on its selected engine, even when the client starts another context on that connection. Before you stop an engine, remove it from routing and wait for the change to propagate. SIGUSR2 marks the engine as not ready but lets it accept requests that still arrive. SIGTERM starts request drain. At that point, the engine rejects new synthesis, including new contexts on existing WebSocket connections. Accepted work can finish until --shutdown-drain-timeout, which defaults to ten minutes. Give the engine time to finish. Set the container stop timeout or Kubernetes terminationGracePeriodSeconds to cover request drain and backend cleanup. The quickstart allows eleven minutes for the default configuration. Your gateway’s stream and idle timeouts must also allow the request duration your application needs. After shutdown, reconnect clients to a ready engine. Check what audio they already received before you retry. An automatic retry can play the same audio twice.