Routing Proxy¶
The routing proxy is the Sardeenz component that sits on the critical path of every inference request. It is a stateless Rust (axum/tokio) service that resolves model names to runner endpoints, handles connection parking when models are sleeping, and forwards traffic to the appropriate runner.
For the overall system context, see the architecture overview. For the proxy ↔ control plane data contract, see packages/contracts/specs/proxy-control-plane.yaml.
For the Rust choice rationale, see ADR-003.
Scope¶
The proxy owns the inference request path — from client connection to runner response. It does not:
- Make orchestration decisions. The control plane decides where models run, when to sleep them, and how to allocate device memory. The proxy reads the output of those decisions (the routing map) and acts on them.
- Health-check runners directly. The control plane owns runner health. The proxy applies its own per-endpoint circuit breaker as a last-resort backstop, but does not poll runners.
- Persist state. The proxy is fully stateless across replicas. Routing state lives in Redis. Circuit breaker and parking state are per-replica and ephemeral.
- Serve admin traffic. Admin routes (
/healthz,/readyz,/metrics) are on a separate port and not part of the inference path.
Overview¶
The proxy exposes six inference endpoints on its primary port (default 0.0.0.0:8080), split
across two protocol-family path prefixes — /openai for the OpenAI-compatible surface and
/oip for the KServe V2 Open Inference Protocol surface (see ADR-021):
| Endpoint | Method | Purpose |
|---|---|---|
/openai/v1/chat/completions |
POST |
OpenAI chat inference (forwarded) |
/openai/v1/completions |
POST |
OpenAI text completion (forwarded) |
/openai/v1/models |
GET |
List openai-protocol models |
/oip/v2/models/{model}/infer |
POST |
KServe V2 (OIP) inference (forwarded) |
/oip/v2/models/{model}/ready |
GET |
KServe V2 readiness (503 for sleeping, no wake) |
/oip/v2/models |
GET |
List oip-protocol models |
The prefix names the protocol family, never the engine — it is stripped before forwarding, so
runners always receive canonical /v1/* or /v2/* paths. Both prefixes are always mounted; a
protocol family with no deployed models simply lists nothing.
Protocol families¶
Each routing entry carries a required protocol tag (openai or oip, ADR-021)
written by the control plane. The proxy compiles in both protocol adapters and always mounts every
prefix; which models appear under which prefix is driven entirely by this per-entry tag — importing
and deploying a runner lights up its surface live, with no proxy restart or config edit. At startup
and on every Redis reconnect, the proxy also publishes its supported protocol set to the
{prefix}:proxy:protocols Redis key, which the control plane's catalog import path checks as a
forward-compat guard (importing a runner whose protocol the running proxy does not advertise
fails fast at import time rather than at request time).
A separate admin server on 0.0.0.0:9099 exposes /healthz, /readyz, and /metrics. The admin port is never exposed outside the cluster.
The proxy is designed to run as multiple stateless replicas behind a load balancer. Replicas share no in-process state — all coordination happens through the Redis routing map.
Request Flow¶
Hot Path Summary¶
Every inference request follows the same steps:
- Read the raw request body (buffered, max 1 MiB by default —
SARDEENZ_PROXY_MAX_BODY_BYTES) - Extract the
modelfield — from the JSON body for/openai/*requests, from the{model}URL path segment for/oip/*requests. The rest of the path (resolve/park/wake/forward/circuit-break) is protocol-agnostic. - Resolve the model's current state from the in-memory routing cache
- If sleeping, park the connection and fire a wake trigger; if starting (wake already in progress), park without triggering
- Filter endpoints through the circuit breaker
- Pick an endpoint via weighted round-robin
- Forward the request and stream the response back
- Record success or failure on the circuit breaker
Scenario 1: Active Model¶
sequenceDiagram
participant Client
participant Proxy
participant Cache as Routing Cache<br/>(in-memory)
participant Runner
Client->>Proxy: POST /openai/v1/chat/completions<br/>{"model": "llama-3", ...}
Proxy->>Proxy: Extract model name from body
Proxy->>Cache: resolve("llama-3")
Cache-->>Proxy: Resolution::Active(entry)<br/>endpoints: [{host, port, weight, healthy}]
Proxy->>Proxy: Filter endpoints through circuit breaker
Proxy->>Proxy: Weighted round-robin pick
Proxy->>Runner: POST /v1/chat/completions<br/>(forwarded, host header stripped)<br/>(prefix stripped)
Runner-->>Proxy: 200 OK (streaming body)
Proxy->>Proxy: record_success(endpoint)
Proxy-->>Client: 200 OK (streaming body)
The routing cache is always local (in-memory). There is no Redis call on the hot path once the cache is populated.
Scenario 2: Sleeping Model¶
sequenceDiagram
participant Client
participant Proxy
participant Cache as Routing Cache<br/>(in-memory)
participant CP as Control Plane
participant Redis
participant Runner
Client->>Proxy: POST /openai/v1/chat/completions<br/>{"model": "llama-3", ...}
Proxy->>Cache: resolve("llama-3")
Cache-->>Proxy: Resolution::Sleeping(entry)
Note over Proxy: check_limits() — verify per-model<br/>and global parking caps
Proxy->>Proxy: increment parked count
alt First request for this model
Proxy->>Proxy: pending_wakes.insert("llama-3")
Proxy->>CP: POST /api/v1/wake<br/>{"modelName": "llama-3"}
CP-->>Proxy: 202 Accepted
else Subsequent requests (thundering herd)
Note over Proxy: pending_wakes already has "llama-3"<br/>— skip wake trigger, park silently
end
loop Until Active or timeout (120s default)
Proxy->>Proxy: watch::Receiver::changed()
Note over Redis,Cache: Control plane updates<br/>routing map → Redis pub/sub<br/>→ cache.replace() → notify.send(())
Proxy->>Cache: get("llama-3")
Cache-->>Proxy: entry.state == Active
end
Proxy->>Proxy: pending_wakes.remove("llama-3")
Proxy->>Proxy: decrement parked count
Proxy->>Cache: get("llama-3") — fresh entry
Cache-->>Proxy: endpoints: [{host, port, ...}]
Proxy->>Proxy: Filter + round-robin pick
Proxy->>Runner: POST /v1/chat/completions<br/>(prefix stripped)
Runner-->>Proxy: Response
Proxy-->>Client: Response
The client connection is held open throughout. The proxy never disconnects the client during parking.
Scenario 3: Multiple Replicas (Weighted Round-Robin)¶
sequenceDiagram
participant Client
participant Proxy
participant Cache as Routing Cache
participant R1 as Runner A<br/>(weight 3)
participant R2 as Runner B<br/>(weight 1)
Client->>Proxy: POST /openai/v1/chat/completions
Proxy->>Cache: resolve("llama-3")
Cache-->>Proxy: Resolution::Active<br/>endpoints: [A(w=3), B(w=1)]
Proxy->>Proxy: Filter through circuit breaker<br/>Both endpoints: Closed → both pass
Note over Proxy: Cumulative weight bands: A=[0,3), B=[3,4)<br/>counter.fetch_add(1) % 4 selects a band
Proxy->>Proxy: counter = 5 → 5 % 4 = 1 → falls in [0,3) → pick A
Proxy->>R1: Forward request
R1-->>Proxy: Response
Proxy->>Proxy: record_success("A:8000")
Proxy-->>Client: Response
Note over Client,Proxy: Next request: counter = 6 → 6 % 4 = 2 → falls in [0,3) → pick A
Note over Client,Proxy: Next request: counter = 7 → 7 % 4 = 3 → falls in [3,4) → pick B
Note over Client,Proxy: Next request: counter = 8 → 8 % 4 = 0 → falls in [0,3) → pick A
The WeightedRoundRobin balancer computes cumulative weight bands over the endpoint list (e.g. weights [3, 1] produce bands A=[0,3), B=[3,4)) and uses a global atomic counter modulo the total weight to select a band via an O(n) scan — no expanded endpoint list is allocated. Weight 0 effectively removes an endpoint from rotation — used for graceful drain.
Connection Parking Protocol¶
When the resolver returns Resolution::Sleeping, the request enters the parking subsystem. The ParkingManager holds the Tokio task at an await point, preserving the client connection without consuming a thread.
Thundering Herd Prevention¶
The pending_wakes map (type Arc<Mutex<HashMap<String, ()>>>) serializes wake triggers:
park("llama-3", fire_wake: true):
1. lock pending_wakes
2. if "llama-3" not present:
insert "llama-3" → ()
drop lock
call wake_client.trigger_wake("llama-3") ← only this task fires the trigger
3. else:
drop lock ← all other tasks skip the trigger
4. subscribe to routing_cache watch channel
5. loop: wait for cache change or timeout
All parked tasks for the same model share the same watch::Receiver subscription — they all wake up when the routing cache is updated, check whether the model is now ACTIVE, and proceed if so.
If the wake trigger call fails, pending_wakes removes the entry so the next parked task can retry.
Starting State¶
When the resolver returns Resolution::Starting, the model's wake is already in progress (the control plane has set state to STARTING in the routing map). The proxy calls park(..., fire_wake: false). No wake trigger is sent — the task simply parks and waits for ACTIVE.
State Transitions During Parking¶
The parking loop checks for the following terminal conditions on every wake from receiver.changed():
| State in cache | Action |
|---|---|
ACTIVE |
Remove from pending_wakes, return Ok(()), proceed to forward |
ERROR |
Remove from pending_wakes, return Err(ModelUnavailable) → 503 |
| Entry removed | Remove from pending_wakes, return Err(ModelNotFound) → 404 |
DRAINING |
Remove from pending_wakes, return Err(ModelUnavailable) → 503 (matches the resolver's up-front fail-fast for draining models) |
SLEEPING |
If the model has already left SLEEPING during this wait, treat as a rollback: remove from pending_wakes, return Err(ModelUnavailable) → 503 (client retry re-triggers cleanly). Still-SLEEPING before the wake has progressed is not a rollback — keep waiting. |
STARTING — and SLEEPING before the model has been observed leaving it — cause the loop to keep waiting; the wake is still in progress.
Timeout and Backpressure¶
Three independent limits protect the proxy from runaway parking:
| Limit | Config key | Default | Scope |
|---|---|---|---|
| Per-model cap | SARDEENZ_PARKING_MAX_PER_MODEL |
1000 | Per model name |
| Global cap | SARDEENZ_PARKING_MAX_GLOBAL |
10000 | Across all models |
| Byte cap | SARDEENZ_PARKING_MAX_BYTES |
1 GiB | Sum of parked request bodies |
| Deadline | SARDEENZ_PARKING_TIMEOUT_SECS |
120 | Per parked request |
Limits are checked before incrementing the count. Excess requests receive a 503 (parking_limit_reached). The deadline uses tokio::time::sleep_until inside a select! with the receiver.changed() future — if the deadline fires first, the task returns Err(ParkingTimeout) → 503 (parking_timeout).
Routing Map¶
Redis Key Structure¶
The control plane maintains the routing map as a Redis hash:
| Key | Type | Description |
|---|---|---|
sardeenz:routing-map |
Hash | One field per model. Field name = model name. Value = JSON-serialized RoutingEntry. |
sardeenz:routing-updates |
Pub/sub channel | Receives RoutingMapUpdate JSON on every routing map change. |
sardeenz:routing-barriers |
Pub/sub channel | Requests an applied-and-quiescent acknowledgement after a destructive routing change. |
sardeenz:routing-barrier-acks:{barrierId} |
Set | Short-lived proxy process ids that have applied and quiesced one cutover. |
sardeenz:proxies:{proxyId} |
String + TTL | Presence lease for a proxy that has loaded the current map and is admitting inference traffic. |
RoutingEntry Format¶
Each hash field value is a JSON-serialized RoutingEntry:
{
"modelName": "meta-llama/Llama-3.1-8B-Instruct",
"state": "ACTIVE",
"protocol": "openai",
"endpoints": [
{
"host": "10.244.1.5",
"port": 8000,
"weight": 1,
"healthy": true,
"runnerId": "runner-abc123"
}
],
"updatedAt": "2026-06-12T10:30:00Z",
"metadata": {
"ownedBy": "platform-team",
"maxModelLen": 131072
}
}
A sleeping model has "state": "SLEEPING" and an empty endpoints array. protocol is a required
top-level field (ADR-021) — it drives which
listing (/openai/v1/models or /oip/v2/models) the entry appears under. The metadata block
remains optional and is passed through to the openai listing response without interpretation.
The updatedAt timestamp is written by the control plane for operator diagnostics and dashboard display. The proxy deserializes it for round-trip fidelity but does not consult it for routing or staleness decisions — cache freshness is determined entirely by Redis pub/sub notifications (see Cache Invalidation below).
The port in each endpoint is the runner's engine (inference) port — where the engine's /v1/* server listens — not the runner's management port. The control plane registers this port (a two-port engine like vLLM serves the runner contract on a separate management port it never publishes to the routing map; a single-server runner reports the same value for both). The proxy forwards to whatever host:port the entry names and needs no knowledge of the distinction.
Model States¶
| State | Proxy action |
|---|---|
ACTIVE |
Forward to endpoints via round-robin |
SLEEPING |
Park connection, fire wake trigger |
STARTING |
Park connection, no wake trigger (already in progress) |
DRAINING |
Reject with 503 (model_unavailable) |
ERROR |
Reject with 503 (model_unavailable) |
Refresh Strategy¶
On startup, the proxy performs a full HGETALL sardeenz:routing-map and populates RoutingMapCache. It then subscribes to sardeenz:routing-updates for incremental updates.
On any routing-update or barrier message, the proxy re-fetches the full hash with HGETALL and replaces the in-memory map atomically. This avoids applying partial or out-of-order deltas — the map is small enough (one entry per deployed model) that a full re-read is safe and simpler.
pub/sub message received:
1. HGETALL sardeenz:routing-map
2. Parse each field: serde_json::from_str::<RoutingEntry>(&value)
3. routing_cache.replace(new_map).await
→ *inner.write() = new_map
→ notify.send(()) ← wakes all parked tasks
Fields that fail JSON parsing are logged as warnings. A previous valid entry is retained when one exists; an invalid new entry is dropped. A parse failure for one model does not block the refresh for others.
For a destructive change (an active model becomes unavailable, or a selectable endpoint is removed, made unhealthy, or assigned weight zero), requests hold a read lease on their selected routing generation through the complete upstream response. Cache replacement takes that generation's write lease, so it waits for those requests before acknowledging a propagation barrier. Endpoint additions and diagnostic-only changes reuse the existing lease and do not wait. The control plane snapshots the TTL-backed proxy presence keys at cutover and stops the old runner only after every pre-cutover proxy has acknowledged, disconnected long enough for its presence to expire, or reconnected and loaded the new map. A disconnected proxy rejects new inference rather than continuing to route from its stale cache.
If the Redis connection drops, start_redis_sync returns an error. The main loop logs the error, marks the proxy as not ready (fails /readyz), and retries the connection after a 5-second back-off. The in-memory cache remains intact during the gap — it may become stale but does not clear.
Scale Assumptions¶
The full-refresh strategy (HGETALL on every pub/sub event) is designed for the following envelope:
| Dimension | Expected range | Notes |
|---|---|---|
| Model count | 10–100 | One routing entry per deployed model |
| Entry size | < 1 KB each | A few endpoints + metadata per model |
| Total map size | < 100 KB | Fits comfortably in a single HGETALL |
| Pub/sub frequency | < 1 event/second sustained | Bursts during batch operations are fine |
When to revisit: If the model count exceeds ~500, or if pub/sub events exceed ~10/second sustained, the full-refresh approach may become a bottleneck. At that point, consider incremental delta application (using the RoutingMapUpdate type already defined in the spec) or per-model key reads instead of HGETALL.
Cache Invalidation¶
There is no TTL-based expiry. The in-memory cache is authoritative between pub/sub updates. Staleness is bounded by the latency of the pub/sub notification path (typically a few milliseconds).
Circuit Breaker¶
The proxy maintains a per-endpoint circuit breaker to protect against failed or degraded runners. The endpoint key is "{host}:{port}".
States¶
stateDiagram-v2
[*] --> Closed : new endpoint
Closed --> Open : failures >= threshold<br/>within window
Open --> HalfOpen : recovery_timeout elapsed
HalfOpen --> Closed : record_success()
HalfOpen --> Open : record_failure()
Closed --> Closed : record_success() or<br/>failures < threshold
| State | is_allowed() |
Description |
|---|---|---|
Closed |
true |
Normal operation. Failures accumulate in a sliding window. |
Open |
false |
Endpoint is excluded from routing. |
HalfOpen |
true |
Probe phase — one request is allowed through to test recovery. |
Thresholds and Recovery¶
| Parameter | Config key | Default | Description |
|---|---|---|---|
| Failure threshold | SARDEENZ_CB_FAILURE_THRESHOLD |
5 | Failures within window that trip the breaker |
| Failure window | SARDEENZ_CB_FAILURE_WINDOW_SECS |
30 | Sliding window for failure counting (seconds) |
| Recovery timeout | SARDEENZ_CB_RECOVERY_TIMEOUT_SECS |
15 | Time in Open before transitioning to HalfOpen |
Failures outside the window are pruned on each record_failure() call. The state check is lazy — Open → HalfOpen transition is computed when state() or is_allowed() is called, not on a timer.
A claimed HalfOpen probe is normally released the instant its outcome is recorded (or if the request is cancelled). As a leak backstop, a probe is also treated as re-claimable after max(recovery_timeout, upstream_timeout) elapses — deliberately not recovery_timeout alone, since a legitimate in-flight probe can run as long as upstream_timeout, and reclaiming it earlier would admit a second probe on top of a still-recovering endpoint. This window is derived internally and has no separate env var.
When All Endpoints Are Open¶
If every endpoint for a model has an open circuit breaker, balancer.pick() returns None. The proxy returns 503 (all_endpoints_unhealthy). The control plane's own health monitoring should detect this condition and update the routing map (e.g., marking the model ERROR or routing to a different worker).
Circuit breaker state is per-replica and not shared across proxy replicas. A failing runner appears independently broken to each replica.
Forwarding Concurrency Limits¶
The proxy applies an optional cap on concurrently forwarded (in-flight upstream) requests, both
globally (SARDEENZ_PROXY_MAX_CONCURRENT_FORWARDS) and per-model
(SARDEENZ_PROXY_MAX_CONCURRENT_PER_MODEL). A request over the limit is rejected immediately with
503 Service Unavailable (overloaded). Both default to 0 (unlimited).
This is a last-resort admission control backstop on the proxy itself, protecting runners from being driven past their real concurrency capacity when upstream traffic bursts faster than autoscaling or eviction can react. It is not a substitute for rate limiting — see Deployment Requirements below.
Why this lives at the forwarding step, not the router. The permit is acquired after parking
resolves (i.e. after a sleeping/starting model's request has already waited for ACTIVE) and
before endpoint selection. A limit checked earlier — e.g. at model resolution, before parking —
would let a parked request hold a forwarding permit for the entire wake duration (up to
SARDEENZ_PARKING_TIMEOUT_SECS). Under load, that permit would never free up fast enough, and the
forwarding limiter would effectively cap the number of models that can wake concurrently instead of
the number of requests actually hitting a runner — a self-inflicted deadlock. Parked requests hold no
forwarding permit; they contend only with the parking limits (SARDEENZ_PARKING_MAX_PER_MODEL,
SARDEENZ_PARKING_MAX_GLOBAL, SARDEENZ_PARKING_MAX_BYTES), which are independent from and enforced
much earlier than the forwarding permit.
Like the circuit breaker, the limiter is per-replica, in-memory state — it is not shared across proxy replicas.
Configuration Reference¶
All configuration is read from environment variables at startup via Config::from_env().
| Variable | Type | Default | Description |
|---|---|---|---|
SARDEENZ_PROXY_LISTEN_ADDR |
SocketAddr |
0.0.0.0:8080 |
Inference server bind address (the legacy SARDEENZ_LISTEN_ADDR name is still read as a fallback) |
SARDEENZ_ADMIN_ADDR |
SocketAddr |
0.0.0.0:9099 |
Admin server bind address (health + metrics) |
SARDEENZ_REDIS_URL |
String |
redis://127.0.0.1:6379 |
Redis/Valkey connection URL |
SARDEENZ_CONTROL_PLANE_URL |
String |
http://127.0.0.1:3000 |
Control plane base URL for wake triggers |
SARDEENZ_LOG_LEVEL |
String |
info |
Log level (trace, debug, info, warn, error) |
SARDEENZ_UPSTREAM_TIMEOUT_SECS |
u64 |
300 |
Timeout for forwarded requests to runners (includes streaming) |
SARDEENZ_REDIS_KEY_PREFIX |
String |
sardeenz |
Prefix for Redis keys and pub/sub channels (e.g. sardeenz:routing-map) |
SARDEENZ_PARKING_TIMEOUT_SECS |
u64 |
120 |
Max time a request can be parked before returning 503 |
SARDEENZ_PARKING_MAX_PER_MODEL |
usize |
1000 |
Max concurrently parked requests per model |
SARDEENZ_PARKING_MAX_GLOBAL |
usize |
10000 |
Max concurrently parked requests across all models |
SARDEENZ_PARKING_MAX_BYTES |
usize |
1073741824 |
Max total bytes of parked request bodies (1 GiB) |
SARDEENZ_PROXY_MAX_BODY_BYTES |
usize |
1048576 |
Max buffered request body size (1 MiB); larger bodies are rejected |
SARDEENZ_API_TOKEN |
String |
unset | Bearer token sent on wake-trigger calls to the control plane (required when its auth is enabled) |
SARDEENZ_CB_FAILURE_THRESHOLD |
u32 |
5 |
Failures within window to trip a circuit breaker |
SARDEENZ_CB_FAILURE_WINDOW_SECS |
u64 |
30 |
Sliding window for circuit breaker failure counting |
SARDEENZ_CB_RECOVERY_TIMEOUT_SECS |
u64 |
15 |
Time before an open circuit transitions to half-open |
SARDEENZ_PROXY_MAX_CONCURRENT_FORWARDS |
usize |
0 (unlimited) |
Max concurrently forwarded (in-flight upstream) requests, across all models |
SARDEENZ_PROXY_MAX_CONCURRENT_PER_MODEL |
usize |
0 (unlimited) |
Max concurrently forwarded requests for a single model |
All socket address and numeric values are validated at startup; a parse failure causes the process to exit immediately.
Metrics Reference¶
Metrics are exposed in Prometheus text format on GET /metrics (admin port). All metric names use the sardeenz_proxy_ prefix.
| Metric | Type | Labels | Description |
|---|---|---|---|
sardeenz_proxy_requests_total |
Counter | model, endpoint, status_code |
Total inference requests, by outcome |
sardeenz_proxy_request_duration_seconds |
Histogram | — | End-to-end request latency, excluding parking wait time |
sardeenz_proxy_active_connections |
Gauge | — | Currently active forwarded connections |
sardeenz_proxy_parked_connections |
Gauge | model |
Currently parked connections, per model |
sardeenz_proxy_wake_triggers_total |
Counter | — | Wake triggers fired to the control plane |
sardeenz_proxy_parking_duration_seconds |
Histogram | — | Time a request spent parked before forwarding |
sardeenz_proxy_circuit_breaker_state |
Gauge | endpoint |
Circuit breaker state: 0=closed, 1=open, 2=half-open |
sardeenz_proxy_routing_parse_errors_total |
Counter | model |
Routing entries that failed to deserialize during Redis sync, per model |
Both histograms are registered with explicit bucket boundaries (seconds) so they render as
Prometheus histograms (_bucket series) rather than summaries — required for the dashboard's
histogram_quantile(rate(..._bucket[5m])) queries:
sardeenz_proxy_request_duration_seconds:0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120sardeenz_proxy_parking_duration_seconds:0.5, 1, 2.5, 5, 10, 30, 60, 120, 300
Health Endpoints¶
Both endpoints are served on the admin port (SARDEENZ_ADMIN_ADDR).
GET /healthz¶
Always returns 200 OK with body ok if the process is running. This is a liveness probe — it does not check Redis connectivity or routing map freshness.
GET /readyz¶
Returns 200 OK with body ready when the proxy has both an active Redis connection and a completed initial routing-map load (routing_map_loaded). Returns 503 Service Unavailable with body not ready if either condition is not met.
The readiness signal is set only after the proxy has subscribed, completed its first
HGETALL sardeenz:routing-map, installed the cache, and written its presence lease. It is cleared when the
Redis connection drops. Kubernetes should use this endpoint for readiness gates — inference
handlers enforce the same readiness check, so a disconnected proxy cannot keep selecting an old
endpoint while a move cutover is in progress.
Relationship to the Control Plane¶
The proxy and control plane communicate through two channels: Redis (shared state) and a direct HTTP call (wake trigger).
Redis Interactions¶
| Redis key | Access pattern | Purpose |
|---|---|---|
sardeenz:routing-map (hash) |
Full read at startup; full re-read on every pub/sub event | Populate and refresh routing cache |
sardeenz:routing-updates (pub/sub) |
Subscription | Trigger ordinary routing cache refresh |
sardeenz:routing-barriers (pub/sub) |
Subscription | Trigger a refresh that must be acknowledged |
sardeenz:proxies:{proxyId} (TTL string) |
Create and heartbeat while connected | Advertise that this process is serving traffic |
sardeenz:routing-barrier-acks:{barrierId} (set) |
Add this process id after cache quiescence | Confirm an old route is no longer in use |
Direct HTTP Call¶
The proxy makes exactly one HTTP call to the control plane: POST /api/v1/wake when a sleeping model needs to wake. This call is:
- Fire-and-forget at the protocol level. The control plane responds
202 Acceptedimmediately; the actual wake-up is asynchronous. - Deduplicated by the proxy. The
pending_wakesmap ensures only one trigger per model per wake event, regardless of how many requests are parked. - Deduplicated by the control plane. The spec declares this endpoint idempotent — duplicate triggers from multiple proxy replicas are safe.
- Timeout-bounded. The wake trigger HTTP call has a 5-second timeout. A timeout causes the triggering task to fail with
ModelUnavailableand removes thepending_wakesentry so subsequent parked tasks can retry.
Wake Trigger Contract¶
POST {SARDEENZ_CONTROL_PLANE_URL}/api/v1/wake
Content-Type: application/json
{
"modelName": "meta-llama/Llama-3.1-8B-Instruct",
"requestId": null
}
The control plane responds 202 Accepted. Any non-2xx response is treated as a failure. After a successful trigger, the proxy does not poll the control plane — it waits for the routing map to reflect ACTIVE via Redis pub/sub.
The full wake trigger schema is defined in packages/contracts/specs/proxy-control-plane.yaml.
Division of Responsibility¶
| Concern | Owner |
|---|---|
| VRAM budget accounting | Control plane |
| Eviction decisions (which model to sleep) | Control plane |
| Runner health polling | Control plane |
| Routing map writes | Control plane |
| Wake orchestration (commands to runner) | Control plane |
| Request routing | Proxy |
| Connection parking | Proxy |
| Per-endpoint circuit breaking | Proxy |
| Client-facing streaming | Proxy |
| Redis pub/sub subscription | Proxy |
Security and Trust Model¶
The proxy trusts the routing map completely. It forwards requests to whatever host:port endpoints appear in the routing map entries retrieved from Redis. This is architecturally intentional — the proxy is a stateless routing layer, not a policy enforcement point.
Implication: compromising Redis or the routing-map writer (the control plane) is equivalent to controlling all request routing. An attacker who can write to sardeenz:routing-map can redirect inference traffic to arbitrary endpoints (SSRF via routing map poisoning).
Trust Boundaries¶
| Trust boundary | What it protects | Required controls |
|---|---|---|
| Redis/Valkey access | Routing map integrity | AUTH/ACLs, network policy, TLS in transit |
| Control plane API | Routing map writes, wake orchestration | Authentication, authorization, network isolation |
| Proxy admin port (9099) | Health/readiness/metrics exposure | Not exposed outside the cluster |
| Proxy ↔ runners | Inference traffic integrity | Network policy; mTLS for sensitive workloads |
Deployment Requirements¶
A Phase 1 deployment must include at minimum:
- Authenticated ingress/gateway in front of the proxy — the proxy does not authenticate clients
- Ingress-level rate limiting and admission control in front of the proxy — the proxy has no
concept of client identity, tenants, or API keys, so it cannot enforce per-tenant quotas or fair
sharing. Per-tenant/per-client rate limiting is the ingress's responsibility, not the proxy's. The
proxy's own
SARDEENZ_PROXY_MAX_CONCURRENT_FORWARDS[_PER_MODEL](see Forwarding Concurrency Limits) is a coarse, identity-blind backstop against runner overload — it complements ingress rate limiting, it does not replace it. - Network policy isolating proxy, control plane, Redis, and runners into a trusted mesh
- Redis AUTH or ACLs preventing unauthorized routing-map access
- Admin port isolation — port 9099 must not be exposed to untrusted networks
These are not optional hardening steps — they are prerequisites for a safe deployment. Without them, the proxy is an open relay to whatever the routing map points at.