Admin Dashboard¶
The admin dashboard is the operator-facing web interface for managing models, monitoring GPU memory utilization, and observing cluster health. It consists of two deployable units: a React single-page application (SPA) and a Fastify backend-for-frontend (BFF) service, packaged as a single container image.
For the overall system context, see the architecture overview. For the dashboard's API data source, see the control plane API spec.
For the TypeScript stack rationale, see ADR-012. For the four-component split rationale (dashboard as BFF + frontend pair), see ADR-002.
Scope¶
The dashboard owns the operator experience for day-to-day cluster management. It does not:
- Make orchestration decisions. The control plane handles placement, eviction, and lifecycle transitions. The dashboard triggers actions (deploy, sleep, wake, delete, and explicit instance move) and displays their results. A move is admin-only from GPU placement: the dashboard selects a compatible worker and exact GPUs, then polls model detail for replacement deployment, cutover, source drain, success, or teardown failure.
- Replace Prometheus/Grafana. The metrics dashboard provides a convenience view of key metrics. Production alerting and deep observability remain in external tooling.
- Enforce RBAC beyond simple role-based auth. A lightweight auth system (JWT, three modes:
none/simple/oauth) is implemented. Per-model permissions and audit logging remain future work. - Manage multiple clusters. Single cluster only.
Architecture¶
┌────────────────────────┐
│ Frontend SPA │
│ (React + PF6) │
│ │
│ ┌─────────────────┐ │
│ │ TanStack Query │ │
│ │ + SSE EventSource│ │
Operator │ └────────┬────────┘ │
(browser) └───────────┼────────────┘
│ /api/*
▼
┌────────────────────────┐
│ BFF (Fastify) │
│ │
│ ┌──────┬──────┬────┐ │
│ │ CP │Redis │Prom│ │
│ │Client│Reader│Clt │ │
│ └──┬───┴──┬───┴──┬─┘ │
└─────┼──────┼──────┼────┘
│ │ │
┌───────────┘ │ └───────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Control │ │ Redis / │ │ Prometheus │
│ Plane API │ │ Valkey │ │ │
└─────────────┘ └─────────────┘ └─────────────┘
Why a BFF?¶
Three reasons justify a backend-for-frontend rather than direct API calls from the browser:
-
Resilience. The dashboard must remain functional during brief control plane restarts. The BFF reads model and worker state directly from Redis when the control plane is unreachable, serving partial but usable responses marked with
source: "redis-fallback". -
Aggregation. Dashboard views combine data from multiple sources (control plane API, Redis key scans, Prometheus queries). Composing these server-side avoids multiple browser-to-backend round trips and keeps Prometheus query knowledge out of the frontend.
-
Auth integration point. When authentication is added in a future phase, the BFF is the natural place for session management and token validation, avoiding CORS complexity with direct control plane calls.
Data Flow¶
Read path (e.g., model list)¶
Browser → GET /api/models
→ BFF tries: controlPlane.listModels()
├─ Success → proxy response to browser
└─ Network error → redis.listModels() → respond with source: "redis-fallback"
Write operations (deploy, sleep, wake, delete) are proxied to the control plane only. They do not fall back to Redis because Redis is a read-only mirror of control plane state.
Real-time updates (SSE)¶
Browser → EventSource(/api/events?token=<jwt>)
→ BFF creates a per-client Redis subscriber on two channels:
├─ {prefix}:routing-updates — model/endpoint state changes
└─ {prefix}:cluster-events — worker lifecycle & memory updates
→ routing-updates: translates RoutingMapUpdate → ClusterEvent
→ cluster-events: forwards ClusterEvent payloads as-is
→ Sends keepalive ping comments every 30 seconds
→ Frontend useEventStream() hook:
├─ Parses ClusterEvent objects
├─ Dispatches to TanStack Query cache invalidation
│ (WORKER_MEMORY_UPDATED invalidation throttled to 1/sec)
└─ Maintains 100-event ring buffer for event feed display
The BFF's SSE relay subscribes to two Redis pub/sub channels:
{prefix}:routing-updates— Published byRoutingMapServiceduring model state transitions and endpoint changes. The BFF translatesRoutingMapUpdatepayloads intoClusterEventobjects. The proxy also subscribes to this channel for routing table sync.{prefix}:cluster-events— Published byReconciliationServiceduring the reconciliation loop. CarriesClusterEventpayloads directly (worker join/leave, memory updates). The BFF forwards these as-is without translation.
The two-channel design keeps the proxy's hot path (routing-updates) free of events it doesn't consume. Each connected browser client gets its own dedicated Redis subscriber connection; the subscriber is created on SSE handshake and cleaned up when the client disconnects.
Design note — per-client subscribers vs. shared fan-out: This implementation uses one Redis subscriber per connected SSE client rather than a single shared subscriber that fans out to all clients. For an admin dashboard with tens of concurrent connections, the per-client model is simpler and entirely adequate — each subscriber is a lightweight pub/sub connection. A shared-subscriber architecture (one Redis connection, in-process fan-out to all SSE responses) would be needed if connection counts grew to hundreds or more, where the per-client approach would create excessive Redis connections. Refactoring to shared fan-out is straightforward if that threshold is ever reached.
Model/endpoint events flow even during control plane restarts, since RoutingMapService publishes to Redis as part of its state transitions. Worker/memory events are published by the reconciliation loop and are therefore only emitted while a leader instance is running.
Metrics path¶
Browser → GET /api/metrics/latency?start=...&end=...&step=...
→ BFF → Prometheus HTTP API (query_range)
→ Transform Prometheus response → chart-ready JSON → browser
The frontend never constructs PromQL queries. The BFF owns the query templates and exposes higher-level endpoints.
Frontend Architecture¶
Technology stack¶
| Layer | Choice | Rationale |
|---|---|---|
| Framework | React 18 | Industry standard, large ecosystem, team familiarity |
| UI library | PatternFly 6 | Red Hat design system, built-in accessibility, consistent with v1 |
| Build tool | Vite | Fast dev server, optimized production builds |
| Server state | TanStack Query v5 | Caching, deduplication, background refresh, optimistic updates |
| Routing | React Router v6 | Client-side routing with nested layouts |
| Testing | Vitest + React Testing Library | Fast, Vite-native test runner |
Route structure¶
| Path | Component | View |
|---|---|---|
/ |
ClusterOverview |
Cluster health dashboard (landing page) |
/models |
ModelList |
Model management table (configured models, running or not) with filtering/sorting |
/models/deploy |
ModelDeploy |
Deploy form (full page) |
/models/:modelName |
ModelDetail |
Model detail with progress/error display |
/workers |
WorkerList |
Worker list table |
/workers/:workerId |
WorkerDetail |
Per-worker GPU memory breakdown |
/gpu-memory |
GpuMemory |
Cluster-wide measured GPU memory view |
/catalog |
RunnerCatalog |
Runner catalog: browse and import runner SIFs |
/playground |
Playground |
Chat playground against the inference endpoint (admin only) |
/metrics |
MetricsDashboard |
Prometheus-backed performance charts |
State management¶
The frontend uses TanStack Query for all server state (data fetching, caching, mutations). There is no global client-side state store. UI-only state (sidebar open/closed, filter selections, modal visibility) lives in component-local useState.
Query invalidation is event-driven: the useEventStream hook listens for SSE events and invalidates the relevant query keys. For example, a MODEL_STATE_CHANGED event invalidates both ['models'] and ['cluster'] query keys, triggering a refetch only for components currently mounted and subscribed to those queries.
SSE reconnection and degraded mode¶
The useEventStream hook connects to /api/events via EventSource (JWT token passed as ?token= since EventSource cannot send headers). The hook implements a two-tier failure model:
CONNECTED → (SSE error) → RECONNECTING → (5 failures) → DEGRADED
↑ ↑ |
+---------- (SSE recovers) +----- (SSE recovers) ---------+
connected— SSE stream is live; events drive cache invalidation in real timereconnecting— temporary failure; retries every 5 secondsdegraded— 5+ consecutive failures; retries slow to every 30 seconds
In degraded state, the DegradedBanner component shows a persistent warning: "Real-time updates unavailable — polling for changes". Query hooks automatically switch to faster polling intervals to compensate. The banner auto-dismisses as soon as the connection recovers.
Shared components¶
| Component | Purpose |
|---|---|
AppLayout |
PF6 Page shell with masthead, sidebar nav, active highlighting |
StateLabel |
Model lifecycle state as a colored PF6 Label with state-appropriate icons |
MemoryVisualization |
Stacked memory bar (used / reserved / available) with tooltips and expandable inline detail panel |
DegradedBanner |
Persistent warning banner shown when control plane is unreachable or SSE is in degraded state |
State color mapping¶
Consistent across all views. These states are tracked per instance (a model may have several
instances/replicas — see ADR-019); the model
list and detail pages show the logical model's aggregate state, derived from its instances by
precedence ACTIVE > STARTING > DRAINING > SLEEPING > PENDING > STOPPING > ERROR (a model with no
instances is STOPPED). ACTIVE outranks ERROR deliberately — one healthy replica masks a
broken one. ModelDetail additionally renders each instance's own state in its instances table.
| State | PF6 Label color | Semantic meaning |
|---|---|---|
ACTIVE |
green | Healthy, serving inference |
SLEEPING |
blue | Weights offloaded, can be woken |
STARTING |
teal | Loading weights, spinner icon |
DRAINING |
orange | Completing in-flight requests |
ERROR |
red | Requires operator intervention |
STOPPING |
grey | Being torn down |
STOPPED |
grey | Configured but not running; can be started or deleted |
PENDING |
yellow | Deployment accepted, not yet placed |
BFF Architecture¶
Technology stack¶
| Layer | Choice | Rationale |
|---|---|---|
| Framework | Fastify 5 | High performance, structured logging, plugin ecosystem |
| Redis client | ioredis | Cluster support, pub/sub, SCAN iteration |
| HTTP client | undici (via fetch) | Node.js native, connection pooling |
| Logging | pino (via Fastify) | JSON structured logging, request ID propagation |
Upstream clients¶
ControlPlaneClient — Typed HTTP proxy to the control plane's admin API. Each method wraps a fetch call, checks the response status, and either returns the body or throws a BffError. Used for all write operations and as the primary read path.
RedisReader — Direct Redis reader for resilience. Uses SCAN to enumerate model names from {prefix}:models:state:* keys, then reads per-model keys (state, worker, memory, required-memory, runner-type, pinned, created-at, inference:last). For workers, reads JSON hashes from {prefix}:workers:*. Also provides getClusterStatus() which aggregates model counts and memory totals from the raw keys. getClusterMemory() reads the per-device snapshot at {prefix}:cluster:memory (written by the control plane's MemoryBudgetService on each budget refresh, TTL 300s). getWorkerDetail(id) reads the full worker record from {prefix}:worker:{id}:detail (written by WorkerPoolService on each heartbeat check, TTL 120s). Only used when the control plane is unreachable.
PrometheusClient — Thin wrapper around the Prometheus HTTP API. Supports query_range (for time-series charts) and query (for instant gauges).
GithubRepoStatsClient — Fetches star/fork counts for the sidebar footer server-side, since the CSP's connect-src is 'self' only and blocks a browser fetch to api.github.com. Caches in memory per BFF process: a success is reused for an hour, a failure (network error, non-2xx, malformed body) is cached for 5 minutes so a disconnected cluster is not retried on every page load. Never throws; GET /api/repo-stats always returns 200 with { stars, forks, fetchedAt }, nulls on any failure. Disabled (nulls only, no fetch) when SARDEENZ_REPO_STATS_URL is empty.
Redis fallback behavior¶
Read routes follow this pattern:
try {
const { status, data } = await controlPlane.listModels();
return reply.code(status).send(data);
} catch {
log.warn('Control plane unavailable, falling back to Redis');
const models = await redis.listModels();
return reply.code(200).send({ models, source: 'redis-fallback' });
}
The source: "redis-fallback" field signals to the frontend that the response is from the fallback path. The DegradedBanner component detects this field across all active queries and shows a persistent warning: "Control plane unreachable — showing cached data". The banner auto-dismisses as soon as fresh data resumes.
Write routes (POST, DELETE) never fall back. They return the upstream error to the client, since Redis is a read-only mirror.
Fallback coverage¶
| Route | Fallback | Degraded behavior |
|---|---|---|
GET /api/models |
Redis | Full model list, may be stale |
GET /api/models/:name |
Redis | Full model detail, may be stale |
GET /api/workers |
Redis | Worker list, may be stale |
GET /api/workers/:id |
Redis | Worker detail, may be stale |
GET /api/cluster/status |
Redis | Aggregate status, may be stale |
GET /api/cluster/memory |
Redis | Per-device memory, may be stale |
GET /api/metrics/* |
None | Metrics unavailable (Prometheus dependency) |
GET /api/events |
None | SSE stream disconnected |
POST/DELETE /api/models/* |
None | Mutating operations fail with 502 |
Health probes¶
| Endpoint | Purpose | Checks |
|---|---|---|
GET /healthz |
Liveness | Always returns 200 |
GET /readyz |
Readiness | Checks control plane, Redis, Prometheus connectivity |
Static file serving¶
In production (NODE_ENV=production), the BFF serves the frontend's static assets from dist/client/ using @fastify/static. A catch-all not-found handler serves index.html for SPA client-side routing. In development, Vite's dev server handles static assets and proxies /api requests to the BFF.
Configuration¶
BFF environment variables¶
| Variable | Default | Description |
|---|---|---|
SARDEENZ_BFF_LISTEN_ADDR |
0.0.0.0:4000 |
BFF listen address and port |
SARDEENZ_CONTROL_PLANE_URL |
http://localhost:3000 |
Control plane base URL |
SARDEENZ_REDIS_URL |
redis://localhost:6379 |
Redis/Valkey connection string |
SARDEENZ_REDIS_KEY_PREFIX |
sardeenz |
Prefix for all Redis keys |
SARDEENZ_PROMETHEUS_URL |
http://localhost:9090 |
Prometheus (or Thanos Querier) base URL |
SARDEENZ_PROMETHEUS_BEARER_TOKEN_PATH |
(empty) | Path to a token file sent as Authorization: Bearer … on every Prometheus request, re-read per request |
SARDEENZ_PROMETHEUS_CA_PATH |
(empty) | PEM CA bundle used to validate the Prometheus endpoint's TLS certificate |
SARDEENZ_PROMETHEUS_TENANT_NAMESPACE |
(empty) | Sent as the namespace query parameter on every Prometheus request (Thanos Querier tenancy) |
SARDEENZ_INFERENCE_URL |
http://localhost:8080 |
Proxy inference base URL used by the Playground |
SARDEENZ_REPO_STATS_URL |
GitHub API URL for rh-aiservices-bu/school-of-sardeenz |
Repo API URL polled for the sidebar star/fork counts; empty string disables the lookup |
SARDEENZ_BFF_MAX_CONCURRENT_INFERENCE_REQUESTS_PER_USER |
4 |
Maximum active chat-completions responses for one identity, per BFF replica |
SARDEENZ_LOG_LEVEL |
info |
Pino log level |
AUTH_MODE |
none |
Authentication mode: none, simple, or oauth |
ADMIN_USERNAME |
admin |
Admin username for simple auth mode |
ADMIN_PASSWORD |
(empty) | Admin password for simple auth mode |
JWT_SECRET |
(empty) | JWT signing secret (required when AUTH_MODE is not none) |
JWT_EXPIRATION_HOURS |
8 |
JWT token expiration in hours |
OAUTH_CLIENT_ID |
sardeenz |
OAuth client ID (for oauth mode) |
OAUTH_CLIENT_SECRET |
(empty) | OAuth client secret (for oauth mode) |
OAUTH_ISSUER_URL |
(empty) | OAuth OIDC issuer URL (for oauth mode) |
K8S_API_URL |
(empty) | Kubernetes API URL for namespace-scoped OAuth RBAC role resolution (required in oauth mode) |
SERVICE_ACCOUNT_TOKEN |
(mounted token) | ServiceAccount token override for OAuth RBAC checks outside Kubernetes |
NAMESPACE |
sardeenz |
Kubernetes namespace for RBAC scope (for oauth mode) |
Frontend environment variables¶
| Variable | Default | Description |
|---|---|---|
VITE_API_URL |
/api |
BFF API base URL (for non-proxied setups) |
Container Image¶
Single container image (containers/dashboard/Dockerfile) using a multi-stage build:
- deps stage —
npm ciwith workspace manifests only (cache-friendly) - build stage — TypeScript compilation (shared types + BFF) and Vite production build (frontend)
- runtime stage — Node.js 22 slim with compiled BFF, frontend static assets, and production dependencies
The BFF serves both the API and the frontend from a single port (4000). This simplifies deployment for an internal admin tool where scaling the frontend independently is unnecessary.
Inference request lifecycle¶
POST /api/inference/chat/completions takes a per-user slot before it begins an upstream request and holds it until the response body ends, including a non-streaming response. In simple and oauth modes the identity is the verified JWT username; in none mode all requests share anonymous. Excess requests receive 429 RATE_LIMITED without contacting the inference proxy. Disconnects abort the upstream request and release the slot. The limiter is an in-memory, per-replica guard, so it protects each BFF process but does not create a cluster-wide cap.
Testing Strategy¶
| Level | Tool | Scope |
|---|---|---|
| Unit (frontend) | Vitest + React Testing Library | API client, hooks, utility functions, component rendering |
| Unit (BFF) | Vitest | Upstream clients (mocked HTTP), route handlers (mocked deps) |
| E2E (workflows) | Playwright | Critical admin workflows against real frontend + BFF with mock upstreams |
| E2E (accessibility) | Playwright + @axe-core/playwright |
WCAG 2.1 AA scans on all key pages using the mock harness |
E2E mock service harness¶
The Playwright tests use a purpose-built mock harness rather than real upstream services:
MockControlPlane(dashboard/e2e/mocks/control-plane.ts) — Fastify server on a random port serving all BFF-facing control plane endpoints with configurable canned responses. Supports stateful scenarios (deploy, delete, sleep, wake) and an SSEpushEvent()API for testing real-time transitions.MockPrometheus(dashboard/e2e/mocks/prometheus.ts) — Fastify server serving/api/v1/query_rangeand/api/v1/querywith pluggable response factories.- Playwright fixtures (
dashboard/e2e/fixtures.ts) — per-test fixture that starts both mock servers on random ports, spawns the BFF process pointed at the mocks, and tears everything down after the test. - Dist freshness guard (
dashboard/e2e/global-setup.ts) — a PlaywrightglobalSetupthat fails the run whendist/client/index.htmlis missing or older than a client source: recursively any file insrc/orpublic/, plusindex.htmlandvite.config.ts.public/is recursive because Vite copies its complete asset tree unchanged into the build.npm run test:e2ebuilds first so it always passes; runningnpx playwright testdirectly against a stale build now errors instead of giving a false-green signal. Bypass withSKIP_DIST_FRESHNESS_CHECK=1.
Prometheus Integration¶
PrometheusClient (dashboard/server/clients/prometheus.ts) authenticates and secures every
request when the corresponding env var is set: SARDEENZ_PROMETHEUS_BEARER_TOKEN_PATH adds an
Authorization: Bearer header (the token file is re-read on every request, since projected
ServiceAccount tokens rotate), SARDEENZ_PROMETHEUS_CA_PATH builds an undici dispatcher that
validates the endpoint's TLS certificate against that CA, and SARDEENZ_PROMETHEUS_TENANT_NAMESPACE
adds a namespace query parameter required by Thanos Querier's tenancy API. All three are unset
by default, which reproduces the original plain, unauthenticated fetch. When either the token
path or the tenant namespace is configured, isHealthy() probes with
GET /api/v1/query?query=vector(1) instead of GET /-/healthy, because the Thanos Querier
tenancy port does not serve the latter. See
deployment/dashboard/README.md for the OpenShift
user-workload monitoring setup that uses these three env vars.
The metrics dashboard exposes ten BFF routes, each backed by Prometheus queries:
| BFF route | Metric(s) queried | Type | Dashboard chart |
|---|---|---|---|
GET /api/metrics/latency |
sardeenz_proxy_request_duration_seconds |
Histogram | p50/p95/p99 latency line chart |
GET /api/metrics/throughput |
sardeenz_proxy_requests_total |
Counter | Throughput (req/s) line chart |
GET /api/metrics/connections |
sardeenz_proxy_active_connections, sardeenz_proxy_parked_connections |
Gauge | Active and parked connection gauges |
GET /api/metrics/parking-duration |
sardeenz_proxy_parking_duration_seconds |
Histogram | p50/p95 parking wait time |
GET /api/metrics/memory |
sardeenz_control_plane_device_memory_bytes |
Gauge | Device memory instant query |
GET /api/metrics/memory-history |
sardeenz_control_plane_device_memory_bytes |
Gauge | Device memory over time (range) |
GET /api/metrics/wake-triggers |
sardeenz_control_plane_wake_triggers_total |
Counter | Wake trigger frequency bar chart |
GET /api/metrics/state-transitions |
sardeenz_control_plane_state_transitions_total |
Counter | State transitions by type |
GET /api/metrics/evictions |
sardeenz_control_plane_evictions_total |
Counter | Evictions over time |
GET /api/metrics/operations |
sardeenz_control_plane_{deploy,sleep,wake,eviction,placement}_duration_seconds_sum, ..._count |
Histogram | Average duration per operation (bars) |
Time ranges map to PromQL step sizes: 15m → 15s, 1h → 60s, 6h → 300s, 24h → 900s, 7d → 3600s. The BFF constructs the PromQL and handles time range parameters; the frontend receives chart-ready arrays. GET /api/metrics/operations is the exception: it ignores step and instead evaluates ten instant queries (sum/count per operation) at end, over a window equal to end - start (minimum 60s), returning an average duration and count per operation rather than a time series — the sparse, one-off nature of deploy/sleep/wake/eviction/placement events made histogram_quantile over a rolling window mostly NaN between events.
Authentication¶
The BFF implements a lightweight auth system controlled by the AUTH_MODE environment variable.
Three modes¶
| Mode | Behavior |
|---|---|
none (default) |
No authentication. authenticate and requireRole decorators are no-ops. All routes are open. Use for local development and air-gapped deployments. |
simple |
Username/password login. POST /api/auth/login validates credentials against ADMIN_USERNAME / ADMIN_PASSWORD and returns a signed JWT. |
oauth |
OIDC authorization code flow. GET /api/auth/callback exchanges the code for tokens, resolves Sardeenz marker Roles through namespace-scoped Kubernetes RBAC, then issues an internal JWT. |
JWT flow¶
- Client authenticates and receives a JWT signed with
JWT_SECRET. - Subsequent requests include the token as
Authorization: Bearer <token>. - SSE routes that cannot send headers accept
?token=<jwt>as a query-parameter fallback. - The
authenticatepreHandler verifies the token;requireRole('admin-readonly')checks therolesclaim.
Route protection¶
Protected routes use { preHandler: [app.authenticate, app.requireRole('admin-readonly')] }. Public routes (/api/health, /api/auth/*, /healthz, /readyz) are exempt.
Internationalization (i18n)¶
The frontend uses react-i18next with a namespace-per-page pattern.
Namespaces¶
| Namespace | Pages / components |
|---|---|
common |
Shared strings: nav labels, action names, status labels, degraded banner |
cluster |
Cluster Overview page |
models |
Model List, Model Detail, Model Deploy pages |
workers |
Worker List, Worker Detail pages |
metrics |
Metrics Dashboard page |
auth |
Login page, OAuth callback page |
catalog |
Runner Catalog page |
playground |
Playground page |
Configuration¶
dashboard/src/i18n.ts initialises i18next with:
- Browser language detection (navigator → htmlTag, no localStorage caching)
- English as the sole locale at present (locale files at
dashboard/src/locales/en/) escapeValue: false(React handles XSS escaping)
New strings go in the appropriate namespace JSON file. New languages add a parallel locales/<lang>/ directory with the same files.
Degraded Mode¶
Two independent conditions can trigger the degraded state indicator:
Redis fallback (control plane unreachable)¶
When the BFF cannot reach the control plane, read routes fall back to Redis and include source: "redis-fallback" in the response body. The DegradedContext on the frontend detects this field across all active TanStack Query results and sets isDegraded = true. DegradedBanner then shows: "Control plane unreachable — showing cached data".
SSE degraded state¶
After 5 consecutive SSE connection failures, useEventStream transitions to degraded status (reconnect backoff 30s instead of 5s). DegradedBanner shows: "Real-time updates unavailable — polling for changes".
Priority¶
Redis fallback takes precedence in the banner (more severe: data staleness) over SSE degraded (less severe: push updates delayed). Both conditions auto-clear as soon as fresh data or a healthy SSE connection resumes — no manual dismissal required.