Skip to content

Sardeenz Documentation

Welcome to the Sardeenz documentation! This directory contains comprehensive guides for understanding, developing, deploying, and using the platform.

Quick Navigation

📚 Core Documentation

Document Description
User Guide Dashboard walkthrough — loading models, benchmarks, cluster management
Architecture Overview High-level system architecture, design decisions, and technical details
Backend Architecture Detailed backend components, flows, and process management
Frontend Architecture Frontend component specs, state management, and API integration
API Guide API reference with code examples for Controller, Proxy, and Cluster APIs
Deployment Guide Container building, OpenShift/Kubernetes deployment, multi-pod cluster
RBAC Setup Kubernetes-native RBAC configuration for OAuth authentication

🛠️ Development Guides

Resource Description
Development Guide Getting started with local development
GPU Setup GPU environment setup (vLLM, kvcached, uv)
GPU-Free Development Inference simulator for development without GPU
PatternFly 6 Guide Complete UI development guide with PatternFly 6
Frontend API Client API client integration for the frontend

🔧 Technical Resources

Resource Description
kvcached GPU memory sharing documentation and setup guides
Specifications Design specifications, planning documents, and data models
Changelog Project change history

🚀 Getting Started

New to Sardeenz? Start here:

  1. Quick Deploy
  2. Pre-built images at quay.io/rh-aiservices-bu/sardeenz

  3. See deployment.md for OpenShift/Kubernetes setup

  4. Use the Dashboard

  5. Read the User Guide for a walkthrough of all dashboard features

  6. Understand the Project

  7. Read the main README for overview

  8. Review architecture.md for system design

  9. Develop Locally

  10. Follow the Development Guide
  11. See GPU Setup for vLLM/kvcached configuration

  12. No GPU? Use the Inference Simulator for GPU-free development

  13. Use the APIs

  14. Explore api-guide.md for API examples
  15. Test with provided code snippets (Python, JavaScript, curl)

Documentation by Role

🧑‍💻 Developers

Building features and fixing bugs?

Key Files:

  • apps/backend/ - Fastify backend source code
  • apps/frontend/ - React + PatternFly dashboard
  • packages/types/ - Shared TypeScript types

🏗️ DevOps Engineers

Deploying and managing the platform?

Key Resources:

  • GPU-enabled OpenShift cluster setup
  • PostgreSQL database (deployed alongside the application)
  • Model storage: HuggingFace cache PVC or local/mounted models (see deployment/README.md)
  • OAuth 2.0 integration with Kubernetes RBAC (see rbac-setup.md)
  • Kubernetes manifests for single/multi-pod StatefulSet (see deployment/)

📊 Data Scientists / ML Engineers

Loading and using models?

Supported Models:

  • Any model compatible with vLLM (Llama, Mistral, Qwen, etc.)
  • HuggingFace format or local paths

🎨 Frontend Developers

Working on the dashboard?

Key Files:

  • apps/frontend/src/components/ - Reusable UI components
  • apps/frontend/src/pages/ - Page-level components
  • apps/frontend/src/hooks/ - Custom React hooks

Document Index

Architecture & Design

Document Topics Covered
architecture.md System overview, technology stack, component architecture, data model, security, performance
architecture/backend-architecture.md Backend components, model loading flows, process management, GPU memory tracking
architecture/frontend-architecture.md Frontend components, state management, routing, API integration, real-time updates
specs/spec.md Feature requirements, priorities, user stories
specs/plan.md Implementation plan, milestones, timeline
specs/data-model.md Entity schemas, relationships, validation rules

API & Integration

Document Topics Covered
api-guide.md Controller API (load/unload/move models), Proxy API (inference), Cluster API (multi-pod), Internal API (inter-pod), authentication, error handling, code examples
specs/contracts/ OpenAPI 3.1 specifications (when available)
Cluster Admin API OpenAPI spec for cluster management endpoints
Internal API OpenAPI spec for inter-pod communication endpoints

Deployment & Operations

Document Topics Covered
deployment.md Container build, Docker Compose, OpenShift deployment, multi-pod cluster deployment, configuration, health checks, monitoring, troubleshooting
rbac-setup.md Kubernetes-native RBAC setup for OAuth mode, Role/RoleBinding configuration, ServiceAccount permissions
kvcached/ kvcached installation, configuration, memory segment management
deployment/ Kubernetes manifests: StatefulSet, PostgreSQL, Services, RBAC, ConfigMap, Secret

Development Guides

Document Topics Covered
development/README.md Local setup, prerequisites, project structure, development commands
dev-setup.md GPU development environment, vLLM setup, kvcached configuration
development/pf6-guide/ PatternFly 6 components, styling standards, testing patterns, troubleshooting
development/frontend-api-client.md Axios setup, API client patterns, error handling

Project Management

Document Topics Covered
CLAUDE.md Project overview, quick start, common commands, active technologies
CHANGELOG.md Project change history, feature additions, bug fixes
specs/quickstart.md Developer quickstart guide

Key Concepts

Controller API

The Controller API manages the lifecycle of model instances:

  • Load models dynamically without downtime
  • Unload models to free GPU memory
  • Query status of all running models
  • Monitor metrics (GPU usage, request counts)

Base URL: http://localhost:3000/api/v1/

Authentication: OAuth 2.0 with RBAC (admin, admin-readonly roles)

Proxy API

The Proxy API provides a unified inference endpoint:

  • OpenAI-compatible format (drop-in replacement)
  • Routes requests to the correct model instance
  • Streaming support via Server-Sent Events (SSE)
  • <50ms routing overhead (performance target)

Base URL: http://localhost:8000/v1/

Compatible with: OpenAI Python SDK, OpenAI JavaScript SDK, curl

Cluster Orchestration

Multi-pod cluster mode enables horizontal scaling across GPU nodes:

  • Leader election via Kubernetes Leases for coordinated management
  • Heartbeat protocol (5s interval) for health monitoring and state sync
  • Unified routing table for cross-pod inference request routing
  • Cross-pod model operations — load, unload, move, sleep, wake on any pod
  • Preset reconciliation — apply model presets across the entire cluster
  • HMAC-SHA256 authentication on all inter-pod communication

See Deployment Guide: Multi-Pod Cluster and API Guide: Cluster API.

Inference Simulator

The inference simulator (inference-sim) enables GPU-free development:

  • Simulates model loading with configurable startup duration
  • Tracks simulated GPU memory allocation
  • Returns echo responses for inference requests
  • Set INFERENCE_BACKEND=inference-sim to enable

See GPU-Free Development.

kvcached

kvcached is a GPU memory sharing tool that enables multiple vLLM instances to coexist on a single GPU:

  • IPC segments for shared KV cache
  • Memory limits enforced per model
  • Dynamic allocation based on demand

See kvcached/README.md for detailed setup instructions.

Model Instance Lifecycle

┌─────────────┐
│   STARTING  │  ← Model loading, process spawning
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   ACTIVE    │  ← Serving requests, healthy
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   STOPPING  │  ← Graceful shutdown
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   STOPPED   │  ← Process terminated, memory freed
└─────────────┘

       ┌─────────────┐
       │   FAILED    │  ← Error during startup or runtime
       └─────────────┘

Performance Targets

Metric Target Status
Proxy Routing Overhead <50ms (p95) 🔴 CRITICAL
Model Load Time <60s (1-3B params) 🟡 Target
Model Unload Time <30s 🟡 Target
Concurrent Models 3-5 (24GB GPU) 🟡 Target
Request Throughput vLLM native + <5% overhead 🟡 Target

FAQ

Can I use this with non-NVIDIA GPUs? No, vLLM and kvcached require NVIDIA GPUs with CUDA support. AMD GPUs (ROCm) are not currently supported.
Is this compatible with the OpenAI Python SDK? Yes! The Proxy API is OpenAI-compatible. Just point your OpenAI client to the proxy URL: 
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
How many models can I run simultaneously? It depends on your GPU memory. Example on a 24GB GPU: - 3-5 small models (1-3B params) - 2-3 medium models (7B params) - 1 large model (13B+ params) + 1-2 small models Use kvcached to optimize memory sharing.
Can I deploy this on CPU-only machines? Production deployment requires NVIDIA GPUs (vLLM needs CUDA). However, for **development and testing**, you can use the [inference simulator](./dev-setup.md#gpu-free-development-inference-simulator) (`INFERENCE_BACKEND=inference-sim`) which runs entirely on CPU with simulated GPU memory and echo responses.
Is persistent storage required? Yes, an App Data PVC (1Gi) is required for the SQLite database that stores benchmarks and memory profiles. For models, you can either: - Use a Model Cache PVC for HuggingFace downloads (optional) - Mount pre-downloaded models via `LOCAL_MODELS_PATH` (optional) - Or use both sources simultaneously See [deployment/README.md Storage Configuration](https://github.com/rh-aiservices-bu/sardeenz/blob/main/deployment/README.md#storage-configuration) for details.
How do I enable authentication? Set the `AUTH_MODE` environment variable: **Simple Mode** (username/password): - `AUTH_MODE=simple` - `ADMIN_USERNAME=admin` - `ADMIN_PASSWORD=` - `JWT_SECRET=` **OAuth Mode** (OpenShift): - `AUTH_MODE=oauth` - `OAUTH_ISSUER_URL=` - `OAUTH_CLIENT_ID=sardeenz` - `OAUTH_CLIENT_SECRET=` - `K8S_API_URL=` - `JWT_SECRET=` Roles are assigned via Kubernetes RoleBindings. See [rbac-setup.md](./rbac-setup.md) for complete RBAC configuration. See [api-guide.md](./api-guide.md#authentication) and [deployment.md](./deployment.md#configuration) for details.

Contributing

This project follows the principles defined in .specify/memory/constitution.md:

  • Type Safety: TypeScript strict mode
  • Performance-First: <50ms routing overhead
  • API-First Design: OpenAPI 3.1 specifications
  • Security by Design: OAuth 2.0 + RBAC
  • Observability: Prometheus metrics, structured logging

Before contributing, read:

  1. Architecture - System design
  2. Specifications - Feature requirements
  3. CLAUDE.md - Development setup

Support

License

To be determined

Last Updated: 2026-05-19 Project Status: Active Development (004-school-orchestration, 006-inference-sim-backend)