Phase 5: MaaS Model Deployment

This phase deploys LLM models and registers them with the MaaS platform. Each model includes the inference workload (LLMInferenceService) and the MaaS control plane resources (MaaSModelRef, MaaSAuthPolicy, MaaSSubscription) that enable API key management, access control, and rate limiting.

All file paths and oc apply commands in this guide are relative to the rhoai-maas-guide repository root. Make sure you have cloned it and are working from its root directory (see Getting Started).
This guide is not a replacement for the official Red Hat OpenShift AI Models as a Service documentation. It is a companion resource with opinionated Kustomize manifests and automation scripts to accelerate deployment.

Prerequisites

  • Phases 1-4 completed (RHOAI installed, MaaS platform running)

  • MaaS Gateway available in openshift-ingress namespace

  • maas-api and maas-controller pods running in redhat-ods-applications

Available Models

simulator (recommended for initial testing)

A CPU-only mock LLM that requires no GPU. It uses the llm-d-inference-sim container to simulate an OpenAI-compatible inference endpoint. This is the best choice for validating the MaaS platform without GPU infrastructure.

  • Runtime: llm-d-inference-sim (CPU-only)

  • Resources: 100m-500m CPU, 256Mi-512Mi memory

  • GPU: Not required

  • Startup time: ~30 seconds

granite-tiny-gpu

Red Hat AI Granite 4.0-h-tiny FP8 Dynamic, a small Granite model (~1B parameters) running on the Red Hat AI Inference Server (vLLM CUDA). Suitable for clusters with modest GPU resources.

  • Runtime: vLLM CUDA (registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0)

  • Model weights: OCI modelcar from registry.redhat.io

  • Resources: 2-4 CPU, 8Gi-24Gi memory, 1x NVIDIA GPU

  • GPU memory: ~8GB required

  • Startup time: 5-15 minutes (image pull + model loading)

gemma

Google Gemma 2 9B IT FP8 running on the Red Hat AI Inference Server (vLLM CUDA). A 9B parameter instruction-tuned model with FP8 quantization, suitable for general-purpose text generation and code tasks. Fits on a single A10G GPU (24GB VRAM).

  • Runtime: vLLM CUDA (registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0)

  • Model weights: OCI modelcar registry.redhat.io/rhelai1/modelcar-gemma-2-9b-it-fp8:1.5

  • Resources: 2-4 CPU, 8Gi-24Gi memory, 1x NVIDIA GPU

  • GPU memory: ~12GB required (FP8 quantized)

  • Startup time: 5-15 minutes (image pull + model loading)

  • Context length: 4096 tokens (configurable via --max-model-len)

Gemma is a US-origin model, making it suitable for deployments with export control requirements. The OCI modelcar is available from registry.redhat.io and works in both connected and disconnected (air-gapped) environments.

gpt-oss-20b

OpenAI gpt-oss-20b running on the Red Hat AI Inference Server (vLLM CUDA). Requires a capable GPU node (A10G or better) with sufficient VRAM.

  • Runtime: vLLM CUDA (registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0)

  • Model weights: OCI modelcar from registry.redhat.io (~8GB download)

  • Resources: 2-4 CPU, 16Gi-60Gi memory, 1x NVIDIA GPU

  • GPU memory: ~16GB required

  • Startup time: 5-15 minutes (image pull + model loading)

simulator-disconnected

A variant of the simulator for air-gapped clusters. Uses oci:// storage URI instead of hf:// because HuggingFace is unreachable in disconnected environments. Functionally identical to the connected simulator - the inference container generates random responses regardless of the model URI.

  • Runtime: llm-d-inference-sim (CPU-only)

  • Resources: 100m-500m CPU, 256Mi-512Mi memory

  • GPU: Not required

  • Startup time: ~30 seconds (plus OCI modelcar init pull from mirror)

Custom Resource Descriptions

Each model deployment creates the following CRDs under maas.opendatahub.io/v1alpha1:

LLMInferenceService (namespace: llm)

Defines the inference workload: the container, model weights, resource requests, health probes, and gateway routing. This is the actual serving pod that handles inference requests.

MaaSModelRef (namespace: llm)

Registers the LLMInferenceService with the MaaS control plane. The MaaS API uses this reference to discover available models and route API requests to the correct inference endpoint.

MaaSAuthPolicy (namespace: models-as-a-service)

Defines who can access the model. The examples here grant access to all authenticated users (system:authenticated group). You can restrict access to specific users or groups by modifying the subjects field.

MaaSSubscription (namespace: models-as-a-service)

Defines rate-limiting tiers for model access. Each model ships with two tiers:

  • Free tier (priority 10): 100 tokens/min for all authenticated users

  • Premium tier (priority 20): 100,000 tokens/min for all authenticated users

Higher-priority subscriptions take precedence. You can adjust limits, windows, and subject groups to match your usage policies.

Deploying a Model

Gateway Pod OOMKill

Deploying a model triggers the Istio gateway to load Kuadrant Wasm extensions, which can push the pod past the default 1Gi memory limit. If you applied the gateway-resources.yaml ConfigMap in Phase 2 Step 5b, this is already handled (the limit is set to 2Gi via parametersRef).

If the gateway pod starts OOMKilling after deploying a model, see Phase 2: Gateway OOMKill Prevention for the fix.

Create the llm namespace if it doesn’t exist and label it for RHOAI:

oc create namespace llm
oc label namespace llm opendatahub.io/generated-namespace=true --overwrite
oc label namespace llm maas.opendatahub.io/gateway-access=true --overwrite
oc label namespace llm opendatahub.io/dashboard=true --overwrite
The maas.opendatahub.io/gateway-access label is required for any namespace that serves models through the MaaS Gateway. Without it, the Gateway will not accept HTTPRoutes from this namespace and the model will not be reachable.
The opendatahub.io/dashboard label makes the llm namespace appear as a Data Science Project in the Red Hat OpenShift AI dashboard.

Deploy the simulator (no GPU required):

oc apply -k manifests/05-maas-models/simulator/

HuggingFace Xet Download Hang

The simulator uses hf://sshleifer/tiny-gpt2 as its model URI. HuggingFace has migrated model storage to the Xet protocol, which can cause the KServe storage-initializer init container to hang indefinitely during download. If the model pod is stuck in Init:0/1 for more than 5 minutes, apply this workaround:

oc patch deployment facebook-opt-125m-simulated-kserve -n llm \
  --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/initContainers/0/env/-","value":{"name":"HF_HUB_DISABLE_XET","value":"1"}}]'

This disables the Xet protocol and falls back to standard HTTP downloads. The patched pod should complete init within 30 seconds. Note that KServe may revert this patch on reconciliation, so reapply if the model is redeployed.

GPU models using OCI modelcar images (granite-tiny-gpu, gpt-oss-20b) are not affected by this issue since they do not download from HuggingFace.

For disconnected (air-gapped) clusters, use the disconnected simulator variant:

oc apply -k manifests/05-maas-models/simulator-disconnected/

Or deploy a GPU model if your cluster has NVIDIA GPUs:

# Gemma 2 9B IT FP8 (12GB+ VRAM, single A10G)
oc apply -k manifests/05-maas-models/gemma/

# Granite 4.0-h-tiny FP8 (8GB+ VRAM)
oc apply -k manifests/05-maas-models/granite-tiny-gpu/

# GPT-oss-20b (16GB+ VRAM, A10G or better)
oc apply -k manifests/05-maas-models/gpt-oss-20b/

Verifying Model Readiness

After deploying, verify that the model is ready:

# Check LLMInferenceService status
oc get llminferenceservice -n llm

# Check that model pods are running
oc get pods -n llm

# Check MaaSModelRef registration
oc get maasmodelref -n llm -o wide

# Check MaaSAuthPolicy and MaaSSubscription
oc get maasauthpolicy -n models-as-a-service
oc get maassubscription -n models-as-a-service

The LLMInferenceService should show a Ready condition. For GPU models, initial startup takes 5-15 minutes while the container image is pulled and model weights are loaded into GPU memory.

Testing Inference

Once the model is ready, test inference through the MaaS API:

# Get the MaaS API domain
CLUSTER_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')
MAAS_URL="maas.${CLUSTER_DOMAIN}"

# Create an API key (requires the maas-api to be running)
API_KEY=$(curl -sk -X POST "https://${MAAS_URL}/maas-api/v1/api-keys" \
  -H "Authorization: Bearer $(oc whoami -t)" \
  -H "Content-Type: application/json" \
  -d '{"name": "test-key", "subscription": "simulator-free", "expiresIn": "1h"}' \
  | jq -r '.key')

# List available models
curl -sk "https://${MAAS_URL}/v1/models" \
  -H "Authorization: Bearer ${API_KEY}"

# Get the model endpoint URL from the listing
MODEL_URL=$(curl -sk "https://${MAAS_URL}/v1/models" \
  -H "Authorization: Bearer ${API_KEY}" | jq -r '.data[0].url')

# Send a chat completion request (uses the model-specific URL path)
curl -sk "${MODEL_URL}/v1/chat/completions" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "facebook/opt-125m",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

For GPU models, replace the model name with granite-4-tiny or gpt-oss-20b as appropriate.

Removing a Model

oc delete -k manifests/05-maas-models/simulator/

Appendix

Directory Structure

Each model follows the same Kustomize layout:

{model}/
  kustomization.yaml          # Aggregates llm/ and maas/ subdirectories
  llm/
    model.yaml                # LLMInferenceService CR
    kustomization.yaml        # Sets namespace: llm (+ namePrefix for simulator)
  maas/
    maas-model.yaml           # MaaSModelRef CR
    maas-auth-policy.yaml     # MaaSAuthPolicy CR (namespace: models-as-a-service)
    maas-subscription-free.yaml     # Free tier MaaSSubscription
    maas-subscription-premium.yaml  # Premium tier MaaSSubscription
    kustomization.yaml        # Lists all MaaS resources

References

Next step

Proceed to Phase 6: Verification.