Phase 0: Disconnected Setup

Additional steps to deploy Red Hat OpenShift AI Models as a Service on an air-gapped (disconnected) OpenShift cluster.

This phase is only needed for disconnected environments where the cluster has no internet access. For connected clusters, skip directly to Phase 1.

Overview

A disconnected OpenShift cluster pulls all container images from a private mirror registry instead of the public registries. Before deploying MaaS, the operator catalogs and container images must be mirrored to that registry.

This guide assumes:

  • OpenShift 4.20+ is already installed in disconnected mode

  • Red Hat OpenShift AI 3.4 operators are installed (rhods-operator, cert-manager, OSSM3, NFD, GPU operator)

  • A mirror registry is running and the cluster trusts its CA

  • ImageDigestMirrorSet (IDMS) and CatalogSources from the RHOAI mirror are already applied

The MaaS guide adds 9 operators and several container images that are NOT part of the base RHOAI mirror set. This phase mirrors those extra components.

Currently, oc-mirror mirrors all images referenced in the RHOAI operator bundle regardless of which DSC components you enable. There is an initiative (RHOAIENG-75890 / OCPSTRAT-3350) to support selective mirroring - so you can mirror only the images for the components you need (e.g. Dashboard + MaaS). Until that lands, this guide mirrors the full set.

MaaS operators to mirror

Operator Package Name Purpose

Red Hat Connectivity Link

rhcl-operator

API gateway, auth (Authorino), rate limiting (Limitador)

Authorino Operator

authorino-operator

Auth engine (RHCL dependency, required >= 1.4.2)

DNS Operator

dns-operator

DNS policy (RHCL dependency, required >= 1.4.1)

Limitador Operator

limitador-operator

Rate limiting engine (RHCL dependency, required >= 1.4.1)

Leader Worker Set

leader-worker-set

Distributed inference coordination

MetalLB

metallb-operator

LoadBalancer service for the MaaS gateway

Cluster Observability Operator

cluster-observability-operator

Monitoring stack

OpenTelemetry

opentelemetry-product

Distributed tracing

Tempo

tempo-product

Trace storage backend

Additional images to mirror

The vLLM CUDA runtime is published under registry.redhat.io/rhaii/ (double 'i', no trailing 's'). Some upstream references incorrectly use rhaiis/ - this will fail on disconnected clusters because the IDMS only covers rhaii/. The model manifests in this guide already use the correct rhaii/ path with a pinned digest. See RHOAIENG-82814.

Images not discoverable through operator bundles:

Image Purpose

registry.redhat.io/rhel9/postgresql-16@sha256:3943…​ (pinned digest)

MaaS PostgreSQL database

registry.redhat.io/rhaii/vllm-cuda-rhel9@sha256:5800…​ (pinned digest)

vLLM CUDA runtime for GPU models

registry.access.redhat.com/rhcl-1/wasm-shim-rhel9@sha256:…​ (pinned digest)

Kuadrant WASM shim for gateway auth and rate limiting (see Step 6: Patch RHCL for WASM shim)

ghcr.io/llm-d/llm-d-inference-sim:v0.7.1

Simulator runtime (CPU-only model)

registry.redhat.io/rhai/modelcar-granite-4-0-h-tiny-fp8-dynamic:3.0

Granite 4.0-h-tiny model weights

registry.redhat.io/rhelai1/modelcar-gpt-oss-20b:1.5

GPT-oss-20b model weights

registry.redhat.io/rhelai1/modelcar-gemma-2-9b-it-fp8:1.5

Gemma 2 9B IT FP8 model weights
The WASM shim image lives on registry.access.redhat.com (not registry.redhat.io). This is a different registry. The WASM shim digest changes with each RHCL operator version - see Step 6: Patch RHCL for WASM shim for how to discover it.

Mirroring workflow

Step 1: Mirror the MaaS content

An ImageSetConfiguration is provided at manifests/00-disconnected/imageset-config-maas.yaml. It includes all 11 operators (Red Hat OpenShift AI base + MaaS) and the additional images listed above.

Before running oc-mirror, discover the WASM shim digest for your RHCL version and update the ImageSetConfiguration:

# If RHCL is already installed on a connected cluster, get the digest:
oc get csv -n openshift-operators \
  -l operators.coreos.com/rhcl-operator.openshift-operators \
  -o jsonpath='{range .items[0].spec.install.spec.deployments[0].spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep RELATED_IMAGE_WASMSHIM

# Replace the placeholder in imageset-config-maas.yaml with the digest

On a connected system (jump box) with oc-mirror v2 and a valid pull secret:

# Set the staging directory (needs ~100 GB free for the MaaS operators)
export WORKDIR=/mnt/mirror/maas-mirror

# Run oc-mirror to disk
oc-mirror --v2 \
  --config manifests/00-disconnected/imageset-config-maas.yaml \
  file://${WORKDIR}
If the RHOAI operators are already mirrored, oc-mirror only downloads the 6 new MaaS operators and the additional images. The run is incremental.

Step 2: Transfer to the disconnected side

Transfer the archive to the disconnected host. Use rsync for resumable transfers:

rsync -avP --append ${WORKDIR}/ user@highside:/mnt/mirror/maas-mirror/

Step 3: Push to the mirror registry

On the disconnected host, push the content into the mirror registry:

oc-mirror --v2 \
  --from /mnt/mirror/maas-mirror \
  docker://<mirror-registry>:<port>/<namespace>

This produces IDMS/ITMS and CatalogSource files in working-dir/cluster-resources/.

Step 4: Apply cluster resources

Apply the generated resources to the cluster:

# Apply the IDMS (triggers a MachineConfig rollout - nodes will reboot)
oc apply -f /mnt/mirror/maas-mirror/working-dir/cluster-resources/idms-oc-mirror.yaml

# Apply the CatalogSource
oc apply -f /mnt/mirror/maas-mirror/working-dir/cluster-resources/cs-*.yaml

If the cluster already has a CatalogSource alias named redhat-operators, update it to point to the new catalog index image (which now includes the MaaS operators):

# Get the new catalog image from the generated CatalogSource
NEW_IMAGE=$(grep 'image:' /mnt/mirror/maas-mirror/working-dir/cluster-resources/cs-redhat-operator-index-*.yaml | awk '{print $2}' | head -1)

# Update the alias
oc patch catalogsource redhat-operators -n openshift-marketplace \
  --type=merge -p "{\"spec\":{\"image\":\"${NEW_IMAGE}\"}}"

Step 5: Create ITMS for modelcar images

oc-mirror creates ImageDigestMirrorSet (IDMS) entries for images referenced by digest, but OCI modelcar images use tags (e.g. :3.0, :1.5). CRI-O needs an ImageTagMirrorSet (ITMS) to redirect tag-based pulls to the mirror registry.

A ready-to-use ITMS manifest is provided at manifests/00-disconnected/itms-maas-modelcars.yaml. It covers rhai/, rhelai1/, registry.access.redhat.com/rhcl-1/, and ghcr.io/llm-d namespaces.

Check if ITMS entries already exist:

oc get imagetagmirrorset -o yaml | grep -E 'registry.redhat.io/rhai|registry.redhat.io/rhelai1'

If no ITMS entries exist, apply the manifest (replace the placeholder with your mirror address):

sed 's|MIRROR_REGISTRY:PORT/NAMESPACE|<your-registry>:<port>/<ns>|g' \
  manifests/00-disconnected/itms-maas-modelcars.yaml | oc apply -f -

Replace <your-registry>:<port>/<ns> with your mirror registry address (the same one used in Step 3).

Applying ITMS triggers a MachineConfig update. Worker nodes will reboot in a rolling fashion. Wait for oc get mcp worker to show UPDATED=True before continuing.
Batch all ITMS/IDMS changes into a single oc apply. Do not add them incrementally - each apply triggers a separate MachineConfig rollout with node reboots.

Step 6: Patch RHCL for WASM shim

The Kuadrant/RHCL WASM shim is loaded at runtime by the Envoy gateway pod using the OCI protocol. This fetch does not go through CRI-O and does not respect IDMS/ITMS/ICSP. On a disconnected cluster, the gateway tries to reach registry.access.redhat.com directly. When it fails, Kuadrant applies a deny-all RBAC filter and every request returns "RBAC: access denied".

The fix is to set the RELATED_IMAGE_WASMSHIM environment variable on the RHCL operator subscription, which tells the operator to serve the shim from the mirrored image via its internal cluster service instead of having the gateway fetch it from the public registry.

First, find the WASM shim digest from the installed RHCL CSV:

WASM_IMAGE=$(oc get csv -n openshift-operators \
  -l operators.coreos.com/rhcl-operator.openshift-operators \
  -o jsonpath='{range .items[0].spec.install.spec.deployments[0].spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep RELATED_IMAGE_WASMSHIM | cut -d= -f2)
echo "WASM shim image: $WASM_IMAGE"

Build the mirrored image reference (replace the source registry with your mirror):

# Extract the digest from the image reference
WASM_DIGEST=$(echo "$WASM_IMAGE" | grep -o 'sha256:.*')

# Build the mirrored reference
MIRROR_REGISTRY="<your-registry>:<port>/<ns>"
WASM_MIRROR="${MIRROR_REGISTRY}/rhcl-1/wasm-shim-rhel9@${WASM_DIGEST}"
echo "Mirrored WASM shim: $WASM_MIRROR"

Patch the RHCL subscription:

oc patch subscription rhcl-operator -n openshift-operators --type=merge -p "{
  \"spec\": {
    \"config\": {
      \"env\": [{
        \"name\": \"RELATED_IMAGE_WASMSHIM\",
        \"value\": \"${WASM_MIRROR}\"
      }]
    }
  }
}"

Wait for the RHCL operator pod to restart with the new env var:

oc get pods -n openshift-operators -l app.kubernetes.io/name=kuadrant-operator -w

After the operator restarts, delete the gateway pod so it picks up the new WASM shim location:

oc delete pod -n openshift-ingress -l istio.io/gateway-name=maas-default-gateway
If the mirror registry uses a self-signed CA, the gateway pod may fail with a TLS certificate error when fetching the WASM shim. Ensure the mirror registry CA is in the cluster’s proxy/trust bundle (oc get proxy/cluster -o jsonpath='{.spec.trustedCA.name}').

Verification

Confirm all MaaS operators are available:

for pkg in rhcl-operator authorino-operator dns-operator limitador-operator \
           leader-worker-set metallb-operator \
           cluster-observability-operator opentelemetry-product tempo-product; do
  echo -n "  $pkg: "
  oc get packagemanifest "$pkg" -n openshift-marketplace \
    -o jsonpath='{.status.catalogSource}' 2>/dev/null && echo || echo "NOT FOUND"
done

All 9 should resolve to a CatalogSource. If any show NOT FOUND, the mirror set is incomplete - check the oc-mirror output for errors.

Verify the gateway pod has no WASM errors:

oc logs -n openshift-ingress -l istio.io/gateway-name=maas-default-gateway --tail=20 | grep -i wasm

If you see cannot fetch Wasm module or applying deny RBAC filter, the WASM shim patch from Step 6: Patch RHCL for WASM shim was not applied or the image is not in the mirror registry.

GPU setup (optional)

GPU MachineSet

If the cluster has no GPU nodes and you plan to deploy GPU models (Gemma, Granite, GPT-oss), create a GPU MachineSet.

A template is provided at manifests/00-disconnected/gpu-machineset.yaml. Before applying, discover the cluster-specific values:

export INFRA_ID=$(oc get -o jsonpath='{.status.infrastructureName}' infrastructure cluster)
export AMI=$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.ami.id}')
export SUBNET=$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.subnet.id}')
export REGION=$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.placement.region}')
export AZ=$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.template.spec.providerSpec.value.placement.availabilityZone}')
export INSTANCE_TYPE=g5.2xlarge
export MS_NAME=${INFRA_ID}-gpu-${AZ}
export IAM_PROFILE=${INFRA_ID}-worker-profile
export SG=${INFRA_ID}-node

envsubst < manifests/00-disconnected/gpu-machineset.yaml | oc apply -f -

Wait for the GPU node to be Ready:

oc get machines -n openshift-machine-api -w
# Once Running, check for GPU labels:
oc get nodes -l nvidia.com/gpu.present=true

The default instance type is g5.2xlarge (1x A10G, 24 GB VRAM) - enough for Gemma 2 9B FP8. Edit the template to use g5.12xlarge (4x A10G, 96 GB VRAM) for larger models.

GPU driver compilation on disconnected

The NVIDIA GPU operator compiles the NVIDIA kernel module at runtime, against the exact kernel running on the GPU node. On OpenShift it does this one of two ways, and which one you get decides whether a disconnected cluster needs anything extra:

  • With the Driver Toolkit (DTK) - the driver pod gets a second container built from the Driver Toolkit image, which ships in the OpenShift release payload and is therefore already mirrored. That image contains the matching kernel sources at /usr/src/kernels/<kernel-version>, so the build downloads nothing. The operator symlinks dnf to /usr/bin/true inside the pod, which makes the dnf install kernel-* calls in the build script inert.

  • Without DTK - the driver container builds alone and its dnf is real. It is UBI-based, and the standard UBI repos do not ship kernel packages, so the build script enables rhocp-<ocp-version>-for-rhel-9-x86_64-rpms and rhel-9-for-x86_64-baseos-eus-rpms and installs kernel-headers, kernel-devel and kernel (with vmlinuz) from them. Both are entitled Red Hat CDN repos and are unreachable from a disconnected cluster. This is the only case that needs a local repo.

Both paths run the same build script and ask for the same packages. The only thing that decides between them is whether the operator can pair the driver with a DTK image whose kernel matches the node. Check that first - on a healthy disconnected cluster it matches and there is nothing to do.

Step 1: Confirm the node and DTK kernels match

# The kernel actually running on the GPU node.
# Do NOT use `uname -r` - that reports the kernel of whatever host you are
# typing on (usually a RHEL bastion), not the RHCOS node.
oc get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kernelVersion}{"\n"}{end}'

# The kernel baked into the Driver Toolkit for the cluster's release payload
DTK_IMAGE=$(oc adm release info --image-for=driver-toolkit)
podman run --rm --entrypoint cat "$DTK_IMAGE" /etc/driver-toolkit-release.json

The two kernel versions must be identical. If no GPU node exists yet, read the kernel from any worker instead - a GPU MachineSet cloned from the worker set boots the same RHCOS.

Nodes lag the release payload while a MachineConfig rollout is in progress, which breaks the match. Confirm oc get mcp shows every pool UPDATED=True, not paused and not degraded, before trusting the comparison - and check after any IDMS/ITMS reboot cycle, not before.
Create the GPU MachineSet by cloning the worker MachineSet and changing only the name, the machineset labels, replicas and instanceType. Leaving providerSpec.value.ami.id untouched makes the kernel match structural instead of coincidental.

Also confirm the DTK image is actually in the mirror. oc image info does not apply the cluster’s IDMS, so check the mirror path directly - IDMS maps quay.io/openshift-release-dev/ocp-v4.0-art-dev to <mirror-registry>/openshift/release:

oc get imagedigestmirrorset -o jsonpath='{range .items[*]}{range .spec.imageDigestMirrors[*]}{.source}{" -> "}{.mirrors[0]}{"\n"}{end}{end}' \
  | grep art-dev

podman manifest inspect <mirror-registry>/openshift/release@${DTK_IMAGE#*@} >/dev/null \
  && echo PRESENT || echo MISSING

If the kernels match and the DTK image is present, go to Step 2 - no local repo is required. If they do not match, fix the drift (converge the MachineConfigPool, or rebuild the GPU MachineSet from the current worker AMI). Do not jump to the fallback: it extracts from the DTK image, so a mismatched DTK yields the wrong kernel-devel and the module still will not load.

Step 2: Confirm the driver built

# Two containers means the DTK route engaged. One means it did not.
oc get pods -n nvidia-gpu-operator -l app=nvidia-driver-daemonset \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{" "}{end}{"\n"}{end}'

# The module is loaded and the GPU is visible
POD=$(oc get pods -n nvidia-gpu-operator -l app=nvidia-driver-daemonset -o name | head -1)
oc exec -n nvidia-gpu-operator "$POD" -c nvidia-driver-ctr -- nvidia-smi

# Capacity exposed to the scheduler
oc get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{.items[*].status.capacity.nvidia\.com/gpu}'

A healthy DTK build shows nvidia-driver-ctr and openshift-driver-toolkit-ctr, and the DaemonSet is named for the RHCOS version (for example nvidia-driver-daemonset-9.6.20260715-0). At this point driver.repoConfig.configMapName in the ClusterPolicy should still be empty - that is expected and correct.

If the driver pod never appears at all, the GPU operator has no labels to match on. Check that NFD is healthy (oc get pods -n openshift-nfd) and that the node carries feature.node.kubernetes.io/pci-10de.present=true. A broken NFD looks exactly like a driver problem but is not one.

Fallback: serve kernel and CUDA RPMs over HTTP

Only needed when Step 2 shows a single-container driver pod - the operator did not pair with DTK despite a matching image being available. This happens on older GPU operator versions where use_ocp_driver_toolkit was unset or false (the field no longer exists in v26.x, where DTK handling is internal), or when the operator cannot resolve a DTK image for that RHCOS version.

The approach is to take the kernel content out of the DTK image, repackage it as RPMs, add the CUDA development packages, and serve them to the driver container over HTTP.

Extract kernel packages from DTK

The obvious shortcut - run dnf download kernel-devel-${KERNEL_VERSION} inside the DTK container and get genuine RPMs - does not work. The DTK image has only the UBI repos configured (ubi-9-baseos, ubi-9-appstream, ubi-9-codeready-builder) and carries no entitlement certificates, and UBI does not ship kernel packages. Downloading on the bastion instead needs a host entitled for the rhocp- and -eus-rpms repos, which a disconnected lab bastion generally is not. That leaves the files already installed inside the DTK image as the only source of kernel content at the node’s exact version - hence extraction.

If you do have an entitled RHEL host that can reach the CDN, download the real kernel-devel, kernel-headers and kernel RPMs at the node’s exact NVR and skip the rpmbuild step. Genuine RPMs are more reliable than hand-rolled ones because the NVRs and file layout are correct by construction.

On a host that can pull from the mirror registry:

DTK_IMAGE=$(oc adm release info --image-for=driver-toolkit)

# Take the version from the NODE, not from the extraction host
KERNEL_VERSION=$(oc get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{.items[0].status.nodeInfo.kernelVersion}')

podman create --name dtk-extract $DTK_IMAGE
podman cp dtk-extract:/usr/src/kernels/${KERNEL_VERSION} /tmp/kernel-src/
podman cp dtk-extract:/usr/include /tmp/kernel-headers/
podman cp dtk-extract:/lib/modules/${KERNEL_VERSION} /tmp/kernel-modules/
podman rm dtk-extract
Build RPMs and create a local repo
yum install -y rpm-build createrepo_c

# No spec files ship with this guide. Write a minimal spec per package that
# installs the extracted trees to their normal locations:
#   kernel-devel   -> /usr/src/kernels/${KERNEL_VERSION}
#   kernel-headers -> /usr/include
#   kernel         -> /lib/modules/${KERNEL_VERSION}, including vmlinuz
# The package NVRs must match ${KERNEL_VERSION} exactly, because the driver
# build script installs them by fully-qualified version.

mkdir -p /tmp/kernel-repo/rpms
cp ~/rpmbuild/RPMS/x86_64/kernel*.rpm /tmp/kernel-repo/rpms/
Download CUDA development packages

The NVIDIA driver container also needs CUDA development packages (cuda-devel, cuda-nvcc, cuda-cudart-devel) to compile. On a connected cluster the driver pod downloads these from developer.download.nvidia.com. On a fully air-gapped cluster that is unreachable, so the CUDA RPMs must be included in the same local repo.

This applies to the fallback path only. A DTK build resolves everything locally and runs nvidia-installer with --no-kernel-modules, so it never reaches developer.download.nvidia.com - one more reason to confirm Step 1 before doing any of this.

On a connected host with access to the NVIDIA CUDA repo:

# Add the NVIDIA CUDA repo (if not already configured)
dnf config-manager --add-repo \
  https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Download the CUDA packages and their dependencies
dnf download --resolve --destdir /tmp/cuda-rpms \
  cuda-devel-12-6 cuda-nvcc-12-6 cuda-cudart-devel-12-6

# Copy them into the same repo directory as the kernel RPMs
cp /tmp/cuda-rpms/*.rpm /tmp/kernel-repo/rpms/
The CUDA version must match what the GPU operator expects. Check the driver container image tag or the CUDA_VERSION env var in the ClusterPolicy to find the right version.

Generate the repo metadata:

createrepo_c /tmp/kernel-repo/rpms/
Serve the repo and configure the GPU operator
# Serve the repo over HTTP (from a host reachable by the GPU node)
cd /tmp/kernel-repo/rpms && python3 -m http.server 8088 &

# Create a ConfigMap with the repo config
oc create configmap kernel-repo-config -n nvidia-gpu-operator \
  --from-literal=local-kernel.repo="[local-kernel]
name=Local Kernel and CUDA Packages
baseurl=http://<bastion-ip>:8088/
enabled=1
gpgcheck=0
priority=1"

# Patch the ClusterPolicy to use the repo
oc patch clusterpolicy gpu-cluster-policy --type=merge \
  -p '{"spec":{"driver":{"repoConfig":{"configMapName":"kernel-repo-config"}}}}'

Replace <bastion-ip> with the IP of the host serving the repo. The GPU driver pod must be able to reach this host on port 8088.

The HTTP repo must stay running for the lifetime of the GPU nodes. The NVIDIA driver pod runs dnf install every time it starts - on node reboots, MachineSet scale-ups, operator upgrades, or ClusterPolicy changes. If the repo is gone, driver compilation fails and the GPU stays unusable. For production use, serve the repo from a persistent service (httpd container, nginx pod, or a proper artifact server) instead of a background python3 process. We opened RHOAIENG-82815 to track finding a better long-term solution for this.
Verify the driver
# Wait for the driver pod to complete compilation
oc get pods -n nvidia-gpu-operator -l app=nvidia-driver-daemonset -w

# Verify GPU is available
oc get nodes -l nvidia.com/gpu.present=true \
  -o jsonpath='{.items[*].status.allocatable.nvidia\.com/gpu}'

Troubleshooting

WASM shim: "RBAC: access denied"

Symptom: All MaaS API requests return "RBAC: access denied". The gateway pod logs show:

error wasm  error in converting the wasm config to local: cannot fetch Wasm module
oci://registry.access.redhat.com/rhcl-1/wasm-shim-rhel9@sha256:...
could not fetch manifest: dial tcp: lookup registry.access.redhat.com: no such host.
applying deny RBAC filter

Cause: The Kuadrant WASM shim is fetched at runtime by the Envoy gateway pod via the OCI protocol. This fetch bypasses CRI-O and does not respect IDMS/ITMS/ICSP. On a disconnected cluster, the public registry is unreachable, the fetch fails, and Kuadrant falls back to a deny-all RBAC filter.

Fix: Apply Step 6 of the Mirroring Workflow to patch the RHCL subscription with RELATED_IMAGE_WASMSHIM pointing to the mirror registry. Then restart the gateway pod.

TLS variant: If you see tls: failed to verify certificate: x509: certificate signed by unknown authority instead of no such host, the WASM shim is reachable but the mirror registry’s CA is not trusted by the gateway pod. Ensure the mirror CA is included in:

  • The cluster proxy trusted CA bundle: oc get proxy/cluster -o jsonpath='{.spec.trustedCA.name}'

  • The additionalTrustedCA in image.config.openshift.io/cluster

Certificates

In a disconnected environment, the mirror registry uses a self-signed CA. The cluster needs to trust this CA for image pulls AND for workloads running inside RHOAI (workbenches, model servers, pipelines).

If RHOAI was deployed using the standard disconnected procedure:

  • The mirror CA is already in the cluster’s image.config (additionalTrustedCA)

  • The DSCInitialization trustedCABundle is set to Managed with the CA in customCABundle

  • The odh-trusted-ca-bundle ConfigMap is propagated to all RHOAI-managed namespaces

Verify:

# Cluster trusts the mirror CA
oc get image.config.openshift.io/cluster -o jsonpath='{.spec.additionalTrustedCA.name}'

# DSCI has the CA bundle
oc get dscinitialization default-dsci -o jsonpath='{.spec.trustedCABundle.managementState}'

# CA bundle propagated to the MaaS namespace
oc get configmap odh-trusted-ca-bundle -n redhat-ods-applications

If any of these are missing, refer to the RHOAI disconnected installation guide, Chapter 8 (Working with certificates).

MaaS Gateway connectivity (AWS)

On AWS, the MaaS gateway creates a LoadBalancer service that defaults to an external NLB with public IPs. In a disconnected VPC, these public IPs are unreachable - both from the high-side host and from pods inside the cluster (the RHOAI dashboard’s maas-ui sidecar fails with EOF errors).

When you run setup-maas.sh with DISCONNECTED=true on AWS, the script automatically patches the Gateway to use an internal NLB:

# What setup-maas.sh does in Phase 2, Step 6:
oc patch gateway maas-default-gateway -n openshift-ingress --type=merge -p '{
  "spec": {
    "infrastructure": {
      "annotations": {
        "service.beta.kubernetes.io/aws-load-balancer-internal": "true"
      }
    }
  }
}'
# Then deletes the service so the controller recreates it with an internal NLB
oc delete svc maas-default-gateway-openshift-default -n openshift-ingress

After the internal NLB is provisioned (1-2 minutes), maas.<cluster-domain> resolves to private IPs reachable from inside the VPC.

If you already deployed MaaS without DISCONNECTED=true, you can apply the patch manually and delete the service. The gateway controller recreates it automatically.

The setup-maas.sh and verify.sh scripts also have a NodePort fallback - if the MaaS gateway URL is unreachable, they discover the NodePort and test via --resolve. You can test manually:

# Find the NodePort for HTTPS
NODEPORT=$(oc get svc maas-default-gateway-openshift-default -n openshift-ingress \
  -o jsonpath='{.spec.ports[?(@.port==443)].nodePort}')

# Find a node's internal IP
NODE_IP=$(oc get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')

# Test via NodePort
CLUSTER_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')
curl -sk --resolve "maas.${CLUSTER_DOMAIN}:${NODEPORT}:${NODE_IP}" \
  "https://maas.${CLUSTER_DOMAIN}:${NODEPORT}/maas-api/v1/health"

Next steps

Once the MaaS operators are available and (optionally) a GPU node is ready, proceed to Phase 1: Prerequisites and run the guide normally. Use the DISCONNECTED=true flag with setup-maas.sh to automatically skip Phase 8 (External Models) and use the disconnected simulator variant:

DISCONNECTED=true ./scripts/setup-maas.sh