Phase 8: External Models (Optional)

This phase requires a completed MaaS installation (through Phase 4) and at least one local model deployed (Phase 5) to validate the gateway is functional.

This phase is NOT compatible with disconnected/air-gapped environments. External models require internet access to reach third-party APIs (OpenAI, Bedrock, Gemini). If you are running in disconnected mode, skip this phase entirely.

The ExternalModel CRD lets you expose third-party LLM APIs (OpenAI, AWS Bedrock, Google Gemini, Azure OpenAI, etc.) through the MaaS gateway. External models get the same API key management, authentication, rate limiting, and usage tracking as locally-served models - your consumers use a single gateway endpoint regardless of where the model runs.

All file paths and oc apply commands in this guide are relative to the rhoai-maas-guide repository root. Make sure you have cloned it and are working from its root directory (see Getting Started).

This guide is not a replacement for the official Red Hat OpenShift AI Models as a Service documentation. It is a companion resource with opinionated Kustomize manifests and automation scripts to accelerate deployment.

How It Works

An external model deployment requires five resources:

Secret - stores the provider API key. Must have the label inference.networking.k8s.io/bbr-managed=true so the BBR ext-proc can inject the credential into upstream requests.
ExternalModel - defines the provider, target model, endpoint, and credential reference. The BBR (Backend Binding Resolver) handles path rewriting and authentication header injection.
MaaSModelRef - registers the external model in the MaaS catalog so it appears in the API and can be governed.
MaaSAuthPolicy - grants access to specific users or groups.
MaaSSubscription - sets rate limits (tokens per hour) and priority.

Step 1: Create the Namespace and Secret

The external models live in their own namespace, separate from the MaaS platform and local models:

oc apply -f manifests/08-external-models/openai/namespace.yaml

The namespace manifest includes the maas.opendatahub.io/gateway-access=true label, which is required for the Gateway to accept HTTPRoutes from this namespace. If you create the namespace manually instead of using the manifest, remember to apply this label: oc label namespace external-models maas.opendatahub.io/gateway-access=true --overwrite

Create the provider credential Secret. This is done imperatively (never stored in manifests):

oc create secret generic openai-api-key \
    --from-literal=api-key="$OPENAI_API_KEY" \
    -n external-models \
    --dry-run=client -o yaml | oc apply -f -

oc label secret openai-api-key -n external-models \
    inference.networking.k8s.io/bbr-managed=true --overwrite

The inference.networking.k8s.io/bbr-managed=true label is required. Without it, the BBR ext-proc will not inject the API key into upstream requests and the provider will reject calls with 401.

Step 2: Apply the ExternalModel CR

oc apply -k manifests/08-external-models/openai/model/

This creates the ExternalModel resource:

apiVersion: maas.opendatahub.io/v1alpha1
kind: ExternalModel
metadata:
  name: gpt-4o-mini
  namespace: external-models
spec:
  provider: openai
  targetModel: gpt-4o-mini
  endpoint: api.openai.com
  credentialRef:
    name: openai-api-key

Step 3: Apply MaaS Governance

oc apply -k manifests/08-external-models/openai/maas/

This creates three resources:

MaaSModelRef - registers gpt-4o-mini in the MaaS catalog
MaaSAuthPolicy - grants all authenticated users access
MaaSSubscription - sets a free tier with 10,000 tokens/hour rate limit

Verification

Check the MaaSModelRef reaches Ready:

oc get maasmodelref gpt-4o-mini -n external-models
# Expected: PHASE = Ready

Check the model appears in the MaaS API:

MAAS_GW="https://maas.$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')"
curl -sk "${MAAS_GW}/maas-api/v1/models" \
    -H "Authorization: Bearer $(oc whoami -t)" | python3 -m json.tool

Create an API key and test inference:

# Create an API key bound to the openai-free subscription
API_KEY=$(curl -sk -X POST "${MAAS_GW}/maas-api/v1/api-keys" \
    -H "Authorization: Bearer $(oc whoami -t)" \
    -H "Content-Type: application/json" \
    -d '{"name": "openai-test", "subscription": "openai-free", "expiresIn": "1h"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['key'])")

# Test inference
curl -sk "${MAAS_GW}/external-models/gpt-4o-mini/v1/chat/completions" \
    -H "Authorization: Bearer ${API_KEY}" \
    -H "Content-Type: application/json" \
    -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Say hello in 3 words."}], "max_tokens": 20}'

A successful response contains a choices array with the model’s reply.

Automated Setup

The setup script handles all of the above in a single command:

./scripts/setup-maas.sh --from-phase 8 \
    --with-external-models \
    --external-model-provider openai \
    --external-model-api-key "$OPENAI_API_KEY"

The --external-model-provider flag selects which provider manifests to deploy (openai, gemini, or bedrock; default: openai).

Other Providers

Google Gemini

Manifests for Google Gemini (gemini-2.5-flash) are included under manifests/08-external-models/gemini/. To deploy Gemini instead of OpenAI, use --external-model-provider gemini.

There is a known incompatibility between the Gemini API and the BBR openai provider translator. The BBR hardcodes the upstream path to /v1/chat/completions, but Gemini’s OpenAI-compatible endpoint requires /v1beta/openai/chat/completions. This means ExternalModel registration and MaaS governance will work correctly, but inference requests will return HTTP 404 from Google’s API. This is tracked in RHOAIENG-68592 and upstream issue #260. Once a gemini provider translator is added to the BBR, Gemini external models will work end-to-end.

To deploy with Gemini (registration only, inference will 404):

./scripts/setup-maas.sh --from-phase 8 \
    --with-external-models \
    --external-model-provider gemini \
    --external-model-api-key "$GEMINI_API_KEY"

AWS Bedrock

Manifests for AWS Bedrock (openai.gpt-oss-20b via the Mantle endpoint) are included under manifests/08-external-models/bedrock/. Bedrock uses the bedrock-openai provider, which routes through the OpenAI-compatible Mantle API - no request body translation is needed.

Use bedrock-mantle.<region>.api.aws, not bedrock-runtime.<region>.amazonaws.com. The BBR translator uses /v1/chat/completions, which is only available on the Mantle endpoint. Using bedrock-runtime will result in 404 errors.

Prerequisites

Bedrock requires a Bedrock API Key (ABSK key), not standard AWS access keys. Create one via IAM service-specific credentials:

# Create an IAM user for Bedrock API access
aws iam create-user --user-name bedrock-api-user

# Attach Bedrock access policy
aws iam attach-user-policy \
    --user-name bedrock-api-user \
    --policy-arn arn:aws:iam::aws:policy/AmazonBedrockLimitedAccess

# Generate a long-term ABSK key (up to 90 days)
aws iam create-service-specific-credential \
    --user-name bedrock-api-user \
    --service-name bedrock.amazonaws.com \
    --credential-age-days 90

The output contains a ServiceSpecificCredential.ServicePassword field starting with ABSK - this is the API key to use.

Verify the ABSK key works before deploying: curl -s "https://bedrock-mantle.us-east-2.api.aws/v1/models" -H "Authorization: Bearer $BEDROCK_API_KEY" | python3 -m json.tool

Manual Deployment

Create the namespace if it does not already exist (shared with other external model providers):

oc apply -f manifests/08-external-models/bedrock/namespace.yaml

Create the credential Secret (never stored in manifests):

oc create secret generic bedrock-api-key \
    --from-literal=api-key="$BEDROCK_API_KEY" \
    -n external-models \
    --dry-run=client -o yaml | oc apply -f -

oc label secret bedrock-api-key -n external-models \
    inference.networking.k8s.io/bbr-managed=true --overwrite

Apply the ExternalModel and MaaS governance resources:

oc apply -k manifests/08-external-models/bedrock/model/
oc apply -k manifests/08-external-models/bedrock/maas/

The ExternalModel CR:

apiVersion: maas.opendatahub.io/v1alpha1
kind: ExternalModel
metadata:
  name: aws-gpt-oss-20b
  namespace: external-models
spec:
  provider: bedrock-openai
  targetModel: openai.gpt-oss-20b
  endpoint: bedrock-mantle.us-east-2.api.aws
  credentialRef:
    name: bedrock-api-key

The endpoint must match your AWS region. Replace us-east-2 with your region if different.

Verify the model reaches Ready:

oc get maasmodelref aws-gpt-oss-20b -n external-models
# Expected: PHASE = Ready

Test inference:

MAAS_GW="https://maas.$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')"

API_KEY=$(curl -sk -X POST "${MAAS_GW}/maas-api/v1/api-keys" \
    -H "Authorization: Bearer $(oc whoami -t)" \
    -H "Content-Type: application/json" \
    -d '{"name": "bedrock-test", "subscription": "bedrock-free", "expiresIn": "1h"}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['key'])")

curl -sk "${MAAS_GW}/external-models/aws-gpt-oss-20b/v1/chat/completions" \
    -H "Authorization: Bearer ${API_KEY}" \
    -H "Content-Type: application/json" \
    -d '{"model": "openai.gpt-oss-20b", "messages": [{"role": "user", "content": "Say hello in 3 words."}], "max_tokens": 20}'

Automated Deployment

./scripts/setup-maas.sh --from-phase 8 \
    --with-external-models \
    --external-model-provider bedrock \
    --external-model-api-key "$BEDROCK_API_KEY"

Known Issues

Gemini path incompatibility (RHOAIENG-68592)

The BBR openai provider hardcodes the upstream path to /v1/chat/completions, but Gemini requires /v1beta/openai/chat/completions. See the Google Gemini section above for details. Tracked in RHOAIENG-68592.

Appendix

Directory Structure

manifests/08-external-models/
  openai/
    kustomization.yaml           # Top-level: namespace + model + maas
    namespace.yaml               # external-models namespace
    model/
      kustomization.yaml
      external-model.yaml        # ExternalModel CR (gpt-4o-mini)
    maas/
      kustomization.yaml
      maas-model.yaml            # MaaSModelRef
      maas-auth-policy.yaml      # MaaSAuthPolicy (system:authenticated)
      maas-subscription.yaml     # MaaSSubscription (10k tokens/hour)
  gemini/
    kustomization.yaml
    namespace.yaml
    model/
      kustomization.yaml
      external-model.yaml        # ExternalModel CR (gemini-2.5-flash)
    maas/
      kustomization.yaml
      maas-model.yaml
      maas-auth-policy.yaml
      maas-subscription.yaml
  bedrock/
    kustomization.yaml
    namespace.yaml
    model/
      kustomization.yaml
      external-model.yaml        # ExternalModel CR (openai.gpt-oss-20b via Mantle)
    maas/
      kustomization.yaml
      maas-model.yaml
      maas-auth-policy.yaml
      maas-subscription.yaml

Supported Providers

The provider field in the ExternalModel spec maps to a BBR payload translator. Currently supported values:

openai - OpenAI API (api.openai.com)
anthropic - Anthropic API
azure-openai - Azure OpenAI Service
bedrock-openai - AWS Bedrock (OpenAI-compatible)

References

Next step

Proceed to Architecture & Request Flow.