Configuring the connection to storage

In node 2, the notebook uploads the model to the S3 storage bucket. You must set the S3 storage bucket keys by using the secret created by the My Storage connection that you set up in Storing data with connections.

You can use this secret in your pipeline nodes without having to save the information in your pipeline code. This is important, for example, if you want to save your pipelines - without any secret keys - to source control.

The name of the secret is aws-connection-my-storage.

If you named your connection something other than My Storage, you can obtain the secret name in the OpenShift AI dashboard by hovering over the help (?) icon in the Connections tab.

My Storage Secret Name

The aws-connection-my-storage secret includes the following fields:

  • AWS_ACCESS_KEY_ID

  • AWS_DEFAULT_REGION

  • AWS_S3_BUCKET

  • AWS_S3_ENDPOINT

  • AWS_SECRET_ACCESS_KEY

You must set the secret name and key for each of these fields.

Prerequisites
Procedure

  1. Remove any pre-filled environment variables.

    1. Select node 2, and then select the Node Properties tab.

      Under Additional Properties, note that some environment variables have been pre-filled. The pipeline editor inferred that you need them from the notebook code.

      Because you do not want to save the value in your pipelines, remove all of these environment variables.

    2. Click Remove for each of the pre-filled environment variables.

      Remove Env Var

  2. Add the S3 bucket and keys by using the Kubernetes secret.

    1. Under Kubernetes Secrets, click Add.

      Add Kubernetes Secret

    2. Enter the following values and then click Add.

      • Environment Variable: AWS_ACCESS_KEY_ID

        • Secret Name: aws-connection-my-storage

        • Secret Key: AWS_ACCESS_KEY_ID

          Secret Form

  3. Repeat Step 2 for each of the following Kubernetes secrets:

    • Environment Variable: AWS_SECRET_ACCESS_KEY

      • Secret Name: aws-connection-my-storage

      • Secret Key: AWS_SECRET_ACCESS_KEY

    • Environment Variable: AWS_S3_ENDPOINT

      • Secret Name: aws-connection-my-storage

      • Secret Key: AWS_S3_ENDPOINT

    • Environment Variable: AWS_DEFAULT_REGION

      • Secret Name: aws-connection-my-storage

      • Secret Key: AWS_DEFAULT_REGION

    • Environment Variable: AWS_S3_BUCKET

      • Secret Name: aws-connection-my-storage

      • Secret Key: AWS_S3_BUCKET

  4. Select File → Save Pipeline As to save and rename the pipeline. For example, rename it to My Train Save.pipeline.

Verification

  • You set the S3 storage bucket keys by using the secret created by the My Storage connection.

Next step

Running your pipeline