Sanity-Check Pipeline

What will the pipeline do?

To make sure that everything works as we would expect it to, and that the model has not been tampered with, we will create a sanity-check pipeline that tests the model through its endpoint.
We will test the response time, the response quality, and that the model hash has not changed.
And to make sure it stays the same over time, we’ll schedule that pipeline.

Deploy a sanity-check pipeline

In the insurance-claim-processing/lab-materials/03/06 folder there are two pipeline files, one sanity_check.pipeline and one sanity_check.yaml file.

The .pipeline file can be opened in Elyra to be visually modified and executed, while the .yaml file can be imported into the pipeline server through the RHOAI Dashboard.
Here we will be running the pipeline through Elyra.

Ad-Hoc execution

Running it through Elyra is the same as doing an ad-hoc execution of the pipeline, as opposed to importing the pipeline which won’t automatically execute it.

  1. Start by going to your running workbench

  2. Navigate to the folder insurance-claim-processing/lab-materials/03/06

  3. Open up the sanity_check.pipeline file

  4. Here we can see that the pipeline consists of 3 checks:

    1. response quality check

    2. response time check

    3. security check

  5. Feel free to peek into each of the python files by double clicking on the nodes to see what they do.

  6. After the tests have been run, we have a final function that will summarize the results and log them.

  7. To run the pipeline, press the Play button in the menu bar.

    elyra sanity pipeline

  8. in the popup, leave the name unchanged and click OK:

    03 07 run pipeline ok

  9. When you get a popup that says Job submission to Data Science Pipelines succeeded, click the link Run details to see how the pipeline is progressing.

Schedule execution

We can also schedule an execution so that the sanity check is executed at regular intervals.
To do that:

  1. Go back to the OpenShift AI Data Science Project

  2. Find the pipeline you just ran

  3. Click the 3 dots at the very end of the line, and click "Create run".

    create run

  4. On the next screen, choose a name, select a Periodic run to trigger every Day and click Create:

    03 06 dailyrun

  5. We can now leave the sanity-check pipeline alone.

  6. It will run daily, and will inform us if anything goes wrong with our LLM.