Set up CI/CD

Managing the lifecycle of a ZenML pipeline with Continuous Integration and Delivery

Until now, we have been executing ZenML pipelines locally. While this is a good mode of operating pipelines, in production it is often desirable to mediate runs through a central workflow engine baked into your CI.

This allows data scientists to experiment with data processing and model training locally and then have code changes automatically tested and validated through the standard pull request/merge request peer review process. Changes that pass the CI and code review are then deployed automatically to production. Here is how this could look like:

Breaking it down

To illustrate this, let's walk through how this process could be set up with a GitHub Repository. Basically we'll be using Github Actions in order to set up a proper CI/CD workflow.

To see this in action, check out the ZenML Gitflow Repository. This repository showcases how ZenML can be used for machine learning with a GitHub workflow that automates CI/CD with continuous model training and continuous model deployment to production. The repository is also meant to be used as a template: you can fork it and easily adapt it to your own MLOps stack, infrastructure, code and data.

Configure an API Key in ZenML

In order to facilitate machine-to-machine connection you need to create an API key within ZenML. Learn more about those here.

zenml service-account create github_action_api_key

This will return the API Key to you like this. This will not be shown to you again, so make sure to copy it here for use in the next section.

Created service account 'github_action_api_key'.

Successfully created API key `default`.

The API key value is: 'ZENKEY_...'

Please store it safely as it will not be shown again.

To configure a ZenML client to use this API key, run:

...

Set up your secrets in Github