Running secured MLOps pipeline in Azure DevOps

Sometimes Data Scientists use “Confidential” business datasets to perform ML experiments and ultimately train models as per the business problem statement. They have been asked to automate the whole process and create the MLOps pipeline, which runs in a highly secured environment (Managed System Identity) and automates consumption of “Confidential Dataset.”

Below is a typical MLOps (Machine Learning Ops) pipeline. Steps in this pipeline can be set up using a YML file and stored in a GIT repository.

Now, there is a defined configuration ,which you need to follow and execute to achieve MLOps pipeline running in Azure in the most secured way. Below are the steps performed on Azure from Infra and Azure DevOps perspective to create a ‘Managed System Identity’ (MSI) enabled MLOps pipeline:

Infra Side

• Create a Linux VM on Azure (usual Ubuntu 18.04, 20.04) to be used as your Azure DevOps build slave.
• Enable System Assigned Managed Identity on Linux VM.
• Login to your Linux VM and install Build Agent on the same (note: Create Personal Access Token [PAT ] in Azure DevOps and which will help to connect to your Azure DevOps Organization).

Once your VM is provisioned on Azure and the necessary DevOps build agent is installed on it, you need to attach this Linux VM to the Azure Machine Learning Workspace so that ultimately when your MLOps pipeline runs, it can submit the experiment to run on Azure Machine Learning (AML) compute.

In order to attach your Self-Hosted Linux VM with AML, navigate to your AML workspace -> IAM -> Create Role Assignment -> Add Hostname of your Azure VM with ‘Contributor’ role.

DevOps Engineering

Next, are steps from DevOps Engineering using Azure DevOps:

- Follow the below steps to set up Agent Pool in your Azure DevOps organization. Your Azure DevOps organization page url will be https://dev.azure.com/<YOUR_ORGANIZATION_NAME>.

- Go to your Azure DevOps Organization page -> Organization Settings -> Agent pool -> New Agent ->. Enter the same Linux VM hostname, which you set up to be used as your Self-Hosted Agent.

- You have a VM and you need to add this VM to your Azure DevOps Agent Pool so that when the MLOps pipeline runs, it can execute all steps on this Self-Hosted Azure VM.

Please note that only the Azure DevOps Organization Admin can set up the Agent Pool. Also, make sure that your Self-Hosted VM is visible online in your Azure DevOps Agent Pool.

- Create Service Connection in Azure DevOps.

- Create Service Connection in Azure DevOps based on Managed Identity.

Your setup is complete now from Azure Infra and DevOps Admin perspective.

Now, you need to jump on creating your MLOps pipeline on Azure DevOps.

Create your YML based Pipeline file and make sure that Azure Subscription Name is the name of your Service Connection:

Once you follow all steps as mentioned above, you can see the final outcome as :

Please note that the image above shows that the MLOps pipeline is running using System Assigned Managed Identity (MI) and running all your pipeline steps in a secured manner.