Integrating Machine Learning With DevOps and Docker

Published in

Analytics Vidhya

6 min readMay 26, 2020

Why most of the machine learning projects are not implemented? Because the changing of the hyper-parameter is a very huge and tedious task. What are hyper-parameters?

Hyperparameters can be classified as model hyperparameters, that cannot be inferred while fitting the machine to the training set because they refer to the model selection task, or algorithm hyperparameters, that in principle have no influence on the performance of the model but affect the speed and quality of the learning process.

So. this article is related to automate our machine learning by changing some of the hyper-parameter.

* Prerequisites for the completion of this project:-

Docker
Jenkins
For smooth working try using Linux based OS
Git
Github.

* What we have to do:-

1. Create container image that’s has Python3 and Keras or NumPy installed using Dockerfile

2. When we launch this image, it should automatically start to train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins

4. Job1: Pull the Github repo automatically when some developers push the repo to Github.

5. Job2: By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing).

6. Job3: Train your model and predict accuracy or metrics.

7. Job4: if metrics accuracy is less than 80%, then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor: If the container where the app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.

* Steps to follow:-

Creating one directory where we push all are files and folder in the redhat8 system.

# mkdir /mlops_pro# cd mlops_pro

2. Now, here we create one Dockerfile for creating an image for running our CNN model.

Saving it and after that building the image using command

#  docker build -t tensorflow:latest .

* Now creating Jobs:-

Job1:- Pull the Github repo automatically when some developers push the repo to Github.

2. Job2:- By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the CNN processing).

This job will run after the successful launch of job1. Now using the above image a container will be launched if the code belongs to CNN.

3. Job3: Train your model and predict accuracy or metrics.

This job will run after the successful completion of job2. For the training model, I have created one task.py file. This file is given in the GitHub repo, link given at the end of the article.

So, you can see the accuracy is almost 84% and this accuracy has been stored in one file called accuracy.txt, which can be used further for twerking our model.

4. Job4: if metrics accuracy is less than 85%, then tweak the machine learning model architecture.

As you can clearly see that my accuracy was almost 84%, so I need to twerk my model for achieving this preferable accuracy.

This job will run after the successful completion of job3.

I have created one twerk.py file for tweaking our model according to the accuracy got in the previous job i.e job3.

In the last image, uploaded above you can clearly see that the accuracy now increased up to 91% by tweaking our model i.e by changing some hyper-parameter.

5. Job5: Retrain the model or notify that the best model is being created.

This job will send the mail to the developer, if we successfully reached the highest accuracy.

# Python code to illustrate Sending mail from# your Gmail accountimport smtplib# creates SMTP sessions = smtplib.SMTP('smtp.gmail.com', 587)# start TLS for securitys.starttls()# Authentications.login("sender_email_id", "sender_email_id_password")# message to be sentmessage = "Message_you_need_to_send"# sending the mails.sendmail("sender_email_id", "receiver_email_id", message)# terminating the sessions.quit()

This is the mail.py file.

Got the mail of getting the highest accuracy for the model.

6. Create One extra job job6 for monitor: If the container where the app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left.