Train Object detection model on Cloud ML Engine

Tensorflow is widely used for Machine learning and Deep learning. There are many models in tensorflow repo for use. But training devices are also a big issue, because these models require a lot of processing power. There are different Cloud environments like Azure, AWS and Google cloud for this purpose where different GPUs are avaiable to use. Google has create special processing devices (TPU) for this purpose, but they are only accessible on Cloud ML Engine. So, people can train models using service/api provided by google as Cloud ML Engine. It is easy to use and takes a little time to train a model using a lot of data. So, we will discuss that how we can train an object detection model using this api on cloud.

1. Create a VM on Cloud

In first steps, we will create a VM in our project. If you don't know how to create a project, please refer to this link. We will create a basic VM, because we are not training our model on this VM. After creating our VM, we need to create a bucket. In Cloud, buckets are used to store data, that is required for training of model, like images, config files, models etc. Buckets can only contain small letters _, or -. After creating a bcuket, create a folder for storing your data, you can directly upload your data on that bucket, just by drag and drop.

2. Intialize APIs for Project

ML Engine API is required to train model, which can be activated using API tab in your project. Open API dashboard and search for Cloud ML Engine and enable that API. It may take some time(seconds) to enable that api. After that you can use that api in next steps.

3. Install Required apps and libraries

ML Engine requires Google cloud CLI for submitting jobs. So we need to setup GCloud CLI on our VM. First click on SSH tab in your VM to open virtual machine. And execuate these commands to install GCloud CLI.

sudo apt-get update

# Create environment variable for correct distribution
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"

# Add the Cloud SDK distribution URI as a package source
echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

# Import the Google Cloud Platform public key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

# Update the package list and install the Cloud SDK
sudo apt-get update && sudo apt-get install google-cloud-sdk

View more details at https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu for installing on docker and initalizing. Follow Initaize the SDK

Next steps are to give roles to that service account. Which can be done by IAM and Admin tab.

4. Setup ML Engine API

Export Some variables for use in commands

# Set Project
gcloud config set project YOUR_PROJECT_ID
# Export Porject name for further use
export PROJECT="YOUR_PROJECT_ID"
# Export Bucket name for further use
export YOUR_GCS_BUCKET="YOUR_BUCKET_NAME"

Next, to give our Cloud TPU access to our project we need to add a TPU-specific service account. First, get the name of your service account with the following command:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
    https://ml.googleapis.com/v1/projects/${PROJECT}:getConfig

When this command completes, copy the value of tpuServiceAccount (it will look something like your-service-account-12345@cloud-tpu.iam.gserviceaccount.com) and then save it as an environment variable:

export TPU_ACCOUNT=your-service-account

Finally, grant the ml.serviceAgent role to your TPU service account:

gcloud projects add-iam-policy-binding $PROJECT  \
    --member serviceAccount:$TPU_ACCOUNT --role roles/ml.serviceAgent

5. Installing TensorFlow Object Detection

If this is your first time using TensorFlow Object Detection, welcome! To install it, follow the instructions here.
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md

Once you’ve installed Object Detection, be sure to test your installation by running the following:

python object_detection/builders/model_builder_test.py

If installation is successfull, you will see result somthing like this

Ran 18 tests in 0.079s

OK