Machine learning isn’t a destination – it’s a journey. To get the most from it, your models need to evolve alongside your needs, enabling you to ask new questions of your data, improve insights, and create models that get smarter as you do.
Even with the best training and data, a model rarely tells you everything you need to know immediately. The key to success is continuous evaluation and improvement.
The Google Cloud AI platform is equipped with capabilities to make that evaluation and improvement simple and streamlined. By automatically reviewing insights from your active models and workloads, and comparing them against a base truth designed by you, it doesn’t just serve you predictions – it tells you how strong those predictions are.
In this blog post, we’ll walk you through the model evaluation process in the AI Platform. In just a few minutes, you’ll learn how to create evaluation jobs for your deployed machine learning (ML) models, learn from feedback, and identify any potential errors that might impact your insights.
Step one: Creating your evaluation job
Once you’ve deployed a trained ML model that’s generating predictions in the AI Platform, you can create an evaluation job for that model. To do that, navigate to the version of the model you want to evaluate in the models page, click the evaluation tab, then click “set up evaluation job.”
Evaluation jobs work by saving some of your online prediction inputs and outputs in a BigQuery table, then evaluating that sample to generate evaluation metrics. For that to happen, you’ll also be asked to define a target BigQuery table, along with all relevant prediction keys, so the Data Labeling Service can extract necessary information from raw prediction inputs.
Step two: Establishing your ground truth
In order to properly evaluate the prediction insights of your ML model, you first need to establish some ground truth labels for the ML task that you’re running. This is where you decide what “good” looks like for your prediction insights.
These labels are used as an answer key and enable the AI platform to compare your model’s prediction results against a basic output expectation. You can either create these labels yourself and manually assign them to your evaluation job or ask Data Labeling Service to do it for you.
Step three: Starting your continuous evaluation job
Your continuous evaluation job is now configured and almost ready to go. But there are a few key APIs you’ll need to enable before you set your job to start:
- AI Platform Training and Prediction API
- Data Labeling Service API
- BigQuery API
With all your variables specified and your APIs enabled, click on the Create button to start your evaluation job. Prediction input and output should begin getting sampled from your model version into your BigQuery table immediately.
Step four: Viewing and comparing your evaluation metrics
You can receive evaluation insights in different formats, depending on your needs. If you’re only evaluating a single model version in isolation, follow this process:
- Open the AI Platform models page in the Cloud Console.
- Click on the name of the model containing the version you want.
- Click on the Evaluation tab.
This will return detailed evaluation insights, including a chart of mean average precision over time. It also displays any errors from your model version’s recent evaluation job runs.
AI Platform Prediction can also group multiple model versions together in a model resource to be evaluated and compared. Each model version in one model should perform the same task, but they may each be trained differently.
Start building smarter machine learning models today
As the name suggests, machine learning is a learning process. The deepest insights and strongest predictions are only achievable by those committed to continuously improving and refining their models to deliver better insights. That’s why we’ve made continuous improvement simple in the Google Cloud AI Platform.
By blending intuitive improvement features with the raw power of NVIDIA® GPUs, NVIDIA and Google Cloud are laying the ideal foundation for ML excellence.
Find out more about the powerful combination of NVIDIA GPUs and Google Cloud, get inspired by AI, and discover how you can redefine possible for your organization at thepoweroftwo.solutions.