AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AI-300 · LESSON 2

Implement machine learning model lifecycle and operations25–30% · 4 topics

Learn the full ML lifecycle: orchestrating training jobs, registering and versioning models, deploying to production and monitoring in production.

2.1Orchestrate model training

Concept

Training runs as jobs. A command job runs one script, a sweep job explores hyperparameters, a pipeline job chains multiple steps (preprocessing → training → evaluation), and AutoML automates model selection and tuning. MLflow logs metrics, parameters and artifacts.

How it works / steps

  1. Log metrics in the training script with mlflow.autolog() or similar
  2. Submit it as a command job specifying environment, compute and input data
  3. In a sweep job, set the search space, sampling (random, grid, Bayesian) and early termination policy (such as Bandit)
  4. Componentize preprocessing, training and evaluation and connect them in a pipeline job
  5. Run pipelines periodically on a schedule or event

Workplace example

A demand forecasting pipeline runs on a schedule every Monday, a sweep explores learning rate and tree depth, and only the best run is passed to the next step.

Common confusions and exam tips

Mini quiz

Q1. You want to record the parameters and accuracy of multiple training jobs and compare them later. What is appropriate?

  1. Application Insights
  2. The job’s standard output log
  3. Experiment tracking with MLflow
  4. Data asset tags
Show answer and explanation

Answer: C Experiment tracking with MLflow

Logging parameters, metrics and artifacts with MLflow allows comparison across jobs. [Other options] Standard output logs are not structured for comparison. Application Insights is for operational monitoring, not experiment comparison. Data asset tags do not record trial metrics.

Q2. For a classification problem where you don’t know which algorithm fits, you want to explore candidates efficiently. Which feature is appropriate?

  1. The Responsible AI dashboard
  2. A sweep job (hyperparameter tuning)
  3. Automated machine learning (AutoML)
  4. A designer pipeline
Show answer and explanation

Answer: C Automated machine learning (AutoML)

AutoML automatically explores combinations of preprocessing and algorithms. [Other options] Sweep jobs explore parameters of a chosen algorithm. The designer builds pipelines by hand. The dashboard is for analyzing models.

Reference links

2.2Implement model registration and versioning

Concept

Models are registered by name and version in the workspace model registry. Registering in MLflow format allows no-code deployment without writing a scoring script or environment. To share across multiple workspaces (development, production), use Azure Machine Learning registries.

How it works / steps

  1. Register as an MLflow model from the best job’s artifacts (the version increments automatically)
  2. Record training data version, evaluation metrics and approval status in tags and properties
  3. Register models, environments and components in the organization’s registry and reference them from the production workspace
  4. Make clear which version goes to production, and keep old versions rather than deleting them to allow rollback

Workplace example

Version 7 of churn-model registered in the development workspace is copied to the organization’s registry, and the production workspace’s endpoint deploys it by specifying the version from the registry.

Common confusions and exam tips

Mini quiz

Q1. A registered model is no longer needed, but you want to keep its history for audits. What is the appropriate action?

  1. Just add a “deprecated” tag to the model
  2. Archive the model version
  3. Move the model to another workspace
  4. Delete only the model’s latest version
Show answer and explanation

Answer: B Archive the model version

Archiving removes it from the list while keeping history. [Other options] Deleting loses history. A tag alone keeps it showing in the list. Moving it to another workspace scatters audit history.

Q2. You want records of training run on a local PC to be kept in the Azure Machine Learning workspace. Which method is most appropriate?

  1. Copy results by hand into the workspace job description
  2. Stop local training and rerun it as a cloud job in the workspace
  3. Register the local mlruns folder as a workspace data asset
  4. Set the MLflow tracking URI to the workspace
Show answer and explanation

Answer: D Set the MLflow tracking URI to the workspace

Pointing the MLflow tracking URI to the workspace records even local training in the workspace. [Other options] Rerunning is unnecessary work. A data asset is not treated as run tracking. Manual copying is error-prone and loses metrics.

Reference links

2.3Deploy machine learning models for production environments

Concept

An endpoint is the entry point for calls; a deployment is the model and compute running behind it. Managed online endpoints serve real-time, low-latency inference; batch endpoints serve asynchronous inference on large data. One endpoint can hold multiple deployments switched by traffic percentage.

How it works / steps

  1. Create a managed online endpoint and choose authentication (key or Microsoft Entra token)
  2. Create a blue deployment (current) and a green deployment (new) and send 10% of traffic to green
  3. Mirror traffic can also send copies of production requests to the new version to validate it without returning responses
  4. If there are no problems, set green to 100% and delete blue
  5. Run large periodic inference on a batch endpoint with data assets as input

Workplace example

A new credit score model runs as green with 10% of traffic for a week; error rate and latency are checked before switching over fully.

Common confusions and exam tips

Mini quiz

Q1. You want to judge order acceptance instantly during screen operations, while processing large data for monthly demand forecasts overnight. Which combination is appropriate?

  1. Make both online endpoints and send many requests overnight for monthly forecasts
  2. Make both batch endpoints and return instant judgments after the job completes
  3. Real-time (online) endpoint for instant judgment, batch endpoint for monthly forecasts
  4. Batch endpoint for instant judgment, real-time (online) endpoint for monthly forecasts
Show answer and explanation

Answer: C Real-time (online) endpoint for instant judgment, batch endpoint for monthly forecasts

Online suits low-latency single requests; batch suits large asynchronous processing. [Other options] The fourth option reverses roles. Bulk processing on online endpoints is costly and heavy. Waiting for batch completion cannot judge instantly.

Q2. You want to continuously check whether the production model’s input data trends have changed since training. What is appropriate?

  1. Set threshold alerts on the endpoint’s request count and latency metrics
  2. Data drift monitoring with alerts and retraining triggers when thresholds are exceeded
  3. Save inference inputs and outputs to storage and check them when problems occur
  4. Record training evaluation metrics and have staff review them quarterly
Show answer and explanation

Answer: B Data drift monitoring with alerts and retraining triggers when thresholds are exceeded

Drift monitoring detects signs of degradation early and leads to retraining. [Other options] Request counts and latency do not show data trends. Quarterly reviews and after-the-fact checks are not continuous monitoring.

Reference links

2.4Monitor and maintain machine learning models in production

Concept

Model monitoring continuously checks whether production inputs and predictions have changed compared with training. Azure Machine Learning model monitoring computes data drift, prediction drift, data quality and feature attribution drift against baseline data (such as training data) and raises alerts when thresholds are exceeded.

How it works / steps

  1. Enable inference data collection (data collector) on the online endpoint
  2. Configure the monitoring baseline (training data or past production data), signals and thresholds
  3. Run monitoring on a schedule and receive alerts via Azure Monitor or email
  4. When drift is detected, investigate the cause and retrain, re-register and redeploy with the pipeline
  5. Also monitor latency and errors with Application Insights

Workplace example

After an alert that the distribution of “contract length” in churn prediction inputs changed greatly, the model is retrained on new data, validated as a green deployment and then switched over.

Common confusions and exam tips

Mini quiz

Q1. Which is the most appropriate mechanism for monthly retraining?

  1. Schedule only the training command job and evaluate quarterly
  2. Have staff run preprocessing, training and evaluation notebooks by hand every month
  3. Run a preprocessing, training, evaluation and registration pipeline on a schedule
  4. Reconfigure an AutoML job by hand every month and register the best model
Show answer and explanation

Answer: C Run a preprocessing, training, evaluation and registration pipeline on a schedule

Scheduling the pipeline retrains, evaluates and registers with the same steps every month. [Other options] Manual runs cause omissions and inconsistent steps. Evaluating on a different cycle means using retrained models unchecked.

Q2. Computing data drift in model monitoring requires baseline data for comparison. What is the most appropriate baseline?

  1. Part of the test data used for evaluation
  2. Randomly generated data with the same columns
  3. The last day of production inference input data
  4. Training data (or validated past production data)
Show answer and explanation

Answer: D Training data (or validated past production data)

Drift is computed by comparison with baseline data (usually training data or production data from a stable period). [Other options] Comparing recent data with itself does not show change since training. A small test set is a biased baseline. Random data has unrelated distributions.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)