AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

MLA-C02 · LESSON 4

Operating, Monitoring, and Securing ML and AI Solutions24% · 3 topics

Learn monitoring of ML and AI model inference and performance, cost and performance optimization, and security of workloads and endpoints.

4.1Monitor inference and performance

4.1 Monitor inference and performance of ML and AI models.

Concept

SageMaker Model Monitor periodically checks data quality, model quality (comparison with ground truth labels), bias drift and feature attribution drift against a baseline. Use A/B testing with production variants to compare multiple models. For generative AI, monitor tokens, latency and traces with CloudWatch generative AI observability.

How it works / steps

  1. Enable data capture on the endpoint
  2. Create a baseline from training data
  3. Check on a Model Monitor schedule and alarm on violations in CloudWatch
  4. Start the retraining pipeline when drift is detected
  5. Compare a new model on part of the traffic with A/B testing

Workplace example

For a price prediction model, Model Monitor detects that the distribution of the input “region” has drifted far from the baseline and automatically starts the retraining pipeline.

Common confusions and exam tips

Mini quiz

Q1. You want to record actual prompts and responses of an app using Bedrock for quality investigations and audits. Which is the most appropriate setting?

  1. Record token counts and latency with Bedrock CloudWatch metrics
  2. Enable CloudTrail data events and store InvokeModel records in S3
  3. Enable Bedrock guardrail trace to record blocked requests
  4. Enable Bedrock model invocation logging to S3 or CloudWatch Logs
Show answer and explanation

Answer: D Enable Bedrock model invocation logging to S3 or CloudWatch Logs

Model invocation logging saves inputs and outputs to CloudWatch Logs or S3. CloudTrail records API calls and does not include the text. [Other options] CloudTrail records API calls without prompt bodies. Metrics are numbers only. Guardrail trace covers only guardrail decisions.

Q2. You are A/B testing with two production variants. Which is the most appropriate way to decide which model is better?

  1. Compare per-variant latency and error rate and pick the lower
  2. Compare offline validation accuracy and raise the weight of the better one
  3. Compare per-variant CloudWatch metrics and business metrics such as purchase rate
  4. Compare purchase rate in the first hour and pick the higher
Show answer and explanation

Answer: C Compare per-variant CloudWatch metrics and business metrics such as purchase rate

SageMaker AI emits metrics per variant. Compare technical and business metrics over a sufficient period. [Other options] Latency and errors alone don't show business impact. One hour is too short. Offline accuracy isn't a comparison on production users.

Reference links

4.2Optimize infrastructure cost and performance

4.2 Optimize and manage cost and performance of ML and AI infrastructure.

Concept

Optimize ML costs in training (Spot training, appropriate instances, early stopping), inference (Inference Recommender, auto scaling, multi-model endpoints, serverless, Savings Plans for SageMaker) and development (automatic notebook shutdown). Make costs visible with Cost Explorer and tags.

How it works / steps

  1. Understand ML costs per project with Cost Explorer and tags
  2. Use Spot training for training
  3. Right-size inference instances with Inference Recommender
  4. Stop or delete unused endpoints and notebooks
  5. Use SageMaker Savings Plans for steady usage

Workplace example

Delete development endpoints unused at night and automatically stop idle Studio notebooks, cutting monthly costs by 40%.

Common confusions and exam tips

Mini quiz

Q1. You want to view Bedrock costs separately for Accounting and other departments. Which is the most appropriate method?

  1. Create application inference profiles per department, tag them and aggregate with cost allocation tags
  2. Print the AWS bill for each department and have each estimate and write in their usage by hand
  3. Use the same model ID for all departments and divide the cost by headcount at month-end
  4. Do not split by department; record everything as used by Accounting
Show answer and explanation

Answer: A Create application inference profiles per department, tag them and aggregate with cost allocation tags

Tagging application inference profiles lets you track Bedrock usage and cost by tag.

Q2. Cost Explorer shows large costs for unused SageMaker Studio apps. What is the most appropriate next step?

  1. Turn off the Cost Explorer view so no one sees the cost graphs
  2. Configure idle auto-shutdown to stop unused apps
  3. Increase the number of Studio users so the apps get used
  4. Change all app instances to the largest type to speed up work
Show answer and explanation

Answer: B Configure idle auto-shutdown to stop unused apps

Studio idle shutdown automatically stops unused apps and reduces costs.

Reference links

4.3Secure workloads and endpoints

4.3 Secure ML and AI workloads and model endpoints.

Concept

Secure ML with IAM (least-privilege, role-based roles via SageMaker Role Manager), network isolation (training and hosting inside a VPC, network isolation mode, VPC endpoints), encryption (KMS for S3, EBS and training traffic) and management of credentials such as Bedrock API keys.

How it works / steps

  1. Create roles for data scientists, MLOps and other roles with SageMaker Role Manager
  2. Place training jobs and endpoints inside a VPC and restrict internet traffic
  3. Encrypt S3 and EBS data with KMS
  4. Control endpoint invocation with IAM
  5. Use short-term Bedrock API keys and manage long-term ones strictly

Workplace example

Run a training job on medical data in network isolation mode, completely blocking outbound traffic, and read data from S3 through a VPC endpoint.

Common confusions and exam tips

Mini quiz

Q1. You want to quickly create SageMaker AI roles with appropriate permissions for each role, such as data scientists and MLOps engineers. Which is the most appropriate method?

  1. Attach the AmazonSageMakerFullAccess managed policy to everyone's role
  2. Generate policies with IAM Access Analyzer as templates for each role
  3. Assign AdministratorAccess in an IAM Identity Center permission set
  4. Create roles for personas and ML activities with SageMaker Role Manager
Show answer and explanation

Answer: D Create roles for personas and ML activities with SageMaker Role Manager

Role Manager helps create least-privilege roles tailored to personas and ML activities. [Other options] FullAccess and AdministratorAccess break least privilege. Access Analyzer policy generation needs past activity, so it doesn't quickly create roles for new personas.

Q2. You train with an algorithm container provided by a third party. You want to prohibit all outbound traffic from the container during training. Which is the most appropriate setting?

  1. Use only one instance to keep traffic volume low
  2. Enable network isolation (EnableNetworkIsolation) on the training job
  3. Write “external traffic prohibited” in the training job name as a marker
  4. Disable logs inside the container so no traffic records remain
Show answer and explanation

Answer: B Enable network isolation (EnableNetworkIsolation) on the training job

With network isolation enabled, training and inference containers cannot connect to external networks.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)