AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

MLA-C02 · LESSON 2

ML Model and Foundation Model (FM) Development24% · 3 topics

Learn how to choose ML and AI modeling approaches, train, fine-tune and customize models, and analyze and evaluate performance.

2.1Choose an appropriate modeling approach

2.1 Choose a modeling approach suited to ML and AI solutions.

Concept

Development effort increases in this order: ready-made AI services (Rekognition, Comprehend and others) → foundation models (Bedrock, JumpStart) → SageMaker AI built-in algorithms (XGBoost, Linear Learner, k-means, DeepAR, BlazingText and others) → your own models. Choose by problem type (classification, regression, time series, clustering, images, natural language), data volume, interpretability and cost.

How it works / steps

  1. Translate the problem into an ML task
  2. Check whether a ready-made AI service or a foundation model is enough
  3. For tabular classification and regression, start with built-in algorithms such as XGBoost
  4. Consider DeepAR and similar for time-series forecasting, and BlazingText and similar for text classification
  5. Choose a simple model if interpretability is required

Workplace example

First try inquiry classification with prompts to a Bedrock foundation model, and build your own classification model with labeled data only for categories where accuracy falls short.

Common confusions and exam tips

Mini quiz

Q1. A team with few ML experts wants to automatically build classification models from tabular data and compare candidates. Which is the most appropriate method?

  1. Pass the table to a Bedrock foundation model and have it classify
  2. Pick a pretrained JumpStart model and fine-tune it on the table
  3. Try preprocessing, algorithms and tuning automatically with SageMaker Autopilot
  4. Search XGBoost hyperparameters with automatic model tuning
Show answer and explanation

Answer: C Try preprocessing, algorithms and tuning automatically with SageMaker Autopilot

Autopilot automatically handles preprocessing, algorithm selection and tuning from tabular data, and lets you compare candidate models. [Other options] Automatic model tuning doesn't choose algorithms or preprocess. JumpStart fine-tuning and foundation models don't build and compare candidates automatically.

Q2. You want to choose a Bedrock foundation model for an assistant. Which is the most appropriate approach?

  1. Compare quality, latency and cost with Bedrock evaluations on your own question set
  2. Pick the top model on public benchmarks and verify it in production
  3. Start with the largest model and switch to smaller ones if cost becomes an issue
  4. Try a few questions in the Bedrock playground and pick the best impression
Show answer and explanation

Answer: A Compare quality, latency and cost with Bedrock evaluations on your own question set

Choose FMs by comparing quality, latency and cost on data from your own tasks. Bedrock evaluations let you compare. [Other options] Public benchmarks don't show quality for your use. Impressions from a few tries aren't reproducible. Starting largest postpones the cost requirement.

Reference links

2.2Train, fine-tune and customize models

2.2 Train, fine-tune and customize models for ML and AI solutions.

Concept

Train models with SageMaker training jobs and tune them with automatic model tuning (hyperparameter optimization: Bayesian, random, grid, Hyperband). Lower costs with Managed Spot Training (with checkpoints), and train large models faster with distributed training. Address overfitting with regularization (L1, L2), dropout and early stopping. Foundation models can be fine-tuned, continually pre-trained and distilled in Bedrock and JumpStart.

How it works / steps

  1. Decide the training script and hyperparameter search ranges
  2. Optimize the objective metric with automatic model tuning
  3. Lower costs with Spot training and checkpoints while preparing for interruptions
  4. Watch for signs of overfitting (the gap between training and validation error) and adjust regularization
  5. Detect problems such as vanishing gradients with SageMaker Debugger

Workplace example

Train an image classification model with Managed Spot Training, saving checkpoints to S3 every 30 minutes so it resumes where it left off if interrupted.

Common confusions and exam tips

Mini quiz

Q1. Training accuracy is 99%, but validation accuracy is only 70%. Which is the most appropriate countermeasure?

  1. Increase the learning rate and train until training accuracy reaches 100%
  2. Make the model even larger and more complex, and greatly increase the number of epochs
  3. Stop using validation data and evaluate the model on training accuracy alone
  4. Strengthen regularization or use early stopping, and also consider adding training data
Show answer and explanation

Answer: D Strengthen regularization or use early stopping, and also consider adding training data

High accuracy only on training data is overfitting. Improve generalization with regularization, dropout, early stopping and more data.

Q2. Accuracy is low on both training and validation data, stuck around 60% on each. Which is the most appropriate countermeasure?

  1. Add features or use a more expressive model, and weaken regularization
  2. Strengthen regularization further so the model fits training data less
  3. Mix validation data into training data so accuracy looks higher
  4. Halve the number of training records to shorten training time
Show answer and explanation

Answer: A Add features or use a more expressive model, and weaken regularization

Low on both is insufficient learning (underfitting). Increase model capacity or features.

Reference links

2.3Analyze and evaluate performance

2.3 Analyze and evaluate the performance of ML and AI systems.

Concept

For classification evaluation, use the confusion matrix, precision, recall, F1 and AUC-ROC; for regression, RMSE, MAE and R². Use MLflow in SageMaker for experiment tracking, and shadow testing (trying the new model on a copy of production traffic) for near-production validation. For foundation models and RAG, measure correctness, relevance and faithfulness with Bedrock evaluations (including knowledge base evaluation).

How it works / steps

  1. Decide evaluation metrics that fit the business goal
  2. Compare training results on validation data and record them in MLflow
  3. Check for overfitting and underfitting with learning curves
  4. Compare under near-production conditions with shadow testing
  5. For generative AI, use Bedrock evaluations or LLM-based evaluation

Workplace example

For a fraud detection model where misses are costly, prioritize recall and lower the threshold, absorbing the increase in false positives with a staff review process.

Common confusions and exam tips

Mini quiz

Q1. The confusion matrix of a fraud detection model shows many false negatives (misses). You want to reduce misses without changing the model. Which is the most appropriate adjustment?

  1. Remove all missed transactions from the evaluation data
  2. Recount false negatives as correct results
  3. Raise the classification threshold to narrow what is judged positive
  4. Lower the classification threshold to widen what is judged positive
Show answer and explanation

Answer: D Lower the classification threshold to widen what is judged positive

Lowering the threshold raises recall and reduces misses, but false positives increase. Decide based on business costs.

Q2. You are running many experiments and lost track of which hyperparameters and data produced the best metrics. Which is the most appropriate method?

  1. Register each model in SageMaker Model Registry and compare version numbers
  2. Log each experiment's parameters, metrics and artifacts in managed MLflow
  3. Write metric values in training job tags and list them with Resource Groups
  4. Send training logs to CloudWatch Logs and search metrics with Logs Insights
Show answer and explanation

Answer: B Log each experiment's parameters, metrics and artifacts in managed MLflow

MLflow experiment tracking records parameters, metrics and models, enabling comparison and reproduction. [Other options] Model Registry manages versions and doesn't compare parameters and metrics. Log searches and tags don't give structured experiment comparison.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)