AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AIP-C01 · LESSON 5

Testing, Validation, and Troubleshooting11% · 2 topics

Learn to implement evaluation systems for generative AI and to troubleshoot generative AI apps.

5.1Implement GenAI evaluation systems

5.1 Implement GenAI evaluation systems.

Concept

Bedrock evaluation includes model evaluation (automatic metrics, human evaluation and LLM-as-a-judge) and RAG evaluation (knowledge base retrieval evaluation: relevance and coverage; retrieve-and-generate evaluation: correctness, faithfulness and helpfulness). Prepare an evaluation dataset (prompts and expected answers) and compare on every change.

How it works / steps

  1. Build an evaluation dataset of representative questions and expected answers
  2. To compare models, measure quality metrics with LLM-as-a-judge evaluation
  3. Evaluate RAG retrieval and generation separately to pinpoint problems
  4. Evaluate safety too (harmfulness, appropriateness of refusals)
  5. Build evaluation into CI/CD as a gate

Workplace example

Before and after changing the chunking method, run knowledge base evaluation and confirm that both context relevance and answer faithfulness improved before applying it to production.

Common confusions and exam tips

Mini quiz

Q1. You want LLM-as-a-judge scoring to be trustworthy. Which method is most appropriate?

  1. Ask the same model that produced the answer only “is it good?” without defined criteria
  2. If scores look unfavorable, keep changing evaluator models and use the one that scores highest
  3. If the evaluator model's score is high, assume quality is good without criteria or human checks
  4. Define clear scoring criteria and calibrate the evaluator's scoring against a subset scored by people
Show answer and explanation

Answer: D Define clear scoring criteria and calibrate the evaluator's scoring against a subset scored by people

Make scoring criteria clear and calibrate against human evaluation. Using a model different from the one that generated answers reduces bias.

Q2. You are building an evaluation dataset. Which approach is most appropriate?

  1. Prepare only 10 easy questions a developer thought of and use them forever
  2. Use examples representative of real user questions, add edge cases, adversarial inputs and unanswerable questions, and prepare expected answers
  3. Build no dataset and check each time with questions thought up on the spot
  4. Collect only questions the model answered correctly into the dataset
Show answer and explanation

Answer: B Use examples representative of real user questions, add edge cases, adversarial inputs and unanswerable questions, and prepare expected answers

Evaluation data should represent production questions and include hard and adversarial examples.

Reference links

5.2Troubleshoot GenAI apps

5.2 Troubleshoot GenAI applications.

Concept

Common problems and fixes: ThrottlingException (429: retry, raise quotas, cross-Region inference, Provisioned Throughput), ValidationException (input format, exceeding token counts), AccessDeniedException (enable model access, IAM), ModelTimeout and slow responses (streaming, shorter output), hallucinations (check RAG retrieval, grounding checks), and guardrails blocking too much (adjust filter strength).

How it works / steps

  1. Classify the type of cause from the error code and message
  2. For throttling, check quotas and usage, then retry and distribute
  3. For access denials, check model access and IAM policies
  4. For answer quality issues, check retrieval results, then the prompt, then the model
  5. Reproduce with traces and logs, and confirm with evaluation after fixing

Workplace example

After a report of a wrong answer, traces show outdated documents ranking at the top, so fix retrieval with a metadata filter that searches only the latest versions.

Common confusions and exam tips

Mini quiz

Q1. The model's JSON output sometimes fails to parse and errors. Which countermeasure is most appropriate?

  1. Show one JSON example in the prompt and ask for the same format
  2. Raise max output tokens so the JSON isn't cut off
  3. Set temperature to 0 so the same format is more likely
  4. Fix the format with structured output or a tool-use schema, and regenerate with the error on failure
Show answer and explanation

Answer: D Fix the format with structured output or a tool-use schema, and regenerate with the error on failure

Define the format with a schema. On failure, passing the validation error and regenerating tends to fix it. [Other options] Temperature or an example reduces failures but doesn't guarantee format. Max tokens only helps when output is truncated. A schema with validation-error retries fixes it reliably.

Q2. Which pairing of an error type returned by Converse or InvokeModel calls and its handling is incorrect?

  1. Transient errors should be retried with exponential backoff
  2. ThrottlingException indicates a malformed request, so retrying is pointless
  3. AccessDeniedException indicates insufficient permissions
  4. ValidationException indicates errors in the request format or parameters
Show answer and explanation

Answer: B ThrottlingException indicates a malformed request, so retrying is pointless

ThrottlingException means too many requests. Retrying with backoff often succeeds.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)