AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AI-300 · LESSON 4

Implement generative AI quality assurance and observability10–15% · 2 topics

Learn evaluation (quality and safety) of generative AI apps and agents, and observability through tracing.

4.1Configure evaluation and validation for generative AI applications and agents

Concept

Evaluation measures output quality and safety with metrics. Foundry has built-in evaluators: AI-assisted quality metrics (groundedness, relevance, coherence, fluency), metrics compared with ground truth (F1, BLEU, ROUGE, similarity), safety metrics (violence, self-harm, hate, sexual, protected material, indirect attacks) and agent metrics (intent resolution, tool call accuracy, task adherence).

How it works / steps

  1. Build an evaluation dataset with representative questions and expected answers
  2. Choose evaluators suited to the use (groundedness and relevance for RAG, tool call accuracy for agents and so on)
  3. Run evaluation in the cloud or with the SDK and compare results with the previous version
  4. Run adversarial tests with tools such as the AI Red Teaming Agent to confirm safety
  5. Build evaluation into CI/CD as a pre-release gate

Workplace example

For an internal policy Q&A bot, answers with groundedness below 4.0 are identified, the retrieval settings and system message are fixed, and the bot is re-evaluated.

Common confusions and exam tips

Mini quiz

Q1. What should you prepare first when starting to evaluate a generative AI app?

  1. A GPU compute cluster for evaluation and a request for its quota
  2. A screen design and color scheme for a dashboard showing evaluation results
  3. A JSONL dataset of all exported production conversation logs, personal information included
  4. A test dataset including questions, context and expected answers, and column mapping to evaluators
Show answer and explanation

Answer: D A test dataset including questions, context and expected answers, and column mapping to evaluators

Evaluation starts with test data and a mapping of each column to evaluator inputs. [Other options] Production logs with personal information left in are inappropriate. Evaluators run without a cluster. Screen design is not a prerequisite for evaluation.

Q2. Which metric measures whether an answer is “supported by the provided context”?

  1. A similarity metric
  2. Relevance
  3. Groundedness
  4. Fluency
Show answer and explanation

Answer: C Groundedness

Groundedness measures whether the answer is based on the given context. [Other options] Relevance is fit to the question, fluency is naturalness of text and similarity is closeness to the expected answer; none measures support by context.

Reference links

4.2Implement observability for generative AI applications and agents

Concept

Observability makes it possible to trace what happened inside apps and agents in production. Traces based on OpenTelemetry record prompts, responses, tool calls, token counts and latency and send them to Application Insights. Continuous evaluation periodically evaluates samples of production traffic.

How it works / steps

  1. Connect Application Insights to the project
  2. Enable tracing in the SDK and record each span (model calls, retrieval, tools)
  3. Monitor token usage, latency and error rate in dashboards
  4. Set up continuous evaluation and alerts to detect quality drops
  5. Adjust what is recorded so logs do not retain too much personal information

Workplace example

In response to complaints that an agent is slow, traces reveal one tool call taking 8 seconds, and a cache is added for that API.

Common confusions and exam tips

Mini quiz

Q1. In a production generative AI app, you want to identify why responses are slow. What is most useful?

  1. The endpoint’s average response time metric and an alert every 5 minutes
  2. Daily token consumption and deployment quota utilization
  3. App error logs and the number of user inquiries
  4. Per-step traces (time breakdown for search, model calls, tool execution and so on)
Show answer and explanation

Answer: D Per-step traces (time breakdown for search, model calls, tool execution and so on)

Breaking down latency per step with traces identifies the bottleneck. [Other options] Average response time does not tell which step is slow. Token volume or quota are possible causes but give no breakdown. Error logs are not a latency breakdown.

Q2. You want to manage generative AI usage costs per department. Which metrics should you monitor?

  1. Evaluation scores and number of safety detections
  2. Request count and average response time
  3. Token consumption and resource usage
  4. Number of deployments and types of models used
Show answer and explanation

Answer: C Token consumption and resource usage

Cost is closely tied to token consumption and resource usage. [Other options] Request counts and response times are not proportional to cost. Evaluation scores measure quality. The number of deployments does not show the cost breakdown.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)