AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AB-620 · LESSON 3

Test and manage agents20–25% · 2 topics

Learn to evaluate agent performance and implement application lifecycle management (ALM) for Copilot Studio agents.

3.1Evaluate agent performance

Concept

Agent evaluation measures quality by automatically scoring answers against a set of test questions. With agent evaluation in Copilot Studio you create a test set (manual, CSV or generated) and evaluate answer quality or match with expected answers. After publishing, use analytics (resolution rate, escalation rate, satisfaction, knowledge usage) to find improvements.

How it works / steps

  1. Create a test set of representative questions and expected answers
  2. Run the evaluation and review pass/fail results and reasons
  3. Fix the causes of failures (missing knowledge, vague instructions, tool descriptions) and re-evaluate
  4. After publishing, monitor resolution and escalation on the analytics dashboard
  5. Check for regressions with the same test set after every change

Workplace example

In a 100-question test set for an IT help agent, a password reset question failed; fix the cause (an outdated procedure document) and confirm it passes on re-evaluation.

Common confusions and exam tips

Mini quiz

Q1. You want to evaluate the agent’s quality continuously. What is the most appropriate way to build a test set?

  1. Prepare pairs of representative questions and expected answers (conditions to meet) and rerun them after every change
  2. Collect only conversations that users rated poorly and use them to reproduce problem answers
  3. Pick random questions from production logs and evaluate with different questions each time
  4. Record the questions developers tried while building and check the results once before the first release
Show answer and explanation

Answer: A Prepare pairs of representative questions and expected answers (conditions to meet) and rerun them after every change

Running a fixed test set repeatedly detects degradation (regressions) caused by changes. [Other options] Different questions each time cannot compare before and after a change. Checking once before release is not continuous evaluation. Poorly rated conversations alone do not cover representative questions.

Q2. Test results showed many incorrect answers. What is the most appropriate way to proceed with improvements?

  1. Add the failed questions to topic trigger phrases and switch to returning fixed answers
  2. Classify failures by cause (missing knowledge, wrong routing, tool failure, insufficient permissions, etc.) and address each
  3. Switch to a larger model with a custom prompt and see whether overall accuracy improves
  4. Loosen content moderation to reduce cases where the agent declined to answer, then retest
Show answer and explanation

Answer: B Classify failures by cause (missing knowledge, wrong routing, tool failure, insufficient permissions, etc.) and address each

The fix depends on the cause. Erasing failed cases defeats the purpose of evaluation. [Other options] Turning questions into topics or changing the model or moderation without identifying causes does not fix root causes such as missing knowledge or permissions.

Reference links

3.2Implement application lifecycle management (ALM) for agents in Copilot Studio

Concept

ALM is how you safely manage and move agents across development, test and production environments. Bundle agents, flows, connectors and environment variables in a Power Platform solution and import it into production as a managed solution. Automate moves with Power Platform pipelines, Azure DevOps or GitHub, and manage source with Git integration.

How it works / steps

  1. Separate development, test and production environments
  2. Put the agent and related components in the same solution
  3. Use environment variables for values that differ by environment and connection references for connections
  4. Import into test and production as managed solutions
  5. Automate moves with pipelines and add approvals

Workplace example

Deploy an agent from development to test with Power Platform pipelines, evaluate it, and after approval deploy it to production as a managed solution.

Common confusions and exam tips

Mini quiz

Q1. You want to move an agent built in the development environment to production. What is the standard method?

  1. Create a copy of the agent in production and match the development settings by hand
  2. Export the agent’s topics with the code editor and paste them into production
  3. Share the development agent with production users and let them use it as is
  4. Include the agent in a solution and move it as a solution
Show answer and explanation

Answer: D Include the agent in a solution and move it as a solution

Solutions are the unit for moving between environments. Manual re-creation causes differences and omissions. [Other options] Pasting topics or reproducing by hand easily misses things. Using the development environment as production breaks environment separation.

Q2. You want to bring an agent that has already been created under ALM management. What is the appropriate action?

  1. Export the agent and rebuild it in a new environment
  2. Move the agent while it remains in the default solution
  3. Move the agent’s topics one by one into a separate solution
  4. Add the existing agent to a solution
Show answer and explanation

Answer: D Add the existing agent to a solution

Existing agents can also be added to a solution. [Other options] Rebuilding is unnecessary. The default solution is not suited as a unit of transport. You need to add the agent itself, not just its topics.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)