AIAI Agent Dev Labaiagentdev-lab.com
Sign in to save progressSign in
Menu

AB-100 · LESSON 3

Deploy AI-powered business solutions40–45% · 4 topics

Learn to analyze, monitor and tune AI solutions, manage testing, design ALM, and design responsible AI, security, governance, risk management and compliance.

3.1Analyze, monitor and tune AI solutions

Analyze, monitor, and tune AI-powered business solutions

Concept

After release, keep improving by monitoring usage (adoption, active users), quality (resolution rate, satisfaction, escalation) and cost (message and token consumption). Use Copilot Studio analytics, the Microsoft 365 Copilot usage dashboard, Azure Monitor and more.

How it works / steps

  1. Prepare a KPI dashboard
  2. Analyze unresolved conversations and reasons for escalation
  3. Improve knowledge, instructions and tools and measure the effect
  4. Monitor cost consumption and set limits and alerts
  5. Review regularly with stakeholders and revisit priorities

Workplace example

Analyze “contract change” conversations with a high escalation rate, add the procedure to knowledge, and cut the escalation rate from 30% to 12% in two weeks.

Common confusions and exam tips

Mini quiz

Q1. For an agent running in production, what should be defined first as the monitoring process?

  1. Number of changes per developer, number of topics, agent publishing frequency and a list of owners
  2. Metrics to monitor (resolution rate, escalation rate, response quality, error rate) and the frequency and owners of reviews
  3. Usage counts per user department and the date the agent was published
  4. Number of agent topics and number of registered knowledge sources
Show answer and explanation

Answer: B Metrics to monitor (resolution rate, escalation rate, response quality, error rate) and the frequency and owners of reviews

Without deciding metrics, frequency and owners, monitoring becomes a formality. [Other options] Usage counts, publish dates, topic counts and change counts are not metrics for monitoring quality or problems.

Q2. Many pieces of user feedback have come in. Which approach turns them into improvements?

  1. Pass feedback to developers in the order received and have them fix in order of arrival
  2. Aggregate positive feedback to report results and handle negative voices individually
  3. Analyze feedback and usage, turn it into a backlog and improve by priority
  4. Aggregate feedback counts by department and address requests from departments with the most first
Show answer and explanation

Answer: C Analyze feedback and usage, turn it into a backlog and improve by priority

Manage feedback as a backlog and prioritize by impact. [Other options] Counts alone do not show the size of impact. Positive voices alone do not lead to improvement. Order of arrival assigns no priority.

Reference links

3.2Manage testing of AI solutions

Manage the testing of AI-powered business solutions

Concept

AI testing evaluates answer quality, safety, bias and accuracy of tool operations in addition to traditional functional testing. Combine automated evaluation with test sets, user acceptance testing (UAT) by business staff, adversarial testing (red teaming) and load testing.

How it works / steps

  1. Build a test set with representative questions, edge cases and requests that must be refused
  2. Decide pass criteria for automated evaluation
  3. Run acceptance testing with business staff
  4. Run adversarial tests such as prompt attacks
  5. Run regression tests after every change

Workplace example

Before production, 10 business staff run two weeks of acceptance testing, automated evaluation confirms a 90% pass rate, and red teaming confirms no information leak paths.

Common confusions and exam tips

Mini quiz

Q1. You want to use AI-based tools to streamline analyzing and tuning agent problems. Which way of thinking is appropriate?

  1. Use AI analysis results as clues, but apply changes only after confirming their effect with evaluation
  2. Report only weekly summaries of AI tool analysis and make no changes to the agent
  3. Automatically apply AI tool improvement suggestions to the production agent without evaluation
  4. Don’t use AI tools since they can be wrong; have staff read conversations one by one to analyze causes
Show answer and explanation

Answer: A Use AI analysis results as clues, but apply changes only after confirming their effect with evaluation

Even AI analysis should be validated with evaluation before changes are applied. [Other options] Manual review of everything is inefficient. Reporting alone does not lead to improvement. Automatic application without evaluation risks lowering quality.

Q2. You want to validate AI features spanning multiple Dynamics 365 apps—Sales, Supply Chain and Finance—from order to invoice. Which test is appropriate?

  1. Checking the display of each app’s screens
  2. Integration tests of invoice processing in the Finance app
  3. End-to-end test scenarios that follow the whole business flow
  4. Unit tests of each AI feature in each app
Show answer and explanation

Answer: C End-to-end test scenarios that follow the whole business flow

For work spanning apps, anything other than E2E scenarios misses defects in the integration points. [Other options] Unit tests, checks within one app or screen display alone cannot validate a business flow spanning multiple apps.

Reference links

3.3Design the ALM process for AI solutions

Design the ALM process for AI-powered business solutions

Concept

ALM separates development, test and production environments and manages and moves changes with solutions and pipelines. Use Power Platform solutions (managed and unmanaged), environment variables and connection references, Power Platform pipelines, and Azure DevOps or GitHub, and manage Foundry assets as code too.

How it works / steps

  1. Decide the environment strategy (development, test, production, handling of the default environment)
  2. Decide solution units and owners
  3. Make per-environment values environment variables and connection references
  4. Automate moves with pipelines and approvals
  5. Manage Foundry prompts and agent definitions in Git too

Workplace example

Copilot Studio agents go through Power Platform pipelines and Foundry agents and prompts through GitHub Actions, both on the same release approval flow.

Common confusions and exam tips

Mini quiz

Q1. You are designing ALM for data used by AI models and agents. Which is appropriate?

  1. Manage model and agent versions in Git and have data reference the latest production state
  2. Manage data versions, sources and update procedures, and map them to model and agent changes
  3. Copy production data to developers’ PCs and have each manage versions
  4. Keep only the latest state of data and overwrite it with each update
Show answer and explanation

Answer: B Manage data versions, sources and update procedures, and map them to model and agent changes

Without managing data versions and sources too, changes in quality cannot be traced. [Other options] Management on personal PCs cannot be controlled. Overwriting leaves no history. Without data versions, the mapping to model changes is unknown.

Q2. You are designing ALM for Copilot Studio agents, connectors and actions. Which is appropriate?

  1. Edit agents directly in the production environment and record changes in a log sheet
  2. Move agents with solutions, and recreate connector connections and URLs by hand in production
  3. Rebuild agents created in the development environment by hand again in production
  4. Bundle them in solutions, absorb environment differences with environment variables and connection references, and move them with pipelines
Show answer and explanation

Answer: D Bundle them in solutions, absorb environment differences with environment variables and connection references, and move them with pipelines

Solutions, environment variables, connection references and pipelines are the basics of Copilot Studio ALM. [Other options] Manual rebuilding creates differences. Direct production edits cannot be controlled. Creating connections by hand cannot absorb environment differences.

Reference links

3.4Design responsible AI, security, governance, risk and compliance

Design responsible AI, security, governance, risk management, and compliance

Concept

Manage AI solution risk by combining responsible AI principles (fairness, reliability and safety, privacy and security, inclusiveness, transparency, accountability), Microsoft Purview (sensitivity labels, DLP, auditing), Power Platform DLP policies and environment governance, and Microsoft Entra ID access management.

How it works / steps

  1. Assess risk by scale of impact and data sensitivity
  2. Organize sensitivity labels and permissions so Copilot does not reference excessive information
  3. Control usable connectors with Power Platform DLP policies
  4. Design content safety settings, human approval and AI-use disclosure
  5. Meet regulations with audit logs and retention policies

Workplace example

Before deploying Copilot, review SharePoint oversharing, restrict access to labeled confidential documents, and limit connectors agents can use with DLP policies.

Common confusions and exam tips

Mini quiz

Q1. Before release, you want to confirm the solution follows responsible AI principles. Which approach is appropriate?

  1. Review from the perspectives of fairness, reliability and safety, privacy, transparency and accountability, and record issues and measures
  2. Review only from security and privacy perspectives and skip the rest since it is internal with few users
  3. Perform a security vulnerability assessment and decide based only on its results
  4. Have developers self-declare compliance with the principles and approve based on that
Show answer and explanation

Answer: A Review from the perspectives of fairness, reliability and safety, privacy, transparency and accountability, and record issues and measures

Reviews from the perspectives of the principles, and records of them, are required. [Other options] Reviews are needed regardless of scale. Self-declaration alone is not objective. A vulnerability assessment is only part of responsible AI perspectives.

Q2. Which is the most appropriate benefit of enabling Managed Environments?

  1. Power Apps licenses needed to use the environment become unnecessary
  2. Agents needed for business are created automatically from requirements
  3. Governance features such as sharing limits, usage insights and solution checks become available
  4. Data policy (DLP) settings are created automatically from usage and become unnecessary
Show answer and explanation

Answer: C Governance features such as sharing limits, usage insights and solution checks become available

Managed Environments provide governance features such as sharing limits, usage analytics and solution checks. [Other options] Managed Environments do not remove the need for licenses or DLP, nor do they create agents automatically.

Reference links

Lessons are written by this site based on public information and do not guarantee exam content. Product specifications change, so check the latest information at the reference links.

Notes (unofficial, original questions, no pass guarantee)