Microsoft AB-620 - Designing and Building Integrated AI Agent Solutions in Copilot Studio

Fixed prepared interactions → Use a curated prompt list; Predefined baseline → Compare against predefined expected outcomes; Repeatable comparison → Use a consistent scoring evaluation matrix.

Comprehensive and Detailed Explanation From Microsoft AB-620 Study Guide: A curated prompt list creates the controlled input population required for repeatability. Predefined expected outcomes establish what a correct or acceptable response should contain, which lets the evaluator judge performance against a baseline rather than against a reviewer ' s memory. A stable scoring matrix ensures that repeated runs are interpreted under the same thresholds and dimensions. Live conversations and post-rollout feedback can enrich the test inventory, but they cannot replace a fixed regression set because the questions, users, and context change. Publishing status confirms deployment, not response quality. General impressions are similarly unsuitable for consistent comparison. The team should select the scoring method according to the response type: semantic comparison for valid paraphrases, exact match for immutable values, keyword checks for mandatory phrases, and custom criteria for compliance or domain quality. Versioning the prompt list, expected answers, rubric, agent version, and knowledge snapshot is essential; otherwise a changed test set can be mistaken for a change in agent performance. Study Guide alignment: Test and manage agents > Evaluate agent performance > Create a test set; Choose an evaluation method.

===============

Comprehensive and Detailed Explanation From Microsoft AB-620 Study Guide: A predefined test set with expected outcomes is the only option that satisfies consistency, grounding comparison, and repeatable regression analysis. The fixed prompts ensure that each version receives the same inputs. Verified expected responses provide the baseline against which meaning, quality, keywords, or exact output can be scored. Rerunning the same set produces comparable results over time, allowing the team to calculate whether a knowledge-base change improved or degraded performance. The manual test pane is valuable for interactive debugging and inspecting the activity map, but it does not provide the same controlled batch or historical comparison. Channel usage metrics describe adoption and operational behavior, not response correctness. Transcript keyword searches can reveal patterns after deployment but do not establish a golden answer or a controlled experiment. The evaluation method should match the content: compare-meaning for legitimate paraphrases, exact match for fixed values, keyword match for required terms, or a custom rubric for domain-specific quality. Results should be retained or exported when a longer audit history is required. Study Guide alignment: Test and manage agents > Evaluate agent performance > Create a test set; Choose an evaluation method; Review test results.

===============

Reusable governed components → Export or package them as a component collection; Environment-specific connections and secrets → Use connection references and environment variables; Automated staged approvals → Use an ALM pipeline.

Comprehensive and Detailed Explanation From Microsoft AB-620 Study Guide: Reusable authored assets should be packaged in a governed component collection so the same topics and prompts can be promoted and consumed without uncontrolled copying. Values that differ by environment must remain outside those authored assets: connection references bind connectors in the target environment, while environment variables supply endpoints, IDs, and other configuration; secret values should use the appropriate secret mechanism. A Power Platform pipeline then moves the solution through defined stages and can impose approvals and validation before production. Hard-coding configuration would make every promotion a source edit, and copying components between agents would create divergent versions. The collection and the solution still need explicit ownership and dependency management, because a shared component update can affect multiple agents. Before each stage, validate current variable values, connections, component versions, and target permissions. After deployment, publish and run the same evaluation set in the target environment so the team verifies not only package transport but also runtime behavior under target data and identity controls. Study Guide alignment: Test and manage agents > Implement application lifecycle management for agents in Copilot Studio > Add existing agents to a solution; Create and use environment variables; Implement Power Platform Pipelines.

===============