The problem? Understanding how AI coding agents interact with the software you build. "Whether you're maintaining an SDK, building a command-line interface, or publishing an MCP server or skills... you need to understand whether agents can discover and use what you ship, and what your team can change." Results can vary, and configurations keep changing, with behaviors influenced by everything from custom instructions and MCP servers to skills and plugins. So after a year of development Microsoft has created Scope, "an open-source platform built on Kubernetes for turning those interactions into evidence product teams can act on." Details are available on... the Microsoft open source blog: Available under the MIT License, Scope supports a continuous Developer Agent Experience (DevAx) learning loop grounded in repeatable experiments. The Azure Cosmos DB team's experience shows what that loop can deliver... "The evaluations led to 20 Agent Kit issues and 27 documentation issues resolved," [says Jay Gordon, Senior Program Manager, Azure Cosmos DB]... These results connect evaluation to decisions a product team can make: which documentation needs clarification, where tooling needs a fix, and whether an update changes agent behavior.... [The gains] demonstrate how targeted changes, informed by observed behavior, can improve the way agents use a product. For more on the shift from building for human developers to supporting agents, watch Your Product Was Built for Humans. Is It Ready for Agents? or join us at AI DevCon NYC conference in November. Scope surfaces everything from incorrect assumptions in documentation (or recurring missed instructions) to whether agents select the intended SDK, initialize it correctly, and use its APIs, leading to clearer descriptions for command-line tool and more actionable errors. Microsoft's open source blog also explains why AI agent evaluation needs an open-source foundation: The challenges behind agent experience aren't unique to one product or organization. SDK maintainers, tool authors, researchers, and agent developers are all asking how agents behave and what helps them succeed. No single team can represent every language, workflow, or integration. An open project can bring more scenarios, more failure cases, and more viewpoints into the evaluation process... Opening Scope's execution and evaluation code gives contributors a way to question the methods, investigate unexpected results, and improve the evaluation itself. A shared foundation also reduces duplicated infrastructure. Agent integrations, reusable criteria, and trace-analysis improvements can help other teams study their own products without rebuilding the same machinery. The goal isn't one universal benchmark or definition of success. It's a common starting point that teams can adapt to their needs. The MIT License lets teams use and extend Scope, while self-managed deployments let them evaluate private workloads without publishing their code or run data.... Scope streams activity as the agent works and retains interactions and workspace snapshots for later review. Your team can follow tool calls, inspect generated files, and connect a failed requirement to the steps that preceded it. Those traces provide observable evidence of what the agent did, helping turn a failure into a concrete documentation, SDK, or tooling improvement.

Read more of this story at Slashdot.