If you ship an SDK, a CLI, or an MCP server, you probably have no reliable way to know whether AI coding agents can actually use what you built. Custom instructions, connected MCP servers, plugins, and skills all influence agent behavior in ways that are hard to observe from the outside.
Microsoft spent a year building a tool to fix that problem. It’s called Scope: an open source platform built on Kubernetes that turns agent interactions into documented evidence a product team can act on. It’s available under the MIT License.
What Scope Evaluates
Scope runs repeatable experiments against your software and captures what the agent actually does. That means:
- Whether the agent selects the intended SDK
- Whether it initializes the SDK correctly
- Whether it uses the APIs as documented
- Whether documentation contains incorrect assumptions or recurring missed instructions
- Whether a software update changes agent behavior
It streams activity as the agent works and retains interaction logs and workspace snapshots for later review. Your team can trace individual tool calls, inspect generated files, and connect a failed requirement to the exact steps that preceded it.
What the Azure Cosmos DB Team Found

The Azure Cosmos DB team ran Scope against their Agent Kit. According to Jay Gordon, Senior Program Manager at Azure Cosmos DB, the evaluations surfaced 20 Agent Kit issues and 27 documentation issues, all of which were resolved.
That’s a concrete output: not a score, not a benchmark, but a list of specific things the team could fix. Clearer command-line tool descriptions, more actionable error messages, documentation that reflects how agents actually read it.
Why MIT and Self-Managed
Microsoft’s reasoning for open sourcing Scope is straightforward. SDK maintainers, tool authors, researchers, and agent developers all face the same question: how do agents behave with my product, and what helps them succeed? No single team can cover every language, workflow, or integration.
Opening the execution and evaluation code lets contributors question the methods and improve the evaluation itself. Self-managed deployments mean teams can evaluate private workloads without publishing their code or run data. The MIT License lets anyone use and extend it without restriction.
Microsoft frames the goal as a shared starting point teams adapt to their needs, not a single universal benchmark. If you maintain developer tooling and want to understand how agents interact with it before your users hit the confusion wall, Scope is worth a look.
