Microsoft open sourced a tool that tests your AI coding agents
Coding agents can write functions and wire up APIs, but they still trip over tasks a human developer would handle without thinking. Microsoft just open sourced a tool designed to catch those failures before they reach production.
It’s called Scope, and it runs the same task through different coding agents, then scores what each one produces. The code is available under the MIT license.
️ What It Does
Scope doesn’t write code. It evaluates the agents that do.
You provide a task, a starting codebase if needed, and an agent setup. Scope then drives existing agents such as GitHub Copilot and Claude Code through that task. Success is defined through criteria organized as a dependency graph. One check can gate another.
A simple example: ask an agent to build a Node.js server, then check whether it includes a working GET /health route that returns JSON.
An LLM based Judge service scores each run against your criteria. Microsoft is upfront about the limitations. The Judge can make mistakes, and important conclusions still warrant human review.
What Gets Captured
Each run preserves a full trace:
- Live logs
- Generated files
- Workspace snapshots
- Tool calls
When a criterion fails, you can trace backward through those steps to find where the agent stalled or made a wrong turn.
Versioned profiles store the agent, model, skills, MCP servers, and extensions used in a run. You can swap one component, rerun the same task, and compare results. Since agent behavior varies from attempt to attempt, repeating a task helps separate recurring problems from one off issues.
Configurable personas let you set the viewpoint and tone of the simulated user. You can review results through the web portal or use the CLI and REST API to feed them into scripts and pipelines.
Where This Matters
Three groups get immediate value from a tool like this:
Docs teams can watch for instructions that agents miss run after run. Fix the page, rerun the scenario, and verify the improvement.
SDK maintainers can check whether an agent picks their SDK in the first place. They can also test whether new sample code or a skill file changes that outcome.
CLI and MCP server authors can examine traces to see where an agent stalled. That often points to a vague tool description or an error message that leaves the agent guessing.
️ Pricing and Requirements
Scope is MIT licensed and free to use. There’s no hosted version, so you run it yourself.
A local setup requires:
- Docker with Compose v2
- Node 22
- pnpm
- mkcert
- GitHub CLI
The default worker calls for an active GitHub Copilot entitlement. The Judge service needs credentials for a model provider. Teams that want a shared deployment can run it on Kubernetes using the provided Helm chart.
Scope lives on GitHub. If you want to contribute, you’ll likely need to sign the Microsoft Contributor License Agreement.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
