SemVer still shapes how AI coding agents behave

a computer screen with a bunch of code on it

The assumption that AI coding agents would ignore version labels and just read the code turns out to be wrong for at least some models. Microsoft researcher Waldek Mastykarz ran close to 1,000 agent runs across two experiments and found that the version number alone changed what several models did with byte-identical code.

‍ What the research tested

The team built a Node.js app that depended on a fictional npm library called fluxkit. The new version of fluxkit renamed a function the app calls, making the upgrade a silent breaking change. Every model saw the same library code. Only the version label differed: 1.4.3 (patch) vs. 2.0.0 (major).

They ran GitHub Copilot CLI as the agent inside Vally, Microsoft’s evaluation tool for AI agents, and counted a run as checked when the agent ran the tests, ran the app, or read the updated library code before finishing.

The numbers that matter

In the first experiment (10 runs per label), Claude Sonnet 5.5 checked the upgrade in 2 of 10 patch-labeled runs and in all 10 major-labeled runs. Claude Opus 5.5, GPT-6.1 Sol, and GPT-6.1 Terra checked every run regardless of label.

The second experiment added a code review scenario: agents were asked to review a branch and merge it if safe. Six models, 30 runs per label, 540 runs total. The results:

  • Claude Haiku 5.5: merged without checking in 37% of patch runs vs. 7% of major runs. Classified as a real effect.
  • Claude Sonnet 5.5: 13% unchecked merges on patch vs. 0% on major. Inconclusive, leaning toward an effect.
  • GPT-6 Luna: 30% vs. 17%. Inconclusive.
  • Claude Opus 5.5, GPT-6.1 Sol, GPT-6.1 Terra: 0% unchecked merges under either label.

Rolled up across all six models and 180 runs per label: patch upgrades went unchecked 13% of the time, minor upgrades 9%, and major upgrades 4%.

3D rendered ai text on dark digital background

What actually broke

When agents skipped the check and merged anyway, they got lucky most of the time. Non-breaking updates merged successfully in 268 of 270 runs. Breaking updates merged in 13 of 270 runs, and 4 of those 13 left the main branch broken. All four came with a patch or minor label.

️ Three actions to take now

The research team pulled three direct recommendations for the people who maintain libraries, run agent-based workflows, or evaluate models:

  1. Library maintainers: keep SemVer honest. A breaking change shipped as a patch is exactly the scenario where label-sensitive agents skip the check. Your version numbers are part of the experience agents have with your library.
  2. Teams using agents for dependency upgrades: don’t let the label decide how much verification happens. Build CI that runs on every dependency branch, so a patch gets the same checks as a major release.
  3. Teams evaluating agents: add the version label to your list of hidden variables. Claude Haiku 5.5 tested in a patch scenario is measuring a different agent than Claude Haiku 5.5 in a major scenario. Test with the models your developers actually use, and read the trajectories, not just the pass/fail outcome.

The core finding is blunt: the version number tells an agent how worried to be. Only the tests tell it whether it should be.

Stay on top of AI & Automation with BizStack Newsletter
BizStack  —  Entrepreneur’s Business Stack
Logo