AI agent harness rules outlive the bugs they were written to fix
Researcher Hassan Hassanzadeh Aliabadi spent six weeks logging every failure produced by three AI coding agents working inside a shared plain-markdown repository under one rule set. The result is a 118-failure ledger with fixes, recurrence data, and a documentation audit run against 47 known stale statements and 20 planted errors.
What the data shows
Stale documentation, what the paper calls drift, was the single largest failure class at 48 of 118 failures. That said, 35 of those 48 surfaced on a single audit day, so the distribution is lumpy rather than steady.
The monthly LLM documentation audit found roughly half of known drift in a single run. Claude Sonnet 5.5 hit a mean of 50% recall on the 47 known stale statements. On precision, at least 93% of its findings were correct, meaning false positives were low even when recall was not.
On rule durability
The researcher replayed five past failures in isolated git clones, with and without their fixes applied. On Sonnet 5.5, the agent finished four of the five tasks. In three of those four, the original failure did not recur regardless of whether the fix was present. Only one failure, the stale-count failure, came back when its automated check was removed. With the check active, it was blocked every time.
One older-model finding stands out: an agent drafted an em-dash while the em-dash lint check was running, and the check caught and rejected it. A written rule about runtime estimates, by contrast, did not stop Sonnet 4.5 from overestimating. Mechanical checks outperformed written rules when enforcement was needed.
The operator takeaway
If you maintain an AGENTS.md or any harness rule set for AI coding workflows, this study suggests two things. First, rules tend to stick around after the failures that prompted them are gone. Harnesses accumulate lessons faster than they retire them. Second, drift in your documentation is real and persistent, and a single audit pass catches only about half of it.
The author’s recommendation: re-test your rules periodically and retire the ones that no longer correspond to active failure modes. The paper is exploratory, based on one repository with three runs per condition, but the directional findings are practical enough to act on now.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
