Poisoned benchmarks can corrupt self-modifying AI coding agents
In 1984, Ken Thompson showed that a poisoned compiler could reinsert its own backdoor even when recompiled from clean source code. A new paper from Franziska Roesner and collaborators asks an uncomfortable question: does the same attack class apply to AI coding agents that rewrite themselves?
The Attack
Self-modifying coding agents evaluate their own performance against benchmarks and use those evaluations to update their own instructions or code. The researchers found that if an adversary supplies poisoned benchmarks during that self-improvement loop, the agent evolves to write vulnerable code on completely unrelated, clean tasks it has never seen before.
Three agents were tested: the Darwin Gödel Machine (with experimental modifications by the researchers), the Self-Improving Coding Agent, and Hyperagents (both used substantively unmodified). All three were successfully compromised in proof-of-concept attacks.
A Concrete Example
With Hyperagents running on Sonnet 4.5, one poisoned benchmark caused the agent to self-evolve instructions that disable HTTPS certificate validation on neutral URL-fetching tasks. The agent was not directly told to do this. It learned the vulnerability pattern through its own self-improvement process and then applied it to unrelated work.
The Persistence Problem
The finding that should concern anyone building or deploying these systems: contamination often persists even after a poisoned agent is subsequently evolved against clean benchmarks. Retraining on good data does not reliably flush the vulnerability out.
The paper identifies specific properties of the vulnerability, the benchmark design, the underlying model, and the agent scaffolding that together enable a successful attack. The researchers argue that self-modifying coding agents need to be designed from the ground up with resilience to this class of input poisoning in mind.
Why This Matters for Operators
If you are using or building with any self-improving agent framework, the attack surface is not just the code the agent writes today. It is the agent’s future versions as well. Benchmark curation and evaluation pipeline integrity are now a security concern, not just a quality concern.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
