Boot probes beat full shells for AI coding agents: new study
If you’re building with AI coding agents, the tools you hand them for self-checking matter more than most configuration decisions. A new controlled study puts hard numbers on exactly which verification tools are worth the token cost.
The Study Setup
Researchers built a minimal coding agent and used the tool list as the single controlled variable. They ran it across six models and eight tool configurations to produce 1,116 web applications. A condition-blind human grader scored every app against a frozen rubric, and automated probes stress-tested the API-visible behaviors.
The Key Findings
The clearest result: without any verification tools, roughly one in seven builds fails to launch at all. A single boot probe eliminates nearly all of those failures at about 35 percent of a full shell’s token cost. That is the cheapest and highest-leverage tool in the set.
A full shell does more, but it costs more too. The study found that a full shell multiplies the no-tools token cost by 2.35x. Whether that multiplier is worth paying depends entirely on what kind of failures your agent is producing.
Screenshots help in cases where errors are visible, such as element placement and interaction bugs. But the gain over a shell is modest and does not survive correction for multiple statistical comparisons. For failures that can only be measured rather than seen, like keeping scrolling smooth across a 100,000-row list, screenshots add nothing at all.
The Operator Takeaway
The core principle the researchers name is simple: a verification tool only helps where its reach covers the way the application actually fails. Match the tool to the failure mode, not the other way around.
- Start with a boot probe. It removes the most common failure (app won’t launch) at the lowest token cost.
- Add a full shell only if your failures are behavioral, not just launch failures.
- Skip screenshots for performance and data-heavy tasks. They cannot see what they cannot render visually.
The full paper covers six models and eight tool configurations, which makes it one of the more thorough empirical looks at agent verification design available right now.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
