Boot probe vs. full shell: what this AI coding agent study found
How much does giving an AI coding agent more self-check tools actually improve what it ships? A new arXiv study ran the controlled experiment most teams skip, and the numbers are worth knowing before you design your next agent pipeline.
The Setup
Researchers built a minimal coding agent where the only variable was its verification surface, defined as the set of tools available for checking its own work. Examples include a linter, a boot probe, a shell, and a screenshot tool. The agent implemented 1,116 web applications across six models and eight tool configurations. A condition-blind human graded every application against a frozen rubric. Automated probes stress-tested API-observable behaviors independently.
Key Findings
The boot probe is the highest-value first dollar spent. Without any tools, about one build in seven fails to launch at all. A single boot probe removes nearly all of those launch failures at roughly 35% of a full shell’s token cost. The full shell, by comparison, multiplies the no-tools token cost by 2.35x.
Screenshots help, but only where failures are visible. For problems like element placement and interaction bugs, screenshots provide a measurable gain over a shell alone. That gain does not survive correction for multiple statistical comparisons, though. For performance problems that can only be measured rather than seen, such as keeping scrolling smooth over a 100,000-row list, screenshots add nothing.
The Operator Takeaway
The core principle the study surfaces: a verification tool only improves output where its reach covers the actual failure mode. Adding more tools indiscriminately does not linearly improve quality. It raises token costs while delivering returns that depend entirely on the mismatch between tool capability and application failure type.
If you are building or evaluating AI coding agents, start with a boot probe before reaching for a full shell. Reserve screenshots for tasks with visible UI outputs. Skip both for pure performance or data-throughput problems where measurement is the only valid signal.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
