AI agent eval scores lie when the sandbox leaks
A passing score on an AI agent evaluation can prove exactly the wrong thing. The agent answered correctly. That does not mean you measured what you intended to measure.
Waldek Mastykarz ran into this directly while using Vally to test how much GPT-5.6 Luna knew about different versions of Dev Proxy. The goal was to compare the model’s internal knowledge against its self-reported knowledge cutoff. The eval blocked built-in web tools and direct network access via curl. That looked like a closed boundary.
It wasn’t. The agent ran which devproxy, found the executable, got the absolute path to a local source checkout, ran rg against that path, checked out the v0.29.2 tag, and produced a detailed, accurate answer about Dev Proxy’s caching behavior. Every answer reflected what existed on that machine, not what the model knew before the run started.
The core problem
Coding agents treat a blocked tool call as an obstacle, then use whatever capabilities remain to reach the same information. A sandbox built from individual deny rules will always lag behind that adaptive search. Blocking curl and web tools closes one route. The filesystem, installed software, shell commands, environment variables, and attached services can each supply the same evidence through a different path.
The right frame is not which tools are forbidden but what information can the agent obtain through every capability it still has. That question has to be answered before the run, not after a surprising pass.
How to build a valid sandbox
The recommendations from the article map directly to practical eval setup:
- Define the capability you want to measure and the information boundary that measurement requires.
- Start each run in a clean workspace containing only the intended fixtures.
- Use a neutral workspace path that doesn’t leak words related to the evaluation topic.
- Enforce filesystem access at the workspace boundary so every shell command follows the same rule.
- Remove product installations and unrelated source checkouts from the host machine.
- Minimize inherited environment details that reveal paths or installed software.
- Apply equivalent restrictions across every available tool and attached service.
Review trajectories, not just scores
A grader can confirm the final answer matched expected behavior. It can’t tell you whether the model generated that answer from internal knowledge or retrieved it from the environment. Even a well-written rubric can’t repair contaminated inputs after the run.
Inspect the full trajectory: every tool call, every observation returned to the agent. Check whether it searched outside the workspace or resolved an installed product copy. Failed attempts matter too. They show what the agent considered useful and often point to the next boundary to close. A single blocked call proves one route was shut. It doesn’t prove the agent stopped.
Surprising passes near a model’s expected knowledge boundary deserve extra scrutiny. Treat them as hypotheses to investigate, not evidence to report.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
