Google: test AI agents on what they ran, not what they said
Google published four evaluation lessons from internal testing of its Google Cloud Developer plugin for AI coding agents. The guidance is aimed at engineers who measure agent behaviour, not those who build the systems. If you run or evaluate any agentic workflow, the findings are worth a close read.
Lesson 1: score the execution path, not the final reply
In an internal suite of 11 prompts, an automated LLM judge marked the plugin as failing three tasks. A review of tool-call logs and transcripts showed the agent had actually passed all three.
One failure came from a Cloud Run cleanup prompt. The test rubric required the agent to check command syntax before taking action. The agent ran a help command first, which satisfied the requirement, but its final message focused on warning the user about permanent data loss rather than confirming it had checked the documentation. The grader recorded a failure. The task had been completed correctly.
Google’s conclusion: tests should score which commands were run and validated, not what the agent wrote at the end.
Lesson 2: preserve turn structure in telemetry
Some agent harnesses batch multiple tool calls into a single assistant turn. In one case, the harness dispatched a skill-loading call and two documentation searches at the same time. Because skill activation is a multi-stage process, the search instructions arrived only after the searches had already run, so the skill had no turn in which its guidance could shape the agent’s actions.
Flattening activity into a simple event sequence hides this kind of concurrent dispatch. Google recommended preserving the original turn structure in telemetry so engineers can see when parallel tool calls caused actions to happen before relevant guidance entered the context window.
Lesson 3: run live tests and plan-only tests side by side
Live runs with sandboxed credentials show whether commands execute correctly, whether parameters are handled properly, and whether an agent shows restraint before destructive tasks. But relying only on live runs can mask reasoning problems, because permission errors may stop execution before the problematic behaviour appears.
Plan-only tests remove credentials and ask the agent to explain what it would do without running anything. Google said about half of its most important findings came from comparing the two conditions. In one onboarding prompt, a live run failed immediately on a project list call, while a plan-only run produced an 8,000-character plan that, according to Google, ignored organisation policies, administrative approvals, and governance requirements. The live permission block had hidden a fundamental reasoning flaw.
Lesson 4: build golden sets as continuous integration baselines
Model architectures, agent harnesses, and plugin standards change regularly. Google recommended treating evaluation the same way you treat CI: durable benchmark sets created by human experts, capturing the full trajectory of each run including reasoning, commands executed, and whether the task completed safely.
Google also warned against the single-prompt demo as evidence of reliability. It called this the “one-shot demo” problem and said robust evaluation requires suites that cover multi-step interactions, permission handling, and the avoidance of damaging outcomes.
“At the end of the day, the only thing that outlasts model churn is a stress-tested eval suite.”
If you are building agent workflows on top of any cloud platform, these four points are a practical checklist for where your current evaluation is probably blind.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
