How to benchmark context compaction in AI coding agents
AI coding agents run long sessions. They search repos, read files, execute builds, analyze failures, and loop back around. Every step adds to the working context, and eventually something has to give.
Context compaction is the process of shrinking that history into something manageable. The problem is that not all compaction strategies are equivalent, and measuring only token reduction misses most of what matters.
Four Strategies Worth Comparing
The article from C# Corner lays out four main approaches:
- Fixed-interval: Compact after every N tool calls. Simple to implement, but ignores actual content density. Eight large tool results might need compaction before 30 small ones do.
- Token-threshold: Compact when context hits a configured size. More adaptive, but token count alone does not tell you how valuable the information is. Two contexts at the same token count can have completely different information density.
- Event-driven: Compact at meaningful workflow boundaries, such as after a test run or after an investigation phase completes. Preserves natural phase separations.
- Structured-state: Instead of summarizing the raw conversation, the system maintains an explicit state object with objective, modified files, architectural decisions, known failures, and remaining work. More deterministic than a free-form summary.
What to Actually Measure
A benchmark that only tracks context size will mislead you. Strategy A might cut context by 60% but cause the agent to re-read files repeatedly. Strategy B might cut only 35% but finish the task with fewer total tool calls. Strategy B wins.
The metrics that matter:
- Context tokens
- Compaction count
- Tool calls (total and rediscovery)
- Model calls
- Task success rate
- Latency
- Cache hit and miss rates
- Cost estimate
Rediscovery rate is the most diagnostic metric. It measures how often the agent spends tool calls recovering information that existed before compaction. A high rediscovery rate means the strategy is removing things the agent still needs.
Start with a No-Compaction Baseline
Before testing any strategy, run the full task corpus with no compaction at all. Record total context tokens, model calls, tool calls, completion time, and task success. Without this baseline, you cannot tell whether compaction actually helped.
The C# Corner piece includes a simple ICompactionStrategy interface in .NET that lets you run the same task workload against different strategies, with TokenThresholdStrategy and ToolCallStrategy implementations shown as examples. The interface pattern matters more than the specific code: it forces you to swap strategies against identical starting conditions.
⚠️ Common Benchmarking Mistakes
- Running different strategies against different tasks (easier workload looks better)
- Measuring only token count and calling it done
- Using only short tasks where compaction has no room to differentiate
- Ignoring failure-heavy workflows, which are exactly where context management breaks down
- Changing model, prompt, tools, and compaction strategy simultaneously, which makes it impossible to isolate cause
On cache effects: compaction changes the context prefix, which can reduce cache reuse depending on your serving infrastructure. A strategy that cuts tokens may simultaneously cut cache efficiency. Measure both separately.
Pro Tip
For long-running agents, the article recommends keeping critical information in explicit structured state outside the raw conversation. Task objective, modified files, architectural decisions, and known failures should not depend on a free-form summary to survive compaction. The real test of any strategy is not how small the context gets. It is whether the agent can keep solving the task correctly without rediscovering what it already learned.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
