Home » Case Study » Local LLMs vs. cloud coding agents: a real benchmark on real hardware

Local LLMs vs. cloud coding agents: a real benchmark on real hardware

Cagri Sarigoz

Published November 11, 2026

Nikolay Girchev had a practical reason to test local models: Anthropic’s consumer terms explicitly bar EEA and Switzerland residents from using Claude.ai, Pro, and Max plans for commercial or business purposes. If you’re a European developer writing work code in Claude Code on a Max subscription, you’re formally in violation. The API, Team, or Enterprise plans are the compliant path, and they cost significantly more.

There were other signals too. A Decrypt writer reported that Claude Opus 5.5 refused to help him regain root access on his own LG TV, redirecting him to Opus 4.8. A researcher who analyzed firmware using ChatGPT had his account banned for “Cyber Abuse” and lost roughly $500 in API balance. OpenAI has not commented publicly. In October, “uncensored” model variants started disappearing from Hugging Face with minimal explanation.

Girchev’s conclusion: the cloud is probably not closing tomorrow, but having a local fallback that runs without a subscription and without someone else’s rules is worth building now. So he ran a benchmark.

The Task

The benchmark used a real debugging scenario, not a toy prompt. The agent received minibank, a roughly 700-line Python multi-currency banking ledger with accounts, currency conversion, transfers, fees, monthly statements, and a CSV export. A DECISIONS.md file documented team constraints. The prompt was written as a manager would actually write it:

Customers are complaining: the monthly statement total doesn’t match the account balance, especially when there were transfers between currencies. Figure it out and fix it. And check whether the same thing breaks anywhere else.

No mention of tests. No mention of DECISIONS.md. The agent had to find those on its own.

The scoring system ran after each agent finished: 20 hidden acceptance tests worth half the score, 7 checks on subtle edge cases (UTC month boundaries, ISO timestamps, fee line items, corrected rate handling) worth 15%, 9 planted wrong fixes that the agent’s own self-written tests had to catch worth 25%, and rules compliance (don’t touch the forbidden module, pass existing tests, update the traitor test) worth 10%. A score of 0 means nothing changed. Below 0 means the agent broke more than it fixed. The time limit was 20 minutes.

☁️ The Cloud Baseline

Three models set the bar:

All three read DECISIONS.md unprompted, avoided the bait converter trap, found the broken CSV export, updated the traitor test, and wrote tests that caught nearly every planted wrong fix. Opus finished in 91 seconds with 7 tool calls. The only difference between a 1.00 and a 0.96 was which single wrong fix each model’s self-written tests failed to pin down.

MacBook Pro M5, 32 GB

This is the hardware most developers already have. Three models ran locally via LM Studio with a 65,536-token context (the default 8,192 context caused silent failures because the agent’s system prompt alone ran about 15k tokens).

Related:Build a Stream Dock plugin with an AI agent in one afternoon

Gemma 4 12B won on the laptop despite being the smallest model. It found the core bug: it took the posted transfer amount instead of recomputing it, and applied that fix to both the statement and the CSV export. It missed UTC month boundaries, ISO timestamps, and the fee line item, wrote no tests, and never read DECISIONS.md. But it did fix the customer-facing bug the manager asked about. That result showed up in only 3 of its 10 runs. In other runs it scored as low as -0.17 with identical settings, so the variance from chance exceeded the variance from any sampling configuration Girchev tried.

Qwen3 Coder 30B A3B ran 50 tool calls, produced a cheerful report about a “minimal and precise fix,” and fell directly into the bait converter trap: it swapped the old converter for the new one but kept recomputing the transfer. By the hidden tests, it had done essentially nothing. It also left a test_debug.py file in the repo root. Qwen3.8 27B was the only laptop model to read DECISIONS.md but made zero edits in 20 minutes.

️ RTX 5080 Gaming Laptop

The gaming laptop ran an RTX 5080 with 16 GB of VRAM and 32 GB of RAM, serving Qwen3.6-35B-A3B over the network via LM Studio. The model weighs 21.7 GB, so part of it always lives in RAM. Out of the box, generation speed dropped from 54 tok/s on short contexts to 9.5 tok/s on 42k-token contexts because LM Studio had moved attention layers to the CPU.

The fix required three LM Studio settings: GPU Offload at maximum (all 40 layers on GPU), MoE expert weights forced to CPU for about 40% of layers (freeing VRAM while keeping attention on the GPU), and the KV cache set to q8_0. Total VRAM in use came to about 14.5 GB. After retuning, the same 42k context ran at 94 tok/s instead of 9.5.

Performance comparison between harnesses on the same model:

Pi’s system prompt is about 2,200 tokens. OpenCode sends roughly 15,000 tokens on the first step alone. On local hardware, that 13k-token difference costs about 13 seconds per step. Pi finished in under five minutes. OpenCode never finished within the limit, reasoned out 55k tokens versus pi’s 11k, and additionally broke the corrected rate in the export. The model got faster after retuning, but not smarter. Both missed UTC months, ISO timestamps, and the fee on every run.

NVIDIA DGX Spark

The Lenovo ThinkStation PGX runs a GB10 chip with 128 GB of unified memory. Models ran in vLLM. Four configurations were tested:

Qwen3.6-35B-A3B (MoE, NVFP4): 0.25

Five runs across two computers, two servers, and two harnesses. Every single run missed UTC months, ISO timestamps, and the fee. The model is consistent in both its strengths and its blind spots.

ThinkingCap-Qwen3.8-27B (dense, 987 seconds, no limit): 0.92

With a 10-minute limit this model produced an empty diff. It spent the first 5.5 minutes having its scout subagent survey the project, correctly named the main bug at minute 8, started one long generation, and was cut off. Without the limit it scored 0.92: all 7 subtle conditions, the traitor test updated, and its own tests catching 7 of 9 wrong fixes. That is Opus-level quality on a task that takes Opus 91 seconds. The 27B needed 987 seconds.

Qwen3.8-Flash-Next 125B (MoE, NVFP4+MXFP8, 10-minute limit): 0.96

The largest model tested locally. It hit the limit during a final re-check, but all edits were already in place by then. The result: 20/20 hidden tests, all 7 subtle conditions, the traitor test updated, and 8 of 9 wrong fixes caught by its own tests. That matches Claude Opus in a clean setup, at six to seven times the wall-clock time. It consumed nearly all 128 GB of the Spark’s unified memory: roughly 96 GiB held by the NVIDIA driver, leaving about 16 GB free for the system while running.

NVIDIA Nemotron 3.5 Super 120B (MoE, NVFP4, 10-minute limit): 0.68

Finished on its own in 533 seconds. Got 18/20 hidden tests and 6/7 subtle conditions. It left the fee inside the transfer line in the statement, did not update the traitor test, and its own tests caught only 4 of 9 wrong fixes. The driver held about 96 GiB for this model as well. Flash-Next is clearly the stronger of the two big MoEs on the Spark.

Cheap Cloud: Chinese Models via OpenRouter

Between local hardware and Opus at $0.30 per run, a large market of cheaper models exists. Girchev tested one or two recent models from eight Chinese vendors plus Devstral, all via OpenRouter with OpenCode and the same time limit.

DeepSeek V4.1 Flash matched Codex’s perfect score in 112 seconds for $0.008 per run. That is 36 times cheaper than Opus on the same task, with a higher score. The September snapshot of the same model scored 0.88, suggesting the October update closed real holes.

Related:Google Modern Web Guidance 1.0: free skill pack for AI coding agents

GLM-5.3 from Zhipu missed only the traitor test update and scored 0.98 for $0.10 a run. MiniMax M3.1 Flash hit 20/20 and all subtle conditions for $0.03, though it added a paragraph to DECISIONS.md, which the rules forbid. Kimi disappointed across both models tested: K3 missed the fee and didn’t update the traitor test, K3.1 Pro passed all hidden tests but wrote only 3 tests of its own. At $0.27 per run that’s approaching Opus pricing for meaningfully weaker test coverage. DeepSeek V4 Pro is the counterintuitive result: the larger, more expensive model from the same vendor scored 0.47, roughly half the Flash score, and missed time zones and the fee. The Pro is from August, the Flash from October.

The Full Picture

The gap between the best Chinese cloud models and the best laptop models is not close. Four models via OpenRouter matched or approached Codex’s perfect score for under $0.20 a run. The best laptop result, Gemma 4 12B at 0.21, fixes the customer-facing bug but misses four of the seven subtle edge cases and writes no tests.

Local hardware can reach cloud-level quality, but the requirements are steep. The dense 27B on the Spark scored 0.92 but needed 987 seconds and wouldn’t finish within a 10-minute limit. The 125B MoE scored 0.96 but consumed nearly all 128 GB of unified memory on a dedicated AI workstation. Speed is constrained by hardware, quantization, and speculative decoding, and all of that gets cheaper over time. The intelligence, at least on a 700-line codebase, appears to already be there in the larger models.

One practical note from Girchev on running these setups: several local models in the test, including Qwen3.6 builds from huihui-ai and Flash-Next, are abliterated versions with refusals removed. The pi harness runs commands without confirmation by default and its documentation explicitly states that watching the transcript does not protect you. A model that cannot refuse paired with an agent that does not ask is a combination better run inside a container on a work machine.

The Verdict

For a solo developer who wants a cheaper, less restricted coding agent today: DeepSeek V4.1 Flash via OpenRouter is the standout result. Perfect score, 112 seconds, $0.008 per run. The other top performers (Xiaomi MiMo V2.6 Pro, Tencent HY4, Qwen3.8 Max Prime) also hit 1.00 but cost more. GLM-5.3 at $0.10 with a 0.98 score is also worth a look.

For a true offline fallback with no external dependencies: a DGX Spark running Qwen3.8-Flash-Next 125B gets you Opus-level results in 10 minutes with no subscription and no one else’s rules. The hardware cost and memory requirements are real constraints. A MacBook with a quantized model is a fallback, not a replacement: it fixes the obvious bug, but it will not catch your edge cases or write the tests that prevent regressions.

Cagri Sarigoz( Founder )

I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.

With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.

At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.

And more than all else, I’m a father to a (always) little girl and a husband.

Contact me at [email protected] for assistance.