Case study · coding agents · local models · 2 to 3 November 2026
Can a local model replace Claude Code?
I built 23 real coding tasks, hid the tests, and gave them to four agents. A free cloud model came within one task of the best, and the model on my laptop solved three in four.
Part of Work · What I'm doing
I use Claude Code every day. I wanted to know how much of that work a free model, or the model running on my own laptop, could do today. So I measured it.
Each task is a small repository with a bug to fix or a feature to add, like the ones in my own projects. The agents work with their real tools, and hidden tests they never see decide pass or fail.
Terms used here
- Coding agent
- A model that reads files, runs commands and edits code in a loop until the job is done.
- Harness
- The program around the model that gives it tools and runs the loop: Claude Code, OpenCode or DeepSeek's harness here.
- Hidden tests
- Tests written before the run and copied in only after the agent stops, so it cannot fit its code to them.
- Reference solution
- My own solution to each task. It proves the hidden tests can be passed.
- Local model
- A model running on my own laptop, through llama.cpp on its Radeon 8060S graphics with 64 GB of shared memory.
- Qwen3.6-35B-A3B
- An open model with 35 billion parameters, of which about 3 billion work on each token, so it runs at a usable speed on a laptop.
- Perft
- A chess test: count every legal position a few moves deep and compare with known totals. One wrong rule changes the count.
- Median
- The middle time when a run's 23 task times are sorted. One very slow task does not move it.
The test
one task, start to finish
- A small real repositorya bug, a feature, a refactor or a slow function, with a plain request
- The agent workswith its own tools, in a fresh copy, until it says it is done or time runs out
- Hidden tests arrivecopied in after the agent stops
- Pass or faila task passes only if every hidden test passes
Twenty everyday tasks cover bug fixes, features, a refactor across three files, a crash to debug from its traceback and a function to make fast. Three hard tasks follow: a chess move generator checked by perft, a regular-expression engine that must agree with Python's own, and a small SQL engine that must return what SQLite returns. Before any run, every task is checked both ways: the starting code fails its hidden tests and my reference solution passes them.
The 23 tasks
- 01 Elo rating after a draw · bug fix
- 02 Chess game header parser · feature
- 03 Private-data detector · bug fix
- 04 Running pace calculator · feature
- 05 Cache with expiry · feature
- 06 Time-weighted investment return · feature
- 07 Spaced-repetition scheduler · feature
- 08 Spanish-format CSV import · bug fix
- 09 One config loader for three modules · refactor
- 10 Async retry with back-off · feature
- 11 Crash from a traceback · debugging
- 12 Near-duplicate finder, made fast · performance
- 13 Broken-link checker for notes · feature
- 14 Free slots across calendars · feature
- 15 Text chunker that hangs · bug fix
- 16 JSON output for a CLI, old output kept · feature
- 17 Validator for tool-call arguments · feature
- 18 Mixture-of-experts routing · feature
- 19 Database migrations · feature
- 20 A failing test suite · bug fix
- 21 Chess move generator, perft-checked · hard
- 22 Regex engine, checked against Python's re · hard
- 23 Mini SQL engine, checked against SQLite · hard
Results
tasks passed, of 23
Same 23 tasks, one run each. The two bottom rows are the same model in two different harnesses.
Task by task
23 tasks × 4 agents
Everyday tasks 01 to 20, hard tasks 21 to 23. Point at a square for the task and its time.
Claude Code passed everything. DeepSeek's only miss was the regex engine, which ran out of time. The local model, in OpenCode, missed four everyday tasks and two hard ones, and finished the regex engine that DeepSeek ran out of time on.
Time
each dot is one task
Hollow dots are failed tasks; dots at the far right ran out of time. Cloud times include the network.
Findings
What the numbers say
- A free model nearly matched the frontier. DeepSeek V4.1 Flash, on OpenCode's free tier, passed 22 of 23 and had the fastest median, 36 s against Claude Code's 52 s. The whole run cost nothing; the same 23 tasks in Claude Code would cost $6.07 at API prices (covered here by my subscription).
- The laptop handles most everyday work. Qwen3.6 on my own machine passed 16 of 20 everyday tasks, for free and with nothing leaving the laptop. It took four to five times longer than the cloud agents, and the hard tier is still beyond it: 1 of 3.
- The harness matters. The same local model passed 17 in OpenCode and 14 in DeepSeek's harness. In three of its misses, DeepSeek's harness stopped before writing the file it was asked for.
- Where the local model slips. Its everyday misses were details: merging overlapping calendars, one error path in a validator, rolling back a failed migration, and a duplicate finder that stayed too slow.
So today I keep Claude Code for the hard work, and the local model is ready for private, everyday tasks.
Limits
What this does not show
- 23 tasks is a small sample. A gap of one or two tasks could change on a second run.
- One run each. The one exception: DeepSeek's regex engine ran out of 25 minutes, so I ran it again with 40, and it ran out again. The code it had written by the first cut-off did pass the hidden tests; a run that runs out of time counts as a fail here.
- Local times are rough. The runner records the power state for every task, and parts of both local runs were on battery, which slows the laptop.
- This benchmark only. The tasks are short, precise and in Python with no outside libraries. Long projects, vague requests and large codebases may rank the agents differently.
Method
How I built it
- The runner is one Python script. For each agent and task it copies the starting code into a fresh git repository, runs the agent there with the task as its prompt, then copies in the hidden test and runs it.
- Real tools. Claude Code with Opus 5.5 on my subscription, OpenCode with DeepSeek V4.1 Flash on its free tier, and OpenCode and DeepSeek's harness with Qwen3.6-35B-A3B (4-bit) served by llama.cpp on the laptop. The score measures model and harness together, which is what replacing Claude Code means.
- A fair start. A check mode runs every task's hidden tests on the starting code, which must fail, and on my reference solution, which must pass.
- Everything kept. Each run writes one line per task, with time, power state, hidden tests passed and the end of the test output, and leaves the work folder and the agent's log for inspection.
Python · Claude Code · OpenCode · DeepSeek harness · llama.cpp (Vulkan) · Qwen3.6