Same model, different hands

Agents write code all the time. Not just when you ask for an app: a script to check something, a fix to a config, a small tool to finish a job. And there's a familiar complaint. An agent framework fumbles a coding job that a dedicated coding tool handles easily, with the very same model behind both.

So before we gave our agents better hands for code, we measured. We took four ways an agent can write code, gave each the same model, the same tasks and the same prompt, and ran them on the same small box our agents live on. Then we scored the results with tests the agents never saw.

What we tested

The harnesses

• Hermes, on its own. The agent our boxes run, writing code with its own shell and file tools. This is the baseline: an all-round agent doing the coding itself.
• pi. A deliberately small coding agent: a very short system prompt and four tools (read, write, edit, bash). MIT licensed.
• Codex CLI. OpenAI's coding agent, pointed at OpenRouter.
• Crush. Charm's terminal coding agent.
• Hermes, handing code to pi. What we're shipping: Hermes stays in charge, hands the coding to a fresh pi session through a skill, then checks the result itself.

The models

The tasks

Each task comes with a few visible tests and a TASK.md. The score comes from a separate set of hidden tests that check the spec, the edge cases and that nothing else broke. The agents never see them.

The setup

• The same box our agents live on: one CPU and 1 GB of memory, with a Hermes agent already running on it, which leaves roughly 480 MB.
• The same prompt for everyone: read TASK.md, do it, work only in this directory, run the tests before you finish.
• Reasoning effort set to medium in every harness. A fresh copy of the task for every run, a 15-minute limit, an unprivileged user.
• Correct means hidden tests passed. Cost is what the OpenRouter key was actually charged for that run. Memory is the peak of the harness's own working memory, sampled twice a second, which is what competes with the agent on a 1 GB box. Time is wall clock.

Results

GPT-6.1 Terra (strong)

DeepSeek V4.1 Flash (mid)

Qwen3.7 Flash (cheap)

Cost is the total for the four tasks. Amber marks a run that missed hidden checks. "Limited" means the model's only provider rejected the requests (HTTP 429) before the harness could change anything, on every attempt; we retried up to three times, two minutes apart. An asterisk marks a run the rate limit cut short partway through; we show the attempt that got furthest. Cost and time leave out the limited runs.

What we learned

1. With a strong model, the harness decides the bill, not the result

All four harnesses got every hidden check right with the strong model, on all four tasks, including the five-module codebase. What differed was the path. Hermes on its own explored more, wrote more and paid for every token of it. pi's tiny prompt and four tools leave almost the whole context for the code: it was the cheapest, and as fast as anything we tested.

2. With a cheap model, the harness starts to matter

The cheap model is where the gaps opened up. Every harness stopped once the visible tests went green, so bugs that only the docstrings described went unfixed everywhere. That one is a model limit. But some harnesses also missed edge cases the spec spelled out, and the focused coding agents did better there than the all-round agent.

3. Rate limits are part of the job

The cheap model had a single provider on OpenRouter, and it throttled hard during our runs. Harnesses handled that very differently. Codex CLI gave up within seconds on every attempt, so we couldn't score it on that model at all. The others usually got through, though a few runs were cut short, and one stopped halfway through an edit and left the code broken. For an agent working unattended, patient retries and checking your own work matter as much as clever prompting.

4. Memory is not the deciding factor, but it counts

Codex CLI is a small native program and used the least memory by far. pi and Hermes each needed around a hundred megabytes while working. All of them fit next to a running agent on a 1 GB box.

What we're doing about it

Every bootkey agent gets coding hands: a skill that hands coding work to a fresh pi session with the agent's own model, then makes the agent check the result itself (run the tests, read the change, try again with the failing output if something is still wrong) before it reports back. Hermes stays the agent you talk to; pi does the typing.

Why a fresh session instead of one big agent that does everything? Because the same engine behaves very differently depending on what else is in its context. A focused session that holds only the task, a short prompt and four tools is exactly the setup that did best here.

Does the hand-off work?

We ran the setup we're shipping on the hardest task: Hermes in charge, with the skill, handing the coding to pi. In every run where the model was reachable, Hermes made exactly one hand-off, waited, then ran the tests itself before answering.

Handing off kept pi's results. On the strong model it also cut the bill by about a fifth against Hermes working alone. On DeepSeek it cost a little more, because Hermes' own turns (reading the task, then checking pi's work) weigh more when the coding itself costs under a cent. Either way it costs more than pi on its own. We think the check is worth paying for: it's the step that catches broken work before you see it.

The cheap model taught us one more thing. On one attempt the provider started rejecting Hermes' own requests while pi was still working, and Hermes gave up. The coding hands now log every call, and we're changing the hand-off so a parent that gets throttled waits and picks the result back up instead of walking away.

Caveats

• One run per harness, model and task. Treat single-check differences as noise and look at the patterns.
• The tasks are small to mid-sized and written for this test. They're not a substitute for large public suites; they're closer to the jobs our agents actually do.
• Hermes was measured as a one-shot command, not as the always-on agent, so its memory figure includes starting up. It runs its shell commands through its always-on service, so the commands it ran (mostly tests) aren't in its figure; for Hermes handing to pi we add pi's own peak.
• Hermes handing to pi ran with Hermes' normal instructions and skills loaded, as it would in production. Hermes alone ran without them.
• Versions as tested: Hermes Agent 0.23.1, pi 0.73.1 (published since under a new package name), Codex CLI 0.167.1, Crush 0.100.0.
• Costs are OpenRouter list prices on the day, read from the key's usage counter after each run.

Run it yourself

The tasks, the hidden tests and the scripts that ran them are in coding-harnesses-2026-11.tar.gz. You need a Linux box with systemd, Python 3, Node and an OpenRouter key. If you get different numbers, tell us at hello@bootkey.ai.