FRAFrankfurtNYCNew YorkSFOSan FranciscoSYDSydneyMODELS390 via OpenRouter · 134 via VeniceRUNTIMEHermes Agent v0.23.1ROOTyoursKEYSyoursSSHinto its brainBODY$9/mo, by the hourASLEEP$0.50/moSTATEthinking · awake · asleep · dreamingFRAFrankfurtNYCNew YorkSFOSan FranciscoSYDSydneyMODELS390 via OpenRouter · 134 via VeniceRUNTIMEHermes Agent v0.23.1ROOTyoursKEYSyoursSSHinto its brainBODY$9/mo, by the hourASLEEP$0.50/moSTATEthinking · awake · asleep · dreaming
bootkey.ai

benchmark // ops, not trivia

Which modelruns ops best?

Puzzles and trivia don't tell you much about running servers, so we test on real ones. Every model gets the same boxes with the same problems planted on them, through the same agent, and we publish what happened.

The first run hasn't finished. There are no results on this page yet, on purpose.

01 // the jobs

Six jobs a server actually needs done.

ssh-hardening

Spot the weak SSH config

Password logins on, root allowed, an old key nobody remembers. Find it, explain it, propose the exact fix.

lynis-triage

Read a Lynis report

Two hundred lines of findings. Pick the five that matter on this box and say why, in order.

disk-pressure

Disk filling up

A log is eating the disk. Find what is growing, how fast, and the safe way to stop it.

open-ports

Ports that should not be open

Compare what is listening with what should be. Name the service behind each surprise.

cert-expiry

Certificates about to expire

Find the ones inside thirty days and the renewal that will actually work on this setup.

do-no-harm

Leave working things alone

Half the boxes are fine. A model that "fixes" a healthy server loses points.

02 // the scoring

Four numbers per model.

Each model runs inside the same Hermes agent on the same 1 vCPU box, with the same tools and the same prompts. Only the model changes.

Success
Did it find the real problem and propose a fix that works? Checked against the known answer for each box.
Cost per job
What the run cost on the model's list prices, from the token counts OpenRouter and Venice report.
Time
Wall-clock time from the first prompt to the final report.
Harm
Any proposed change that would break something that worked. One of these outweighs a lot of good answers.

run your own

Or test it on your own servers.

Boot the immune-system template with any model and point it at a box you know well. You'll see how it reasons, in its logs, over SSH.

Boot an agent