bootkey.ai

research // benchmark · work in progress

What does your agent forget between Monday and Thursday?

· 5 min read

An agent on bootkey lives for weeks. You tell it something on Monday, and on Thursday you change your mind. Between those two days it has been asleep, woken for its duties, talked to other people, and started a fresh conversation more than once. Whatever it still knows on Thursday, it knows because it wrote it down.

Most memory benchmarks don’t test that. They test recall inside one long conversation, or they ask about a fact planted earlier and check whether it comes back. We wanted to know something harder: whether an agent carries its own decisions from one session to the next, and whether it changes the right ones when someone takes something back.

So we built a benchmark for it, and the code is now public: InterruptedMemoryArena on GitHub.

A group trip, one traveler at a time

The tasks come from the group travel planner in MemoryArena. Several people plan a trip together. The first traveler gets a full plan: transport, three meals, attractions and a place to sleep, for every day. Each traveler after that states their own wishes, and many of them point back at someone earlier: “I’ll have dinner wherever Anna has dinner on day 2.”

In our version every traveler is a separate session with a fresh conversation. The agent has a machine of its own, and between sessions it keeps only what it chose to save there: notes, a file of decisions, whatever it likes. Nothing is carried over for it.

Then people change their minds

On top of that we add interruptions, adapted from the taxonomy in InterruptBench and aimed at memory across sessions:

  • Retract. Right after the agent answers, the traveler takes back one request in the same conversation.
  • Revise. In a later session, an earlier traveler now wants to join someone else for a meal or a night.
  • Add. In a later session, an earlier traveler adds a request they forgot.
  • Undo. In a later session, an earlier traveler takes back a revision they made in an even earlier session.

Joins are live links. If Ben has dinner wherever Anna does, and Anna later changes her restaurant, Ben’s dinner moves too, and so does anyone who joined Ben. One sentence from one traveler can change three plans the agent wrote two sessions ago. The undo is the nastiest: the agent has to remember not only what changed, but what it was before.

Because every change is a join, the right answer is always exact. We replay the events in order, follow the links, and get the plan every traveler should end up with.

Keeping it fair

The point is to compare agents, so everything except the agent is held still.

  • Every agent gets the same prompts, word for word. Only the harness around the model differs.
  • Every model call goes through one proxy that fixes the model, the provider and the reasoning effort, records the tokens and cost of every turn, and retries rate-limited calls the same way for everyone. No agent wins or loses on its own retry logic.
  • The agents look things up and hand in plans through one small command, trip. A network fence lets them reach that and the proxy, and nothing else.
  • Each chain starts in a fresh home on the same kind of machine.

What we measure

We score each traveler the way MemoryArena does, with a fuzzy name match on every field of the final plan: whether each traveler passes, how many of the slots that differ from the first traveler are right, and whether the whole chain passes.

Then come the measures this benchmark exists for. For every slot an interruption moved, did the agent update it, or keep the old answer? For every revision that was undone, did it restore the original, or keep the withdrawn change? We also separate memory from planning. Was a plan that was right at the end of its own session still right at the end of the chain? After which session did things first go wrong? And does every join agree with the agent’s own answer for the traveler it follows, even when that answer is wrong?

Where it stands

We are running it now on Hermes Agent and PicoClaw, and on two private setups: Hermes with bootkey’s coding hands, and our own agent. We will publish results once the MemoryArena and InterruptBench authors have had their say; we have written to both. Until then the repository holds our code only, under Apache 2.0, and you need MemoryArena’s dataset (CC BY 4.0) to build the tasks.

If you build agents and want to know what yours forgets between Monday and Thursday, it is all there to run.