Memory benchmark
This page describes the Memory benchmark: the questions, the graph schema, how Agentic GraphRAG answers are scored, and the result of the committed lite run. For the commands, see Run the benchmarks. For the audit trail and the limits of these numbers, see Benchmarks.
The dataset
BEAM (paper) tests memory over a long chat between a user and an assistant. Each conversation has many sessions, and each session has many messages. After the chat, the benchmark asks probing questions about it. Each question has a rubric: a list of facts that a correct answer must state.
The harness reads the 100K-token split of the dataset Mohammadta/BEAM at a pinned revision. It makes sure that the hash of the file matches before it uses the file. It ingests the chat only. The plan, the profile, and the reference answers that the dataset holds beside the chat never reach Agentic GraphRAG.
One assistant message in conversation 4 turned into 345,211 spaces of filler. The harness cuts that message after its last sound text and adds a short marker. It makes sure that the hash of the original matches first.
The probing questions fall into ten abilities:
| Ability | What the question asks |
|---|---|
abstention | A fact that the chat never states. The right answer says so. |
contradiction_resolution | A point where two statements of the user disagree. |
event_ordering | The order of events in the chat. |
information_extraction | One fact from the chat. |
instruction_following | A request that must follow an earlier standing rule of the user. |
knowledge_update | A fact that the user changed during the chat. |
multi_session_reasoning | An answer that joins facts from several sessions. |
preference_following | A request where an earlier preference of the user shapes the answer. |
summarization | A summary of a part of the chat. |
temporal_reasoning | A question about time between events. |
Lite and full
| Set | Questions | Conversations | Tokens |
|---|---|---|---|
| Lite | 2 | 1 (conv17), 3 messages | 1,279 |
| Full | 100, ten for each ability | 5 (conv1, conv4, conv6, conv13, conv17), whole sessions | 568,987 |
The lite set is a smoke test, not a sample of the benchmark. It takes 3 messages of conversation 17, so a run takes a few minutes. Each question of the benchmark names the messages that hold its answer (source_chat_ids), and the harness ingests only those messages for the two lite questions:
conv17:information_extraction:0asks which days the kids have afterschool activities. Message 18 of session 1 holds the answer.conv17:multi_session_reasoning:0asks how many scenes the user had filmed by July 5 and how many were left. Messages 156 and 157 of session 3 hold the answer. Despite its name, the question needs only one session.
A slice keeps the session number and the session date, and it tags each message with its id. The harness pins the hash of each slice, and the fixture lists the message ids. Both lite questions are in the full set. Each conversation gets its own Neo4j service and its own graph. In full mode each session of a conversation is one document.
Four questions of the full set have a known defect in their rubric. They carry the flag known_bad_rubric. The lite set has none.
The graph schema
See benchmarks/schemas/memory.py. The schema has these entity types: Person, Place, Organization, Tool, Topic, Event, Fact, Instruction, Preference, Measure, and Session.
A statement is a node, not a relation, because a relation carries no time and no order. A newer Fact SUPERSEDES an older one. Two opposing facts stay linked by CONTRADICTS. A standing rule (Instruction or Preference) points at the topic that triggers it. A Session node holds the only clock: one date for each session. The order of messages comes from the order of the chunks.
A schema that fits the corpus helps Agentic GraphRAG, so these numbers do not compare with systems that use another schema.
How answers are scored
A judge model reads the answer and the rubric. It scores each rubric item 1, 0.5, or 0. The score of a question is the mean of its item scores. The score of an ability is the mean over its questions.
Each question gets these metrics:
score: the score of the question.score_excluding_known_bad: the same score, except that a question with the flagknown_bad_rubrichas no value. The group mean therefore leaves it out.event_f1andevent_tau_norm: forevent_orderingonly. The harness splits the answer into lines and matches them to the rubric items.event_f1is the F1 of that match.event_tau_normis the normalized rank correlation of the two orders. Thescoreof the question isevent_f1timesevent_tau_norm.
The scorer is ported from the benchmark, with its prompts kept. The judge is the judge model of the run. A judge reply that does not parse scores 0 for its item, and the question gets the flag parse_error.
The benchmark reports only the rank correlation for event ordering. Its score is stricter, so do not compare it with a published event ordering number.
Lite mode makes about 2 judge calls for each question, because it has no event-ordering question. Full mode allows for 12 on average, because an event-ordering question compares each answer line with each unused rubric item. It can need about 90 calls.
The judge is the same model as the agent by default. The scores are therefore biased upward. See Benchmarks.
Results of the lite run
The run used gemma4:31b for extraction, the agent, and the judge, on one clean commit, with the default chunking of about 1,000 tokens.
| Question | Ability | score |
|---|---|---|
conv17:information_extraction:0 | information_extraction | 1.00 |
conv17:multi_session_reasoning:0 | multi_session_reasoning | 1.00 |
Both answers match every rubric item, no question failed, and no question carries a flag. Two questions cannot rank anything. The run shows that ingest, answering, and grading work end to end.
The set cannot test memory over a long chat. The graph holds 3 messages, so a search returns all of the text, and a question never has to find a fact among many sessions. Use the full set for that.
The graph has 21 entities and 20 relations from 3 chunks. No chunk gave an empty extraction.
Print the record with:
uv run python -m benchmarks report --domain memory
Cost and time
| Phase | Model calls | Tokens |
|---|---|---|
| Ingest: extraction | 3 | 9k |
| Ingest: resolution and merge | 2 | 13k |
| Agent, 2 questions | 13 | 49k |
| Judge, 2 questions | 3 | 3k |
| Total | 21 | 74k |
The run took 153 seconds. The agent made 6 and 7 calls for the two questions. max_llm_pairs was 10, which is the most entity pairs per label that resolution sends to the model. The default is 500. A graph this small has few pairs to compare, so resolution made only 2 calls.
These figures are floors. The cost of a larger slice grows faster than its size, because resolution compares every new entity with the whole graph. The spend cap of the lite set is 40 calls and 120,000 tokens, and the dry-run estimate is 24 calls and 78,000 tokens. The cap uses the estimate before a run and does not stop a run in progress.
A run of the old lite set (all 5 sessions of conversation 17, 10 questions) made 1,018 calls and used 13.3M tokens in 115 minutes. Nobody ran the full set. The dry-run estimate of the full set is 11,323 calls and 89 million tokens, and its spend cap is 17,000 calls and 133.9 million tokens. The estimate is linear in the chunk count. A real run can therefore cost more. Before you start one, run make bench-dry DOMAIN=memory MODE=full. Then read the estimate.
Terms of use
The BEAM code is under the MIT license. The BEAM data is under CC BY-SA 4.0, and the share-alike term applies to adapted text. It therefore applies to the trace of a run. A trace holds passages of the chat. Before you share one, read benchmarks/results/NOTICE.