--- title: "Agent memory benchmarks are baked" description: "Agent memory benchmark scores are recipes: answering model, judge, prompts and scoring rule. Here is how ours was made, including the numbers that lower it." url: https://surrealdb.com/blog/agent-memory-benchmarks-are-baked date: 2026-10-14 authors: "Tobie Morgan Hitchcock" --- # Agent memory benchmarks are baked ![Agent memory benchmarks are baked](https://cdn.surrealdb.com/kmdjeifs3wjzfrhbfdla902j.auto) Every agent memory vendor publishes a benchmark score. While most of these scores are true, very few of them measure the same thing. We read the test harnesses (the code that runs a benchmark), prompts, judge settings and per-question output files behind the published scores on [BEAM](https://arxiv.org/abs/2510.27246) and [LoCoMo](https://snap-research.github.io/locomo/), two benchmarks for agent memory, and we did the same for our own. A memory benchmark score is the result of several choices: which model writes the answers, which model judges them, which prompts they get and which scoring rule turns the judgements into a number. Change any one of these and the score changes. Without them, a headline score is hard to interpret. This post describes what we found when we ran SurrealDB Agent Memory on these benchmarks, what we changed in our own run, and every number behind our headline score, including the ones that lower it. ## A score is a recipe BEAM gives a memory system long conversations, up to ten million tokens, and then asks questions about them. An answering model writes each answer from what the memory system retrieves. For each question, the benchmark lists the points a correct answer must contain, called rubric items. A judge model scores each rubric item 0, 0.5 or 1, the question's score is the average of its items, and the headline is the mean across all questions. Each step has settings that a vendor chooses. Who answers? Who judges? Does the judge see the question? Does the answering prompt know what kind of question it is? Does half credit count? How is event ordering scored? While none of these settings is dishonest on its own, a headline score rarely says how they were set. ## Five ways a score gets easier All of these practices appear in published runs. The code and output files for each are linked from our [benchmarks page](https://surrealdb.com/agent-memory/benchmarks). **1. The answering prompt includes parts of the answer.** BEAM stores metadata with each question, such as the topics a correct answer has to put in order. [Hindsight](https://hindsight.vectorize.io/)'s April 2026 harness and [Exabase](https://exabase.io/)'s published prompt add fields from this metadata to the answering prompt. In Hindsight's own outputs, 66 of 70 event-ordering answers name every expected topic, and none of those topics appears in the question. [HydraDB](https://hydradb.com/) says it reused Hindsight's original prompts unchanged, and it has published no outputs. **2. The question category chooses the prompt.** BEAM sorts its questions into categories, such as event ordering and instruction following. Hindsight, Exabase and [Cognee](https://www.cognee.ai/) choose an answering prompt based on the question's category, and [Honcho](https://honcho.dev/) sends instruction-following questions through a separate pipeline. In production, a memory system does not know in advance what kind of question it is about to answer. **3. The scoring rule gets more lenient.** The BEAM paper scores event ordering with Kendall tau, a measure of how closely the answer's order matches the correct order. A rubric average only checks whether each event is mentioned, in any order. [Mem0](https://mem0.ai/) and Cognee use the rubric average. We use Kendall tau, as the paper does. Event ordering is our lowest-scoring category, so this choice lowers our mean, but it is the only rule of the two that checks whether the order is correct, which is what the category exists to test. Scored by rubric average, the same answers would read 79.3% instead of 77.2%. **4. The retrieval budget grows.** Mem0's headline on BEAM 1M (the version with conversations of one million tokens) gives the answering model up to 200 memories. Mem0’s own files show a score of 64.1% with 200 memories and 60.4% with 50. Our published run gives the answering model the top 50 memories. **5. On LoCoMo, the hardest questions are removed and the judge accepts near misses.** LoCoMo groups its questions into five categories. Category 5 holds the adversarial questions, which cannot be answered from the conversation, so a correct response says that the information is not there. Following the Mem0 paper, most published LoCoMo scores leave this category out. Mem0's judge also counts an answer as correct when it gives one correct item out of a list, or a date up to 14 days away from the right one. In April 2026, an [audit by Penfield Labs](https://dev.to/penfieldlabs/we-audited-locomo-64-of-the-answer-key-is-wrong-and-the-judge-accepts-up-to-63-of-intentionally-33lg) found that LoCoMo's judge accepted 62.81% of answers that were deliberately wrong but on the right topic. The choice of models also affects the score. Most vendors use models from the same family to answer and to judge, and many use a stronger answering model than the original papers did. Some headline scores have no per-question files behind them at all. ## Counting only full marks A BEAM mean hides partial credit. A question where every rubric item gets half marks scores 0.5, and it raises the mean just as a correct answer does. So we also counted something simpler: the share of questions where every rubric item got full marks. We call this the strict score. None of the vendors we compared publishes one, so we calculated it from their own per-question files wherever those files exist. On BEAM 1M: | System | Headline mean | Strict | Answering model | Judge | | ---------------------- | ------------- | ------------- | ---------------------- | ----------------------------------- | | SurrealDB Agent Memory | 77.2% | 57.1% | Gemini 3.8 Flash | Gemini 3.5 Flash Lite | | Exabase | 75.0% | 48.0% | Gemini 3 Flash Preview | Gemini 2.5 Flash-Lite | | Hindsight | 73.9% | 48.4% | Gemini 3.1 Pro Preview | Gemini 2.5 Flash-Lite | | Mem0 (top-200) | 64.1% | 40.1% | GPT-5 | GPT-5 | | Honcho | 63.1% | 41.4% | Claude Haiku 4.5 | Claude Sonnet 4.5 (harness default) | | HydraDB | 82.0% | Not countable | Not stated | GPT-5.4 | Every strict score we could count is between about 20 and 27 points below its headline mean. A gap of this size is expected, because the mean gives credit for answers that are partly correct. Publishing both numbers shows how much of each score comes from that partial credit. Read the table as a record of how each score was produced. The rows use different answering models, different judges and different retrieval budgets, so they cannot be ranked against each other. ## What we did in our own run We followed the BEAM paper wherever we could, and we list every place where we did not. - **Same judge prompt.** The judge prompt is the official BEAM prompt. The judge model is Gemini 3.5 Flash Lite instead of the GPT-4.1 Mini that the authors' code uses, so the judge model is one of the ways our run differs from the paper. - **Nothing extra in the prompts.** The question category is never given to the answering model, and the judge sees the answer and the rubric but not the question. - **Half marks counted, and shown separately.** The mean counts 0.5 scores, as the paper specifies. The strict score of 57.1% counts only full marks. - **Our own prompt and retrieval.** The run answers with the product's own prompt and retrieval, instead of the paper's baseline prompt with the top 5 results. - **Ten non-default settings.** We switched on ten retrieval and chat settings that the product does not enable by default. Eight change retrieval: `SPECTRON_RETRIEVAL_TURN_ORDER`, `SPECTRON_RETRIEVAL_CLAIM_CONFLICTS`, `SPECTRON_RETRIEVAL_GROUNDING_CHECK`, `SPECTRON_RETRIEVAL_EPISODIC_TREE`, `SPECTRON_RETRIEVAL_EVENT_DIGESTS`, `SPECTRON_RETRIEVAL_TREE_BREADTH`, `SPECTRON_RETRIEVAL_GROUNDING_PROVENANCE` and `SPECTRON_RETRIEVAL_TREE_ADAPTIVE_BUDGET`. Two change how the chat answer is written: `SPECTRON_CHAT_ANSWER_CONTRACTS` and `SPECTRON_CHAT_STANDING_INSTRUCTIONS`. - **Event ordering scored by Kendall tau.** As in the paper, the score checks whether the order is correct. We score 49.7% in this category, and 1 of its 70 questions gets full marks. - **Failures score zero.** When a judge call fails twice, the rubric item scores 0, and a score outside the range 0 to 1 is clamped to that range. The BEAM reference code stops scoring the whole conversation instead. On LoCoMo, our strict score is 75.8% across all five categories, including the adversarial one, over 1,986 questions. Under Mem0's own protocol, which leaves out the adversarial category and uses the more lenient judge, the same system scores 92.6% with GPT-5 answering and judging, and 90.1% with GPT-OSS 120B answering and GPT-4o judging. We publish all three numbers together so that readers can compare them directly. Our 700 BEAM 1M answers are published on the [benchmarks page](https://surrealdb.com/agent-memory/benchmarks), each with its question and rubric, so you can grade them with any judge you like. The harness commands are not public yet. ## Be your own judge Before you trust any memory benchmark score, including ours, ask five questions: 1. Which model answered, and which model judged? 2. Did the prompt know the question's category, or any part of the answer? 3. Does the headline include partial credit, and what is the strict score? 4. How much context did the answering model get? 5. Are the per-question files published, so that you can check the count yourself? A score that comes with answers to all five can be checked. A score without them has to be taken on trust. Every run, judge, protocol and vendor comparison is on our [benchmarks page](https://surrealdb.com/agent-memory/benchmarks), with sources linked for every claim.