01 |BEAM 1M
Headline and strict. Side by side, for every vendor we could check.
A headline BEAM score is a mean: each rubric item earns 0, 0.5 or 1, and the scores are averaged. A strict score counts only the questions where every item is fully met. None of the vendors below publishes one, so we counted it from their own per-question files wherever those files exist.
SurrealDB Agent Memory
Answered by Gemini 3.8 Flash Judged by Gemini 3.5 Flash Lite
Exabase
Answered by Gemini 3 Flash Preview Judged by Gemini 2.5 Flash-Lite
Hindsight
Answered by Gemini 3.1 Pro Preview Judged by Gemini 2.5 Flash-Lite
Mem0
Answered by GPT-5 Judged by GPT-5
Honcho
Answered by Claude Haiku 4.5 Judged by Claude Sonnet 4.5 (harness default)
HydraDB
Answering model not stated Judged by GPT-5.4
Our 77.2 scores event ordering by Kendall tau, as the paper does. Our run used ten retrieval and chat settings that the product does not enable by default. Strict shares for other vendors are our count from each vendor's own published per-question files: a question counts when every rubric item scored 1, or, where only a question score is stored, when it scored 1. Headlines are as each vendor published them. The systems used different answering models, judges and event-ordering rules, so the rows show how each score was made rather than a ranking.
EVERY RUN
The published runs we checked. One tab for each split.
| System | Strict | Mean | Pass | Answering model | Judge | Event ordering | Retrieval | Per-question files |
|---|---|---|---|---|---|---|---|---|
| SurrealDB rubric average | 59.9% | 79.3% | 85.7% | Gemini 3.8 Flash | Gemini 3.5 Flash Lite | Rubric average | Top 50 memories | Download |
| SurrealDB Kendall tau | 57.1% | 77.2% | 82.9% | Gemini 3.8 Flash | Gemini 3.5 Flash Lite | Kendall tau | Top 50 memories | Download |
| Exabase | 48.0% | 75.0% | 85.3% | Gemini 3 Flash Preview | Gemini 2.5 Flash-Lite | Kendall tau | ~18,900 context tokens | Yes |
| Hindsight | 48.4% | 73.9% | 83.6% | Gemini 3.1 Pro Preview | Gemini 2.5 Flash-Lite | Kendall tau | ~23,900 context tokens | Yes |
| Mem0 top-200 | 40.1% | 64.1% | 70.1% | GPT-5 | GPT-5 | Rubric average | Top 200 memories | Yes |
| Mem0 top-50 | 35.9% | 60.4% | 67.1% | GPT-5 | GPT-5 | Rubric average | Top 50 memories | Yes |
| Honcho | 41.4% | 63.1% | 73.3% | Claude Haiku 4.5 | Claude Sonnet 4.5 (harness default) | Kendall tau | Agentic, up to 20 tool calls | Yes |
| HydraDB | - | 82.0% | - | Not stated | GPT-5.4 | Not stated | Not stated | None |
Strict shares are our count from each vendor's own per-question files. Every pass rate is the vendor's own published count of correct answers; the pass bar is a rubric average of 0.5.
HydraDB gives its 82 as an unweighted mean of per-category scores. It does not state its answering model or its retrieval settings, and there are no per-question files behind it.
Event ordering by rubric average scores whether each event is named, not whether the order is right, so it gives higher scores, even when the events are not in the correct order.
Pass is the share of questions that score at least 0.5. A question with half marks or more counts as a full pass, where Mean counts only the marks it earned, so Pass reads higher than Mean. It is also more lenient than Strict, which counts only the questions where every rubric item is fully met.
How the run was made
The run used our benchmark harness, which lives in the Agent Memory repository. That repository is private today, and the exact invocation behind the published run was not recorded. We will publish the commands beside the answers file.
Ingest the conversations
Load the 1M conversations once, into a cached store.
Answer every question
Answer all 700 questions over that store, and record every answer.
Judge the answers
Score every recorded answer against its rubric with the judge model.
02 |THE PAPER AND US
What we kept from the paper. And what we changed.
Our harness replicates the BEAM paper's judge. Where the run departs from the paper in a way that moves the score, it is stated here, including the one departure that lifts our own score.
Same judge prompt
The judge prompt is the official BEAM prompt. The judge model is gemini-3.5-flash-lite, not the gpt-4.1-mini the authors' code uses, so the model is one of the differences from the paper.
Nothing added
The question category never reaches the answerer, and the question never reaches the judge.
Half marks kept
Each rubric item scores 0, 0.5 or 1, as the paper specifies, and the mean averages them. The strict share counts only full marks.
Our prompt and retrieval
The run answers with the product's own chat prompt over its own retrieval, not the paper's baseline prompt over a top-5. Ten retrieval and chat settings were switched on that the product does not enable by default.
Event ordering
Event ordering is scored by Kendall tau, whether the order is right, as the paper does. It scores 49.7 here, our weakest ability, and 1 of its 70 questions is fully correct.
Failures score zero
A judge call that fails twice scores its item 0, and an out-of-range score is clamped to between 0 and 1. The reference code abandons the conversation instead.
03 |HOW WE REPORT
Four rules for every number. Applied to every run on this page.
No score without its judge
Every score names the model that answered, the model that judged and the protocol it ran under.
The strict score sits beside the mean
Partial credit stays visible, rather than folded into one figure.
No ranking claims
The systems on this page used different models, judges and protocols, so it compares how scores were made, not who came first.
The changes from the paper are listed
Where the run departs from the paper in a way that moves the score, the change is stated above, including the one that lifts our own score.
WHAT LIFTS A SCORE
Eleven practices that shape a memory score. Who does what, with the evidence.
Each practice appears in public benchmark code, results or write-ups, and each is sourced below. None of it says anything about intent. Our own column is marked as plainly as everyone else's.
| Practice | Mem0 | Zep | Hindsight | Exabase | HydraDB | Honcho | Cognee | ByteRover | SurrealDB |
|---|---|---|---|---|---|---|---|---|---|
| How it is scored | |||||||||
| Event ordering scored by rubric average | - | - | - | - | - | - | - | ||
| A pass rate beside the mean | - | - | - | - | - | - | - | - | |
| Lenient LoCoMo judging | - | - | - | - | |||||
| The adversarial category dropped | - | - | - | ||||||
| Excluded questions counted in the score | - | - | - | - | - | - | - | - | |
| What the system is given | |||||||||
| Answer metadata in the answer prompt | - | - | - | - | - | - | |||
| The question category picks the answer prompt | - | - | - | - | |||||
| A larger retrieval budget | - | - | - | - | - | - | - | - | |
| Same-family answering and judging models | - | ||||||||
| How it is reported | |||||||||
| A small subset | - | - | - | - | - | - | - | - | |
| Headlines without per-question files | - | - | - | - | |||||
The evidence, practice by practice
How it is scored
What the system is given
How it is reported
GET STARTED
Give your agents a memory you can measure. Every score here names its judge, its models and its protocol, with the strict score beside the mean.
