TECHNICAL DEEP DIVE

Agent Memory,
under the hood

Join the waitlist Book a demo

Compounds. How memory builds on itself as agents work.

Fast at scale. How recall stays quick as the store grows.

Defensible. How every answer carries the facts behind it.

01 |ANATOMY

The six parts of an agentic system. One decides what the model knows.

1 The agent

LLM · tools · loop

2 The context window

Working memory - where the traditional agent forgets

Capture

Recall · inject

Agent Memory

3 The context layer

“What does the model need to know right now?”

gathers · filters · packages

4 Memory

Everything the agent learned

episodic · identity · knowledge · context · instructions · uncertainty

5 The knowledge graph

How everything is connected

entities · typed edges · properties

6 Storage

One multi-model database

documentgraphvectorfull-text

Agent Memory is the bottom four: the context layer, memory, the knowledge graph and storage.

02 |MEMORY MODEL

Six kinds of memory. One raw record, five typed categories.

Episodic

The raw conversational record: sessions and turns as authored, in order. Every extracted fact cites back to it by byte span.

Identity

Durable facts about the principal: name, role, employer, preferences. Long retention, low decay, surfaced in the prompt-ready profile.

Knowledge

What the principal has learnt or shared: project facts, observations, references. Medium retention; decays without reinforcement.

Context

What is going on now: active topics, recent intents, the working set. Short retention, replaced quickly, anchors the next turn.

Instructions

Behavioural memory, such as 'always answer in British English'. Applied when the prompt is assembled rather than at retrieval.

Uncertainty

Explicit 'not yet known' rows, written when confidence is under the floor or sources contradict. Gaps stay visible.

03 |DOCUMENT PIPELINE

Documents become structure. Every format, one graph.

Multi-modal ingest

An ingestion profile per Context, from text only to full multimodal, trades completeness against cost. Lift it later without re-architecting.

Originals in object storage

PDFs and media live in S3-compatible storage; the database holds the indexable state. Backups, deletion and cost scale on object-store economics.

Content-addressed

A Blake3 hash is the identity, and a cache of extracted text lets you rechunk and re-embed without re-running the parsers.

Content-aware chunking

Passages are first-class rows with their own embeddings, byte spans into the original and edges to the entities in them. Audio and video chunk by time code.

One reconciler

Facts from documents pass through the same supersession-and-uncertainty function as facts from turns. A document can contradict a turn, and the contradiction is recorded.

Keyword graph

A RAKE pass, with no model call, produces keyword nodes with PMI-scored edges to chunks and entities: cheap structural recall for the rare terms vector search underweights.

04 |HYBRID RETRIEVAL

Eight signals, one ranking. Every result explains its score.

VECTORS

Vector recall

Dense embeddings on entities, attributes and chunks, plus native vectors for images and audio when enabled.

LEXICAL

Lexical (BM25)

BM25 over chunk text and entity surface forms catches the exact phrases and rare terms embeddings underweight.

GRAPH

Graph traversal

One or two hops from a seed entity surface related facts even when their wording diverges.

KEYWORDS

Keyword bridges

RAKE keyword nodes connect chunks and entities that share rare terms but sit far apart in embedding space.

DOCUMENTS

Document links

Section embeddings and chunk-to-chunk and document-to-document edges surface related sections of related documents.

RANKING

Personalised PageRank

A graph walk biased towards the query's seed nodes, scored as one feature in the fused ranker.

GEO

Geographic recall

Within-radius, inside-polygon and nearest-k predicates compose with the other signals. 'Acme within 50km of Berlin in Q3' is one query.

TRACES

Trace-derived features

Retrieval reads its own history: rows useful for similar queries are boosted, rows tied to corrections demoted.

05 |TIERED QUERIES

Four tiers, one trace. Cheap reads stay cheap.

Tier 1. Direct lookup

Typed questions, such as 'what is my role at Acme?', fetched by key from the entity graph. No embeddings, no model call. Sub-millisecond.

Tier 2. Response reuse

A semantic match against prior answers, keyed on the facts they cited. Supersede one fact and every dependent answer is invalidated. Tens of milliseconds.

Tier 3. Hybrid retrieval

All eight signals fused into one ranking, then synthesised by the model. Hundreds of milliseconds, with per-signal scores on the trace.

Tier 4. Full-context fallback

When retrieval runs thin: more candidates, deeper traversal, an optional query rewrite, a longer window. The expensive path, taken explicitly.

06 |AUDIT & TIME

Every fact has a source and a time. The audit trail is the data.

Provenance as data

Every entity, attribute, relation, instruction and uncertainty row carries a source: kind, reference, trust, byte span, location and derived_from. Reconciliation compares it, calibration weights it, supersession audits it.

Three independent clocks

System time, when the substrate changed; known time, when Agent Memory first believed a fact; valid time, when it held in the world. Each queryable on its own via VERSION and as_of.

Traces are memory

Retrieval, decision and response traces are graph nodes. The ranker reads its own history, lineage downgrades trust, and any answer walks back to the bytes behind it.

Supersede rather than delete

A changed fact is superseded with valid_until set, or aged out with a reason. Its history stays queryable. forget is an explicit verb, with --purge for hard removal.

07 |AUTONOMOUS UNDERSTANDING

Memory that improves between conversations. Reflection, elaboration, consolidation.

Reflection

On demand. POST /reflect runs a model pass over retrieved context and can persist the synthesis as new facts, with their own provenance kind and a lower default trust.

Elaboration

In the background. A job finds entities that share context but no relation; a model proposes the link and the reconciler accepts, supersedes or marks it uncertain.

Consolidation

Asynchronously. Recent facts are pooled and each is created, updated with the delta recorded, or superseded. Every observation tracks its inputs and proof count.

THE PLATFORM

Everything an application and its agents know. Five surfaces, one engine.

Database

Document, graph, vector, time-series and relational in one engine.

Five data models as dotted tiles: documents, graph, vector, time-series and relational

Agent Memory

What an agent learns, with its source and its time, in the same engine.

A timeline of remembered facts, each with its source

Cloud

Managed clusters in the regions you choose, scaled on demand.

Clusters in three regions on a world map, each running or scaling

Studio

Query, explore and design the schema from the browser.

MCP

Every model that speaks MCP reaches the database and the memory directly.

Three models connected through MCP to the database and Agent Memory

IN PRODUCTION

Trusted at scale. Samsung, Nvidia, Verizon, Tencent and Walmart run on SurrealDB.

14,000+

Developers building on SurrealDB Cloud

4M+

Developers building on SurrealDB worldwide

FROM THE TEAMS

SurrealDB gives us a foundation where we can unify semantic search, knowledge graphs, and AI-driven decision making without stitching together multiple systems. Collapsing responsibility into SurrealDB has become our default engineering posture.

Justin Foley

VP of Engineering, Later

GET STARTED

Be first to build with Agent Memory. Join the waitlist for early access: a free Sandbox plan and paid plans from $30 a month at launch.

Join the waitlist Back to the overview

Agent Memory, under the hood