--- title: "Agent Memory, under the hood | SurrealDB" description: "The technical treatment of the Agent Memory engine: memory categories, the document pipeline, hybrid retrieval, tiered queries and audit." url: https://surrealdb.com/agent-memory/deep-dive --- # Agent Memory, under the hood The full technical treatment of the agent memory engine: how conversations and documents compound into one governed company brain, how recall stays fast and cheap at scale, and how every answer can be defended when it reaches a stakeholder. [Join waitlist](https://surrealdb.com/agent-memory/get-started) [Book a demo](https://surrealdb.com/contact/sales) Works with · [+29 more](https://surrealdb.com/agent-memory#integrations) ![Claude Desktop and Claude Code](https://surrealdb.com/assets/static/claude.qeceRvhT.svg)![Cursor](https://surrealdb.com/assets/static/cursor.CffcCNKv.svg)![OpenAI Codex](https://surrealdb.com/assets/static/openai.CNSrSktx.svg)![OpenClaw](https://surrealdb.com/assets/static/openclaw.CzSOJJfQ.svg)![Hermes](https://surrealdb.com/assets/static/hermes.u8A4-1Lq.svg)![VS Code](https://surrealdb.com/assets/static/vscode.ofeWxTsy.svg)![Windsurf](https://surrealdb.com/assets/static/windsurf.Ww98i45f.svg)![Zed](https://surrealdb.com/assets/static/zed.M8dgM-za.svg)![LangChain](https://surrealdb.com/assets/static/langchain.B2qiXaj0.svg)![Vercel AI SDK](https://surrealdb.com/assets/static/vercel.4jR_VMI9.svg)![Cloudflare](https://surrealdb.com/assets/static/cloudflare.CGYztXwG.svg)![n8n](https://surrealdb.com/assets/static/n8n.CjHLK_Mt.svg)![Zapier](https://surrealdb.com/assets/static/zapier.V_eYzQMe.svg) SOC 2 Type 2 GDPR Cyber Essentials Plus ISO 27001 1. ![Babcock](https://surrealdb.com/assets/static/babcock.lo4rnVg1.svg) 2. ![ING](https://surrealdb.com/assets/static/ing.X3I6S3_V.svg) 3. ![British Airways](https://surrealdb.com/assets/static/british-airways.KEsZiwV-.svg) 4. ![Nvidia](https://surrealdb.com/assets/static/nvidia.DaIEuMil.svg) 5. ![Apple](https://surrealdb.com/assets/static/apple.D5pq4flY.svg) 6. ![SpaceX](https://surrealdb.com/assets/static/spacex.CQJEk-IL.svg) 7. ![Samsung](https://surrealdb.com/assets/static/samsung.CH-vQgnb.svg) 8. ![adidas](https://surrealdb.com/assets/static/adidas.DdTC5qhk.svg) 9. ![Tencent](https://surrealdb.com/assets/static/tencent.paQmLxyy.svg) 10. ![Alibaba](https://surrealdb.com/assets/static/alibaba.B16idgfM.svg) 11. ![PolyAI](https://surrealdb.com/assets/static/poly-ai.c3w_fAg6.svg) 12. ![Later](https://surrealdb.com/assets/static/later.Ds736jFO.svg) 13. ![Verizon](https://surrealdb.com/assets/static/verizon.BI7CajdX.svg) 14. ![Liberty Mutual](https://surrealdb.com/assets/static/liberty-mutual.B7qOU1pd.svg) 15. ![Walmart](https://surrealdb.com/assets/static/walmart.BjDg_Sr8.svg) 16. ![Carrier](https://surrealdb.com/assets/static/carrier.D21gC6NX.svg) 17. ![Saks Fifth Avenue](https://surrealdb.com/assets/static/saks-fifth-avenue.COIDpLSb.svg) 18. ![San Francisco Compute Company](https://surrealdb.com/assets/static/sfcc.B7jlImq4.svg) 19. ![Shield AI](https://surrealdb.com/assets/static/shield-ai.pINZ0KJr.svg) 20. ![Wix](https://surrealdb.com/assets/static/wix.DvHhmoBi.svg) THE ANATOMY ## Anatomy of an agentic system Every agentic system is made of the same six parts. The one that decides what the agent actually knows on any given turn is the context layer - its whole job is answering one question: what does the model need to know right now? 1 The agent LLM · tools · loop 2 The context window Working memory - where the traditional agent forgets Capture Recall · inject Agent Memory 3 The context layer “What does the model need to know right now?” gathers · filters · packages 4 Memory Everything the agent learned episodic · identity · knowledge · context · instructions · uncertainty 5 The knowledge graph How everything is connected entities · typed edges · properties 6 Storage - one multi-model database document graph vector full-text The bottom four parts - the context layer, memory, the knowledge graph, and storage - are exactly what Agent Memory provides. 01 |MEMORY MODEL ## Six typed memory categories, not one bucket Experiential memory is not a single store. It is six typed sub-stores - one raw record and five extracted categories built on top - each with its own schema, lifecycle, retrieval weight, and prompt-injection path. ### Episodic The raw conversational record. Sessions and turns as authored, in order, with anaphora intact. The source of truth every extracted category cites back to via byte-level spans. ### Identity Durable facts about the principal: name, role, employer, preferences, long-lived attributes. Long retention, low decay, surfaced into the prompt-ready profile. ### Knowledge Things the principal has learnt or shared, distinct from authoritative documents. Project facts, observations, references. Medium retention, decays without reinforcement. ### Context What is going on right now: active topics, recent intents, the working set for the current conversation. Short retention, replaced rapidly, anchors the next turn. ### Instructions Behavioural memory, not factual. 'Always answer in British English', 'never call me by my first name'. Applied at prompt-assembly time, not at retrieval. ### Uncertainty Explicit 'we don't know yet' rows. Emitted by the reconciler when confidence is below the floor, by cross-provenance contradictions, by ungrounded references. Gaps are visible, not papered over with hallucinations. 02 |DOCUMENT PIPELINE ## Documents become structure - not a blob in a vector store This is where the company brain gets built. Plain text, markdown, code, JSON, PDFs, images, audio and video flow through a proper knowledge-representation pipeline, landing in the same graph as everything your agents learn in conversation. Both the content index and the structural index that retrieval reads from are built at write time - so the cost ladder downstream stays tractable. ### Multi-modal ingest Configurable per Context via an IngestionProfile (TextOnly → MultimodalFull) that trades completeness against cost. Pay for what you actually need; lift it later without re-architecting. ### Object store for originals PDFs and media originals live in S3-compatible storage. The database holds the structured indexable state. Backups, GDPR deletion, and storage cost scale on object-store economics, not OLTP-row economics. ### Content-addressed Content addressing works exactly as in the ingest stage above - the Blake3 hash is the identity - and an extracted-text cache lets you rechunk and reembed without re-running expensive parsers. ### Content-aware chunking Passages are first-class chunk rows with their own embeddings, byte spans into the original artefact, and edges to the entities extracted from them. Time-coded segments for audio and video answer 'where in this recording' instead of 'this file is somehow relevant'. ### Same reconciler as turns Document-extracted facts flow through the same supersession-and-uncertainty function as turn-extracted facts. A document can contradict a turn and produce explicit uncertainty - identical guarantees, regardless of source. ### Keyword graph + understanding A non-LLM RAKE pass produces first-class keyword nodes with PMI-scored edges to chunks and entities - cheap structural recall for rare-term queries that vector search underweights. Document-level summaries, summary embeddings, and document-to-document links sit in the same graph. 03 |HYBRID RETRIEVAL ## Eight signals fused into one auditable ranking Embeddings-only retrieval has known failure modes - near-duplicates dominate top-k, rare-term queries miss the right chunk, structure between facts is invisible. Agent Memory's structural index is built at write time and read cheaply on every retrieval. Per-feature scores ride on every trace, so any result is auditable as a weighted combination of signals - not a black-box top-k. The result is memory that stays coherent along five axes at once - semantic, lexical, relational, temporal, and spatial. Vector-only memory hits walls on four of the five; row-store memory hits walls on all five. The three clocks behind the temporal axis are covered in full under Audit and time below. ### Vector recall Dense embeddings on entities, attributes and chunks - plus modality-native vectors for images and audio when enabled. ### Lexical (BM25) BM25 over chunk text and entity surface forms catches the exact-phrase and rare-term queries embeddings underweight. ### Graph traversal One or two hops from a seed entity surface related facts even when surface forms diverge. The graph is the substrate, not a sidecar. ### Keyword bridges RAKE-derived keyword nodes connect chunks and entities that share rare or discriminative terms but sit far apart in embedding space. PMI-scored so common terms don't dominate. ### Document links Section embeddings, chunk-to-chunk and document-to-document edges surface related sections of related documents - not only the single closest chunk. ### Personalised PageRank Graph-walk scoring biased toward the query's seed nodes - a structural feature in the fused ranker, not a separate pipeline. ### Geographic recall Within-radius, inside-polygon and nearest-k spatial predicates compose with semantic and graph signals in the same ranker. 'Discussions about Acme within 50km of Berlin in Q3' is one query, not three pipelines. ### Trace-derived features Retrieval reads its own history. Rows useful for similar queries get boosted; rows associated with corrections get demoted. The ranker learns from the trace graph in place. chunk embedding vector text extracted_from attribute text confidence valid_from has_source source kind trust belongs_to entity type name embedding vector relation kind (verb) · valid_from · valid_until entity type name embedding vector 04 |TIERED QUERIES ## Cheap questions are cheap. Expensive questions are explicit. Most memory layers run a vector search on every request. Agent memory does not. Reads route through a four-tier cost and latency ladder gated by query understanding at the front. Every tier emits a trace recording which path was taken, why, and what it returned - so the cost story is observable per Context, not buried in a flat per-request average. ### Tier 1 · Direct lookup Typed questions - 'what is my role at Acme?', 'when did I join?' - fetched directly from the entity and attribute graph by key. No embeddings, no LLM, no ranking. Sub-millisecond. ### Tier 2 · Response reuse Semantic match against prior responses with entity-aware invalidation. The cache key is the query plus the set of facts the prior answer cited - when any cited fact is superseded, every dependent response is invalidated. Tens of milliseconds. No LLM call. ### Tier 3 · Hybrid retrieval Vector, BM25, graph traversal, keyword bridges, section embeddings, Personalised PageRank, geographic recall and trace-derived features fused into a single ranking, then synthesised by the LLM. Hundreds of milliseconds. Auditable per-feature scores ride on every retrieval trace. ### Tier 4 · Full-context fallback When hybrid retrieval is thin or low-confidence, fall back to a broader sweep: more candidates, deeper graph traversal, optionally a HyDE-style query rewrite, longer context window. Higher token cost and latency - explicit as the tier-4 escalation, not buried in a flat per-request average. Search Natural language query Generate embeddingsProvider-pluggable Vector searchchunks HNSW · COSINE Entity searchentities HNSW · COSINE Fact chain traversalentity → relation → entity 3-depth recursive DFS Success Memories + facts 05 |AUDIT & TIME ## Every fact knows where it came from - and when it was true The risk teams weigh most with agentic AI is a confident wrong answer reaching a stakeholder. Agent Memory's answer is structural: provenance is a stored field on every fact-bearing row, three independent clocks answer three different questions, and traces are first-class nodes that feed back into ranking. The audit story is queryable substrate state - not a logging pipeline bolted on. ### Provenance as data Every entity, attribute, relation, instruction, and uncertainty row carries a source object: kind (turn, document, reflection, elaboration, consolidation), ref, trust, lexical span, location, and derived_from. Reconciliation compares provenance; calibration weights it; supersession audits it - down to the bytes. ### Three independent clocks System time (the MVCC byte-level audit clock - 'what did the substrate look like at instant T?'), known time (when Agent Memory first believed a fact), and valid time (when the assertion held in the world). Different questions, all queryable independently via VERSION and as_of. ### Traces are memory retrieval_trace, decision_trace, and response_trace are first-class graph nodes, not external observability data. The unified ranker reads its own history; supersession lineage downgrades source trust; any answer walks back to the bytes that produced it. ### Supersede, don't delete Memories are not overwritten - they are superseded with valid_until set, or aged out with a reason recorded. The history of how a fact changed is queryable. forget is an explicit first-class verb, distinct from natural ageing, with a --purge option for hard removal. 06 |AUTONOMOUS UNDERSTANDING ## Memory that improves between conversations Three named mechanisms generate new memory at different times, all flowing through the same reconciler with explicit provenance. Agent memory does not just store what agents tell it - the substrate deepens its own understanding between interactions, so the tribal knowledge tenured staff feed it compounds into shared memory every teammate's agent can recall. Reflection On-demand synthesis. POST /reflect runs an LLM pass over retrieved context and optionally persists the synthesised answer as new facts - with their own provenance kind and a lower default trust, so calibration stays honest. Elaboration Background sweep. A job walks the substrate looking for entities and attributes that share context but no explicit relation. An LLM proposes the link; the reconciler accepts, supersedes, or surfaces it as uncertainty. Consolidation Belief crystallisation. An async job pools recent facts and decides to create, update (delta recorded), or mark superseded. Each observation tracks its derived inputs and proof count, so the evolution of belief is replayable. GET STARTED ## Be first to build with agent memory Join the waitlist for early access - a free Sandbox plan and self-serve plans from $30 a month at launch. ![Samsung](https://surrealdb.com/assets/static/4c58b81e7b3c9466.C_Hv0eml.svg)![NVIDIA](https://surrealdb.com/assets/static/nvidia.DaIEuMil.svg)![Apple](https://surrealdb.com/assets/static/f7dc2519e0d212bc.Cn8MYAK7.svg)![Verizon](https://surrealdb.com/assets/static/18b99996c689000f.B5PQ-nI9.svg)![Tencent](https://surrealdb.com/assets/static/401d8346058682c8.DqM87mst.svg) SOC 2 Type 2 GDPR Cyber Essentials Plus ISO 27001 [Join waitlist](https://surrealdb.com/agent-memory/get-started) [Back to the overview](https://surrealdb.com/agent-memory) ```json {"@context":"https://schema.org","@type":"Organization","name":"SurrealDB","url":"https://surrealdb.com","logo":"https://surrealdb.com/assets/static/logo.BG7_TG2b.svg","description":"SurrealDB is the unified data layer for AI. A multi-model database for documents, graphs, vectors, and time-series.","foundingDate":"2022","legalName":"SurrealDB Ltd","identifier":{"@type":"PropertyValue","propertyID":"GB-COH","value":"13615201"},"address":{"@type":"PostalAddress","streetAddress":"3rd Floor, 1 Ashley Road","addressLocality":"Altrincham","addressRegion":"Cheshire","postalCode":"WA14 2DT","addressCountry":"GB"},"contactPoint":[{"@type":"ContactPoint","contactType":"customer support","email":"support@surrealdb.com","url":"https://surrealdb.com/contact","availableLanguage":"English"},{"@type":"ContactPoint","contactType":"sales","email":"info@surrealdb.com","url":"https://surrealdb.com/contact","availableLanguage":"English"},{"@type":"ContactPoint","contactType":"security","email":"security@surrealdb.com","url":"https://surrealdb.com/.well-known/security.txt","availableLanguage":"English"},{"@type":"ContactPoint","contactType":"legal","email":"legal@surrealdb.com","url":"https://surrealdb.com/legal","availableLanguage":"English"}],"hasCertification":[{"@type":"Certification","name":"SOC 2 Type 2"},{"@type":"Certification","name":"GDPR"},{"@type":"Certification","name":"Cyber Essentials Plus"},{"@type":"Certification","name":"ISO 27001"}],"owns":[{"@type":"SoftwareApplication","name":"SurrealDB","url":"https://surrealdb.com/surrealdb"},{"@type":"SoftwareApplication","name":"Agent Memory","url":"https://surrealdb.com/agent-memory"}],"knowsAbout":["multi-model databases","document databases","graph databases","vector search","time-series databases","SurrealQL","Agent Memory","real-time databases","embedded databases","context layer","graph ontology","distributed database","knowledge graphs","distributed transaction protocols","highly-scalable databases"],"sameAs":["https://www.wikidata.org/wiki/Q124316308","https://github.com/surrealdb/surrealdb","https://twitter.com/surrealdb","https://www.youtube.com/@surrealdb","https://www.linkedin.com/company/surrealdb","https://discord.gg/surrealdb","https://www.reddit.com/r/surrealdb","https://www.instagram.com/surrealdb","https://medium.com/surrealdb","https://dev.to/surrealdb"]} ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://surrealdb.com"},{"@type":"ListItem","position":2,"name":"Agent Memory","item":"https://surrealdb.com/agent-memory"},{"@type":"ListItem","position":3,"name":"Deep dive","item":"https://surrealdb.com/agent-memory/deep-dive"}]} ```