Keywords and BM25

SurrealDB Agent Memory combines two statistical retrieval mechanisms - RAKE keyword extraction and BM25 full-text search - to complement dense vector retrieval. Together, they ensure that exact terms, product identifiers, technical phrases, and domain vocabulary are reliably findable even when their embedding representation is weak.

RAKE keyword extraction

During document ingestion, SurrealDB Agent Memory applies RAKE (Rapid Automatic Keyword Extraction) to each document. RAKE is a statistical, language-agnostic algorithm that identifies multi-word keyphrases by exploiting word co-occurrence and word frequency without requiring a language model or training data.

SurrealDB Agent Memory extracts the top-5 keyphrases per document with a RAKE score of 0.8 or above. These keyphrases are stored as keyword nodes in the knowledge graph and linked to their source documents via knowledge_has_keyword edges.

Because RAKE runs without an LLM, it adds negligible latency to the ingestion pipeline and incurs no token cost.

Why keyword extraction matters

Embedding models represent meaning holistically. A query for "MLWK3LL/A" (a product SKU) may produce a low-similarity result even against a document that contains that exact string, because the embedding model has not learned to weight arbitrary identifier strings meaningfully. BM25 and keyword graph lookups have no such blind spot - they operate on token identity, not semantic proximity.

Keywords extracted by RAKE also serve as seeds for hybrid_graph retrieval. When a query matches a keyword node, the retrieval pass can expand to all documents linked to that keyword before the final scoring step.

Keyword endpoints

Keywords for a document

Retrieve the keyphrases extracted from a specific document:

import os
from surrealdb.memory import AsyncMemory

memory = AsyncMemory(
    context="acme-prod",
    endpoint=os.environ["AGENT_MEMORY_ENDPOINT"],
    api_key=os.environ["AGENT_MEMORY_API_KEY"],
)

response = await memory.documents.keywords.for_document(doc.id)
for kw in response.keywords:
    print(kw.text, kw.score)
# RETURN POLICY       1.8
# UNOPENED ITEMS      1.5
# 30 DAYS             1.2
# PURCHASE DATE       1.1
# INTERNATIONAL ORDERS 0.9

import { AgentMemory } from "@surrealdb/memory";

const memory = new AgentMemory({
    context: "acme-prod",
    endpoint: process.env.AGENT_MEMORY_ENDPOINT,
    apiKey: process.env.AGENT_MEMORY_API_KEY,
});

const keywords = await memory.documents.keywords.forDocument(doc.id);
for (const kw of keywords) {
    console.log(kw.text, kw.score);
}

List keywords across the corpus

List all keywords in the Context, optionally filtered by minimum document count:

# Keywords that appear in at least 3 documents, sorted by frequency
page = await memory.documents.keywords.list(
    min_document_count=3,
    sort="-document_count",
)
for kw in page.keywords:
    print(kw.text, kw.document_count)

Get a keyword by its normalised form

detail = await memory.documents.keywords.get("RETURN POLICY")
print(detail.text)            # RETURN POLICY
print(detail.document_count)  # 12
print([(d["id"], d["title"], d["score"]) for d in detail.documents])

A keyword has no score of its own. The RAKE score belongs to the link between a keyword and a document, so it appears on each entry in documents and on the per-document keyword list.

Vector search over keywords

Find keywords semantically similar to a query string. This uses the keyword text embeddings rather than the chunk embeddings, making it useful for building keyword-expansion pipelines:

similar = await memory.documents.keywords.search("refund policies", k=10)
for kw in similar.results:
    print(kw.text, kw.score)
# RETURN POLICY     0.94
# REFUND WINDOW     0.91
# EXCHANGE POLICY   0.88

BM25 full-text search

SurrealDB Agent Memory indexes all chunk text in a full-text index using one analyzer, knowledge_text, which every BM25 index in the schema shares:

DEFINE ANALYZER knowledge_text TOKENIZERS blank, class, camel, punct
	FILTERS lowercase, snowball(english);

The four tokenizers together split text on:

• •

  blank: whitespace

• •

  class: changes of character class, such as a letter followed by a digit

• •

  camel: camelCase boundaries

• •

  punct: punctuation characters

The tokens then pass through two filters:

• •

  lowercase: case-normalisation for consistent matching

• •

  snowball(english): English stemming (e.g. returning → return, purchases → purchas)

An identifier such as MLWK3LL/A is therefore split into its letter, digit and punctuation runs, and a query containing the same identifier is split the same way, so the two match. Inflected natural-language terms match through stemming.

BM25 is invoked automatically when you use mode="bm25" or mode="hybrid" in the query endpoint. The BM25 score contributes to the RRF fusion in hybrid mode.

# Pure BM25 - best for exact-term matching
response = await memory.documents.query(
    "MLWK3LL/A MacBook Pro",
    mode="bm25",
    k=10,
)

Keyword graph in hybrid_graph retrieval

When mode="hybrid_graph" is used, keywords serve as intermediate nodes in the graph-density rerank. The retrieval pipeline:

1. 1.

   Runs hybrid (vector + BM25) to produce initial candidates.

2. 2.

   Embeds the query and finds the top-k matching keyword nodes.

3. 3.

   Expands from those keyword nodes through knowledge_has_keyword edges to include additional chunks that share relevant keywords.

4. 4.

   Boosts the score of candidates reachable through those keyword bridges and other graph-density signals.

This means a query containing "returns" will boost any chunk connected to the keywords RETURN POLICY, RETURN WINDOW, and REFUND - even if the chunk's embedding or term frequency score would not have ranked it highly on its own.

Scope gating: keyword listing, BM25 candidate reads, and graph-density expansion (PageRank hops over knowledge_has_keyword and typed relation edges) honour the same scope visibility model as unified recall - edges and keyword records outside the caller’s grant are not traversed.

When to use keyword versus vector search

In practice, hybrid or hybrid_graph are the right defaults for most production queries. Use bm25 directly only when you have strong reason to believe exact-match precision matters more than recall - for example, in lookup-style queries where the answer is a specific document known to contain an exact string.