> Full SurrealDB documentation index: https://surrealdb.com/docs/llms.txt

# Keywords and BM25

Keyword extraction and full-text search. How both work in SurrealDB Agent Memory's knowledge layer.

SurrealDB Agent Memory combines two statistical retrieval mechanisms - RAKE keyword extraction and BM25 full-text search - to complement dense vector retrieval. Together, they ensure that exact terms, product identifiers, technical phrases, and domain vocabulary are reliably findable even when their embedding representation is weak.

## RAKE keyword extraction

During document ingestion, SurrealDB Agent Memory applies RAKE (Rapid Automatic Keyword Extraction) to each document. RAKE is a statistical, language-agnostic algorithm that identifies multi-word keyphrases by exploiting word co-occurrence and word frequency without requiring a language model or training data.

SurrealDB Agent Memory extracts the **top-5 keyphrases per document** with a RAKE score of **0.8 or above**. These keyphrases are stored as keyword nodes in the knowledge graph and linked to their source documents via `knowledge_has_keyword` edges.

Because RAKE runs without an LLM, it adds negligible latency to the ingestion pipeline and incurs no token cost.

### Why keyword extraction matters

Embedding models represent meaning holistically. A query for "MLWK3LL/A" (a product SKU) may produce a low-similarity result even against a document that contains that exact string, because the embedding model has not learned to weight arbitrary identifier strings meaningfully. BM25 and keyword graph lookups have no such blind spot - they operate on token identity, not semantic proximity.

Keywords extracted by RAKE also serve as seeds for `hybrid_graph` retrieval. When a query matches a keyword node, the retrieval pass can expand to all documents linked to that keyword before the final scoring step.

## Keyword endpoints

### Keywords for a document

Retrieve the keyphrases extracted from a specific document:

```python
import os
from surrealdb.memory import AsyncMemory

memory = AsyncMemory(
    context="acme-prod",
    endpoint=os.environ["AGENT_MEMORY_ENDPOINT"],
    api_key=os.environ["AGENT_MEMORY_API_KEY"],
)

response = await memory.documents.keywords.for_document(doc.id)
for kw in response.keywords:
    print(kw.text, kw.score)
# RETURN POLICY       1.8
# UNOPENED ITEMS      1.5
# 30 DAYS             1.2
# PURCHASE DATE       1.1
# INTERNATIONAL ORDERS 0.9
```

```javascript
import { AgentMemory } from "@surrealdb/memory";

const memory = new AgentMemory({
    context: "acme-prod",
    endpoint: process.env.AGENT_MEMORY_ENDPOINT,
    apiKey: process.env.AGENT_MEMORY_API_KEY,
});

const keywords = await memory.documents.keywords.forDocument(doc.id);
for (const kw of keywords) {
    console.log(kw.text, kw.score);
}
```

### List keywords across the corpus

List all keywords in the Context, optionally filtered by minimum document count:

```python
# Keywords that appear in at least 3 documents, sorted by frequency
page = await memory.documents.keywords.list(
    min_document_count=3,
    sort="-document_count",
)
for kw in page.keywords:
    print(kw.text, kw.document_count)
```

### Get a keyword by its normalised form

```python
detail = await memory.documents.keywords.get("RETURN POLICY")
print(detail.text)            # RETURN POLICY
print(detail.document_count)  # 12
print([(d["id"], d["title"], d["score"]) for d in detail.documents])
```

A keyword has no score of its own. The RAKE score belongs to the link between a keyword and a document, so it appears on each entry in `documents` and on the per-document keyword list.

### Vector search over keywords

Find keywords semantically similar to a query string. This uses the keyword text embeddings rather than the chunk embeddings, making it useful for building keyword-expansion pipelines:

```python
similar = await memory.documents.keywords.search("refund policies", k=10)
for kw in similar.results:
    print(kw.text, kw.score)
# RETURN POLICY     0.94
# REFUND WINDOW     0.91
# EXCHANGE POLICY   0.88
```

## BM25 full-text search

SurrealDB Agent Memory indexes all chunk text in a full-text index using one analyzer, `knowledge_text`, which every BM25 index in the schema shares:

```surql
DEFINE ANALYZER knowledge_text TOKENIZERS blank, class, camel, punct
	FILTERS lowercase, snowball(english);
```

The four tokenizers together split text on:

- **`blank`**: whitespace
- **`class`**: changes of character class, such as a letter followed by a digit
- **`camel`**: camelCase boundaries
- **`punct`**: punctuation characters

The tokens then pass through two filters:

- **`lowercase`**: case-normalisation for consistent matching
- **`snowball(english)`**: English stemming (e.g. `returning` → `return`, `purchases` → `purchas`)

An identifier such as `MLWK3LL/A` is therefore split into its letter, digit and punctuation runs, and a query containing the same identifier is split the same way, so the two match. Inflected natural-language terms match through stemming.

BM25 is invoked automatically when you use `mode="bm25"` or `mode="hybrid"` in the query endpoint. The BM25 score contributes to the RRF fusion in hybrid mode.

```python
# Pure BM25 - best for exact-term matching
response = await memory.documents.query(
    "MLWK3LL/A MacBook Pro",
    mode="bm25",
    k=10,
)
```

## Keyword graph in hybrid_graph retrieval

When `mode="hybrid_graph"` is used, keywords serve as intermediate nodes in the graph-density rerank. The retrieval pipeline:

1. Runs hybrid (vector + BM25) to produce initial candidates.
2. Embeds the query and finds the top-k matching keyword nodes.
3. Expands from those keyword nodes through `knowledge_has_keyword` edges to include additional chunks that share relevant keywords.
4. Boosts the score of candidates reachable through those keyword bridges and other graph-density signals.

This means a query containing "returns" will boost any chunk connected to the keywords `RETURN POLICY`, `RETURN WINDOW`, and `REFUND` - even if the chunk's embedding or term frequency score would not have ranked it highly on its own.

**Scope gating:** keyword listing, BM25 candidate reads, and graph-density expansion (PageRank hops over `knowledge_has_keyword` and typed relation edges) honour the same scope visibility model as unified recall - edges and keyword records outside the caller’s grant are not traversed.

## When to use keyword versus vector search

| Query type | Recommended mode |
|---|---|
| Product SKU, model number, API key fragment | `bm25` |
| Exact phrase from a document | `bm25` |
| Natural-language question (paraphrase likely) | `vector` or `hybrid` |
| Mixed: named entities + prose context | `hybrid` |
| Graph-connected topic exploration | `hybrid_graph` |
| Keyword discovery and corpus analysis | Keyword endpoints |

In practice, `hybrid` or `hybrid_graph` are the right defaults for most production queries. Use `bm25` directly only when you have strong reason to believe exact-match precision matters more than recall - for example, in lookup-style queries where the answer is a specific document known to contain an exact string.
