Migrate from a vector store | Migrate

Vector stores are a common first step for adding memory to AI agents. They work well for simple document retrieval but fall short when you need structured extraction, provenance tracking, temporal validity, or conflict detection. This guide covers what you gain by migrating to SurrealDB Agent Memory and how to execute the migration.

What you gain

Capability	Vector store	SurrealDB Agent Memory
Document retrieval	Semantic similarity search	Hybrid (semantic + BM25 + graph traversal)
Structured extraction	None - raw chunks	Entities, attributes, and relations
Provenance	None - chunk origin only	Every fact traces back to the turn or document that produced it
Correction tracking	None	Supersession chains with full history
Temporal validity	None	`valid_from` / `valid_until` on every attribute
Conflict handling	None	A contradiction between a document and a conversation keeps both values and raises an `uncertainty`
Scope-based isolation	Manual metadata filtering	Hierarchical scope paths, enforced by grants on every read
State inspection	None	Query memory state directly

Some typical reasons to migrate are: contradictions accumulating with no way to tell which fact is current, no way to correct or expire a fact, no queryable view of what the agent knows, or metadata filtering that no longer scales.

Migration steps

1. Export your existing documents

Export the source documents from your vector store. For most systems, this means exporting the original text rather than the embeddings - SurrealDB Agent Memory re-indexes everything through its own pipeline.

# Example: export from Pinecone
import pinecone

index = pinecone.Index("my-index")
# Fetch all vectors with metadata (Pinecone requires pagination)
results = index.query(
    vector=[0.0] * 1536,
    top_k=10000,
    include_metadata=True,
)
documents = [
    {"text": r.metadata["text"], "source": r.metadata.get("source", "")}
    for r in results.matches
]

// Example: export from a generic vector store
const documents = await vectorStore.fetchAll({ includeMetadata: true });
const exportedDocs = documents.map(doc => ({
    text: doc.metadata.text,
    source: doc.metadata.source ?? "",
}));

2. Upload documents

Upload your exported documents to SurrealDB Agent Memory. Each document is processed through the ingestion pipeline: chunked, keyword-extracted, and passed to extraction, which writes entities and attributes to the Context's graph. The title, source and other metadata go in a metadata part as one JSON object; separate form fields are ignored.

import json
import os
import httpx

client = httpx.Client(
    base_url=f"{os.environ['AGENT_MEMORY_ENDPOINT']}/api/v1/my-context",
    headers={"Authorization": f"Bearer {os.environ['AGENT_MEMORY_API_KEY']}"},
)

for doc in documents:
    client.post(
        "/documents",
        data={"metadata": json.dumps({
            "title": doc.get("source") or "Imported document",
            "source": doc.get("source", ""),
        })},
        files={"file": (
            "document.txt",
            doc["text"].encode(),
            "text/plain",
        )},
    )

for (const doc of exportedDocs) {
    const formData = new FormData();
    formData.append("metadata", JSON.stringify({
        title: doc.source || "Imported document",
        source: doc.source || "",
    }));
    formData.append("file", new Blob([doc.text], { type: "text/plain" }), "document.txt");

    await fetch(`${process.env.AGENT_MEMORY_ENDPOINT}/api/v1/my-context/documents`, {
        method: "POST",
        headers: { "Authorization": `Bearer ${process.env.AGENT_MEMORY_API_KEY}` },
        body: formData,
    });
}

SurrealDB Agent Memory's ingestion pipeline handles chunking internally. You do not need to replicate the chunking strategy from your vector store.

3. Handle the metadata gap

Vector stores often have minimal, inconsistent, or missing metadata. SurrealDB Agent Memory's pipeline extracts structure from content, so missing metadata matters less. If you have source information available, add it to the documents before ingestion.

If your vector store metadata includes a title, source, date, or other descriptive values, include them in the metadata part. A date goes in observedAt, which dates the facts extracted from the document, and other values can go in labels as key=value strings:

client.post(
    "/documents",
    data={"metadata": json.dumps({
        "title": "Return policy",
        "observedAt": "2024-01-15T00:00:00Z",
        "labels": ["doc_type=policy"],
    })},
    files={"file": ("policy.md", content, "text/markdown")},
)

4. Migrate conversational memory

If your vector store also held per-user conversational memory (previous chat turns or extracted facts stored as vectors), re-ingest them as SurrealDB Agent Memory turns:

# For each user, create a session and ingest their history
for user_id, history in user_histories.items():
    session = await memory.sessions.create(
        scopes=[f"org/my-org/user/{user_id}"],
    )
    for message in history:
        await memory.remember(message["content"], session_id=session.id, role=message["role"])

for (const [userId, history] of Object.entries(userHistories)) {
    const session = await memory.sessions.create({
        scopes: [`org/my-org/user/${userId}`],
    });
    for (const message of history) {
        await memory.remember(message.content, { sessionId: session.id, role: message.role });
    }
}

The extraction pipeline runs on each turn and re-derives structured entities and attributes from the conversation history. You do not need to manually map old vector metadata to SurrealDB Agent Memory's schema.

Coexistence strategy

You do not need to cut over immediately. Run SurrealDB Agent Memory and your vector store in parallel for a period:

Write to both - record turns in SurrealDB Agent Memory and continue writing to the vector store.
Read from SurrealDB Agent Memory first - use its context retrieval as primary; fall back to the vector store if it returns nothing.
Validate - compare the quality of responses with SurrealDB Agent Memory context versus vector store context over a sample of real queries.
Cut over - once satisfied, remove the vector store read path.

async def retrieve_context(user_id: str, query: str) -> str:
    # Try Agent Memory first; the context is one string ready for a prompt
    ctx = await memory.query_context(query, lens=[f"org/my-org/user/{user_id}"])
    if ctx.context:
        return ctx.context

    # Fall back to vector store during transition
    chunks = vector_store.search(query=query, user_id=user_id, top_k=6)
    return "\n\n".join(c["text"] for c in chunks)

Timeline

Week	Activity
1	Export documents; begin document upload
2	Begin recording new conversations as SurrealDB Agent Memory turns
3-4	Coexistence: SurrealDB Agent Memory primary, vector store fallback
5	Validate response quality; remove fallback
6+	Re-ingest historical conversational memory if needed