Migrate from a vector store | Migrate

Vector stores are a common first step for adding memory to AI agents. They work well for simple document retrieval but fall short when you need structured extraction, provenance tracking, temporal validity, or authority-based conflict resolution. This guide covers what you gain by migrating to SurrealDB Agent Memory and how to execute the migration.

What you gain

Capability	Vector store	SurrealDB Agent Memory
Document retrieval	Semantic similarity search	Hybrid (semantic + BM25 + graph traversal)
Structured extraction	None - raw chunks	Entities, attributes, and relations
Provenance	None - chunk origin only	Every fact traces back to the turn that produced it
Correction tracking	None	Supersession chains with full history
Temporal validity	None	`valid_from` / `valid_until` on every attribute
Authoritative precedence	None	Authoritative pillar wins over Experiential assertions when they conflict
Scope-based isolation	Manual metadata filtering	Dimensional scope with floor matching
State inspection	None	Query memory state directly

The decision to migrate usually comes when one of the following becomes a pain point: contradictions silently accumulating, inability to correct or expire facts, no insight into what the agent actually knows, or metadata filtering becoming unmanageable.

Migration steps

1. Export your existing documents

Export the source documents from your vector store. For most systems, this means exporting the original text rather than the embeddings - SurrealDB Agent Memory re-indexes everything through its own pipeline.

# Example: export from Pinecone
import pinecone

index = pinecone.Index("my-index")
# Fetch all vectors with metadata (Pinecone requires pagination)
results = index.query(
    vector=[0.0] * 1536,
    top_k=10000,
    include_metadata=True,
)
documents = [
    {"text": r.metadata["text"], "source": r.metadata.get("source", "")}
    for r in results.matches
]

// Example: export from a generic vector store
const documents = await vectorStore.fetchAll({ includeMetadata: true });
const exportedDocs = documents.map(doc => ({
    text: doc.metadata.text,
    source: doc.metadata.source ?? "",
}));

2. Upload documents to authoritative knowledge

Ingest your exported documents into SurrealDB Agent Memory's authoritative knowledge layer. Each document is processed through the ingestion pipeline: chunked, keyword-extracted, and linked to knowledge nodes.

import os
import httpx

client = httpx.Client(
    base_url="https://spectron.surrealdb.com/api/v1/my-context",
    headers={"Authorization": f"Bearer {os.environ['SPECTRON_API_KEY']}"},
)

for doc in documents:
    client.post(
        "/documents",
        files={"file": (
            "document.txt",
            doc["text"].encode(),
            "text/plain",
        )},
        data={
            "title": doc.get("source", "Imported document"),
            "source_url": doc.get("source", ""),
        },
    )

for (const doc of exportedDocs) {
    const formData = new FormData();
    formData.append("file", new Blob([doc.text], { type: "text/plain" }), "document.txt");
    formData.append("title", doc.source || "Imported document");
    formData.append("source_url", doc.source || "");

    await fetch("https://spectron.surrealdb.com/api/v1/my-context/documents", {
        method: "POST",
        headers: { "Authorization": "Bearer mgmt_..." },
        body: formData,
    });
}

SurrealDB Agent Memory's ingestion pipeline handles chunking internally. You do not need to replicate the chunking strategy from your vector store.

3. Handle the metadata gap

Vector stores often have minimal, inconsistent, or missing metadata. SurrealDB Agent Memory's pipeline extracts structure from content, so missing metadata is less critical - but it is worth enriching documents before ingestion if you have source information available.

If your vector store metadata includes document type, date, or author information, include it in the document upload:

client.post(
    "/documents",
    files={"file": ("policy.md", content, "text/markdown")},
    data={
        "title": "Return policy",
        "content_type": "policy",
        "authored_at": "2024-01-15T00:00:00Z",
    },
)

4. Migrate conversational memory

If your vector store also held per-user conversational memory (previous chat turns or extracted facts stored as vectors), re-ingest them as SurrealDB Agent Memory turns:

# For each user, create a session and ingest their history
for user_id, history in user_histories.items():
    session = await memory.sessions.create(
        scopes=[f"org/my-org/user/{user_id}"],
    )
    for message in history:
        await memory.remember(message["content"], session_id=session.id, role=message["role"])

for (const [userId, history] of Object.entries(userHistories)) {
    const session = await memory.sessions.create({
        scopes: [`org/my-org/user/${userId}`],
    });
    for (const message of history) {
        await memory.remember(message.content, { sessionId: session.id, role: message.role });
    }
}

The extraction pipeline runs on each turn and re-derives structured entities and attributes from the conversation history. You do not need to manually map old vector metadata to SurrealDB Agent Memory's schema.

Coexistence strategy

You do not need to cut over immediately. Run SurrealDB Agent Memory and your vector store in parallel for a period:

Write to both - record turns in SurrealDB Agent Memory and continue writing to the vector store.
Read from SurrealDB Agent Memory first - use its context retrieval as primary; fall back to the vector store if it returns nothing.
Validate - compare the quality of responses with SurrealDB Agent Memory context versus vector store context over a sample of real queries.
Cut over - once satisfied, remove the vector store read path.

async def retrieve_context(user_id: str, query: str) -> str:
    # Try Spectron first
    ctx = await session.context(query=query)
    if ctx.items:
        return ctx.formatted

    # Fall back to vector store during transition
    chunks = vector_store.search(query=query, user_id=user_id, top_k=6)
    return "\n\n".join(c["text"] for c in chunks)

Timeline

Week	Activity
1	Export documents; begin authoritative knowledge ingestion
2	Begin recording new conversations as SurrealDB Agent Memory turns
3-4	Coexistence: SurrealDB Agent Memory primary, vector store fallback
5	Validate response quality; remove fallback
6+	Re-ingest historical conversational memory if needed