Announcing Snowflake integration for SurrealDB: from data warehouse to context layer

Start with the question an agent cannot answer

A procurement agent gets asked: if this component slips two weeks, which suppliers are exposed, and which of them have we already escalated with?

Answering it takes four different things at once.

The facts: suppliers, parts, purchase orders, lead times. Those are modelled, governed and clean, and they live in Snowflake.

The relationships: component to assembly to programme to customer, several hops deep, resolved in milliseconds because the agent will traverse them dozens of times while it reasons.

The unstructured history: escalation emails, support notes, meeting summaries, retrieved by meaning rather than by key.

And the agent's own memory: what it concluded last Tuesday, which supplier a human overrode it on, which paths it has already ruled out.

A data warehouse serves the first one beautifully. It was never built for the other three. So teams assemble the rest: Postgres for state, a vector database for embeddings, a graph database for relationships, a memory service bolted on top, and a pile of sync jobs holding the four in rough agreement. Every one of those hops is latency the agent pays on every loop, and a place where the four stores disagree.

Truth at rest, and truth in use

What is emerging is a split in responsibilities.

The warehouse or lakehouse is the system of record. Governed, audited, historically complete, optimised for scanning enormous volumes when someone asks a question about the business.

The context layer is different work. It holds the slice of that truth an application or agent reasons over right now, plus everything the agent accumulates while reasoning. Its demands are almost the inverse of a warehouse: small reads at single-digit milliseconds, constant small writes, relationship traversal, semantic search, and recency.

Agent memory is the clearest example. Memory is a write path. An agent that cannot record what it learned is starting from zero on every invocation, and a virtual warehouse is the wrong place to absorb thousands of tiny writes per session. Worse, when memory lives in a separate service from the facts, the agent cannot ask a single question that spans both. "What did I conclude about the suppliers for this part?" turns into three round trips and a join in application code.

SurrealDB is built for that layer: documents, graph edges, relational tables, time series and vectors in one engine, one query language, reads and writes from the same place. Facts and memory sit next to each other because they are the same store.

Which left one gap. Getting the facts out of the warehouse was still your problem.

Introducing Snowflake integration in

Surreal Sync reads directly from Snowflake. Point it at an account, a warehouse and a schema:

surreal-sync from snowflake \
  --account myorg-myaccount \
  --user my_user \
  --private-key-path ./rsa_key.p8 \
  --warehouse COMPUTE_WH \
  --database MY_DB \
  --schema PUBLIC \
  --to-namespace my_namespace \
  --to-database my_database

Every base table becomes a SurrealDB table, every row a record. Types map automatically: VARIANT and OBJECT become objects, ARRAY becomes an array, timestamps keep their absolute instant, NUMBER(p,s) keeps its precision. Use --tables for a subset, --dry-run to inspect conversions before anything is written. Rows stream a result partition at a time, so table size is not bounded by memory.

No export bucket, no Airflow DAG, no glue code. Snowflake joins PostgreSQL, MySQL, MongoDB and Neo4j as a first-class Surreal Sync source.

What the data becomes

Warehouse foreign keys become real relationships:

FOR $po IN (SELECT id, SUPPLIER_ID FROM PURCHASE_ORDERS) {
    RELATE $po.id->supplied_by->type::thing('SUPPLIERS', $po.SUPPLIER_ID);
};

The multi-hop question becomes one line instead of a recursive CTE:

SELECT NAME, ->supplied_by->PARTS->used_in->PROGRAMMES AS exposure
FROM SUPPLIERS;

And the agent's semantic history sits on the same records, reachable in the same query:

DEFINE INDEX escalation_idx ON escalations
    FIELDS embedding HNSW DIMENSION 768;

SELECT text, ->about->SUPPLIERS.NAME AS supplier
FROM escalations
WHERE embedding <|5|> $query_vector;

Back to the original question. Governed facts from Snowflake, traversal for exposure, semantic recall over escalations, and the agent's own memory written straight back alongside all of it. One store, one query, one round trip.

Scope

The Snowflake source is ingestion-only: a single full snapshot of the tables you select, with no change data capture or incremental mode yet. For a migration, or for refreshing the working set behind an agent on a schedule, that is the right shape. Continuous replication is on the roadmap.

FAQ

Does this replace Snowflake?
No. Snowflake stays the system of record and the analytics engine. This moves the slice your applications and agents reason over into a layer built for that job.

Is it a live sync?
Not yet. Each run takes a full snapshot of the selected tables.

Can I import just one table?
Yes, pass a comma-separated list to --tables.

How does it authenticate?
Key-pair (JWT) auth over the Snowflake SQL REST API, using an unencrypted PKCS#8 key registered on the Snowflake user.

Can I transform rows on the way in?
Yes, with a TOML transform pipeline via --transforms-config.

Try it

The Snowflake migration guide has the full type mapping, record ID options and every flag. Surreal Sync is on GitHub.

Start with --dry-run, see what lands, then run it for real. Tell us what you build in Discord.