Storage no one else has

Q: What is quorum consensus?

Quorum consensus coordinates distributed transactions without a single leader. Each availability zone has its own write node. Transactions commit once a quorum of zones acknowledges. This eliminates the single-leader bottleneck and provides lower latency than traditional Raft or Paxos-based systems.

Q: What object storage providers are supported?

The storage engine supports Amazon S3, Google Cloud Storage, and Azure Blob Storage. Data is stored in commodity object storage, making it portable across cloud providers.

Q: How does scale-to-zero work?

Because the storage engine separates compute from storage, the database engine is stateless. When no queries are running, compute instances can shut down entirely - data persists in object storage. When demand returns, new compute nodes restore state from object storage and replay the recent transaction log to rejoin the cluster. Cold-start time depends on the size of the transaction log delta, not the size of the dataset. You only pay for the compute you use.

Q: How does instant branching work?

The storage engine can clone a petabyte-scale dataset in seconds by creating a logical branch that shares the underlying object storage data. Changes on the branch are isolated. This enables Git-like workflows for databases - branch for testing, experiment freely, merge or discard.

The distributed storage engine for SurrealDB. Gen 3 architecture - quorum consensus, compute-storage separation, and object storage - the high-availability foundation beneath the context layer.

Get started

GEN 3 ARCHITECTURE

Storage redesigned from first principles

Most distributed databases were designed around provisioned disks attached to compute nodes. SurrealDB's next-generation storage architecture is different: transactional data is placed directly in commodity cloud object storage, while the database runs as stateless, elastic compute on top.

This separates compute from storage without depending on a proprietary cloud database storage tier. Data can live in S3-compatible object storage, whether that is provided by a major cloud, a private cloud, or another compatible environment.

Gen 1: Single machine

Compute and storage live on one box. Scaling means buying a bigger box.

Examples: self-hosted MySQL or PostgreSQL.

Gen 2: Proprietary separation

Storage and compute are separated, but the storage layer is tied to a proprietary cloud database service. Data becomes coupled to one vendor's database platform.

Examples: Aurora, AlloyDB.

Gen 3: Object storage

Transactional data lives in commodity object storage, with stateless compute running on top. Storage and compute scale independently, and the architecture avoids dependence on proprietary cloud database storage tiers.

Client

Load balancer layer

Request routed to SurrealDB write node, or SurrealDB read-proxy node

Availability Zone A

Highly-scalable read compute

SurrealDBread node

SurrealDBwrite node

Availability Zone B

Highly-scalable read compute

SurrealDBread node

SurrealDBwrite node

Availability Zone C

Highly-scalable read compute

SurrealDBread node

SurrealDBwrite node

Object storage (S3 / GCS / AFS / durable cold tier)

QUORUM CONSENSUS

No single leader. No split brain.

The storage engine uses quorum-based consensus. Each availability zone has its own write node, writes scale horizontally, and transactions commit once a quorum acknowledges. Lower latency than leader-based replication.

1 transaction → 1 quorum decision

Every transaction is encapsulated within a quorum consensus decision. Transaction consensus is performed at the writing node, allowing writes to horizontally scale across all write nodes in a cluster.

Quorum transaction

(majority commit)

Quorum consensus

Storage enginedata storage

WRITE PATH

How a write transaction flows

From client request through the compute layer, into quorum consensus, and down to durable object storage.

SurrealDB Node

Query engine

Transaction engine

Storage engine(local disk and block storage)

Transaction engine

Periodic storage data compaction and compacted data storage

Asynchronous transaction log storage

Object storage (S3 / GCS / AFS / durable cold tier)

RESILIENCE

Node failure and recovery

When a node fails, a replacement restores from object storage and replays the transaction log. Recovery time is independent of dataset size.

Client

Load balancer layer

Request routed to SurrealDB write node, or SurrealDB read-proxy node

Availability Zone A

Write queries are handled by write nodes

SurrealDBread proxy node

SurrealDBwrite node

Availability Zone B

Node B acknowledges writes.
Persistence succeeds, quorum fulfilled.

SurrealDBwrite node

Availability Zone C

Node C fails to acknowledge writes.
Persistence fails, but quorum fulfilled.

SurrealDBwrite node

LSM tree layers are retrieved from object storage so the query requests ranges which are not in the local storage cache

LSM tree layers are synced to object storage after range compaction

Transaction log written asynchronously to object storage

After the transaction log syncs from object storage, the node syncs the latest writes from a SurrealDB write node, and forms part of the quorum

On node recovery, data is restored from object storage, and the transaction log on object storage is tailed for the latest writes

Object storage (S3 / GCS / AFS / durable cold tier)

Behaviour:

Quorum commit: transaction is committed once a quorum acknowledges the writes.

Durable log: committed writes are safe via a replicated write-ahead-log, with asynchronous durability to object storage for fast node recovery and secondary-region disaster recovery.

Catch-up: failed node restarts and replays transaction log from object storage (for reduced cost) and from cross-availability zone (for recent transactions).

Node crash: a new node catches up with the cluster, regardless of the existing local state, using transaction-log replay before joining the quorum.

Outcome:

Transaction commit succeeds with quorum majority

Client receives transaction success confirmation

SurrealDB write node in Availability Zone C catches up with cluster after restart

No split-brain / consistent ordering

Significant reduction in cross-availability zone traffic

IN PRACTICE

What Gen 3 storage delivers

Storage costs drop

Object storage costs a fraction of provisioned disk. The total dataset can far exceed the local storage of any running instance.

Cross-region replication

Data flows through object storage, not between nodes. The most expensive line item in traditional distributed databases is structurally reduced.

Scale to zero

Compute is stateless. Scale down to zero when idle - data persists in object storage. When demand returns, nodes restore from the durable store in seconds.

Instant recovery

A crashed node restores from object storage and replays the transaction log. Recovery time is independent of dataset size.

Instant branching

Clone a petabyte-scale database in seconds for development, testing, or experimentation. Git-like workflows for production data.

Built-in disaster recovery

Object storage is the durable tier (99.999999999% durability). No separate backup infrastructure. No snapshot schedules.

THE FULL STACK

From object storage to agent memory

No specialist agent database owns its storage layer. No memory middleware controls how data is persisted. The storage engine is the reason the context layer can scale down to zero, branch instantly, and recover from failures by design. No other product on the market owns this vertical.