Deployment & storage layer considerations

So far, throughout this course, we have been talking about data.

• Why data models matter
• How to manipulate data
• How to connect data
• How to index data

We’ll still talk about data in this last lesson about storage layer and deployment considerations. The difference is we’ll be talking about it from the perspective of a manager of databases rather than a user of databases.

We’ll briefly cover

• Why the separation of storage and compute matters
• The different ways of running SurrealDB and the supported key-value storage engines

Why the separation of storage and compute matters

The separation of storage and compute has been a trend that first became popular with analytical databases (OLAP) and which has since made its way into operational databases (OLTP).

The key reason for separating storage and compute is so that you can scale your resources separately.

This has several benefits such as:

• Less wasted capacity as scale is optimised for each workload
• Greater cost efficiency
• Greater flexibility
• Better fault tolerance

This has also allowed us to offer various key-value storage engines you can choose from, depending on your needs.

The different ways of running SurrealDB and the supported key-value storage engines

There are three ways of running SurrealDB

• Directly in the browser
• As a single-node server or in embedded devices
• As a multi-node cluster in the cloud

There are also five storage engines, each engine supporting a transaction-based approach, with the ability to support the reading and writing of individual keys and key ranges within each transaction.

Let’s go through each of these.

Running SurrealDB directly in the browser

Running SurrealDB directly in the browser is the way you have been interacting with SurrealDB throughout the course, as the embedded Surrealist interface we’ve been using is made possible by running SurrealDB in WebAssembly (WASM).

The storage engine we’ve been using to persist the data is called IndexedDB, a large-scale, NoSQL storage system that allows storage of just about anything in your browser. In terms of security, it’s important to point out that IndexedDB lives inside your browser (Chrome, Safari, Firefox, etc) in a sandboxed protected environment.

SurrealDB first serialises both keys and values into a Uint8Array, utilising IndexedDB as a binary key-value store. This offers good performance, and the ability to use the features that SurrealDB offers when running in alternative ways.

Running SurrealDB on a single-node server or in embedded devices

SurrealDB is easy to install and run, whether you are running everything on your laptop, on an IoT device or as a single-node server in the cloud.

You’ll find installation and running instructions for Linux, macOS, Windows and Docker in our documentation, where you can install and run SurrealDB with one command each.

The storage engines you can choose between are RocksDB (default) and SurrealKV.

RocksDB

• Was originally a fork of Google’s LevelDB but was then developed and maintained by Meta (Facebook) entirely in C++.
• It’s optimised for fast, low-latency storage such as flash drives and high-speed disk drives using the log-structured merge-tree data structure (LSM).
• LSM data structures are highly efficient for write-heavy workloads as they minimise the number of disk writes and require less space than B-Trees, as they don’t require constant reorganisation of data.

SurrealKV

• SurrealKV is our own storage engine developed by us from scratch, written entirely in Rust.
• It’s made to support the power and flexibility of SurrealDB, opening the door to exciting use cases such as versioned queries over time, immutable data querying, historic aggregate query analysis, and versioned graph queries. Something that has never been done before!
• We designed a new data structure, an Immutable Versioned Adaptive Radix Trie (VART) to achieve this, which you can read more about on our blog, or reach out to us for a chat.
• It’s still actively evolving, and as such, there might be changes to the file format, APIs, and feature set in future releases until reaching stability. That is why RocksDB is still our default engine, but we encourage you to experiment with SurrealKV.

Running SurrealDB on a multi-node cluster in the cloud

For highly-available and highly-scalable setups, SurrealDB can be run on top of a TiKV cluster (default) or a FoundationDB cluster, with the ability to horizontally scale to 100+ terabytes of data.

TiKV

• TiKV is a distributed and transactional key-value database from PingCap, built in Rust
• It uses the Raft consensus algorithm and RocksDB (LSM) as the underlying storage engine

FoundationDB

• FoundationDB is a distributed and transactional key-value database from Apple, built in C++
• It uses the Paxos consensus algorithm and a B-tree based on SQLite as its underlying storage engine

The highly-available deployment of your storage layer (TiKV, or FoundationDB) is the stateful part of the system. SurrealDB clusters do not have state. Any necessary communication is either avoided by design, or communicated via the storage layer which maintains consistency.

This makes deploying SurrealDB really convenient, since your fault tolerance is equal to the number of replicas (minus 1, the bare minimum number of instances). Even though we do have bootstrapping, by design, nodes can come and go as they please without much interference.

You can see the details of how to run a multi-node cluster and deploy it in our documentation.

Summary

We’ve covered quite a lot, and as we’ve seen, SurrealDB was designed from the start to have unparalleled deployment flexibility, ranging from embedded devices to distributed multi-node clusters and even embedded directly in your website using WebAssembly (WASM).

If you want the power and flexibility of SurrealDB without the pain of managing infrastructure, sign up for Surreal Cloud for a fully managed solution where you can focus on building your application, and we’ll take care of the rest.