q(80)/cl93uia03ggs73c8trqg.webp)
Apr 25th, 2023
Clustered SurrealDB for 1.0.0-beta9
by Hugh Kaznowski, 7 min read
In this post, I will show you how to set up a distributed SurrealDB cluster that shares a distributed TiKV cluster. This architecture allows you to scale your operations to improve writes and reads and seamlessly continue operations during failures.
Introduction and architecture overview
Users of SurrealDB can pick which Key-Value storage engines they want to use. That means that for single deployment, you can use RocksDB or in-memory storage; for distributed storage, you can use TiKV and FoundationDB.
We will deploy a cluster of TiKV that includes 3 TiKV instances (the KV engine) and 3 PD instances (placement driver, a resource tracking service). In addition to the above configuration, we will deploy three nodes of SurrealDB that will point to their respective KV engines. Typically you would want the SurrealDB instances not tied to individual TiKV instances, but that would require a load balancer - something beyond the scope of this article.
q(80)/ch3ufgftjugc73fa8vkg.webp)
Setting up the environment
Because we need access to 6 machines, we will simplify this setup using LXC - a lightweight Linux container system that makes nodes seem like fully-fledged computers.
An important note: LXC does not play nice with docker. There are ways around that, but I removed docker from my host machine for this example. It’s a VM; usually, you wouldn’t host this environment this way anyway.
Let’s start by running and configuring our first LXC container for usage.
hugh@hugh-VirtualBox:~$ lxc launch ubuntu: lxc-node-tikv-1
root@lxc-node-tikv-1:~# sudo useradd -m hugh root@lxc-node-tikv-1:~# sudo adduser hugh sudo
root@lxc-node-tikv-1:~# sudo passwd hugh
root@lxc-node-tikv-1:~# tiup cluster template > topology.yaml
Great!
We now have our initial node ready.
We need to modify our topology file to reflect the actual topology we will have.
Here is the sample I have from editing the topology.yaml
we just exported.
That is actually my entire topology.yaml file. I removed TiDB and all the monitoring - we aren’t using that for this example.
We will create a snapshot from the image to simplify our setup and start the installation. We will then create instances that automatically have SSH, the hugh account with a known password, and a sudo group permission without password authentication. Don’t do this in production - this is a highly insecure setup for many reasons.
Instance published with fingerprint: b8841a679a59f98f3c23ba6c8795c84942f19170b4a8c41eb102130467c4cca6
We can now start our cluster from the first node we configured. TiUp will connect to all the other nodes via SSH and password authentication and install the services that way.
root@lxc-node-tikv-1:~# tiup cluster start tikv-test --init
At this point, you should have a running TiKV cluster. All that remains is to put SurrealDB instances on the PD nodes. I will demonstrate this only for a single PD node, as the rest are identical.
root@lxc-node-pd-1:~# curl --proto '=https' --tlsv1.2 -sSf https://install.surrealdb.com | sh -s -- --nightly
root@lxc-node-pd-1:~# PATH=/root/.surrealdb:$PATH root@lxc-node-pd-1:~# surreal sql --ns testns --db testdb -u root -p root --conn tikv://lxc-node-pd-1:2379
Under a real scenario, you would have the SurrealDB nodes separate from the PD nodes, and the connections would be load balanced across the entire PD node pool.
Takeaways
As you can see, it is possible to set up SurrealDB in a cluster so that writes and reads can scale. Failure of a single node would have minimal disruption to the rest of the work while keeping your data intact. Backups can be performed against the TiKV cluster to ensure you can recover in the event of serious failures.
Hopefully, you found this guide helpful, and I look forward to hearing what you get up to with it!
q(80)/clcv8o5ol7tc73e91qeg.webp)
q(80)/cl93hl203ggs73c8trdg.webp)