Kubernetes operator

Available since: v4.0.0

The SurrealDB operator runs SurrealDS groups on Kubernetes. You describe a group in one SurrealDS resource, and the operator creates the nodes, their storage, their configuration and the Services that clients connect to.

When you finish this guide, you will have a three-node SurrealDS group, each node in its own zone, that you can query with the Surreal CLI.

How the operator runs a group

For a SurrealDS resource named <name>, the operator creates:

Node ids start at 1. Each node is its own StatefulSet with one replica, because each node has a fixed identity and a fixed zone.

Requirements

• •

  Kubernetes 1.34 or later, with kubectl and helm.

• •

  Zone labels on your Kubernetes nodes. Each node that can run SurrealDS needs a topology.kubernetes.io/zone label. By default the operator gives each SurrealDS node a zone that no other SurrealDS node has, so a group of three needs three zones. To choose the zones yourself, see spec.shale.zones.

• •

  A StorageClass with volumeBindingMode: WaitForFirstConsumer. The operator creates each volume before its pod, and a class that binds at once can create the volume in a zone other than the pod's.

• •

  An object store bucket and credentials that can read, write, list and delete in it. See What a deployment needs.

• •

  A licence key. See Licensing.

• •

  openssl, to make the TLS certificates.

• •

  The Surreal CLI, to connect when the group is running.

• •

  Optional: Gateway API. Only needed if you publish the group with spec.routes.

Check the zone labels of your Kubernetes nodes:

kubectl get nodes -L topology.kubernetes.io/zone

Install the operator

Install the chart into its own namespace:

helm repo add surrealdb https://helm.surrealdb.com
helm repo update
helm install surrealdb-operator surrealdb/surrealdb-operator -n surrealdb-operator-system --create-namespace

The chart installs the SurrealDS CustomResourceDefinition (surrealds.operator.surrealdb.com), the operator Deployment, its ServiceAccount, and two ClusterRoles: one to manage the resources of a group, and one to read Kubernetes nodes for their zone labels.

Check that the operator is running:

kubectl -n surrealdb-operator-system get pods

Chart values

Upgrade the operator

Helm installs a chart's CustomResourceDefinitions on the first install only, and never upgrades them. Apply the new definition before you upgrade the chart:

helm repo update
helm pull surrealdb/surrealdb-operator --untar
kubectl apply --server-side -f surrealdb-operator/crds/
helm upgrade surrealdb-operator surrealdb/surrealdb-operator -n surrealdb-operator-system

An operator upgrade does not restart running nodes unless it must change their pod template for another reason. Uninstalling the chart leaves the CustomResourceDefinition and your SurrealDS resources in place.

Create a group

The steps below create a group named surrealds in the namespace surrealds. Change both names as you need.

kubectl create namespace surrealds

1. Store the object store credentials

For an S3-compatible store, the Secret holds AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The operator passes every key in this Secret to the nodes as an environment variable.

kubectl -n surrealds create secret generic surrealds-s3 \
  --from-literal=AWS_ACCESS_KEY_ID='<access-key-id>' \
  --from-literal=AWS_SECRET_ACCESS_KEY='<secret-access-key>'

For Azure Blob Storage, store AZURE_STORAGE_ACCOUNT_KEY instead, or use AKS workload identity with no Secret.

2. Make the TLS certificates

Nodes authenticate each other with mutual TLS. Each group needs one certificate authority, and one certificate and private key for each node. The certificate of node <id> must carry the DNS name shale-<id>, and the private key must be in PKCS#8 format.

Save this script as surrealds-certs.sh. It works with OpenSSL and with the LibreSSL that macOS includes. It creates the certificate authority once, and creates a certificate only for a node that does not have one yet, so you can run it again to add nodes.

#!/usr/bin/env bash
# Usage: surrealds-certs.sh <output-directory> <node-count>
set -euo pipefail
out="$1"; count="$2"; days=3650
mkdir -p "$out"
if [ ! -f "$out/ca.key" ]; then
  openssl req -x509 -newkey rsa:2048 -nodes -sha256 -days "$days" \
    -subj "/CN=surrealds-ca" -keyout "$out/ca.key" -out "$out/ca.crt" 2>/dev/null
fi
for i in $(seq 1 "$count"); do
  [ -f "$out/node-$i.crt" ] && continue
  printf 'subjectAltName=DNS:shale-%s\nextendedKeyUsage=serverAuth,clientAuth\nbasicConstraints=CA:FALSE\n' "$i" > "$out/node-$i.ext"
  openssl req -new -newkey rsa:2048 -nodes -sha256 -subj "/CN=shale-$i" \
    -keyout "$out/node-$i.rsa.key" -out "$out/node-$i.csr" 2>/dev/null
  openssl pkcs8 -topk8 -nocrypt -in "$out/node-$i.rsa.key" -out "$out/node-$i.key"
  openssl x509 -req -in "$out/node-$i.csr" -CA "$out/ca.crt" -CAkey "$out/ca.key" -CAcreateserial \
    -sha256 -days "$days" -extfile "$out/node-$i.ext" -out "$out/node-$i.crt" 2>/dev/null
  rm -f "$out/node-$i.rsa.key" "$out/node-$i.csr" "$out/node-$i.ext"
done
chmod 600 "$out"/*.key
echo "Wrote ca.crt and node-1 to node-$count certificates in $out"

Run it for three nodes, and store everything except the certificate authority's private key in a Secret:

bash surrealds-certs.sh ./surrealds-tls 3
kubectl -n surrealds create secret generic surrealds-tls \
  --from-file=ca.crt=surrealds-tls/ca.crt \
  --from-file=surrealds-tls/node-1.crt --from-file=surrealds-tls/node-1.key \
  --from-file=surrealds-tls/node-2.crt --from-file=surrealds-tls/node-2.key \
  --from-file=surrealds-tls/node-3.crt --from-file=surrealds-tls/node-3.key

Keep ca.key outside the cluster in a safe place. You need it to make certificates for nodes that you add later.

Each node trusts its peers by their exact certificates, not only by the certificate authority. A new certificate for an existing node therefore changes that node's identity for its peers. The script uses a ten-year validity for this reason. Do not replace the certificate of an existing node without guidance from SurrealDB support.

3. Store the root credentials

The root user is the first user of the database. The Secret keys are username and password by default.

kubectl -n surrealds create secret generic surrealds-root \
  --from-literal=username=root \
  --from-literal=password='<a-strong-password>'

4. Store the licence key

kubectl -n surrealds create secret generic surrealdb-license --from-literal=key='<your-licence-key>'

5. Create the SurrealDS resource

Save this as surrealds.yaml. Replace the bucket, prefix, region, endpoint and storage class with your own values.

apiVersion: operator.surrealdb.com/v1alpha1
kind: SurrealDS
metadata:
  name: surrealds
  namespace: surrealds
spec:
  image: surrealdb/surrealdb-enterprise:v4.0.0-beta.1
  replicas: 3
  rootCredentials:
    secretRef:
      name: surrealds-root
  env:
    - name: SURREAL_LICENSE_KEY
      valueFrom:
        secretKeyRef:
          name: surrealdb-license
          key: key
    - name: SURREAL_LICENSE_STATE_PATH
      value: /var/lib/shale/license-state.json
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
    limits:
      memory: 8Gi
  shale:
    volumeClaimTemplate:
      storageClassName: <your-storage-class>
      resources:
        requests:
          storage: 50Gi
    tlsSecretRef:
      name: surrealds-tls
    s3:
      bucket: <your-bucket>
      prefix: surrealds
      region: <your-region>
      # Leave out `endpoint` for Amazon S3. Set it for any other S3-compatible store.
      # endpoint: https://<your-s3-endpoint>
      credentialsSecretRef:
        name: surrealds-s3

Apply it:

kubectl apply -f surrealds.yaml

Set a memory limit. Each node sizes its caches from the container's memory limit, and without a limit it uses the allocatable memory of the Kubernetes node that it runs on.

6. Wait for the group

kubectl -n surrealds get surrealds surrealds -w

The Ready column becomes True when every node serves. A new group can take a few minutes. Node 1 creates the database, and the other nodes join it.

Show each node's zone and readiness:

kubectl -n surrealds get surrealds surrealds -o jsonpath='{range .status.nodes[*]}{.id}{"\t"}{.zone}{"\t"}{.statefulSet}{"\t"}{.ready}{"\n"}{end}'

Connect to the group

Clients connect to the Service surrealdb-<name> on port 8000. From your machine, forward the port:

kubectl -n surrealds port-forward svc/surrealdb-surrealds 8000:8000

Then connect with the root credentials:

surreal sql -e ws://127.0.0.1:8000 -u root -p '<a-strong-password>'

Applications in the cluster connect to ws://surrealdb-surrealds.surrealds.svc.cluster.local:8000.

Publish the group through a Gateway

To publish the client Service outside the cluster, add one or more routes. Each route becomes an HTTPRoute that sends traffic to the Service on port 8000. The cluster needs the Gateway API and a Gateway.

spec:
  routes:
    - gatewayRef:
        name: <your-gateway>
        namespace: <gateway-namespace>
      hostnames:
        - surrealds.example.com

Each route takes either hostnames or matches, not both. If the Gateway API is not installed, a resource with routes reports HTTPRouteReady=False with the reason GatewayAPINotInstalled, and it does not become ready.

Operate the group

Change the configuration

The operator applies a change to one node at a time, in node order, and continues to the next node only after the previous node is ready again. Changes that restart nodes this way include the image, spec.env, probes, extraAnnotations, and the contents of the TLS, object store credentials and root credentials Secrets. While nodes wait for their turn, the StatefulSetReady condition reports RollingConfig.

These fields cannot change after the group exists: spec.shale.s3 (all of it), spec.shale.scope, spec.shale.tlsSecretRef and the zone of each node. A change to one of them sets SpecValid=False.

Add nodes

1. 1.

   Make certificates for the new nodes. Run the certificate script again with the new node count. It keeps the existing certificates.

2. 2.

   Add the new certificates to the TLS Secret, and keep the existing keys. The nodes restart one at a time to read the new Secret.

3. 3.

   Raise spec.replicas.

bash surrealds-certs.sh ./surrealds-tls 5
kubectl -n surrealds create secret generic surrealds-tls \
  --from-file=ca.crt=surrealds-tls/ca.crt \
  $(for i in 1 2 3 4 5; do printf -- '--from-file=surrealds-tls/node-%s.crt --from-file=surrealds-tls/node-%s.key ' "$i" "$i"; done) \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n surrealds patch surrealds surrealds --type merge -p '{"spec":{"replicas":5}}'

Each new node needs a zone. Without spec.shale.zones, each new node needs a zone that no other SurrealDS node in the group has. To put more nodes than you have zones into a group, list a zone for every node in spec.shale.zones. A zone can repeat in that list.

Remove nodes

Lower spec.replicas. The operator removes the nodes with the highest ids. Their volumes stay, so delete the PersistentVolumeClaims yourself when you no longer need them. A group that has had more than one node cannot go to one node.

Pause and resume

Set spec.replicas to 0 to stop every node. The volumes and the object store keep the data. Set it back to the previous value to start the group again.

Grow the commit log volumes

Raise spec.shale.volumeClaimTemplate.resources.requests.storage. The StorageClass must allow volume expansion. The operator refuses to shrink a volume.

Delete the group

kubectl -n surrealds delete surrealds surrealds

This deletes every resource the operator created for the group, including the PersistentVolumeClaims. The operator never deletes data in the object store. To remove the database completely, delete the bucket prefix yourself.

Status and troubleshooting

The resource reports these conditions. All of them except StorageResizeComplete must be True for Ready to be True.

kubectl -n surrealds describe surrealds surrealds
kubectl -n surrealds logs surrealdb-surrealds-shale-1-0

Common causes of a group that does not become ready:

• •

  SpecValid=False. A Secret is missing or incomplete. For example, the TLS Secret has no certificate for one of the nodes. The operator checks again every minute, and at once when a named Secret changes.

• •

  ZonesResolved=False with InsufficientZones. There are fewer zone-labelled Kubernetes nodes than SurrealDS nodes. Label more nodes, or set spec.shale.zones.

• •

  A pod stays Pending. No Kubernetes node in its zone has capacity, or its volume was created in another zone. Check the StorageClass binding mode.

• •

  A node does not start. Read its log. A licence problem stops the process with a message that names the cause. See Licensing.

A node does not open its client port until it has recovered its log and joined the group, so the readiness probe fails with connection refused while a node starts. That is expected. The startup probe allows two minutes for this before the liveness probe starts. Each node also starts with SURREAL_STARTUP_OPERATION_TIMEOUT=1200s, so that SurrealDB waits for a long recovery.

SurrealDS reference

The resource is SurrealDS in the API group operator.surrealdb.com/v1alpha1. Its short name is sds.

Spec fields

shale fields

Default probes

Azure Blob Storage

Set s3.kind: azure, set s3.bucket to the container and s3.account to the storage account. For AKS workload identity, do not set credentialsSecretRef. Instead, label the pods and annotate the ServiceAccount:

spec:
  extraPodLabels:
    azure.workload.identity/use: "true"
  serviceAccountAnnotations:
    azure.workload.identity/client-id: <managed-identity-client-id>
  shale:
    s3:
      kind: azure
      bucket: <container>
      prefix: surrealds
      account: <storage-account>

One-node groups

A resource with replicas: 1 runs one node with no peers. It still needs the TLS Secret, with ca.crt, node-1.crt and node-1.key. There is one copy of the data, on the node's volume, until the node uploads it to the object store. A restart or a drain stops the database until the node is back. You can raise replicas later.