Deploy on Azure Kubernetes Service (AKS)

What is AKS?

Azure Kubernetes Service is a managed Kubernetes service offered by the Microsoft Azure platform.

This article will guide you through setting up a highly available SurrealDB cluster backed by TiKV on an Azure Kubernetes Service cluster.

What is TiKV?

TiKV is a cloud-native transactional key/value store built by PingCAP and that integrates well with Kubernetes thanks to their TiDB operator.

Prerequisites

In order for you to complete this tutorial you'll need:

An account on Microsoft Azure
The Azure CLI
kubectl to manage the Kubernetes cluster
helm to install SurrealDB server and TiKV
Surreal CLI to interact with the SurrealDB server

Note

Provisioning the environment in your Azure account will create resources and there will be cost associated with them. The cleanup section provides a guide to remove them, preventing further charges.

Note

This guide was tested in westeurope region and it follows TiKV best practices for scalability and high availability.

This tutorial is intended for production workloads using the standard tier. If you want to create a dev/test environment, you should root for the free tier and change the cluster and node pool configuration (no zone, fewer nodes).

Create an AKS Cluster

$ az login
$ az account list

Create a new resource group:

Create new group

$ az group create --name rg-surrealdb-aks --location westeurope

Run the following command to create a new AKS cluster:

Create new cluster

$ az aks create \
    --resource-group rg-surrealdb-aks \
    --location westeurope \
    --name surrealdb-aks-cluster \
    --generate-ssh-keys \
    --load-balancer-sku standard \
    --node-count 3 \
    --zones 1 2 3 \
    --enable-addons monitoring \
    --tier standard

After creation finishes, get credentials to configure kubectl to connect to the new cluster:

Get AKS cluster credentials

$ az aks get-credentials --resource-group rg-surrealdb-aks --name surrealdb-aks-cluster

You can verify the connection to the cluster with the following command:

Display cluster nodes

$ kubectl get nodes

NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-33674805-vmss000000   Ready    agent   2m35s   v1.26.6
aks-nodepool1-33674805-vmss000001   Ready    agent   2m36s   v1.26.6
aks-nodepool1-33674805-vmss000002   Ready    agent   2m34s   v1.26.6

Create cluster node pools

Note

In order to speed things up, the following commands can be executed in parallel.

Create a TiDB Operator and monitor pool:

Create operator and monitor

$ az aks nodepool add --name admin \
    --resource-group rg-surrealdb-aks \
    --cluster-name surrealdb-aks-cluster \
    --zones 1 2 3 \
    --node-count 1 \
    --labels dedicated=admin

Create a PD node pool:

Add node pool

$ az aks nodepool add --name pd \
    --resource-group rg-surrealdb-aks \
    --cluster-name surrealdb-aks-cluster \
    --node-vm-size Standard_F4s_v2 \
    --zones 1 2 3 \
    --node-count 3 \
    --labels dedicated=pd \
    --node-taints dedicated=pd:NoSchedule

Create a TiKV node pool:

Add node pool

$ az aks nodepool add --name tikv \
    --resource-group rg-surrealdb-aks \
    --cluster-name surrealdb-aks-cluster \
    --node-vm-size Standard_E8s_v4 \
    --zones 1 2 3 \
    --node-count 3 \
    --labels dedicated=tikv \
    --node-taints dedicated=tikv:NoSchedule \
    --enable-ultra-ssd

Deploy TiDB operator

Now that we have a Kubernetes cluster, we can deploy the TiDB operator. TiDB operator is a Kubernetes operator that manages the lifecycle of TiDB clusters deployed to Kubernetes. You can deploy it following these steps:

Install CRDS:

CRDS installation

$ kubectl create -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.5.0/manifests/crd.yaml

Install TiDB Operator Helm chart:

Helm chart installation

$ helm repo add pingcap https://charts.pingcap.org
$ helm repo update
$ helm install \
  -n tidb-operator \
  --create-namespace \
  tidb-operator \
  pingcap/tidb-operator \
  --version v1.5.0

Verify that the pods are running:

Verify TiDB operator

$ kubectl get pods --namespace tidb-operator -l app.kubernetes.io/instance=tidb-operator

NAME                                       READY   STATUS    RESTARTS   AGE
tidb-controller-manager-67d678dc64-qf6p2   1/1     Running   0          60s
tidb-scheduler-68555ffd4-l2ssf             2/2     Running   0          60s

Create TiDB cluster

Now that we have the TiDB Operator running, it's time to define a TiDB Cluster and let the Operator do the rest.

Create a local file named tikv-cluster.yaml with this content:

TiDB cluster definition

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: sdb-datastore
spec:
  version: v6.5.0
  timezone: UTC
  configUpdateStrategy: RollingUpdate
  pvReclaimPolicy: Delete
  enableDynamicConfiguration: true
  schedulerName: default-scheduler
  topologySpreadConstraints:
  - topologyKey: topology.kubernetes.io/zone
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 0
    replicas: 3
    storageClassName: managed-csi-premium
    requests:
      cpu: 500m
      storage: 10Gi
      memory: 1Gi
    config: |
      [dashboard]
        internal-proxy = true
      [replication]
        location-labels = ["topology.kubernetes.io/zone", "kubernetes.io/hostname"]
        max-replicas = 3
    nodeSelector:
      dedicated: pd
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: pd
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/component
              operator: In
              values:
              - pd
          topologyKey: kubernetes.io/hostname
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 0
    replicas: 3
    storageClassName: managed-csi-premium
    requests:
      cpu: 1
      storage: 10Gi
      memory: 2Gi
    config: {}
    nodeSelector:
      dedicated: tikv
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: tikv
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/component
              operator: In
              values:
              - tikv
          topologyKey: kubernetes.io/hostname
  tidb:
    replicas: 0

Create the TiDB cluster:

Apply TiDB cluster

$ kubectl apply -f tikv-cluster.yaml

Check the cluster status and wait until it's ready:

Verify TiDB cluster

$ kubectl get tidbcluster

$ kubectl get pods

NAME                                       READY   STATUS    RESTARTS   AGE
sdb-datastore-discovery-7d7d684d88-4v4ws   1/1     Running   0          4m2s
sdb-datastore-pd-0                         1/1     Running   0          4m2s
sdb-datastore-pd-1                         1/1     Running   0          4m2s
sdb-datastore-pd-2                         1/1     Running   0          4m2s
sdb-datastore-tikv-0                       1/1     Running   0          3m12s
sdb-datastore-tikv-1                       1/1     Running   0          3m12s
sdb-datastore-tikv-2                       1/1     Running   0          3m12s

Deploy SurrealDB

Now that we have a TiDB cluster running, we can deploy SurrealDB using the official Helm chart. The deployment will use the latest SurrealDB Docker image and make it accessible on internet.

Get the TiKV PD service URL:

Get service URL

$ kubectl get service sdb-datastore-pd

NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
sdb-datastore-pd   ClusterIP   10.0.161.101   <none>        2379/TCP   5m27s

$ export TIKV_URL=tikv://sdb-datastore-pd:2379

Install the SurrealDB Helm chart with the TIKV_URL defined above and with auth disabled so we can create the initial credentials:

SurrealDB Helm chart installation

$ helm repo add surrealdb https://helm.surrealdb.com
$ helm repo update
$ helm install \
    --set surrealdb.path=$TIKV_URL \
    --set surrealdb.auth=false \
    --set ingress.enabled=false \
    --set service.type=LoadBalancer \
    --set service.port=80 \
    --set service.targetPort=8000 \
    --set image.tag=latest \
    surrealdb-tikv surrealdb/surrealdb

Wait until the Loadbalancer resource has an EXTERNAL-IP assigned:

Wait for Loadbalancer address

$ kubectl get service surrealdb-tikv

NAME             TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)          AGE
surrealdb-tikv   LoadBalancer   10.0.38.191   20.13.45.154   80:30378/TCP   6m34s

Connect to the cluster and define the initial credentials:

Connect and define credentials

$ export SURREALDB_URL=http://$(kubectl get service surrealdb-tikv -o json | jq -r .status.loadBalancer.ingress[0].ip)

$ surreal sql -e $SURREALDB_URL
> DEFINE USER root ON ROOT PASSWORD 'StrongSecretPassword!' ROLES OWNER;

Verify you can connect to the database with the new credentials:

$ surreal sql -u root -p 'StrongSecretPassword!' -e $SURREALDB_URL
> INFO FOR ROOT
[{ accesses: {  }, namespaces: { ns: 'DEFINE NAMESPACE ns' }, nodes: { "0e87c953-68d7-40e1-9090-3dfc404af25e": 'NODE 0e87c953-68d7-40e1-9090-3dfc404af25e SEEN 1742870304370 ACTIVE' }, system: { available_parallelism: 14, cpu_usage: 5.905298233032227f, load_average: [1.2802734375f, 1.5400390625f, 1.71484375f], memory_allocated: 13492047, memory_usage: 187547648, physical_cores: 14, threads: 32 }, users: { root: "DEFINE USER root ON ROOT PASSHASH '...' ROLES OWNER DURATION FOR TOKEN 1h, FOR SESSION NONE" } }]

Now that the initial credentials have been created, enable authentication:

Update SurrealDB Helm chart

$ helm upgrade \
    --set surrealdb.path=$TIKV_URL \
    --set surrealdb.auth=true \
    --set ingress.enabled=false \
    --set service.type=LoadBalancer \
    --set service.port=80 \
    --set service.targetPort=8000 \
    --set image.tag=latest \
    surrealdb-tikv surrealdb/surrealdb

Cleanup

Run the following commands to delete the Kubernetes resources and the AKS cluster.

Cleanup commands

$ helm uninstall surrealdb-tikv
$ helm -n tidb-operator uninstall tidb-operator
$ az aks delete --name surrealdb-aks-cluster --resource-group rg-surrealdb-aks
$ az group delete --resource-group rg-surrealdb-aks