Buckets | Files | Database

A bucket is named storage that backs file values (f"bucket:/path"). You have a few options when defining a bucket:

Memory for non-persistent storage.
A folder on disk for persistence.
Cloud object storage: Amazon S3 or an S3-compatible service, Google Cloud Storage, or Azure Blob Storage.
A global backend driven by environment variables.

Once a bucket is defined, you can then read and write to files inside it through the file functions.

Full statement detail: DEFINE BUCKET. For how pointers look in queries, see Working with files.

Example usage

A bucket backend can be set as "memory" for non-persistent in-memory storage, as "file:/" followed by the path for storage on disk, or as an object storage URL for Amazon S3, an S3-compatible service, Google Cloud Storage or Azure Blob Storage.

Memory backend

The simplest way to experiment with a bucket for files is by using the memory backend:

DEFINE BUCKET my_bucket BACKEND "memory";

Once this is defined, my_bucket can be accessed by using a file pointer: a path prefixed by an f.

-- Create a file by adding some content
f"my_bucket:/my_book.txt".put("Once there were four children whose names were Peter,
  Susan,
  Edmund,
  and Lucy.");
-- Copy it to a new file name
f"my_bucket:/my_book.txt".copy("lion_witch_wardrobe.txt");
-- Read the file as bytes
f"my_bucket:/lion_witch_wardrobe.txt".get();
-- Cast the bytes to a string
<string>f"my_bucket:/lion_witch_wardrobe.txt".get();

File backend

A file backend can be chosen for a bucket by typing "file:" and then the rest of the path, if necessary.

DEFINE BUCKET my_bucket BACKEND "file:/some_directory";
DEFINE BUCKET my_bucket BACKEND "file:/some_directory";

A check will then be made to see if the SURREAL_BUCKET_FOLDER_ALLOWLIST environment variable contains the path, without which the following error will be generated.

'File access denied: /some_directory'

The following command can be used to start running an instance in which a bucket with a file backend can be defined.

# Unix
SURREAL_BUCKET_FOLDER_ALLOWLIST="/" surreal db start --user root --pass secret

# Windows (PowerShell)
$env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "/" 
surreal db start --user root --pass secret

Cloud object storage backend

Available since: v3.3.0

A bucket can keep its files in object storage by giving BACKEND the URL of the store. The scheme selects the service: s3://, s3+https:// or s3+http:// for Amazon S3 and S3-compatible services such as MinIO, Backblaze B2, Wasabi and Cloudflare R2, gs:// or gcs:// for Google Cloud Storage, and az:// or azure:// for Azure Blob Storage.

-- Amazon S3: with no host, requests go to the AWS endpoint for the region
DEFINE BUCKET invoices BACKEND "s3:/acme-invoices?region=eu-west-2&prefix=surrealdb/invoices";

-- An S3-compatible service: the host is the endpoint, and the bucket is the first path segment
DEFINE BUCKET uploads
  BACKEND "s3+http://surrealdb:strongPassword@localhost:9000/uploads";

-- Google Cloud Storage: the host is the bucket
DEFINE BUCKET reports BACKEND "gs://acme-reports?prefix=surrealdb&service_account=/etc/surrealdb/gcs-service-account.json";

-- Azure Blob Storage: the host is the storage account, then the container
DEFINE BUCKET media BACKEND "az://acmemedia/uploads?prefix=surrealdb";

Credentials can go in the URL, but when they are left out the server uses the standard credential sources for each provider, such as the AWS_*, GOOGLE_* and AZURE_* environment variables or an instance role. The URL is stored with the bucket definition and returned by INFO FOR DB, so credentials from the environment stay out of the schema. The DEFINE BUCKET reference lists every URL parameter and credential source.

These backends exist in native builds only. A WebAssembly build of the engine has the memory backend alone. The memory and file backends are unchanged.

Global backend

A server can also have one global bucket, set with the SURREAL_GLOBAL_BUCKET environment variable, which accepts the same URLs as BACKEND, including the cloud object storage schemes. A root user can give a bucket a global:// backend, followed by the path of the part of the global bucket that the bucket serves:

DEFINE BUCKET books BACKEND "global://library/books";

-- Stored as /library/books/my_book.txt in the global bucket
f"books:/my_book.txt".put("Once there were four children whose names were Peter, Susan, Edmund, and Lucy.");

Before SurrealDB 4.0, a bucket defined without BACKEND used the global bucket under a prefix made from internal namespace and database identifiers and its name. From 4.0, BACKEND is required, and a bucket defined that way keeps its files. The DEFINE BUCKET reference describes the global bucket in full, and also how to define a bucket at the root or in a namespace instead of in a database.

Setting permissions on buckets

By default, the permissions on a bucket will be set to FULL unless otherwise specified.

DEFINE BUCKET my_bucket BACKEND "memory";
INFO FOR DB;

Output

{
  accesses: {},
  analyzers: {},
  apis: {},
  buckets: {
    my_bucket: "DEFINE BUCKET my_bucket BACKEND 'memory'
      PERMISSIONS FULL"
  },
  configs: {},
  functions: {},
  models: {},
  modules: {},
  params: {},
  sequences: {},
  tables: {},
  users: {}
}

You can set permissions on buckets to control who can perform operations on the files stored in them using the PERMISSIONS clause. In the clause three additional variables are available:

$action: The action to be executed (put, get, head, delete, copy, rename, exists, list, presign_get, presign_put)
$file: The file pointer of the file to be accessed
$target: The target file pointer in copy/rename operations

-- Set permissions for the bucket
DEFINE BUCKET admin_bucket BACKEND "memory"
  PERMISSIONS WHERE $auth.admin = true

Serving files over HTTP

From SurrealDB 4.0, the files in a bucket can also be read and written over HTTP, through the /bucket route. A request authenticates with a session, or with a presigned URL that lets a browser or another service download or upload one file without a session. The bucket's PERMISSIONS clause applies to both.