DEFINE BUCKET

Available since: v3.0.0

The DEFINE BUCKET statement lets you create a bucket that can hold files.

Statement syntax

DEFINE BUCKET [ OVERWRITE | IF NOT EXISTS ] @name
  [ ON [ ROOT | NAMESPACE | DATABASE ] ]
  BACKEND @string
  [ READONLY ]
  [ PERMISSIONS @expression ]
  [ COMMENT @string ]

Example usage

A bucket backend can be set as "memory" for non-persistent in-memory storage, as "file:/" followed by the path for storage on disk, or as an object storage URL for Amazon S3, an S3-compatible service, Google Cloud Storage or Azure Blob Storage.

Memory backend

The simplest way to experiment with a bucket for files is by using the memory backend:

DEFINE BUCKET my_bucket BACKEND "memory";

Once this is defined, my_bucket can be accessed by using a file pointer: a path prefixed by an f.

-- Create a file by adding some content
f"my_bucket:/my_book.txt".put("Once there were four children whose names were Peter, Susan, Edmund, and Lucy.");
-- Copy it to a new file name
f"my_bucket:/my_book.txt".copy("lion_witch_wardrobe.txt");
-- Read the file as bytes
f"my_bucket:/lion_witch_wardrobe.txt".get();
-- Cast the bytes to a string
<string>f"my_bucket:/lion_witch_wardrobe.txt".get();

-------- Query --------

b"4F6E6365207468657265207765726520666F7572206368696C6472656E2077686F7365206E616D657320776572652050657465722C20537573616E2C2045646D756E642C20616E64204C7563792E"

-------- Query --------

'Once there were four children whose names were Peter, Susan, Edmund, and Lucy.'

File backend

A file backend can be chosen for a bucket by typing "file:" and then the rest of the path, if necessary.

DEFINE BUCKET my_bucket BACKEND "file:/some_directory";

A check will then be made to see if the SURREAL_BUCKET_FOLDER_ALLOWLIST environment variable contains the path, without which the following error will be generated.

'File access denied: /some_directory'

The following command can be used to start running an instance in which a bucket with a file backend can be defined.

# Unix
SURREAL_BUCKET_FOLDER_ALLOWLIST="/" surreal db start --user root --pass secret

# Windows (PowerShell)
$env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "/" 
surreal db start --user root --pass secret

A file: bucket cannot store a content type. From SurrealDB 4.0, a write that names one fails, rather than storing the file without it.

A file key must stay inside the bucket's folder. A key with a . or .. segment, an empty segment such as //, or a trailing / is refused, and so is a key that leaves the folder through a symbolic link.

f"my_bucket:/../notes.txt".put("text");
//- "Operation for bucket `my_bucket` failed: object key '/../notes.txt' contains a path-traversal segment"

f"my_bucket:/drafts//notes.txt".put("text");
//- "Operation for bucket `my_bucket` failed: object key '/drafts//notes.txt' contains an empty segment or a trailing separator"

Cloud object storage backends

Available since: v3.3.0

A bucket can keep its files in Amazon S3 or an S3-compatible service (such as MinIO, Backblaze B2, Wasabi or Cloudflare R2), in Google Cloud Storage, or in Azure Blob Storage. The scheme of the BACKEND URL selects the service:

These backends exist in native builds of SurrealDB only. A WebAssembly build of the engine has the memory backend alone, so a cloud or file: URL fails there as an unsupported backend. The memory and file backends are unchanged.

DEFINE BUCKET checks the URL when it runs, so a malformed URL or an unknown scheme is an error at that point. No request reaches the service until a file operation runs, so a wrong key or a missing bucket shows up there instead.

Amazon S3 and S3-compatible services

s3:/<bucket>[?<parameters>]
s3://[<access_key>:<secret_key>@]<endpoint>[:<port>]/<bucket>[?<parameters>]

The bucket name is the first path segment. The host, when present, is a custom endpoint for an S3-compatible service, and requests to it use path-style addressing. With no host, requests go to the AWS endpoint for the region, unless the server's environment sets AWS_ENDPOINT. Because the host is always an endpoint, s3://my-bucket names an endpoint and no bucket, and fails; for AWS S3 itself, write s3:/my-bucket.

Credentials in the URL take precedence. Without them, the standard AWS credential sources apply: the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables, a web identity token (AWS_WEB_IDENTITY_TOKEN_FILE with AWS_ROLE_ARN), an ECS task role or EKS pod identity, or an EC2 instance profile. The shared credentials file (~/.aws/credentials) is not read.

-- AWS S3, with credentials from the environment or an instance role
DEFINE BUCKET invoices
	BACKEND "s3:/acme-invoices?region=eu-west-2&prefix=surrealdb/invoices";

-- MinIO running locally over plain HTTP, with credentials before the host
DEFINE BUCKET uploads
	BACKEND "s3+http://surrealdb:strongPassword@localhost:9000/uploads";

Google Cloud Storage

gs://<bucket>[?<parameters>]

The bucket name is the host. gcs:// is an alias for gs://.

Without service_account or service_account_key, the standard Google Cloud credential sources apply: the GOOGLE_SERVICE_ACCOUNT, GOOGLE_SERVICE_ACCOUNT_KEY or GOOGLE_APPLICATION_CREDENTIALS environment variables, the application default credentials written by gcloud auth application-default login, or the metadata server on Google Cloud compute.

DEFINE BUCKET reports
	BACKEND "gs://acme-reports?prefix=surrealdb&service_account=/etc/surrealdb/gcs-service-account.json";

Azure Blob Storage

az://<account>/<container>[?<parameters>]

The storage account name is the host and the container name is the first path segment. azure:// is an alias for az://.

Without access_key or sas_token, the standard Azure credential sources apply: the AZURE_STORAGE_ACCOUNT_KEY environment variable, a service principal or workload identity configured through the AZURE_* environment variables, or a managed identity.

DEFINE BUCKET media BACKEND "az://acmemedia/uploads?prefix=surrealdb";

Global backend

The global bucket is one store for the whole server, set with the SURREAL_GLOBAL_BUCKET environment variable, which accepts the same URLs as BACKEND. A bucket uses part of it through a global:// backend followed by a path. The bucket serves the objects under that path, and SurrealDB adds no prefix of its own:

-- The whole global bucket
DEFINE BUCKET everything BACKEND "global://";

-- Only the objects under /app/uploads in the global bucket
DEFINE BUCKET uploads BACKEND "global://app/uploads";

Buckets whose paths overlap see the same objects. With the two buckets above, f"uploads:/a.txt" and f"everything:/app/uploads/a.txt" are the same file.

The global bucket holds the files of every namespace and database, so only a root user can give a bucket a global:// backend, on DEFINE BUCKET and on ALTER BUCKET. A namespace or database user gets the error Only a root user can give a bucket a `global://` backend. A path that contains . or .. as a segment, a backslash or a control character is refused.

When SURREAL_GLOBAL_BUCKET_ENFORCED is true, global:// is the only backend a bucket can have. Namespace and database users then cannot define buckets at all, so a root user defines them.

Buckets defined without a backend

Before SurrealDB 4.0, a DEFINE BUCKET statement could leave out BACKEND, and the bucket then used the global bucket under an automatic prefix. The prefix is made from the internal identifiers of the namespace and the database, which are numbers rather than their names, followed by the name of the bucket. From SurrealDB 4.0, BACKEND is required, and a statement without it fails:

DEFINE BUCKET my_bucket;

'DEFINE BUCKET needs a BACKEND, such as BACKEND "global://" for the global bucket'

A bucket that an earlier release defined this way keeps its files. SurrealDB serves it from the same prefix as before, and INFO FOR DB shows that prefix as an explicit global:// path:

INFO FOR DB.buckets;

{ my_bucket: "DEFINE BUCKET my_bucket BACKEND 'global://1/0/my_bucket' PERMISSIONS FULL" }
``` An [import](/docs/reference/cli/db/commands/import) of an older export that contains such a bucket also still works, and the bucket uses the prefix of the namespace and database it is imported into.

## Setting permissions on buckets

By default, the permissions on a bucket will be set to FULL unless otherwise specified.

```surql
DEFINE BUCKET my_bucket BACKEND "memory";
INFO FOR DB;

{
  accesses: {},
  analyzers: {},
  apis: {},
  buckets: {
    my_bucket: "DEFINE BUCKET my_bucket BACKEND 'memory' PERMISSIONS FULL"
  },
  configs: {},
  functions: {},
  models: {},
  modules: {},
  params: {},
  sequences: {},
  tables: {},
  users: {}
}

You can set permissions on buckets to control who can perform operations on the files stored in them using the PERMISSIONS clause. In the clause three additional variables are available:

• •

  $action: The action to be executed (put, get, head, delete, copy, rename, exists, list, presign_get, presign_put)

• •

  $file: The file pointer of the file to be accessed

• •

  $target: The target file pointer in copy/rename operations

-- Set permissions for the bucket
DEFINE BUCKET admin_bucket BACKEND "memory"
  PERMISSIONS WHERE $auth.admin = true

A bucket's PERMISSIONS clause reads records without checking the caller's permissions, in the same way as a table permission clause. In the example above, $auth.admin resolves even when the record user may not select their own record.

Record users and guests can never list the files in a bucket, whatever its PERMISSIONS clause allows. For them, file::list() fails with an error such as "You don't have permission to list this file in the `my_bucket` bucket".

Permissions for presigned URLs

Available since: v4.0.0

file::presign_get and file::presign_put check the clause with $action set to presign_get or presign_put, which are separate from get and put. A clause that allows get therefore does not allow a URL to be created. The clause runs once, when the URL is created, and not when the URL is used, so a URL keeps working until it expires even if the clause changes. See Presigned URLs.

For these two actions, the clause can also read the options of the URL:

The clause below lets each record user upload to and download from a folder named after their own record ID, through short-lived URLs only, with uploads of PNG and JPEG images up to 5 MiB that cannot replace an existing file.

DEFINE BUCKET avatars BACKEND "memory"
  PERMISSIONS WHERE
       ($action IN ["get", "put"] AND $file.key().starts_with("/u/" + <string>$auth.id + "/"))
    OR ($action = "presign_get"
        AND $file.key().starts_with("/u/" + <string>$auth.id + "/")
        AND $expires <= 15m)
    OR ($action = "presign_put"
        AND $file.key().starts_with("/u/" + <string>$auth.id + "/")
        AND $expires <= 5m
        AND $max_size <= 5242880
        AND $overwrite = false
        AND $content_type IN ["image/png", "image/jpeg"]);

$content_type IN [...] is false when $content_type is NONE, so the clause above also refuses an upload URL that names no type.

Root and namespace buckets

Available since: v4.0.0

A bucket belongs to a database unless the statement says otherwise. ON ROOT defines a bucket at the root, and ON NAMESPACE defines one in the namespace in use. ON DATABASE is the default and can be left out.

A file pointer reaches the buckets at the level of the session that uses it: the database's buckets when a database is selected, otherwise the namespace's buckets when a namespace is selected, and otherwise the root's buckets. A file pointer cannot reach a bucket at another level, so a session with a database selected cannot use a root bucket, even when a bucket of the same name exists at the root.

-- In a root session with no namespace selected
DEFINE BUCKET shared ON ROOT BACKEND "memory";
f"shared:/a.txt".put("root");
<string> f"shared:/a.txt".get();

'root'

INFO FOR ROOT and INFO FOR NS list the buckets of their level under buckets, in the same way as INFO FOR DB:

INFO FOR ROOT.buckets;

{ shared: "DEFINE BUCKET shared ON ROOT BACKEND 'memory' PERMISSIONS FULL" }

ALTER BUCKET and REMOVE BUCKET take the same ON clause, and without it they change or remove a database bucket.

A root or namespace bucket follows these rules:

• •

  Guests and record users can never use a bucket above the database level, whatever its PERMISSIONS clause allows. A system user with a role at the bucket's level or above is judged by that role and the PERMISSIONS clause, as for a database bucket.

• •

  Presigned URLs can be created only for database buckets, because a signature belongs to one namespace and database.

• •

  A root bucket over the global bucket, such as DEFINE BUCKET cloud ON ROOT BACKEND "global://", gives a root session access to the whole global bucket.