# SurrealDB documentation - full corpus > The complete SurrealDB documentation as a single markdown document. Each page starts with a "Source:" line naming its canonical URL. The page index is at https://surrealdb.com/docs/llms.txt, and every page is also available individually by appending `.md` to its URL. In code examples, a fenced block whose title contains "output" holds the result of the block above it rather than runnable code, and a qualifier such as "Sample output" or "Possible output" means the value shown is one of several possible ones. Inside a block, a "//-" comment marks a value the statement above it returned, while "--" is an ordinary comment. --- Source: https://surrealdb.com/docs/build/ai-agents # AI agents MCP servers, Agent Skills, and framework integrations for SurrealDB. The ways SurrealDB fits into AI tooling, and what it gives an agent. SurrealDB connects to AI tooling in three places: the assistant you write code with, the framework your application is built on, and the database your agent stores its knowledge in. This page covers each one and where to go next. ## Ways to integrate **[MCP](/docs/build/ai-agents/mcp.md)** - Give an AI tool a set of tools to call. The hosted [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) connects Claude, Cursor, and other clients to your SurrealDB Cloud account with one URL, so an assistant can deploy instances, query their data, and read logs while you work. A database you run yourself publishes the same data tools through [Embedded MCP](/docs/build/ai-agents/mcp/embedded.md). _(since v3.1.0)_ **[Agent Skills](/docs/build/ai-agents/agent-skills.md)** - Install packaged knowledge into a coding agent. The official skills cover SurrealQL, vector search, and the Python SDK, so generated queries match how SurrealDB actually behaves instead of guessing at an API. **[AI frameworks](/docs/build/ai-agents/ai-frameworks.md)** - Use SurrealDB from the agent library you already have. LangChain, LlamaIndex, CrewAI, PydanticAI, Agno, and others connect through maintained integrations for vector stores, memory, and retrieval. **[SurrealDB Agent Memory](/docs/agent-memory.md)** - A memory and knowledge layer for agents, running in front of SurrealDB. Use it when an agent needs to remember across sessions, and to keep straight what was said, what is true now, and what used to be true, without designing that model yourself. ## What the database gives an agent An agent needs somewhere to keep state, a way to query it that fits the question being asked, and retrieval fast enough to answer in the loop. SurrealDB does all three in one engine, so you are not keeping a document store, a graph database, and a vector index in sync. **Memory** - Store conversation context, artefacts, and entity records as documents, and model the relationships between users, tasks, tools, and outcomes as a graph. Agents can then traverse those connections rather than re-reading everything. Partition memory per tenant or per session while keeping one query model. **Retrieval** - Combine structured filters with [vector indexes and similarity search](/docs/learn/data-models/vector-search/vector-indexes.md) so a lookup can be semantic, exact, or both. That covers RAG and hybrid retrieval without a second system. **Knowledge graphs** - Follow links between concepts, permissions, and resources inside the database, so a question can reach supporting facts in a few hops instead of several round-trips through application code. **Tools the agent can call** - Express logic in SurrealQL, including [database functions](/docs/reference/query-language/functions/database-functions.md) for reusable server-side behaviour and [HTTP functions](/docs/reference/query-language/functions/database-functions/http.md) for reaching external APIs from the query layer. Because schemas and indexes can change as your workflows change, the same deployment serves low-latency reads for live agents and batch jobs for backfills or evaluation. ## Next steps - [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) - connect your AI tool to SurrealDB Cloud - [Agent Skills](/docs/build/ai-agents/agent-skills.md) - install SurrealQL and SDK knowledge into a coding agent - [AI frameworks overview](/docs/build/integrations/ai-frameworks/overview.md) - compare the framework integrations - [Build an AI agent](/docs/explore/tutorials/tutorials/build-an-ai-agent.md) - a worked example, end to end --- Source: https://surrealdb.com/docs/build/ai-agents/agent-skills # Agent Skills Official SurrealDB agent skills for use in agentic coding workflows. AI coding agents perform best when they have the right context for the job. Each Agent Skill is a self-contained package of instructions, reference material, and resources that an agent can pick up and apply whenever the task calls for it. SurrealDB publishes a set of official skills that give your agent deep knowledge of SurrealQL, vector search, the Python SDK, and more. Built on the [Agent Skills](https://agentskills.io/) open standard, they work across all major coding agents including Claude Code, GitHub Copilot, Cursor, Cline, and many others. ## Installation ### Using the Skills CLI The quickest way to get started is to install every SurrealDB skill in one go: ```bash npx skills add surrealdb/agent-skills ``` If you only need a subset, pick the ones relevant to your project: ```bash npx skills add surrealdb/agent-skills --skill surrealql ``` ### Manual setup Alternatively, clone the repository and copy the skills into your agent's context directory: ```bash git clone https://github.com/surrealdb/agent-skills.git ``` After cloning, move the `skills/` folder to wherever your coding agent looks for context files. Check your agent's documentation for the exact path. ## Available skills ### SurrealQL A comprehensive reference for the SurrealQL query language, including syntax, schema design, graph traversals, and idiomatic patterns. ```bash npx skills add surrealdb/agent-skills --skill surrealql ``` > [!NOTE] > Useful for: > · Authoring SurrealQL queries > · Defining and maintaining schemas > · Navigating graph relationships and working with record IDs > · Transitioning from traditional SQL to SurrealQL > · Configuring live queries for real-time data ### SurrealDB Vector Covers vector search in SurrealDB, including creating HNSW indexes, running KNN queries, and applying similarity scoring. ```bash npx skills add surrealdb/agent-skills --skill surrealdb-vector ``` > [!NOTE] > Useful for: > · Setting up HNSW vector indexes on tables > · Running KNN queries with distance operators > · Implementing semantic search, RAG pipelines, or recommendation engines > · Fine-tuning HNSW parameters (EFC, M, M0, distance function, type) ### SurrealDB Python Guidance for using the SurrealDB Python SDK in both client/server mode (WebSocket) and embedded mode (in-memory or file-based). ```bash npx skills add surrealdb/agent-skills --skill surrealdb-python ``` > [!NOTE] > Useful for: > · Connecting to SurrealDB from Python applications > · Working with the `surrealdb` package (synchronous and asynchronous) > · Running SurrealDB embedded in Python without a separate server > · Performing CRUD operations from Python code ## Discovering community skills The [skills.sh directory](https://skills.sh) catalogues skills published by the wider community. You can also search directly from the command line: ```bash npx skills find QUERY ``` ## Further reading - [SurrealDB Agent Skills](https://github.com/surrealdb/agent-skills) - [Agent Skills documentation](https://agentskills.io/home) --- Source: https://surrealdb.com/docs/build/ai-agents/ai-frameworks # AI frameworks Framework integrations for AI and machine learning libraries. Use SurrealDB for vectors, memory, and pipelines. SurrealDB integrates with widely used AI and machine learning frameworks so you can use a single database for features, training or evaluation pipelines, and agent orchestration. Each integration page explains setup, typical patterns, and links to upstream documentation where helpful. Available integrations: - [Agno](/docs/build/integrations/ai-frameworks/agno.md) - [CAMEL](/docs/build/integrations/ai-frameworks/camel.md) - [CrewAI](/docs/build/integrations/ai-frameworks/crewai.md) - [Dagster](/docs/build/integrations/ai-frameworks/dagster.md) - [Google Agent Development Kit](/docs/build/integrations/ai-frameworks/google-agent.md) - [Hermes](/docs/build/integrations/ai-frameworks/hermes.md) - [LangChain](/docs/build/integrations/ai-frameworks/langchain.md) - [LlamaIndex](/docs/build/integrations/ai-frameworks/llamaindex.md) - [PydanticAI](/docs/build/integrations/ai-frameworks/pydantic-ai.md) - [smolagents](/docs/build/integrations/ai-frameworks/smolagents.md) For a comparative table and links to deeper guides, start from the [AI frameworks overview](/docs/build/integrations/ai-frameworks/overview.md). --- Source: https://surrealdb.com/docs/build/ai-agents/mcp # SurrealDB MCP Server Connect Claude, Cursor, and other AI tools to SurrealDB Cloud. One URL is all the configuration needs. The SurrealDB MCP Server connects AI tools to your SurrealDB Cloud account. Add one URL to Claude, Cursor, or any other MCP client, sign in with your Surreal ID, and your assistant can work on your databases alongside you: deploy an instance, query the data inside it, look into why something is slow, and keep track of what it all costs. ```text https://mcp.surrealdb.com ``` The server is hosted for you, so there is nothing to install and nothing to keep running. It acts as you: it sees only the organisations you belong to, respects the role you hold in each one, and asks you to confirm anything that cannot be undone. > [!NOTE] > This page covers the hosted server for SurrealDB Cloud. To give an assistant tools against a database you run yourself, see [Embedded MCP](/docs/build/ai-agents/mcp/embedded.md). ## What is MCP? [Model Context Protocol](https://modelcontextprotocol.io) is an open standard for connecting AI assistants to outside tools and data. A server publishes a set of tools, and a client such as Claude or Cursor calls them on your behalf. Instead of copying a connection string into a chat window and pasting query results back out, your assistant works with SurrealDB directly. ## What you can do - **Deploy and manage instances** - create an instance, pause one you are not using, resize it as traffic grows, or upgrade it to a newer version of SurrealDB. - **Work with your data** - ask a question in plain language and let the assistant write and run the SurrealQL, or have it create tables and seed records while you build. - **Look into problems** - pull up instance status, metrics, and logs next to the query you were just writing. - **Manage your team** - invite people to an organisation, change what they can do, and withdraw invitations. - **Keep an eye on cost** - check usage and invoices, and estimate what an instance will cost before you create it. - **Give agents long-term memory** - create and manage [SurrealDB Agent Memory](/docs/agent-memory.md) contexts, then store and recall memories from them. ## Connect your AI tool Most clients need nothing but the URL. Follow the guide for your tool: - [MCP in Claude](/docs/build/ai-agents/mcp/claude.md) - [MCP in Cursor](/docs/build/ai-agents/mcp/cursor.md) For anything else, add `https://mcp.surrealdb.com` as a remote MCP server. The usual shape is: ```json { "mcpServers": { "surrealdb": { "url": "https://mcp.surrealdb.com" } } } ``` Some clients differ: VS Code uses a `servers` object with `"type": "http"`, and Windsurf and Antigravity use `serverUrl` in place of `url`. Once the server is connected, ask your assistant to list your organisations. If it comes back with them, you are set up. ## Signing in ### With your Surreal ID If your client supports sign-in, this is all it takes: the client opens a browser window, you sign in with the same Surreal ID you use for SurrealDB Cloud, and you approve the connection. No credentials go into a config file, and you can disconnect from the client whenever you like. A signed-in connection can use every tool the server offers, limited only by your role in each organisation. ### With a personal access token Use a token when your client cannot open a browser, or when something runs unattended. Create one in the [account portal](https://account.surrealdb.com/tokens), tick the permissions it should carry, and pass it to your client as a header: ```json { "mcpServers": { "surrealdb": { "url": "https://mcp.surrealdb.com", "headers": { "Authorization": "Bearer " } } } } ``` A token carries only the permissions you selected when you created it: | Permission | Lets your assistant | | --- | --- | | `read:cloud` | See your organisations, instances, SurrealDB Agent Memory contexts, usage, and logs | | `write:cloud-instances` | Deploy, resize, pause, upgrade, and delete instances | | `query:cloud-instances` | Read and write the data inside your instances | | `write:cloud-organization` | Create organisations, and manage members, roles, and invitations | | `write:cloud-billing` | Update billing details and manage SurrealDB Agent Memory plans | | `write:cloud-spectron` | Create SurrealDB Agent Memory contexts and manage who may use them | | `query:spectron-contexts` | Store and recall agent memory in a SurrealDB Agent Memory context | Include `read:cloud` alongside any of the others. Without it your assistant cannot look up which organisation or instance to act on, so the write permissions have nothing to work with. > [!IMPORTANT] > A personal access token stands in for your whole account and does not expire on its own. Give it only the permissions the job needs, keep it out of files you commit, and delete it in the account portal when you are finished with it. ## Available tools You never call the tools by hand. Your assistant picks what it needs, and most clients show you the call before it runs. Read-only tools are marked as safe, so a client can approve them without asking you every time. | Group | What your assistant can do | | --- | --- | | **Profile and organisations** | Check who you are signed in as, list the organisations you can work in, and create or rename one | | **Members and invitations** | See who has access to an organisation, invite someone, change their role, or withdraw an invitation | | **Instances** | List instances and their details, deploy a new one, pause, resume, resize, upgrade, adjust backups, or delete | | **Instance data** | Run SurrealQL and record operations inside a running instance | | **Monitoring and usage** | Read an instance's status, metrics, logs, and usage | | **Billing** | Fill in billing details, check whether an organisation can deploy yet, and read usage and invoices | | **Catalogue** | Look up the regions, instance types, and SurrealDB versions available to you | | **Terms** | Fetch the SurrealDB Cloud terms and record your acceptance | | **SurrealDB Agent Memory contexts** | Create and configure memory contexts, and control who may use them | | **SurrealDB Agent Memory** | Store, search, recall, and reflect on what an agent has learned | The regions, instance types, and versions are also offered as MCP resources, for clients that prefer them to tool calls. Clients with a prompt menu get a **Deploy a Cloud Instance** wizard, which walks through the choices, prices the result, and waits for your go-ahead. To see the exact tool surface your client has, ask your assistant which SurrealDB tools it can use. ## Working with your data The same connection that manages an instance can also query it. Ask a question, and your assistant chooses the namespace and database, writes the SurrealQL, and shows you what came back. [Example usages](/docs/build/ai-agents/mcp/examples.md) has prompts to try. Two things have to be true first. The instance must be running, so resume it if you paused it. And it must be on **SurrealDB 3.1 or later**, the version that began answering these calls. If it is older, ask your assistant to upgrade it. Inside an instance, these are the tools your assistant works with: | Tool | What it does | | --- | --- | | `use` | Choose the namespace and database to work in | | `query` | Run SurrealQL | | `select`, `create`, `insert`, `upsert`, `update`, `delete`, `relate` | Read and change records without writing a full statement | | `run` | Call a database function | | `list` | List namespaces, databases, tables, indexes, and users | | `info` | Describe the schema, or the engine itself | A database you run yourself publishes the same set. See [Embedded MCP](/docs/build/ai-agents/mcp/embedded.md) for how they behave there. SurrealDB Agent Memory contexts work the same way. Your assistant can `remember`, `recall`, `reflect`, `forget`, `upload`, and `inspect` against a context you own, described in the [SurrealDB Agent Memory MCP tools reference](/docs/agent-memory/integrations/mcp-server/tools-reference.md). ## Deploying an instance Creating an instance costs money, so the server never quietly does it. Your assistant first checks whether the organisation is ready to deploy, and reports anything standing in the way: no free instances left on the plan, billing details missing, or no payment method on file. Clearing those blockers is a conversation. The assistant asks for the billing contact and address it needs, then hands you a secure checkout link so you can add a card yourself. Card details never pass through the assistant. Once the payment goes through, it can check readiness again and deploy. Ask for an estimate before it creates anything. The assistant can price a configuration, show you the monthly cost, and wait for you to agree. ## What the server will not do - **Delete an instance on a guess.** Deletion is permanent and takes the instance's backups with it, so the assistant has to name the instance exactly as it stands right now. A stale or mistaken name is refused. - **Handle your card.** Payment details are only ever entered by you, on a secure checkout page. No tool accepts a card number. - **Accept terms for you.** Your assistant can fetch the documents and link you to them; the agreement is yours to give. - **Show a secret twice.** New SurrealDB Agent Memory keys and access tokens appear once, when they are created. Store them somewhere safe there and then. Clients often let you auto-approve tools to save clicks. Reserve that for the read-only ones, and keep the confirmation step on anything that changes or removes something. ## Troubleshooting | What you see | What to do | | --- | --- | | Your client says the server needs authentication | Sign in again from the client, or check the personal access token still exists in the [account portal](https://account.surrealdb.com/tokens) | | A tool reports a missing permission | Your token was created without it. Permissions are fixed once a token exists, so create a new token with that box ticked | | Sign-in is refused because your email is not verified | Verify the email address on your Surreal ID, then sign in again | | Your assistant loses the connection after a while | Idle connections are dropped. Reconnect in the client and carry on | | An instance cannot be queried | It is paused or still starting. Wait until it reports as ready, or ask your assistant to resume it | | An instance says it needs a newer version | Data tools need SurrealDB 3.1 or later. Ask your assistant to upgrade the instance | | Your assistant cannot find an organisation you named | It is looking for the organisation's identifier rather than its display name. Ask it to list your organisations first | ## Next steps - [MCP in Claude](/docs/build/ai-agents/mcp/claude.md) - set it up in Claude Code, Claude Desktop, or the Claude app - [MCP in Cursor](/docs/build/ai-agents/mcp/cursor.md) - set it up in Cursor - [Example usages](/docs/build/ai-agents/mcp/examples.md) - prompts that show what the server can do - [Embedded MCP](/docs/build/ai-agents/mcp/embedded.md) - the MCP server inside SurrealDB, for databases you run yourself - [Agent Skills](/docs/build/ai-agents/agent-skills.md) - teach your assistant SurrealQL and SDK patterns to go with these tools --- Source: https://surrealdb.com/docs/build/ai-agents/mcp/claude # MCP in Claude Add the SurrealDB MCP Server to Claude Code, Claude Desktop, or the Claude app. Claude can reach your SurrealDB Cloud account through the [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md). Setup takes a minute, and you sign in with your Surreal ID in the browser, so there are no credentials to paste. ## Claude Code Add the server: ```bash claude mcp add --transport http surrealdb https://mcp.surrealdb.com ``` Run `/mcp` inside Claude Code, choose **surrealdb**, and authenticate. A browser window opens for you to sign in and approve the connection. Check it worked: ```bash claude mcp list ``` **surrealdb** should be listed as connected. Ask Claude which SurrealDB organisations it can see, and it should answer with yours. To connect it for one project rather than everywhere, add `--scope project`. The entry is written to `.mcp.json` in the repository, which you can commit for your team. Everyone still signs in as themselves. ## Claude Desktop and the Claude app Open **Settings → Connectors → Add custom connector** and fill in: | Field | Value | | --- | --- | | Name | `SurrealDB` | | URL | `https://mcp.surrealdb.com` | Claude opens a browser window for you to sign in. Once it closes, start a new conversation and open the tools menu in the composer. The SurrealDB tools appear there. ## Using a personal access token instead If you would rather not sign in through the browser, or you are running Claude somewhere without one, create a token in the [account portal](https://account.surrealdb.com/tokens) and pass it as a header: ```bash claude mcp add --transport http surrealdb https://mcp.surrealdb.com \ --header "Authorization: Bearer " ``` Tick only the permissions the work needs. [Signing in](/docs/build/ai-agents/mcp.md#signing-in) lists what each one allows. ## Try it > Show me the SurrealDB instances in my organisation, and tell me which of them are paused. Claude lists your organisations, picks the one you meant, and reports each instance with its state. [Example usages](/docs/build/ai-agents/mcp/examples.md) has more to try, including deploying an instance and querying its data. ## Removing it ```bash claude mcp remove surrealdb ``` In Claude Desktop and the Claude app, open **Settings → Connectors** and remove the SurrealDB connector. ## Troubleshooting | What you see | What to do | | --- | --- | | `claude mcp list` shows the server as failed | Run `/mcp` and authenticate. Until you sign in, only the sign-in tool works | | The browser window never opens | Sign in from `/mcp` again, and check nothing is blocking pop-ups | | Claude cannot see the tools in Claude Desktop | Quit and reopen the app completely, then start a new conversation | | A tool reports a missing permission | You are using a personal access token created without it. Create a new token with that box ticked | --- Source: https://surrealdb.com/docs/build/ai-agents/mcp/cursor # MCP in Cursor Add the SurrealDB MCP Server to Cursor and sign in with your Surreal ID. With the [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) connected, Cursor's agent can reach your SurrealDB Cloud account while you work: check what is deployed, query a database, and read logs without leaving the editor. ## Add the server Cursor reads `~/.cursor/mcp.json` for every project, or `.cursor/mcp.json` for one project. Add a `surrealdb` entry: ```json { "mcpServers": { "surrealdb": { "url": "https://mcp.surrealdb.com" } } } ``` If the file already lists other servers, add this entry alongside them. ## Sign in Open **Settings → MCP**, find **surrealdb**, and click **Connect**. Cursor opens a browser window where you sign in with your Surreal ID and approve the connection. The indicator beside the server turns green when it is ready. Nothing about your account is stored in `mcp.json`, so a project-level file is safe to commit. Everyone who opens the project signs in as themselves. ## Check it worked In the chat panel, ask: > Which SurrealDB Cloud organisations can you see? Cursor should answer with your organisations. To see the whole tool surface, ask which SurrealDB tools it has available. ## Using a personal access token instead For a machine that cannot open a browser, create a token in the [account portal](https://account.surrealdb.com/tokens) and add it as a header: ```json { "mcpServers": { "surrealdb": { "url": "https://mcp.surrealdb.com", "headers": { "Authorization": "Bearer " } } } } ``` Give the token only the permissions you want the agent to have. [Signing in](/docs/build/ai-agents/mcp.md#signing-in) lists what each one allows. > [!WARNING] > A token in a project-level `.cursor/mcp.json` will be committed with the repository. Keep tokens in the global `~/.cursor/mcp.json` instead. ## Try it > Create a `task` table in my dev instance with a title and a done flag, insert two rows, then show me the ones that are not done. Cursor picks the instance, sets the namespace and database, and runs the SurrealQL, showing you each call as it goes. [Example usages](/docs/build/ai-agents/mcp/examples.md) has more, including deploying an instance and investigating a slow query. ## Removing it Delete the `surrealdb` entry from `mcp.json`. Cursor stops offering the tools in new sessions. ## Troubleshooting | What you see | What to do | | --- | --- | | The indicator stays red | Click **Connect** in **Settings → MCP** and complete the sign-in. Check the URL has no trailing path | | Tools do not appear in chat | Reload the window after editing `mcp.json` | | The agent asks you to authenticate on every request | Sign in through **Settings → MCP** rather than answering in chat, so Cursor stores the connection | | A tool reports a missing permission | You are using a personal access token created without it. Create a new token with that box ticked | --- Source: https://surrealdb.com/docs/build/ai-agents/mcp/embedded # Embedded MCP Connect AI agents and editors to a SurrealDB server you run yourself, through the built-in MCP server over HTTP or stdio. _(since v3.1.0)_ SurrealDB ships a built-in [Model Context Protocol](https://modelcontextprotocol.io) server, so agents and editors can list schema, run SurrealQL, and change records through a standard set of tools. The same access control applies as on [`/sql`](/docs/reference/rest-api/http-protocol.md#sql) and RPC: `DEFINE USER` permissions, table `PERMISSIONS`, and server capability flags all decide what a tool can do. This page covers the server inside the SurrealDB binary, for databases you run yourself. For SurrealDB Cloud, the hosted [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) reaches the same data tools remotely and adds organisation, instance, and billing management on top. Both publish the **same data tools**. What differs is how your editor connects and who shares the database. ## When to use `surreal mcp` vs `surreal start` | | **`surreal mcp`** (stdio) | **`surreal start`** + **`/mcp`** (HTTP) | | --- | --- | --- | | **How it works** | Your editor spawns SurrealDB as a child process; MCP runs over stdin and stdout | You run a server; the editor connects to `http://…/mcp` | | **Database** | Embedded in the MCP process (default `memory`, or a local file path) | The same instance your app, CLI, or SurrealDB Studio uses | | **Authentication** | Owner-level access on every tool call - no login step | Normal SurrealDB auth (Bearer JWT, HTTP Basic, …) | | **Best for** | Learning MCP, solo local development, quickest editor setup | Shared databases, teams, remote instances, production patterns | | **Editor config** | `command` and `args` in MCP settings | `url` and auth headers | **Use `surreal mcp`** when you want the lowest-friction path on a machine you trust: paste a config, restart the editor, and experiment. Think of it as a self-contained database for your assistant. **Use `surreal start` and `/mcp`** when the agent should work against a database that already exists, or one that other clients share. That is the right model for least-privilege users, TLS, audit logging, and anything beyond trusted solo development. > [!TIP] > New to MCP? Start with `surreal mcp` in your coding assistant, then move to HTTP once you want the agent on the same instance as your application. ## Transports ### HTTP (`/mcp`) When you run `surreal start`, the server exposes **`POST /mcp`** on the same bind address as the REST API. Authenticate with the same headers you use elsewhere, for example `Authorization: Bearer ` or HTTP Basic. ```bash surreal start --user root --pass secret --bind 127.0.0.1:8000 memory # MCP endpoint: http://127.0.0.1:8000/mcp ``` For a non-loopback hostname (a public FQDN, a Kubernetes service name, or a load-balancer host), the transport rejects the request with `403 Forbidden: Host header is not allowed` until you opt in. Set `SURREAL_MCP_ALLOWED_HOSTS` to your hostnames, or `SURREAL_MCP_ALLOW_ALL_HOSTS=true` behind a trusted proxy. See [Configuration](#configuration). > [!WARNING] > Run `/mcp` behind TLS in production. The session header acts like a bearer token for the life of the session: anyone who holds it can repeat tool calls as the same user until the session expires, five minutes after the last request by default. ### Stdio (`surreal mcp`) For local editor integrations, use the dedicated subcommand. It runs the MCP server in the same process as an embedded datastore: ```bash surreal mcp --user root --pass secret --ns main --db main memory ``` Every tool call over stdio runs with owner-level access. There is no network handshake to attach credentials to, so there is nothing to narrow the permissions with. **Do not expose this entry point to untrusted users.** See [`surreal mcp`](/docs/reference/cli/surrealdb-cli/commands/mcp.md) for its flags and environment variables, including `SURREAL_MCP_NS`, `SURREAL_MCP_DB`, and the shared `SURREAL_MCP_*` limits below. ## Connect an editor For stdio, the editor launches SurrealDB itself, so the config is a command and its arguments. **Cursor** `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project): ```json { "mcpServers": { "surrealdb": { "command": "surreal", "args": ["mcp", "--user", "root", "--pass", "secret", "--ns", "main", "--db", "main", "memory"] } } } ``` Restart Cursor, then check that **Settings → MCP** shows `surrealdb` as connected. **VS Code** `.vscode/mcp.json` uses a `servers` object and a `type` field: ```json { "servers": { "surrealdb": { "type": "stdio", "command": "surreal", "args": ["mcp", "--user", "root", "--pass", "secret", "--ns", "main", "--db", "main", "memory"] } } } ``` Reload the window after editing the file, and enable agent mode in your MCP-capable extension. **Claude Desktop** `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, or `%APPDATA%\Claude\claude_desktop_config.json` on Windows: ```json { "mcpServers": { "surrealdb": { "command": "surreal", "args": ["mcp", "--user", "root", "--pass", "secret", "--ns", "main", "--db", "main", "memory"] } } } ``` Quit and reopen Claude Desktop completely, then start a new conversation. If the editor cannot find the binary, use the full path from `which surreal` (macOS and Linux) or `where surreal` (Windows) as `command`. > [!NOTE] > `--user` and `--pass` create the root user on a **new** datastore, the same as [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md). They do not authenticate each MCP call, because stdio runs every call as owner. Prefer an `env` block (`SURREAL_USER`, `SURREAL_PASS`) over literals in `args` so credentials stay out of committed config. To keep data between sessions, replace `memory` with a [file-backed path](/docs/reference/cli/surrealdb-cli/commands/start.md#datastore-configuration) such as `rocksdb://tmp/surreal-mcp`. For HTTP, point the editor at a server you are already running and attach credentials as headers: ```json { "mcpServers": { "surrealdb": { "url": "http://127.0.0.1:8000/mcp", "headers": { "Authorization": "Basic " } } } } ``` Once connected, ask the assistant which SurrealDB tools it has. It should list `query`, `select`, `use`, and the rest of the set below. ## Published tools `tools/list` publishes these tools, and the names are stable: | Tool | Purpose | | --- | --- | | `query` | Run SurrealQL and return serialised results | | `gql` | Run an [ISO GQL](/docs/learn/querying/gql/overview.md) query (on by default from 3.3.0; on 3.2.x needs `--allow-experimental gql`) | | `graphql` | Run a [GraphQL](/docs/learn/querying/graphql/overview.md) query against the configured schema | | `select`, `create`, `insert`, `upsert`, `update`, `delete`, `relate` | Data manipulation helpers | | `run` | Call a database function with typed arguments | | `list` | List namespaces, databases, tables, indexes, users, … | | `use` | Select the namespace and database context | | `info` | Schema or engine information for a scope | Legacy names such as `list_tables`, `use_database`, and `version` are no longer published. Use `list` and `use` instead. When an agent inspects the schema with `info` or `list`, any [`COMMENT`](/docs/reference/query-language/statements/define/overview.md#comments-on-definitions) on tables and fields is included. Put query-relevant detail in those comments (record-ID conventions, how to compare a field, graph paths) so the agent does not have to guess from names alone. Each tool also carries annotation hints that clients use to decide what to auto-approve. `gql` is annotated as neither read-only nor safe, because the GQL dialect parses `INSERT`, `SET`, `REMOVE`, and `DELETE` as well as reads. > [!NOTE] > Before SurrealDB 3.3.0, `gql` was annotated as read-only and non-destructive, so a client could auto-approve a GQL statement that modified data. ## Protocol versions _(since v3.3.0)_ The server states which MCP specification revisions it implements rather than inheriting them from the SDK it links against, and it advertises **`2026-07-28`**. An unknown version in a request degrades to that revision. `2026-07-28` removes the `initialize` handshake and protocol sessions: a request carries its own protocol version, identity, and capabilities, and anything spanning requests must be named on each one. Earlier handshake-based revisions are served on the same endpoint, so clients built against them keep working unchanged. ### Selecting a namespace and database per call Because a stateless request has no session to hold a selection, every tool except `use` takes optional `namespace` and `database` arguments. Resolution takes the first value it finds, filling each part independently: 1. The tool call's own `namespace` and `database` arguments 2. The `surreal-ns` and `surreal-db` request headers 3. The handshake session's `use` selection, on the revisions that still have one 4. The server's configured defaults A call may therefore override the database while inheriting the namespace. A request that names its own scope never changes the connection's state, so the override cannot leak into a later call. `use` stays in `tools/list` under every revision so the tool surface does not vary by protocol version. Under `2026-07-28` it returns an error pointing at the per-call arguments instead of selecting anything. > [!NOTE] > The server also identifies itself to clients as `surrealdb`. Before 3.3.0 it reported the name of the MCP SDK crate it was built with. ## Configuration MCP-specific limits are read once from the environment, with the prefix `SURREAL_MCP_`. HTTP body size uses the server-wide cap. | Variable | Default | Effect | | --- | --- | --- | | `SURREAL_MCP_QUERY_TIMEOUT_SECS` | 60 | Outer timeout on each tool execution (`0` disables) | | `SURREAL_MCP_MAX_RESULT_BYTES` | 256 KiB | Cap on serialised tool output (`0` disables) | | `SURREAL_MCP_RUN_MAX_ARGS` | 64 | Maximum arguments to `run` | | `SURREAL_MCP_PARAMS_MAX_KEYS` | 256 | Maximum top-level keys in parameter objects | | `SURREAL_MCP_PARAMS_MAX_QL_BYTES` | 4 KiB | Maximum byte length of a `$ql` string inside a `*_data` payload | | `SURREAL_MCP_SCHEMA_RESOURCE_MAX_TABLES` | 200 | Cap on tables enriched in the database schema resource | | `SURREAL_MCP_ALLOWED_HOSTS` _(since v3.2.1)_ | loopback only | Exact `Host` values accepted for HTTP `/mcp` (replaces the loopback default) | | `SURREAL_MCP_ALLOW_ALL_HOSTS` _(since v3.2.1)_ | `false` | Accept any `Host` (trusted-proxy escape hatch; overrides the allowlist) | | `SURREAL_HTTP_MAX_MCP_BODY_SIZE` | 4 MiB | Maximum HTTP body size for `/mcp` | Full tables live under [Environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md). [Observability metrics](/docs/manage/observability/metrics.md) include `surrealdb.mcp.*` counters and histograms from 3.1.0. ## Security checklist - Prefer a least-privilege `DEFINE USER` (for example a custom role with table-level `PERMISSIONS`) over root credentials for agent clients. - Lock down capabilities (`--deny-funcs`, `--allow-net`, …) so a hijacked session cannot reach `http::*` or other high-risk functions. - Set `--allow-origin` explicitly for browser-based MCP clients, and avoid `*` in production. - On a public hostname, set `SURREAL_MCP_ALLOWED_HOSTS`, or `SURREAL_MCP_ALLOW_ALL_HOSTS` only behind a trusted proxy. The default allowlist is loopback-only. - Forward the `surrealdb::mcp::audit` tracing target to your SIEM. Audit records include the tool name, subject, namespace, database, and outcome, but never query text or row payloads. ## Troubleshooting | Symptom | Things to check | | --- | --- | | The editor shows the server as disconnected | Is `surreal` on your `PATH`? Use the full path as `command`, or run the same command in a terminal to see the error | | The tools list is empty | Restart the editor after editing MCP config, and confirm the binary is 3.1 or later | | `403 Forbidden: Host header is not allowed` | HTTP `/mcp` accepts loopback hosts by default. Set `SURREAL_MCP_ALLOWED_HOSTS` to your hostname, or `SURREAL_MCP_ALLOW_ALL_HOSTS=true` behind a trusted proxy. Needs 3.2.1 or later | | A tool call times out | The default timeout is 60 seconds (`SURREAL_MCP_QUERY_TIMEOUT_SECS`). Narrow the query | | A result comes back truncated | Output is capped at 256 KiB by default. Paginate or aggregate in SurrealQL | | Permission errors over HTTP | The user lacks rights for the operation. Sign in with different credentials, or adjust `DEFINE USER` and table `PERMISSIONS` | ## Next steps - [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) - the hosted server for SurrealDB Cloud instances - [Example usages](/docs/build/ai-agents/mcp/examples.md) - prompts to try once an assistant is connected - [`surreal mcp` CLI reference](/docs/reference/cli/surrealdb-cli/commands/mcp.md) - flags and environment variables - [Agent Skills](/docs/build/ai-agents/agent-skills.md) - installable SurrealQL and SDK skills for coding agents - [AI frameworks](/docs/build/integrations/ai-frameworks/overview.md) - using SurrealDB from LangChain, CrewAI, and others --- Source: https://surrealdb.com/docs/build/ai-agents/mcp/examples # Example usages Prompts that show what an assistant can do with the SurrealDB MCP Server, from deploying an instance to querying its data. These are prompts to try once you have connected the [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) to your AI tool. Each one explains what the assistant does behind the prompt, so you know what to expect before you ask. ## See what you have > What SurrealDB instances do I have, and which ones are paused? The assistant lists your organisations, picks the one you meant or asks which, and reports every instance with its region, size, version, and state. A good first prompt, because it confirms the connection is working. > Which of my instances is on an old version of SurrealDB? It reads the version of each instance and compares them against the versions available to deploy. ## Deploy an instance > I need a small instance in Frankfurt for a side project. What would it cost, and can my organisation deploy it? The assistant looks up the regions and instance types available to you, puts a configuration together, prices it, and checks whether the organisation can deploy at all: free instances left on the plan, billing details filled in, payment method on file. You get the cost and any blockers before anything is created. > Go ahead and deploy it, and tell me how to connect once it is up. Only after you say so does it create the instance. It then reports the new instance and how to reach it. If billing is not set up yet, it asks for the contact details it needs and hands you a secure checkout link to add a card yourself. Clients with a prompt menu also offer a **Deploy a Cloud Instance** wizard, which walks the same path step by step. ## Query and shape your data > In my dev instance, create a `task` table with a title and a done flag, add a couple of rows, then show me the ones that are not done. The assistant selects the namespace and database, writes the SurrealQL, and shows you both the statements and the results. The instance has to be running and on SurrealDB 3.1 or later. > How many orders did we take last week, grouped by status? It inspects the schema to find the right table and fields, then writes the aggregate query rather than asking you for the shape of your data. > Something looks wrong with the `customer` table. What does its schema actually say? It describes the table, its fields, and its indexes, which is often faster than opening a dashboard to check a definition. If you have put operational detail in [`COMMENT`](/docs/reference/query-language/statements/define/overview.md#comments-on-definitions) clauses, that text comes back with the schema and steers the next query. ## Investigate a problem > My production instance felt slow this morning. Have a look and tell me what you find. The assistant checks the instance's status, pulls metrics for the window you describe, and reads the logs around it. Because it can also query the database, it can follow a hunch about a missing index in the same conversation. > Compare CPU and memory over the last day against the same period last week. It requests both windows and summarises the difference. ## Manage your team > Invite alex@example.com to the Acme organisation as a developer, and show me who else has access. It lists the roles the organisation offers, sends the invitation with the role you chose, and reports current members along with any invitations still outstanding. ## Keep costs in check > What have we spent on SurrealDB this month, and which instance accounts for most of it? The assistant reads usage for the organisation and for each instance, and pulls up recent invoices. > Pause anything in the sandbox organisation that nobody has touched this week. It works out which instances are idle from their usage, tells you what it plans to pause, and waits for you to confirm. Paused instances keep their data. ## Give an agent long-term memory > Set up a SurrealDB Agent Memory context for this project in Frankfurt, then remember that we settled on event sourcing for the orders service. The assistant creates the [SurrealDB Agent Memory](/docs/agent-memory.md) context, registers a scope path for the project, and stores the decision in it. > What do we already know about the orders service? It recalls what has been stored for that project and answers from it, so a decision made weeks ago still turns up in today's conversation. ## Getting better results - **Name the instance or organisation** when you have more than one. Otherwise the assistant has to stop and ask. - **Ask for a plan first** on anything expensive or irreversible: "tell me what you would change, then wait". - **Let it check before it deploys.** Readiness and cost checks cost nothing, and they stop a deploy failing halfway. - **Keep approvals on write tools.** Auto-approving read-only tools saves clicks; auto-approving the rest removes the pause that lets you catch a mistake. ## Next steps - [SurrealDB MCP Server](/docs/build/ai-agents/mcp.md) - what each group of tools does, and how signing in works - [Agent Skills](/docs/build/ai-agents/agent-skills.md) - teach your assistant idiomatic SurrealQL to go with these tools - [Embedded MCP](/docs/build/ai-agents/mcp/embedded.md) - the same data tools against a database you run yourself --- Source: https://surrealdb.com/docs/build/embedding # Embedding SurrealDB How to embed SurrealDB into your application. Detailed instructions for each supported programming language. Instead of connecting to a remote server, you can run SurrealDB directly inside your application process. Embedded mode gives you the full query engine with no network overhead, which is useful for local-first applications, edge deployments, and testing. ## Embedding languages The following languages are supported: - [.NET](/docs/build/embedding/by-language/dotnet.md) - [Go](/docs/build/embedding/by-language/golang.md) - [JavaScript](/docs/build/embedding/by-language/javascript.md) - [Python](/docs/build/embedding/by-language/python.md) - [Rust](/docs/build/embedding/by-language/rust.md) ## Browser embedding options When embedding SurrealDB in web browsers, you have two options: - **IndexedDB**: SurrealDB can be configured to use IndexedDB to store and persist data within the web browser. SurrealDB first serializes both keys and values into a Uint8Array, utilizing IndexedDB as a binary key-value store - offering good performance, and with the ability to offer all of the functionality and features that SurrealDB offers when running in alternative ways. - **SDK**: Alternatively, you can use the SurrealDB SDK to connect to a remote SurrealDB instance instead of using IndexedDB for local persistence. ## Storage - [Storage engines](/docs/build/embedding/storage-engines.md) - which engine an embedded database can use, and what each gives up --- Source: https://surrealdb.com/docs/build/embedding/by-language/dotnet # .NET The documentation for embedding SurrealDB within .NET has been moved to the .NET SDK documentation. SurrealDB can be run as an embedded database within your .NET application, allowing you to use SurrealDB without running a separate server process. This is ideal for desktop applications, testing, local development, and edge computing scenarios. ## Embedded database options SurrealDB supports multiple types of embedded storage in .NET: - **In-memory database** (`mem://` or `SurrealDbMemoryClient`) - Fastest performance with data stored in RAM. Perfect for testing, caching, or temporary data. Data is lost when the connection closes. - **File-based database** (`rocksdb://`, `surrealkv://`, `SurrealDbRocksDbClient`, or `SurrealDbKvClient`) - Persistent storage on disk using RocksDB or SurrealKV storage engines. Data persists across connections and application restarts. ## Quick example ```csharp using SurrealDb.Net; // In-memory database using var db = new SurrealDbMemoryClient(); await db.Use("main", "main"); var person = await db.Create("person", new Person { Name = "John Doe" }); Console.WriteLine(person); // File-based persistent database (RocksDB) using var db = new SurrealDbRocksDbClient("mydb"); await db.Use("main", "main"); var company = await db.Create("company", new Company { Name = "TechStart" }); Console.WriteLine(company); ``` For complete documentation, installation instructions, examples, best practices, and troubleshooting, see the [.NET SDK embedding guide](/docs/reference/dotnet/embedding.md). --- Source: https://surrealdb.com/docs/build/embedding/by-language/golang # Golang The documentation for embedding SurrealDB within Go can be found in the Go SDK and surrealdb.c.go documentation. SurrealDB can be run [as an embedded database](/docs/reference/golang/embedding.md) within your Go application, allowing you to use SurrealDB without running a separate server process. This is ideal for desktop applications, testing, local development, and edge computing scenarios. ## Embedded database options SurrealDB supports multiple types of embedded storage in Go: - **In-memory database** (`mem://`) - Fastest performance with data stored in RAM. Perfect for testing, caching, or temporary data. Data is lost when the connection closes. - **File-based database** (`surrealkv://` or `rocksdb://`) - Persistent storage on disk using the SurrealKV storage engine. Data persists across connections and application restarts. The RocksDB backend requires a separate manual build, detailed in [the RocksDB build guide](https://github.com/surrealdb/surrealdb.c.go/blob/main/docs/rocksdb.md) ## Quick example ```csharp package main import ( "context" "fmt" "log" surrealdb "github.com/surrealdb/surrealdb.c.go" ) type Person struct { ID surrealdb.RecordID[string] `cbor:"id,omitempty"` Name string `cbor:"name"` Age int64 `cbor:"age"` } func main() { ctx := context.Background() db, err := surrealdb.Open(ctx, "mem://") if err != nil { log.Fatal(err) } defer db.Close() db.Use(ctx, "main", "main") db.Query(ctx, "CREATE $rid CONTENT $content", map[string]any{ "rid": surrealdb.NewRecordID("person", "alice"), "content": Person{Name: "Alice", Age: 30}, }) results, _ := surrealdb.Query[Person](ctx, db, "SELECT * FROM person", nil) for _, p := range results[0].Values() { fmt.Printf("%s: %s (age %d)\n", p.ID, p.Name, p.Age) } } ``` For complete documentation, installation instructions, examples, best practices, and troubleshooting, see the [Go SDK embedding guide](/docs/reference/golang/embedding.md). --- Source: https://surrealdb.com/docs/build/embedding/by-language/javascript # JavaScript Embedding SurrealDB in JavaScript SurrealDB is designed to be run in many different ways and in many environments. Due to the [separation of the storage and compute](/docs/learn/data-models/architecture.md) layers, SurrealDB can be run in embedded mode, from within your JavaScript environments. You can embed SurrealDB in both browser and server environments. In browser environments using the [Wasm engine](/docs/reference/javascript/engines/wasm.md), SurrealDB can be run as an in-memory database, or it can persist data using IndexedDB. In server environments using the [Node.js engine](/docs/reference/javascript/engines/node.md), SurrealDB can be run as an embedded database, backed by either an in-memory engine or [SurrealKV](/docs/running/file-backed.md). In this document, we will cover how to embed SurrealDB in both browser and server environments. ## Browser In browser environments, using the [Wasm engine](/docs/reference/javascript/engines/wasm.md), you can run SurrealDB in-memory or with IndexedDB persistence. For more information on how to embed SurrealDB in browser environments, please see the [Wasm engine](/docs/reference/javascript/engines/wasm.md) documentation. ## Server In server environments, you can use the [Node.js engine](/docs/reference/javascript/engines/node.md) to run SurrealDB as an embedded database. For more information on how to embed SurrealDB in server environments, please see the [Node.js engine](/docs/reference/javascript/engines/node.md) documentation. --- Source: https://surrealdb.com/docs/build/embedding/by-language/python # Python The documentation for embedding SurrealDB within Python has been moved to the Python SDK documentation. SurrealDB can be run as an embedded database within your Python application, allowing you to use SurrealDB without running a separate server process. This is ideal for desktop applications, testing, local development, and edge computing scenarios. ## Embedded database options SurrealDB supports two types of embedded storage in Python: - **In-memory database** (`mem://` or `memory`) - Fastest performance with data stored in RAM. Perfect for testing, caching, or temporary data. Data is lost when the connection closes. - **File-based database** (`file://`, `rocksdb://`, or `surrealkv://`) - Persistent storage on disk using the SurrealKV storage engine. Data persists across connections and application restarts. ## Quick example ```python from surrealdb import AsyncSurreal # In-memory database async with AsyncSurreal("mem://") as db: await db.use("main", "main") person = await db.create("person", {"name": "John Doe"}) print(person) # File (RocksDB)-based persistent database async with AsyncSurreal("rocksdb://mydb") as db: await db.use("main", "main") company = await db.create("company", {"name": "TechStart"}) print(company) ``` For complete documentation, installation instructions, examples, best practices, and troubleshooting, see the rest of this guide. --- Source: https://surrealdb.com/docs/build/embedding/by-language/rust # Rust Embedding SurrealDB in Rust SurrealDB can be run as an embedded database within your Rust application, allowing you to use SurrealDB without running a separate server process. This is ideal for desktop applications, testing, local development, and edge computing scenarios. ## Embedded database options SurrealDB supports multiple types of embedded storage in Rust: - **In-memory database** (`Mem`) - Fastest performance with data stored in RAM. Perfect for testing, caching, or temporary data. Data is lost when the connection closes. - **File-based database** (`RocksDb` or `SurrealKV`) - Persistent storage on disk using RocksDB or SurrealKV storage engines. Data persists across connections and application restarts. ## Quick example ```rust use surrealdb::engine::local::Mem; use surrealdb::Surreal; // In-memory database let db = Surreal::new::(()).await?; db.use_ns("main").use_db("main").await?; let person = db.create("person").content(Person { name: "John Doe" }).await?; println!("{:?}", person); // File-based persistent database (RocksDB) use surrealdb::engine::local::RocksDb; let db = Surreal::new::("./mydb").await?; db.use_ns("main").use_db("main").await?; let company = db.create("company").content(Company { name: "TechStart" }).await?; println!("{:?}", company); ``` For complete documentation, installation instructions, examples, best practices, and troubleshooting, see the [Rust SDK embedding guide](/docs/reference/rust/embedding.md). --- Source: https://surrealdb.com/docs/build/embedding/storage-engines # Storage engines What a storage engine does for SurrealDB. How to think about choosing one. SurrealDB sits on top of a storage engine - the component that persists keys and values on disk (or in memory) inside transactions. The query engine speaks SurrealQL; the storage engine decides how durability, concurrency, and replication behave underneath. Why it matters: different engines optimise for different deployment shapes. A single embedded process cares about latency and simplicity; a distributed cluster cares about fault tolerance and horizontal capacity. Picking an engine is not a one-off aesthetic choice - it ties your operational playbook to concrete behaviours (compaction, backup, recovery). Typical options you will see in SurrealDB deployments include RocksDB for mature local disk storage in write-heavy server workloads, SurrealMX (`memory`) for in-process or server in-memory workloads, and SurrealKV (beta) for embedded and local-first workloads where lower resident memory and simpler in-process behaviour are priorities. In-memory modes are ideal for tests and scratch work, and can also be used in embedded deployments when you enable Redis-like SurrealMX persistence. See [Deployment models](/docs/manage/self-hosted/deployment-models.md) for how engines map to hosting models. How to choose: start from your hosting model (embedded, single server, cluster) and read about [deployment models](/docs/manage/self-hosted/deployment-models.md) together with [architecture](/docs/learn/data-models/architecture.md). Then follow the embedding guides for your language under [embedding SurrealDB](/docs/build/embedding.md) for what your SDK supports today - capabilities vary by platform and release. --- Source: https://surrealdb.com/docs/build/integrations # Integrations Integrations connecting SurrealDB to AI frameworks and embeddings. Also agents and data tools. This section collects guides for wiring SurrealDB into the rest of your stack: AI and agent frameworks, embeddings and model providers, data movement (ELT, automation, and similar), and related topics. Each page focuses on a specific tool or pattern so you can find a starting point quickly. This section assumes a certain level of familiarity with SurrealDB itself. If more general orientation and tutorials are needed first, be sure to start with [What is SurrealDB?](/docs/what-is-surrealdb.md) and the [Querying](/docs/learn/querying.md) section of the docs. ## Agent rules - [Agent rules](/docs/build/integrations/agent-rules.md) - rule files that teach an assistant your project's conventions ## Authentication - [Better Auth](/docs/build/integrations/authentication/better-auth/overview.md) - use SurrealDB as the database behind Better Auth - [Getting started](/docs/build/integrations/authentication/better-auth/getting-started.md) - wire the adapter up and run your first sign-in - [Plugins](/docs/build/integrations/authentication/better-auth/plugins.md) - which Better Auth plugins the adapter supports - [Transactions & limitations](/docs/build/integrations/authentication/better-auth/transactions-and-limitations.md) - what the adapter cannot do yet, and why ## Data management - [Data management integrations](/docs/build/integrations/data-management/overview.md) - the pipeline and testing tools that connect to SurrealDB - [Airbyte](/docs/build/integrations/data-management/airbyte.md) - move data in and out with an Airbyte connector - [Fivetran](/docs/build/integrations/data-management/fivetran.md) - managed pipelines into SurrealDB - [n8n](/docs/build/integrations/data-management/n8n.md) - automate workflows against a database - [Qyrus](/docs/build/integrations/data-management/qyrus.md) - test-data generation ## Embeddings providers - [Embeddings provider integrations](/docs/build/integrations/embeddings-providers/overview.md) - which providers are covered, and what each needs - [OpenAI](/docs/build/integrations/embeddings-providers/openai.md) - embed with OpenAI models - [Mistral](/docs/build/integrations/embeddings-providers/mistral.md) - embed with Mistral models - [Fastembed](/docs/build/integrations/embeddings-providers/fastembed.md) - embed locally, without an API call - [Python quickstart](/docs/build/integrations/embeddings-providers/python-quickstart.md) - end to end from Python - [Rust quickstart](/docs/build/integrations/embeddings-providers/rust-quickstart.md) - end to end from Rust --- Source: https://surrealdb.com/docs/build/integrations/agent-rules # Agent rules Agent rules for working with SurrealDB. We've prepared a set of agent rules to help your agents generate better code. - [SurrealQL](https://github.com/surrealdb/docs.surrealdb.com/blob/main/public/integrations/agent-rules/surrealql.mdc) - [Vector indexes and queries](https://github.com/surrealdb/docs.surrealdb.com/blob/main/public/integrations/agent-rules/surrealdb-vector.mdc) - [SurrealDB Python SDK](https://github.com/surrealdb/docs.surrealdb.com/blob/main/public/integrations/agent-rules/surrealdb-python.mdc) - [SurrealDB running embedded in Python](https://github.com/surrealdb/docs.surrealdb.com/blob/main/public/integrations/agent-rules/surrealdb-python-embedded.mdc) Place these files in your project directory under `.cursor/rules`. If you are using a different IDE, take a look at the following instructions. ## How to configure agent rules on main IDEs: - [Cursor rules](https://cursor.com/docs/context/rules) - [Zed rules](https://zed.dev/docs/ai/overview) - [OpenCode rules](https://opencode.ai/docs/rules/) If you are working as a team, consider using the Cursor standard and configure your specific IDE accordingly. For example, for OpenCode you can configure your `opencode.json` like this: ```json { "$schema": "https://opencode.ai/config.json", "instructions": [".cursor/rules/*.md"] } ``` --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/agno # Agno This section contains information about the Agno framework and how to integrate it with SurrealDB. [Agno](https://github.com/agno-agi/agno) is a python framework for building multi-agent systems with shared memory, knowledge and reasoning. ## Setup ```shell docker run --rm \ --pull always \ -p 8000:8000 \ surrealdb/surrealdb:latest \ start \ --user root \ --pass secret ``` or ```shell ./cookbook/scripts/run_surrealdb.sh ``` ## Example ```python from agno.agent import Agent from agno.embedder.openai import OpenAIEmbedder from agno.knowledge.pdf_url import PDFUrlKnowledgeBase from agno.vectordb.surrealdb import SurrealDb from surrealdb import Surreal # SurrealDB connection parameters SURREALDB_URL = "ws://localhost:8000" SURREALDB_USER = "root" SURREALDB_PASSWORD = "secret" SURREALDB_NAMESPACE = "main" SURREALDB_DATABASE = "main" # Create a client client = Surreal(url=SURREALDB_URL) client.signin({"username": SURREALDB_USER, "password": SURREALDB_PASSWORD}) client.use(namespace=SURREALDB_NAMESPACE, database=SURREALDB_DATABASE) surrealdb = SurrealDb( client=client, collection="recipes", # Collection name for storing documents efc=150, # HNSW construction time/accuracy trade-off m=12, # HNSW max number of connections per element search_ef=40, # HNSW search time/accuracy trade-off ) def sync_demo(): """Demonstrate synchronous usage of SurrealDb""" knowledge_base = PDFUrlKnowledgeBase( urls=["https://agno-public.s3.amazonaws.com/recipes/ThaiRecipes.pdf"], vector_db=surrealdb, embedder=OpenAIEmbedder(), ) # Load data synchronously knowledge_base.load(recreate=True) # Create agent and query synchronously agent = Agent(knowledge=knowledge_base, show_tool_calls=True) agent.print_response( "What are the 3 categories of Thai SELECT is given to restaurants overseas?", markdown=True, ) if __name__ == "__main__": # Run synchronous demo print("Running synchronous demo...") sync_demo() ``` ## Developer resources * View [Cookbook (Sync)](https://docs.agno.com/vectordb/surrealdb) * View [Cookbook (Async)](https://docs.agno.com/vectordb/surrealdb) * View [Agno documentation](https://docs.agno.com/vectordb/surrealdb) --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/camel # CAMEL This section contains information about the Camel framework and how to integrate it with SurrealDB. [🐫 CAMEL](https://www.camel-ai.org) is an open-source community dedicated to finding the scaling laws of agents. We believe that studying these agents on a large scale offers valuable insights into their behaviours, capabilities, and potential risks. To facilitate research in this field, we implement and support various types of agents, tasks, prompts, models, and simulated environments. ## Setup You can run SurrealDB locally or start with a [free SurrealDB Cloud account](/docs/manage/instances.md). For local, two options: 1. [Install SurrealDB](/docs/running/installation.md) and run [SurrealDB](/docs/running/in-memory.md). Run in-memory with: ```sh surreal start -u root -p secret ``` 2. [Run with Docker](/docs/running/docker.md). ```sh docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start ``` ## Example ```python import os from camel.storages.vectordb_storages import ( SurrealStorage, VectorDBQuery, VectorRecord, ) def main(): url = os.getenv("SURREAL_URL", "ws://localhost:8000/rpc") table = os.getenv("SURREAL_TABLE", "tb") vector_dim = int(os.getenv("SURREAL_VECTOR_DIM", 4)) namespace = os.getenv("SURREAL_NAMESPACE", "ns") database = os.getenv("SURREAL_DATABASE", "db") user = os.getenv("SURREAL_USER", "user") password = os.getenv("SURREAL_PASSWORD") # Raise an error if password is not set if not password: raise ValueError( "Environment variable SURREAL_PASSWORD is not set. " "Please set it before running." ) # Initialize the SurrealStorage instance with provided parameters storage = SurrealStorage( url=url, table=table, namespace=namespace, database=database, user=user, password=password, vector_dim=vector_dim, ) # Clear existing data in storage storage.clear() # Print the current status after clearing print("[Step 1] After clear:", storage.status()) vec1 = VectorRecord(vector=[1, 2, 3, 4], payload={"name": "test_1"}) vec2 = VectorRecord(vector=[5, 6, 7, 8], payload={"name": "test_2"}) vec3 = VectorRecord(vector=[9, 10, 11, 12], payload={"name": "test_3"}) vec4 = VectorRecord(vector=[13, 14, 15, 16], payload={"name": "test_4"}) storage.add([vec1, vec2, vec3, vec4]) print("[Step 2] After add:", storage.status()) res = storage.client.query_raw( "SELECT * FROM lyz_tb WHERE payload.name = 'test_3';" )["result"][0]["result"][0]["id"].id print("[Step 3] Query Result ID for 'test_3':", res) storage.delete(ids=[res]) print("[Step 4] After delete 'test_3':", storage.status()) res = storage.query( VectorDBQuery(query_vector=[1.1, 2.1, 3.1, 4.1], top_k=2) ) print("[Step 5] Vector Query Result:", res) ``` The output should look like this: ```text [Step 1] After clear: vector_dim=4 vector_count=0oceanbasech [Step 2] After add: vector_dim=4 vector_count=4 [Step 3] Query Result ID for 'test_3': lov5h16x6uog7l2xtsqp [Step 4] After delete 'test_3': vector_dim=4 vector_count=3 [Step 5] Vector Query Result: [VectorDBQueryResult(record=VectorRecord(vector=[], id='9803ae3a-18da-4152-a522-48e1939a3604', payload={'name': 'test_2'}), similarity=0.027665393972965968), VectorDBQueryResult(record=VectorRecord(vector=[], id='de3d085c-ed1b-4d23-9d12-d9fc64ae1e00', payload={'name': 'test_1'}), similarity=0.00010404203326297434)] ``` ## Resources * [Camel documentation](https://docs.camel-ai.org/) --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/cocoindex # CocoIndex Incremental data pipelines for AI agents with declarative SurrealDB graph and vector targets. [CocoIndex](https://cocoindex.io) is an incremental indexing framework for AI agents and LLM applications. You declare what should exist in a target store - documents, embeddings, knowledge graphs - and CocoIndex keeps it in sync, reprocessing only the delta on each run. The SurrealDB connector writes to normal tables, relation (graph edge) tables, and vector indexes. CocoIndex tracks declared records across runs: it upserts changes, skips unchanged records, and removes records that are no longer declared. Related tables in the same database reconcile inside a single atomic transaction. ## How it works 1. **Declare sources** - Walk local files, S3, Google Drive, and other connectors; transform with chunking, embeddings, or LLM extraction. 2. **Declare targets** - Mount SurrealDB `TableTarget` and `RelationTarget` states with optional `TableSchema` (SCHEMAFULL) or schemaless tables. 3. **Reconcile** - On each run, CocoIndex compares the declared state with the previous run and applies upserts and deletions. Schema and vector indexes can be managed automatically when `managed_by` is `"system"`. 4. **Query** - Use SurrealQL for full-text search, graph traversals, and vector similarity on the resulting data. All data can be manually queried at its namespace and database in the same way as with any other SurrealDB instance. ## Key capabilities - **Incremental sync** - Memoised pipeline steps and target-state reconciliation avoid reprocessing unchanged inputs. - **Graph-native writes** - Relation tables with polymorphic `from` / `to` endpoints map cleanly to SurrealDB `RELATE` edges. - **Schema lifecycle** - Optional `TableSchema` with `ColumnDef`; CocoIndex can define fields and drop undeclared columns on re-run. - **Vector indexes** - Declare HNSW indexes on embedding fields; metric and dimension changes trigger index recreation. Pipelines can embed locally with `SentenceTransformerEmbedder` (Rust uses [FastEmbed](https://crates.io/crates/fastembed) under the hood) so you can exercise vector targets without requiring an API key. - **Python and Rust** - Pipelines are typically authored in Python; the Rust SDK exposes the same target-state model for native binaries and examples. ## Local embeddings CocoIndex pipelines that write vectors to SurrealDB often use `SentenceTransformerEmbedder` - models run on your machine and download once, similar to the zero-key graph demos below. The Rust SDK loads them via FastEmbed; Python uses the `sentence-transformers` library with the same Hugging Face model names. The full [`conversation_to_knowledge`](https://github.com/cocoindex-io/cocoindex/tree/main/examples/conversation_to_knowledge) example uses this for entity resolution. To sanity-check local embeddings with SurrealDB directly - or to pick a model before you wire up a CocoIndex pipeline - see the [FastEmbed integration guide](/docs/build/integrations/embeddings-providers/fastembed.md). It covers ONNX models, vector dimensions, and worked examples in Python and Rust without an API key. ## Podcast → knowledge graph CocoIndex's flagship SurrealDB example is [`conversation_to_knowledge`](https://github.com/cocoindex-io/cocoindex/tree/main/examples/conversation_to_knowledge), in which podcast episodes become a graph of **sessions**, **statements**, **people**, **technologies**, **organisations**, and **mention** edges. | Input | What happens | |-------|----------------| | `input/*.txt` (YouTube URLs) | **yt-dlp** downloads audio → **AssemblyAI** transcribes with speaker labels → **LLM** extracts claims and entities → entity resolution → SurrealDB graph | | `input/*.json` (pre-transcribed) | Skips download/transcription; still uses LLM extraction in the full example | The zero-key demos below use the same interview with musician and YouTuber Rick Beato and Alice in Chains guitarist Jerry Cantrell ([YouTube link](https://www.youtube.com/watch?v=vBlfo0GVqqE)) with a pre-transcribed [`input/sample.json`](#appendix-samplejson). Curated statements stand in for LLM extraction so you can see reconciliation without API keys. CocoIndex does **not** fetch transcripts in these demos. When connecting with SurrealDB Studio or `surreal sql`, use the **same namespace and database** your program configures - for example `cocoindex` / `beato_cantrell`, unless you set that explicitly. ## Getting started Start SurrealDB: ```bash title="Run SurrealDB" surreal start --user root --pass secret ``` **Python** Install CocoIndex with the SurrealDB extra: ```bash title="Install" pip install "cocoindex[surrealdb]" ``` Copy [`input/sample.json`](#appendix-samplejson) into an `input` directory next to `main.py`. The script mounts `session`, `statement`, `person`, `tech`, and `org` tables plus `session_statement`, `person_session`, `person_statement`, and polymorphic `statement_mentions` relations - the same shape as the full podcast example. ```bash title="Run (twice - second pass reconciles away the Sabbath branch)" export COCOINDEX_DB=/tmp/cocoindex_beato export SURREALDB_URL=ws://127.0.0.1:8000/rpc export SURREALDB_NS=cocoindex export SURREALDB_DB=beato_cantrell INCLUDE_SABBATH=1 python main.py INCLUDE_SABBATH=0 python main.py ``` `COCOINDEX_DB` stores CocoIndex's local change-tracking state between runs. `INCLUDE_SABBATH=1` declares the Tony Iommi / Black Sabbath influence branch; setting it to `0` on the second run removes those nodes and edges. ```python title="main.py" """Rick Beato × Jerry Cantrell → SurrealDB knowledge graph (no API keys).""" from __future__ import annotations import asyncio import json import os from collections.abc import AsyncIterator from dataclasses import dataclass from pathlib import Path from typing import Any import cocoindex as coco from cocoindex.connectors import surrealdb SURREAL_DB = coco.ContextKey[surrealdb.ConnectionFactory]("surreal_db") SESSION_ID = 100 INCLUDE_SABBATH = os.environ.get("INCLUDE_SABBATH", "1") != "0" @dataclass class Session: id: int youtube_id: str name: str transcript: str description: str | None = None date: str | None = None @dataclass class Statement: id: int statement: str @dataclass class Entity: id: str name: str def statement_seeds(include_sabbath: bool) -> list[dict[str, Any]]: seeds = [ { "id": 200, "text": "Songwriting begins with collecting riffs; most Alice in Chains songs start from a riff idea curated over time.", "speakers": ["Jerry Cantrell"], "persons": ["Jerry Cantrell"], "techs": ["Riffs", "Songwriting"], "orgs": ["Alice in Chains"], }, # ...talk box, Seattle scene, and recording-vibe statements... ] if include_sabbath: seeds.append( { "id": 204, "text": "Tony Iommi and Black Sabbath were foundational guitar influences; Dave Jerden helped expand Alice in Chains' studio sound.", "speakers": ["Jerry Cantrell"], "persons": ["Jerry Cantrell", "Tony Iommi", "Dave Jerden"], "techs": ["Guitar tone"], "orgs": ["Black Sabbath", "Alice in Chains"], } ) return seeds @coco.lifespan async def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]: builder.provide( SURREAL_DB, surrealdb.ConnectionFactory( url=os.environ.get("SURREALDB_URL", "ws://127.0.0.1:8000/rpc"), namespace=os.environ.get("SURREALDB_NS", "cocoindex"), database=os.environ.get("SURREALDB_DB", "beato_cantrell"), credentials={ "username": os.environ.get("SURREALDB_USER", "root"), "password": os.environ.get("SURREALDB_PASS", "secret"), }, ), ) yield @coco.fn(memo=True) async def declare_graph() -> None: session_data = json.loads(Path("input/sample.json").read_text()) transcript = "\n".join( f"({u['speaker']}) {u['text']}" for u in session_data["utterances"] ) seeds = statement_seeds(INCLUDE_SABBATH) session_table = await surrealdb.mount_table_target( SURREAL_DB, "session", await surrealdb.TableSchema.from_class(Session) ) statement_table = await surrealdb.mount_table_target( SURREAL_DB, "statement", await surrealdb.TableSchema.from_class(Statement) ) entity_schema = await surrealdb.TableSchema.from_class(Entity) person_table = await surrealdb.mount_table_target(SURREAL_DB, "person", entity_schema) tech_table = await surrealdb.mount_table_target(SURREAL_DB, "tech", entity_schema) org_table = await surrealdb.mount_table_target(SURREAL_DB, "org", entity_schema) session_statement = await surrealdb.mount_relation_target( SURREAL_DB, "session_statement", session_table, statement_table ) person_session = await surrealdb.mount_relation_target( SURREAL_DB, "person_session", person_table, session_table ) person_statement = await surrealdb.mount_relation_target( SURREAL_DB, "person_statement", person_table, statement_table ) statement_mentions = await surrealdb.mount_relation_target( SURREAL_DB, "statement_mentions", statement_table, [person_table, tech_table, org_table], ) session_table.declare_record( row=Session( id=SESSION_ID, youtube_id=session_data["id"], name=session_data["title"], description=session_data.get("description"), transcript=transcript, date=session_data.get("date"), ) ) people: set[str] = {"Rick Beato", "Jerry Cantrell"} techs: set[str] = set() orgs: set[str] = set() for seed in seeds: statement_table.declare_record( row=Statement(id=seed["id"], statement=seed["text"]) ) session_statement.declare_relation(from_id=SESSION_ID, to_id=seed["id"]) for speaker in seed["speakers"]: people.add(speaker) person_statement.declare_relation(from_id=speaker, to_id=seed["id"]) people.update(seed["persons"]) techs.update(seed["techs"]) orgs.update(seed["orgs"]) for person in seed["persons"]: statement_mentions.declare_relation( from_id=seed["id"], to_id=person, to_table=person_table ) for tech in seed["techs"]: statement_mentions.declare_relation( from_id=seed["id"], to_id=tech, to_table=tech_table ) for org in seed["orgs"]: statement_mentions.declare_relation( from_id=seed["id"], to_id=org, to_table=org_table ) for name in people: person_table.declare_record(row=Entity(id=name, name=name)) person_session.declare_relation(from_id=name, to_id=SESSION_ID) for name in techs: tech_table.declare_record(row=Entity(id=name, name=name)) for name in orgs: org_table.declare_record(row=Entity(id=name, name=name)) app = coco.App(coco.AppConfig(name="beato_cantrell_demo"), declare_graph) if __name__ == "__main__": asyncio.run(app.update()) ``` For LLM extraction, entity resolution, and live YouTube ingestion, follow CocoIndex's [podcast-to-knowledge-graph tutorial](https://cocoindex.io/docs/examples/podcast-to-knowledge-graph/) and the [`conversation_to_knowledge`](https://github.com/cocoindex-io/cocoindex/tree/main/examples/conversation_to_knowledge) source. The [SurrealDB connector reference](https://cocoindex.io/docs/connectors/surrealdb) covers connection setup, `TableSchema.from_class`, vector indexes, and relation tables in full. For local embedding models, see [FastEmbed](/docs/build/integrations/embeddings-providers/fastembed.md). **Rust** The Rust SDK is not yet published on [crates.io](https://crates.io); use a path or git dependency on the [CocoIndex repository](https://github.com/cocoindex-io/cocoindex) (`rust/sdk/cocoindex`, feature `surrealdb`). Copy [`input/sample.json`](#appendix-samplejson) into an `input` directory next to your Rust project and depend on the local SDK: ```toml title="Cargo.toml" [dependencies] cocoindex = { path = "../cocoindex/rust/sdk/cocoindex", features = ["surrealdb"] } tokio = { version = "1", features = ["macros", "rt-multi-thread"] } serde = { version = "1", features = ["derive"] } serde_json = "1" tempfile = "3" ``` ```bash title="Run" export SURREALDB_URL=127.0.0.1:8000 export SURREALDB_NS=cocoindex export SURREALDB_DB=beato_cantrell cargo run ``` The program runs twice in one invocation - the second pass drops the Tony Iommi / Black Sabbath branch to show reconciliation. It mounts the same tables and relations as the Python example above. For LLM extraction, entity resolution, and live YouTube ingestion, use the full [`conversation_to_knowledge` Rust example](https://github.com/cocoindex-io/cocoindex/tree/main/examples/rust/conversation_to_knowledge) (`ASSEMBLYAI_API_KEY`, `OPENAI_API_KEY`, `yt-dlp`). For vector indexes and schema evolution tests, see the [SDK SurrealDB tests](https://github.com/cocoindex-io/cocoindex/blob/main/rust/sdk/cocoindex/tests/surrealdb_target.rs). For local embedding models outside CocoIndex, see [FastEmbed](/docs/build/integrations/embeddings-providers/fastembed.md). Code: ```rust //! Rick Beato × Jerry Cantrell → SurrealDB knowledge graph (no API keys). //! //! Loads a pre-transcribed interview JSON (`input/sample.json`, sourced from //! YouTube captions for https://www.youtube.com/watch?v=vBlfo0GVqqE) and //! declares sessions, statements, people, gear/techniques, bands, and mention //! edges. Curated claims stand in for LLM extraction in the full CocoIndex pipeline. use std::collections::HashSet; use std::path::PathBuf; use std::sync::LazyLock; use cocoindex::prelude::*; use cocoindex::surrealdb::{self, ColumnDef, Graph, TableSchema}; use serde::Deserialize; static GRAPH: LazyLock> = LazyLock::new(|| ContextKey::new("main_db")); const PERSON: &str = "person"; const TECH: &str = "tech"; const ORG: &str = "org"; const SESSION_ID: i64 = 100; const STMT_RIFFS: i64 = 200; const STMT_TALKBOX: i64 = 201; const STMT_SEATTLE: i64 = 202; const STMT_VIBE: i64 = 203; const STMT_SABBATH: i64 = 204; #[derive(Debug, Clone, Deserialize)] struct LocalInput { id: String, title: String, #[serde(default)] description: Option, #[serde(default)] date: Option, utterances: Vec, } #[derive(Debug, Clone, Deserialize)] struct Utterance { speaker: String, text: String, } /// Curated claims - in the full CocoIndex pipeline an LLM extracts these from the transcript. struct StatementSeed { id: i64, text: &'static str, speakers: &'static [&'static str], persons: &'static [&'static str], techs: &'static [&'static str], orgs: &'static [&'static str], } fn statement_seeds(include_sabbath_branch: bool) -> Vec { let mut seeds = vec![ StatementSeed { id: STMT_RIFFS, text: "Songwriting begins with collecting riffs; most Alice in Chains songs start from a riff idea curated over time.", speakers: &["Jerry Cantrell"], persons: &["Jerry Cantrell"], techs: &["Riffs", "Songwriting"], orgs: &["Alice in Chains"], }, StatementSeed { id: STMT_TALKBOX, text: "The I Want Blood sessions leaned on talk box tones, including a Heil talk box and a Jeff Beck-style bag talk box.", speakers: &["Jerry Cantrell"], persons: &["Jerry Cantrell", "Jeff Beck"], techs: &["Talk box", "Heil talk box", "Guitar layering"], orgs: &[], }, StatementSeed { id: STMT_SEATTLE, text: "The early Seattle scene had real camaraderie between bands, and Alice in Chains rarely stockpiled unused songs.", speakers: &["Jerry Cantrell"], persons: &["Jerry Cantrell"], techs: &[], orgs: &["Alice in Chains", "Seattle grunge scene"], }, StatementSeed { id: STMT_VIBE, text: "Recording is about catching the vibe, not perfection - rock and roll finds the audience it is meant to reach.", speakers: &["Jerry Cantrell"], persons: &["Jerry Cantrell"], techs: &["I Want Blood"], orgs: &[], }, ]; if include_sabbath_branch { seeds.push(StatementSeed { id: STMT_SABBATH, text: "Tony Iommi and Black Sabbath were foundational guitar influences; Dave Jerden helped expand Alice in Chains' studio sound.", speakers: &["Jerry Cantrell"], persons: &["Jerry Cantrell", "Tony Iommi", "Dave Jerden"], techs: &["Guitar tone"], orgs: &["Black Sabbath", "Alice in Chains"], }); } seeds } fn load_session() -> Result { let path = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("input/sample.json"); let raw = std::fs::read_to_string(&path) .map_err(|e| Error::engine(format!("read {}: {e}", path.display())))?; serde_json::from_str(&raw).map_err(|e| Error::engine(format!("parse sample.json: {e}"))) } fn transcript_text(session: &LocalInput) -> String { session .utterances .iter() .map(|u| format!("({}) {}", u.speaker, u.text)) .collect::>() .join("\n") } async fn build_graph(app: &App, session: &LocalInput, include_sabbath_branch: bool) -> Result<()> { let seeds = statement_seeds(include_sabbath_branch); let transcript = transcript_text(session); let session = session.clone(); let stats = app .run(move |ctx| { let seeds = seeds; let session = session; let transcript = transcript; async move { let session_target = surrealdb::mount_table_target_with_schema( &ctx, &GRAPH, "session", Some(TableSchema::new([ ("youtube_id", ColumnDef::new("string")), ("name", ColumnDef::new("string")), ("description", ColumnDef::new("string").nullable()), ("transcript", ColumnDef::new("string")), ("date", ColumnDef::new("string").nullable()), ])?), ) .await?; let statement_target = surrealdb::mount_table_target_with_schema( &ctx, &GRAPH, "statement", Some(TableSchema::new([("statement", ColumnDef::new("string"))])?), ) .await?; let entity_schema = TableSchema::new([("name", ColumnDef::new("string"))])?; let person_target = surrealdb::mount_table_target_with_schema( &ctx, &GRAPH, PERSON, Some(entity_schema.clone()), ) .await?; let tech_target = surrealdb::mount_table_target_with_schema( &ctx, &GRAPH, TECH, Some(entity_schema.clone()), ) .await?; let org_target = surrealdb::mount_table_target_with_schema( &ctx, &GRAPH, ORG, Some(entity_schema), ) .await?; let session_statement = surrealdb::mount_relation_target( &ctx, &GRAPH, "session_statement", &session_target, &statement_target, ) .await?; let person_session = surrealdb::mount_relation_target( &ctx, &GRAPH, "person_session", &person_target, &session_target, ) .await?; let person_statement = surrealdb::mount_relation_target( &ctx, &GRAPH, "person_statement", &person_target, &statement_target, ) .await?; let statement_mentions = surrealdb::mount_relation_target_many( &ctx, &GRAPH, "statement_mentions", &[&statement_target], &[&person_target, &tech_target, &org_target], None, ) .await?; session_target.declare_record( &ctx, SESSION_ID, &serde_json::json!({ "youtube_id": session.id, "name": session.title, "description": session.description, "transcript": transcript, "date": session.date, }), )?; let mut people: HashSet<&str> = HashSet::from(["Rick Beato", "Jerry Cantrell"]); let mut techs: HashSet<&str> = HashSet::new(); let mut orgs: HashSet<&str> = HashSet::new(); for seed in &seeds { statement_target.declare_record( &ctx, seed.id, &serde_json::json!({ "statement": seed.text }), )?; session_statement.declare_relation(&ctx, SESSION_ID, seed.id)?; for speaker in seed.speakers { people.insert(speaker); person_statement.declare_relation(&ctx, *speaker, seed.id)?; } for person in seed.persons { people.insert(person); } for tech in seed.techs { techs.insert(tech); } for org in seed.orgs { orgs.insert(org); } for person in seed.persons { statement_mentions.declare_relation_between( &ctx, "statement", seed.id, PERSON, *person, )?; } for tech in seed.techs { statement_mentions.declare_relation_between( &ctx, "statement", seed.id, TECH, *tech, )?; } for org in seed.orgs { statement_mentions.declare_relation_between( &ctx, "statement", seed.id, ORG, *org, )?; } } for name in people { person_target.declare_record( &ctx, name, &serde_json::json!({ "name": name }), )?; person_session.declare_relation(&ctx, name, SESSION_ID)?; } for name in techs { tech_target.declare_record(&ctx, name, &serde_json::json!({ "name": name }))?; } for name in orgs { org_target.declare_record(&ctx, name, &serde_json::json!({ "name": name }))?; } Ok(()) } }) .await?; println!(" {stats}"); Ok(()) } async fn print_counts(graph: &Graph) -> Result<()> { println!( " session: {}, statement: {}, person: {}, tech: {}, org: {}, mentions: {}", graph.count("session").await?, graph.count("statement").await?, graph.count(PERSON).await?, graph.count(TECH).await?, graph.count(ORG).await?, graph.count("statement_mentions").await?, ); Ok(()) } fn env_or(key: &str, default: &str) -> String { std::env::var(key).unwrap_or_else(|_| default.to_string()) } fn print_query_hints() { println!("\nInspect in SurrealDB Studio (or surreal sql with `surreal sql --user root --pass secret --ns cocoindex --db beato_cantrell`):"); println!(" namespace: cocoindex"); println!(" database: beato_cantrell"); println!("\nExample queries:"); println!(" SELECT name FROM person;"); println!(" SELECT statement FROM statement WHERE statement CONTAINS 'riff';"); println!(" SELECT statement FROM statement WHERE statement CONTAINS 'talk box';"); println!( " SELECT id, ->statement_mentions->org.name AS bands FROM statement WHERE ->statement_mentions->org;" ); } #[tokio::main] async fn main() -> Result<()> { let session = load_session()?; println!( "Loaded transcript: {} ({} utterances)\n", session.title, session.utterances.len() ); let graph = Graph::connect( &env_or("SURREALDB_URL", "127.0.0.1:8000"), &env_or("SURREALDB_NS", "cocoindex"), &env_or("SURREALDB_DB", "beato_cantrell"), &env_or("SURREALDB_USER", "root"), &env_or("SURREALDB_PASS", "secret"), ) .await?; let dir = tempfile::tempdir()?; let app = Environment::builder() .db_path(dir.path().join("cocoindex_lmdb")) .provide_key(&GRAPH, graph.clone()) .build() .await? .app("beato_cantrell_demo") .await?; println!("Run 1 - full graph (includes Tony Iommi / Black Sabbath branch)"); build_graph(&app, &session, true).await?; print_counts(&graph).await?; println!("\nRun 2 - narrower extraction (Sabbath influence branch reconciled away)"); build_graph(&app, &session, false).await?; print_counts(&graph).await?; print_query_hints(); Ok(()) } ``` ## Inspect the graph After either demo completes, open SurrealDB Studio with namespace **cocoindex** and database **beato_cantrell**. The schema designer shows the session-centric graph CocoIndex declared: ![SurrealDB Studio schema designer showing session, statement, person, tech, and org tables with session_statement, person_session, person_statement, and statement_mentions relation tables after the Beato - Cantrell demo.](~/assets/img/integrations/cocoindex_schema.png) Example queries: ```surql title="Example queries" SELECT name FROM person; SELECT statement FROM statement WHERE statement CONTAINS 'riff'; SELECT statement FROM statement WHERE statement CONTAINS 'talk box'; SELECT ->statement_mentions->org.name AS bands FROM statement WHERE statement CONTAINS 'Alice'; ``` After the second run (`INCLUDE_SABBATH=0` in Python, or the built-in second pass in Rust), the Tony Iommi / Black Sabbath statement and its mention edges are gone - reconciliation removed anything no longer declared. ## Appendix: sample.json Create an `input` directory next to your demo program and save the following as `input/sample.json`. Utterance excerpts were polished from YouTube auto-captions for the [Rick Beato × Jerry Cantrell interview](https://www.youtube.com/watch?v=vBlfo0GVqqE).
Show input/sample.json ```json title="input/sample.json" { "id": "vBlfo0GVqqE", "title": "Jerry Cantrell: Creating the Iconic Sound of Alice In Chains", "channel": "Rick Beato", "description": "Rick Beato interviews Jerry Cantrell about Alice in Chains, songwriting, guitar tones, the Seattle scene, and the solo album I Want Blood.", "date": "2024-09-17", "utterances": [ { "speaker": "A", "text": "Jerry, welcome. I'm listening to your new solo record and it has all these great riffs. Do you just sit around and put ideas on your phone? How do you keep track of everything?" }, { "speaker": "B", "text": "I'm probably a collector and curator of riffs. It always starts there. Sometimes I'll hum something into a phone, but most songs begin when I stumble across a riff and think, that's cool." }, { "speaker": "A", "text": "Walk me through how a riff becomes an Alice in Chains song." }, { "speaker": "B", "text": "The music always comes first to me. It begins with the riff and then it develops into what I think is a piece of music that makes a great song." }, { "speaker": "B", "text": "We leaned heavily on the talk box on this record - the Heil talk box and some older flavors I had not hit that hard in a while, plus a Jeff Beck-style bag talk box I had never played before." }, { "speaker": "A", "text": "Let's talk about the Seattle scene when Dirt and Jar of Flies were happening." }, { "speaker": "B", "text": "Seattle at that time had a real camaraderie between bands. We were all pushing each other, but Alice in Chains always put out what we wrote - no pile of unused B-sides sitting around." }, { "speaker": "A", "text": "Your new solo album is called I Want Blood. How is that different from a band record?" }, { "speaker": "B", "text": "It's a collaborative thing whether I'm in the band or on a solo record. There's a whole group of people and a producer - it's still a band effort, just with my name on the cover." }, { "speaker": "B", "text": "It's not about perfection, it's about catching the vibe. Rock and roll is not supposed to be for everybody - it finds the people it is meant to speak to." }, { "speaker": "A", "text": "Who were the guitar influences that shaped your sound?" }, { "speaker": "B", "text": "Tony Iommi and Black Sabbath were huge for me early on - that dark, heavy feel. Jimmy Page too, and the way producers like Dave Jerden helped expand what we could do in the studio." } ] } ```
--- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/crewai # CrewAI How to plug SurrealDB into CrewAI as a memory and vector-search layer. [CrewAI](https://github.com/joaomdmoura/crewAI) lets you orchestrate role-playing AI agents that collaborate to complete complex tasks. This guide shows how to use SurrealDB as the memory backend for CrewAI, giving agents: - **Entity memory** for domain objects (products, people, places) - **Short-term memory** for recent conversations - **Vector search** to recall relevant information by semantic similarity The sample below creates two agents that recommend music festival trips: 1. **Researcher** - finds festivals and saves them to SurrealDB 2. **Planner** - queries those saved festivals to build a weekend itinerary ## Install ```bash # CrewAI, SurrealDB Python SDK, and an embedder (OpenAI here; swap if you like) pip install "crewai[tools]" surrealdb openai # (optional) run SurrealDB locally - single binary, no deps docker run --pull always -p 8000:8000 surrealdb/surrealdb:latest \ start --user root --pass secret file:/data/db ```` SurrealDB v2 ships native HNSW indexes, so you get ANN vector search without an extra service. ## SurrealDB storage adapter Create `src/mycrew/surreal_storage.py`: ```python from __future__ import annotations import asyncio, hashlib, logging, os, threading from typing import Any, Dict, List, Optional import openai from crewai.memory.storage.rag_storage import RAGStorage from surrealdb import Surreal, AsyncSurreal logger = logging.getLogger(__name__) _EMBED_DIM = 1536 # OpenAI text-embedding-3-small # ────────────────────────── embeddings ────────────────────────── def _embed(text: str, api_key: Optional[str] = None, model: str = "text-embedding-3-small", dimensions: int = _EMBED_DIM) -> List[float]: """Return an L2-normalised embedding vector.""" resp = openai.Embedding.create( model=model, input=[text], api_key=api_key or os.getenv("OPENAI_API_KEY"), dimensions=dimensions, ) vec = resp["data"][0]["embedding"] norm = sum(x * x for x in vec) ** 0.5 or 1.0 return [x / norm for x in vec] # ─────────────────────────── storage ──────────────────────────── class SurrealStorage(RAGStorage): """CrewAI RAGStorage backend backed by SurrealDB v2.""" _lock = threading.Lock() _tables_ready: set[str] = set() def __init__( self, typ: str, *, allow_reset: bool = True, embedder_config: Optional[Dict[str, Any]] = None, crew: Optional[Any] = None, url: str = "ws://localhost:8000/rpc", namespace: str = "crew", database: str = "memories", user: str = "root", password: str = "secret", ): super().__init__(typ, allow_reset, embedder_config, crew) self.url, self.ns, self.db = url, namespace, database self.user, self.pw = user, password self.table = f"mem_{typ.replace('-', '_')}" self._adb = AsyncSurreal(url) # async client self._sdb = Surreal(url) # sync client (DDL) self._embed_api_key = (embedder_config or {}).get("api_key") # ─────────── helpers ─────────── def _run(self, coro): """Run *coro* in any context (sync script or async crew).""" try: loop = asyncio.get_running_loop() if loop.is_running(): return asyncio.ensure_future(coro) except RuntimeError: pass return asyncio.run(coro) # ─────────── sync API (called by CrewAI) ─────────── def save(self, value: Any, metadata: Dict[str, Any]): self._run(self._save_async(value, metadata)) def search( self, query: str, limit: int = 3, filter: Optional[Dict[str, Any]] = None, score_threshold: float = 0.0, ): return self._run( self._search_async(query, limit, filter, score_threshold) ) def reset(self): self._run(self._reset_async()) # ─────────── async internals ─────────── async def _save_async(self, value: Any, metadata: Dict[str, Any]): await self._ensure_schema() vec = _embed(str(value), self._embed_api_key) rec = { "id": hashlib.sha1(str(value).encode()).hexdigest(), "text": value, "metadata": metadata or {}, "embedding": vec, } async with self._adb as db: await db.signin({"username": self.user, "password": self.pw}) await db.use(self.ns, self.db) await db.create(self.table, rec) async def _search_async( self, query: str, limit: int, filter: Optional[Dict[str, Any]], score_threshold: float, ): await self._ensure_schema() vec = _embed(query, self._embed_api_key) filter_params, where_clause = {}, "" if filter: conds = [] for i, (k, v) in enumerate(filter.items()): pname = f"f{i}" conds.append(f"metadata.{k} == ${pname}") filter_params[pname] = v where_clause = " AND " + " AND ".join(conds) async with self._adb as db: await db.signin({"username": self.user, "password": self.pw}) await db.use(self.ns, self.db) rs = await db.query( f""" SELECT *, vector::distance::knn() AS score FROM {self.table} WHERE embedding <|{limit}|> $vec{where_clause} ORDER BY score ASC; """, {"vec": vec, **filter_params}, ) rows = rs[0]["result"] return [ { "id": r["id"], "metadata": r["metadata"], "context": r["text"], "score": r["score"], } for r in rows if (score_threshold == 0 or r["score"] <= score_threshold) ] async def _reset_async(self): async with self._adb as db: await db.signin({"username": self.user, "password": self.pw}) await db.use(self.ns, self.db) await db.query(f"REMOVE TABLE {self.table};") self._tables_ready.discard(self.table) # ─────────── table / index setup ─────────── async def _ensure_schema(self): if self.table in self._tables_ready: return with self._lock: if self.table in self._tables_ready: return self._sdb.signin({"username": self.user, "password": self.pw}) self._sdb.use(self.ns, self.db) self._sdb.query( f""" DEFINE TABLE {self.table} SCHEMALESS; DEFINE FIELD id ON {self.table} TYPE string; DEFINE FIELD text ON {self.table} TYPE string; DEFINE FIELD metadata ON {self.table} TYPE object; DEFINE FIELD embedding ON {self.table} TYPE array; DEFINE INDEX IF NOT EXISTS {self.table}_vec_idx ON {self.table} FIELDS embedding HNSW DIMENSION {_EMBED_DIM} DIST COSINE; """ ) self._tables_ready.add(self.table) ``` ## Wire it into a crew Create `src/mycrew/crew.py`: ```python import asyncio, logging from typing import Optional from crewai import Agent, Task, Crew from crewai.memory.entity.entity_memory import EntityMemory from crewai.memory.short_term.short_term_memory import ShortTermMemory from mycrew.surreal_storage import SurrealStorage, _EMBED_DIM logging.basicConfig(level=logging.INFO) async def create_crew( openai_api_key: Optional[str] = None, surreal_url: str = "ws://localhost:8000/rpc", ) -> Crew: embed_cfg = {"api_key": openai_api_key} researcher = Agent( name="GeoResearcher", role="Geo-intelligence analyst", goal="collect up-to-date info on travel destinations", ) planner = Agent( name="TripPlanner", role="Personal itinerary planner", goal="craft a perfect 48-hour city break", ) entity_mem = EntityMemory( storage=SurrealStorage("entity", url=surreal_url, embedder_config=embed_cfg) ) short_mem = ShortTermMemory( storage=SurrealStorage("short-term", url=surreal_url, embedder_config=embed_cfg) ) t1 = Task( description="Find vibrant European cities with art festivals in June 2025.", expected_output="Top 3 candidate cities with festival names and dates.", agent=researcher, ) t2 = Task( description="Design a relaxed 2-day itinerary for the best candidate.", expected_output="Detailed schedule with cafes, galleries, evening events.", agent=planner, context=[t1], ) crew = Crew( memory=True, entity_memory=entity_mem, short_term_memory=short_mem, ) crew.add_agents(researcher, planner) crew.add_tasks(t1, t2) return crew async def main(): crew = await create_crew() result = await crew.run() print(result) if __name__ == "__main__": asyncio.run(main()) ``` Run the crew: ```bash python -m mycrew.crew ``` ## Customising further | Piece | How to tweak it | | ----------------- | ----------------------------------------------------------------------------------------------- | | Embedding model | Swap `_embed()` for JinaAI, Instructor, or a local transformer; adjust `_EMBED_DIM`. | | Similarity metric | Change `DIST COSINE` to `L2` or `DOT` to match your embeddings. | | Metadata filters | Extend the simple `filter` mapping to support CrewAI’s richer operators (`$gt`, `$in`, …). | | Remote / Cloud DB | Replace `ws://localhost:8000/rpc` with `wss:///rpc` and supply a token/password. | ## Resources * [Vector Search reference guide](/docs/learn/data-models/vector-search/overview.md) * [Using SurrealDB as a Vector Database](/docs/learn/data-models/vector-search/overview.md) * [Python SDK docs](/docs/reference/python.md) * [DEFINE INDEX statement](/docs/reference/query-language/statements/define/indexes.md) --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/dagster # Dagster This section contains information about the Dagster framework and how to integrate it with SurrealDB. Dagster is a powerful tool for building data pipelines and workflows. It is a popular choice for data engineers and data scientists. ## Install ```bash title="Install" pip install dagster surrealdb openai # or swap out OpenAI for any embedder # optional - launch a local SurrealDB daemon docker run -p 8000:8000 surrealdb/surrealdb:latest \ start --user root --pass secret file:/data/db ``` ## A Dagster "SurrealResource" ```python title="src/dagster_surreal.py" import dagster as dg from surrealdb import Surreal from typing import List, Sequence, Optional import hashlib, json, os, contextlib _EMBED_DIM = 1536 # match your embedder def embed(text: str) -> List[float]: """Tiny helper - replace with your preferred model.""" import openai resp = openai.Embedding.create( model="text-embedding-3-small", input=[text], dimensions=_EMBED_DIM, api_key=os.getenv("OPENAI_API_KEY"), ) return resp["data"][0]["embedding"] class SurrealConfig(dg.Config): url: str = dg.Field(str, default_value="ws://localhost:8000/rpc") namespace: str = dg.Field(str, default_value="dagster") database: str = dg.Field(str, default_value="vector") user: str = dg.Field(str, default_value="root") password: str = dg.Field(str, default_value="secret") @dg.resource(config_schema=SurrealConfig) class SurrealResource: """ A very small wrapper that exposes .add() and .query() like dagster-qdrant. """ def __init__(self, context): cfg: SurrealConfig = context.resource_config self.url = cfg.url self.namespace = cfg.namespace self.database = cfg.database self.user = cfg.user self.password = cfg.password # - helpers ---------------------------------------------------------- def _ensure_table(self, table: str): with Surreal(self.url) as db: db.signin({"username": self.user, "password": self.password}) db.use(self.namespace, self.database) db.query( """ DEFINE TABLE $table SCHEMALESS; DEFINE FIELD id ON $table TYPE string; DEFINE FIELD text ON $table TYPE string; DEFINE FIELD embedding ON $table TYPE array; DEFINE INDEX IF NOT EXISTS ${table}_vec ON $table FIELDS embedding HNSW DIMENSION $dim DIST COSINE; """, { "table": table, "dim": _EMBED_DIM } ) # - public API ------------------------------------------------------- def add(self, collection_name: str, documents: Sequence[str]): self._ensure_table(collection_name) with Surreal(self.url) as db: db.signin({"username": self.user, "password": self.password}) db.use(self.namespace, self.database) for doc in documents: rec = { "id": hashlib.sha1(doc.encode()).hexdigest(), "text": doc, "embedding": embed(doc), } db.create(collection_name, rec) def query( self, collection_name: str, query_text: str, limit: int = 3, score_threshold: float = 0.4, ): self._ensure_table(collection_name) vec = embed(query_text) with Surreal(self.url) as db: db.signin({"username": self.user, "password": self.password}) db.use(self.namespace, self.database) result = db.query( """ SELECT text, vector::similarity::cosine(embedding, $vec) AS score FROM $table WHERE embedding <|$limit|> $vec ORDER BY score DESC """, { "table": collection_name, "vec": vec, "limit": limit } ) rows = result[0]["result"] return [r for r in rows if r['score'] <= score_threshold or score_threshold == 0] # Use the resource as a context-manager inside assets @contextlib.contextmanager def get_client(self): try: yield self finally: pass # No need to close since we use context managers for each operation ``` --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/google-agent # Google Agent Google Agent is a framework for building and deploying agents. Vertex AI Agent Builder provides a powerful framework for developing and deploying intelligent agents in Google Cloud. This integration is particularly valuable since any agent running on Vertex AI Agent Engine or the open-source ADK runtime can be [surfaced through Agentspace](https://cloud.google.com/products/agentspace), making it easily accessible to your organisation. ## TL;DR *Build the agent with ADK → deploy to Agent Engine → "Add to Agentspace"* - the SurrealDB retrieval tool keeps working unchanged, giving your Gemini-based agent sub-second RAG over your own vector index, all inside Google Cloud. Below is a concise walkthrough that shows: 1. creating a SurrealDB-powered retrieval tool 2. wiring it into an **ADK** agent 3. deploying that agent to Agent Engine 4. registering it in Agentspace so employees can chat with it from the Agentspace UI. ## Prerequisites **Bash** ```bash pip install google-adk google-genai surrealdb # ADK + Gemini SDK + SurrealDB docker run -p 8000:8000 surrealdb/surrealdb:latest \ start --user root --pass secret file:/data/db # optional local DB export GOOGLE_API_KEY= ``` **PowerShell** ```powershell pip install google-adk google-genai surrealdb # ADK + Gemini SDK + SurrealDB docker run -p 8000:8000 surrealdb/surrealdb:latest ` start --user root --pass secret file:/data/db # optional local DB $env:GOOGLE_API_KEY = "" ``` ## Prepare SurrealDB (one-time DDL) ```surql DEFINE TABLE kb_docs SCHEMALESS; DEFINE FIELD id ON kb_docs TYPE string; DEFINE FIELD text ON kb_docs TYPE string; DEFINE FIELD source ON kb_docs TYPE string; DEFINE FIELD embedding ON kb_docs TYPE array; -- 1 536-D cosine HNSW (matches Gemini embeddings) DEFINE INDEX IF NOT EXISTS kb_vec ON kb_docs FIELDS embedding HNSW DIMENSION 1536 DIST COSINE; ``` ## Custom "Surreal Retrieve" tool ```python from typing import List, Dict, Any, Optional from google import genai from surrealdb import AsyncSurreal from surrealdb.exception import SurrealException import os import json import hashlib import logging from dataclasses import dataclass from contextlib import asynccontextmanager # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) @dataclass class SurrealConfig: """Configuration for SurrealDB connection.""" url: str = "ws://localhost:8000/rpc" namespace: str = "agent" database: str = "demo" username: str = "root" password: str = "secret" class SurrealRetriever: """SurrealDB-powered retrieval tool for Google Agent.""" def __init__( self, config: Optional[SurrealConfig] = None, api_key: Optional[str] = None ): """Initialize the retriever. Args: config: Optional SurrealDB configuration api_key: Optional Google API key """ self.config = config or SurrealConfig() self.api_key = api_key or os.environ.get("GOOGLE_API_KEY") if not self.api_key: raise ValueError("Google API key is required") genai.configure(api_key=self.api_key) self.embedder = genai.Client().models.get_embedding_model("models/embedding-001") @asynccontextmanager async def _get_connection(self) -> AsyncSurreal: """Get a database connection with proper authentication. Yields: Connected AsyncSurreal instance Raises: SurrealException: If connection fails """ db = AsyncSurreal(self.config.url) try: await db.signin({ "username": self.config.username, "password": self.config.password }) await db.use(self.config.namespace, self.config.database) yield db except SurrealException as e: logger.error(f"Failed to connect to SurrealDB: {str(e)}") raise SurrealException(f"Connection failed: {str(e)}") finally: await db.close() async def _embed(self, text: str) -> List[float]: """Generate embeddings for text. Args: text: Text to embed Returns: List of embedding values Raises: Exception: If embedding generation fails """ try: return self.embedder.embed(content=text).embedding except Exception as e: logger.error(f"Failed to generate embeddings: {str(e)}") raise Exception(f"Embedding generation failed: {str(e)}") async def retrieve(self, question: str, k: int = 4) -> str: """Retrieve top-k passages related to question from SurrealDB. Args: question: Search query k: Number of results to return Returns: Formatted string of matching passages Raises: SurrealException: If database operations fail """ try: qv = await self._embed(question) async with self._get_connection() as db: result = await db.query(""" SELECT text, source, vector::similarity::cosine(embedding, $vec) AS score FROM kb_docs WHERE embedding <|$k|> $vec ORDER BY score DESC """, { "vec": qv, "k": k }) rows = result[0]["result"] return "\n".join( f"- {r['text']} (src: {r['source']})" for r in rows ) except SurrealException as e: logger.error(f"Retrieval failed: {str(e)}") raise SurrealException(f"Retrieval failed: {str(e)}") async def ingest(self, docs: List[Dict[str, str]]) -> None: """Ingest documents into SurrealDB. Args: docs: List of documents with text and source Raises: SurrealException: If database operations fail """ try: async with self._get_connection() as db: for doc in docs: rec = { "id": hashlib.sha1(doc["text"].encode()).hexdigest(), "text": doc["text"], "source": doc["source"], "embedding": await self._embed(doc["text"]), } await db.create("kb_docs", rec) logger.info(f"Ingested {len(docs)} documents") except SurrealException as e: logger.error(f"Ingestion failed: {str(e)}") raise SurrealException(f"Ingestion failed: {str(e)}") async def ensure_schema(self) -> None: """Ensure the required table and index exist. Raises: SurrealException: If schema operations fail """ try: async with self._get_connection() as db: await db.query(""" DEFINE TABLE IF NOT EXISTS kb_docs SCHEMALESS; DEFINE FIELD id ON kb_docs TYPE string; DEFINE FIELD text ON kb_docs TYPE string; DEFINE FIELD source ON kb_docs TYPE string; DEFINE FIELD embedding ON kb_docs TYPE array; DEFINE INDEX IF NOT EXISTS kb_vec ON kb_docs FIELDS embedding HNSW DIMENSION 1536 DIST COSINE; """) logger.info("Schema ensured successfully") except SurrealException as e: logger.error(f"Schema setup failed: {str(e)}") raise SurrealException(f"Schema setup failed: {str(e)}") # Example usage async def main(): retriever = SurrealRetriever() try: # Ensure schema exists await retriever.ensure_schema() # Ingest sample documents await retriever.ingest([ { "text": "Nikola Tesla patented the first practical AC induction motor in 1888.", "source": "wiki/Tesla" }, { "text": "The Wright brothers achieved powered flight on 17 Dec 1903.", "source": "wiki/Wright" }, ]) # Test retrieval result = await retriever.retrieve( "Who invented the AC motor and when?", k=2 ) print(result) except Exception as e: logger.error(f"Error: {str(e)}") raise if __name__ == "__main__": import asyncio asyncio.run(main()) ``` ## Build an ADK agent that uses the tool ```python from typing import Optional from adk import Agent, Tool from surreal_tool import SurrealRetriever import asyncio import logging from contextlib import asynccontextmanager # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Initialize retriever retriever = SurrealRetriever() @Tool async def knowledge_search(query: str) -> str: """Search SurrealDB for passages relevant to the query. Args: query: Search query Returns: Formatted string of matching passages """ try: return await retriever.retrieve(query, k=4) except Exception as e: logger.error(f"Knowledge search failed: {str(e)}") return f"Error searching knowledge base: {str(e)}" @asynccontextmanager async def get_agent(): """Get an agent instance with proper setup and teardown. Yields: Configured Agent instance """ try: agent = Agent( name="Surreal-Facts-Bot", llm="gemini-pro", instructions=""" You are a helpful assistant that must cite retrieved passages verbatim. When knowledge is required, call knowledge_search(). """, tools=[knowledge_search], ) logger.info("Agent created successfully") yield agent except Exception as e: logger.error(f"Failed to create agent: {str(e)}") raise async def main(): """Run the agent.""" try: async with get_agent() as agent: response = await agent.chat("Who invented the AC motor and when?") print(response) logger.info("Agent execution completed successfully") except Exception as e: logger.error(f"Agent execution failed: {str(e)}") raise if __name__ == "__main__": asyncio.run(main()) ``` Run locally (`python agent.py`) to verify the agent calls your SurrealDB tool. ## Deploy to **Vertex AI Agent Engine** Create `agent.yaml` (full schema in the ADK docs): ```yaml name: surreal-facts entrypoint: agent:agent # python-module:object requirements: - surrealdb - google-genai - google-adk - google-cloud-aiplatform ``` ```bash gcloud ai agent-engines deploy surreal-facts \ --agent-spec=agent.yaml \ --region=us-central1 ``` ## Publish in **Agentspace** Agentspace can list any agent that's running on Agent Engine ([Google Cloud][3], [Medium][4]). * **Console path:** *Agentspace → Agent Gallery → "Add agent" → Source: Agent Engine* * **CLI (preview):** ```bash gcloud ai agentspace agent-galleries add-agent \ --agent-engine=projects/$PROJECT/locations/us-central1/agentEngines/surreal-facts ``` Once added, employees will see **"Surreal Facts Bot"** in the gallery and can chat with it from the Agentspace UI; every time it needs knowledge it silently calls your SurrealDB index. ## Operational notes
Topic Guidance
Networking Run SurrealDB on Cloud Run, GKE, or a VM inside the same VPC; use Private Service Connect so the Agent Engine can reach it.
Security Protect the DB with DEFINE ACCESS tokens or basic auth; Agentspace inherits IAM & VPC-SC you already set in Agent Builder (Google Cloud).
Hybrid search Mix Boolean filters with the <|K|> operator (WHERE source ~ 'wiki' AND embedding <|3|>).
Updates New docs can be inserted at any time; SurrealDB's HNSW index updates incrementally, or run REBUILD INDEX during low-traffic windows (SurrealDB).
## Resources - [Google Next 25 Updates: ADK, Agentspace, Application Integration](https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/Google-Next-25-Updates-ADK-Agentspace-Application-Integration/ta-p/898343) - [DEFINE INDEX statement](/docs/reference/query-language/statements/define/indexes.md) - [Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) - [A Foundational Framework for Agentic AI Ecosystems: Enabling](https://medium.com/google-cloud/a-foundational-framework-for-agentic-ai-ecosystems-enabling-development-discovery-and-2aeb120949f6) - [Introduction to Google Agentspace](https://cloud.google.com/agentspace/docs/overview) - [REBUILD statement](/docs/reference/query-language/statements/rebuild.md) --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/hermes # Hermes Give a Hermes agent a persistent filesystem in SurrealDB, as a toolset it calls explicitly and as a memory provider that files and recalls every turn. [Hermes](https://hermes-agent.nousresearch.com) is Nous Research's terminal agent. [SurrealDB filesystem](https://github.com/surrealdb/surrealfs) gives it somewhere durable to keep its work: a `file` table in SurrealDB, plus the tools to hand that table to a model. Files and folders, full-text and semantic search, and everything queryable with SurrealQL, because it is just a table. **SurrealDB filesystem** ships two Hermes plugins. They install separately and either one works on its own: - **The toolset** registers 14 `surrealfs_*` tools and a bundled skill, so the agent reads and writes the filesystem when it decides to. - **The memory provider** files every completed turn into the same filesystem and searches it for context before each turn. They are better together: the provider's recall surfaces the notes the agent wrote with the tools. > [!NOTE] > This page covers **SurrealDB filesystem** on a SurrealDB instance you run. For hosted memory with fact extraction and semantic recall, see the [SurrealDB Agent Memory integration for Hermes](/docs/agent-memory/integrations/frameworks/hermes.md), which is a separate provider. ## Requirements - Python 3.12+ - A SurrealDB 3.x server, [installed locally](/docs/running/installation.md) or on [SurrealDB Cloud](/docs/manage/instances.md) - A Hermes install Install Hermes if you do not have it, and follow the [installation guide](https://hermes-agent.nousresearch.com/docs/getting-started/installation) through to `Installation Complete`: ```bash curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash ``` ## Prepare a database Start SurrealDB with a persistent backend, either [installed directly](/docs/running/installation.md): ```bash surreal start --user root --pass secret rocksdb://surrealfs.db ``` or [with Docker](/docs/running/docker.md): ```bash docker run --rm --pull always -p 8000:8000 -v ./data:/data \ surrealdb/surrealdb:latest start --user root --pass secret rocksdb://data/surrealfs.db ``` Then define the `file` table. The schema command reads the same `SURREALDB_*` variables as everything else, and it is safe to re-run: ```bash python -m surrealfs.schema ``` Add `--include-user` for the optional `user` table and record access that the `file` table's `owner` permissions need for multi-tenancy, or `--print` to dump the DDL without connecting to anything. A file's `parent` is a record link and its `path` is a computed field derived from the parent chain: ```surql DEFINE FIELD path ON file COMPUTED ( ($this.parent.path || '') + '/' + $this.filename ) TYPE string; ``` Nothing stores a path, so moving or renaming a folder is a single `UPDATE` and every descendant's path follows for free. A folder is a row with no content, `hash` is maintained by an event, and the HNSW and BM25 indexes behind the two search modes are part of the same schema. ## Connection settings Both plugins read the same environment variables and default to a local server, so the defaults already work against `surreal start` on your own machine. | Variable | Default | Purpose | | --- | --- | --- | | `SURREALDB_URL` | `ws://localhost:8000/rpc` | Server to connect to | | `SURREALDB_USER` / `SURREALDB_PASS` | `root` / `root` | Credentials | | `SURREALDB_NAMESPACE` / `SURREALDB_DATABASE` | `surrealfs` / `demo` | Where the `file` table lives | | `SURREALFS_SEMANTIC` | unset | `1` adds a vector arm to search and recall | Put these in `$HERMES_HOME/.env` rather than your shell profile. Hermes cron jobs run with a sanitised environment, so a scheduled [indexer](#keep-the-vectors-current) reads its connection details from that file. ## The toolset Hermes runs from its own virtualenv, so install the package there rather than into your project. `surrealfs` is not on PyPI yet, so install it from Git: ```bash uv pip install --python ~/.hermes/hermes-agent/venv/bin/python \ "surrealfs @ git+https://github.com/surrealdb/surrealfs.git" hermes plugins enable surrealfs hermes chat --toolsets surrealfs ``` Installing the package is what makes the plugin discoverable, because it registers through the `hermes_agent.plugins` entry point. Nothing in the plugin imports Hermes: the plugin contract is a `register` function and a manifest, so there is no extra dependency to install. To run from a checkout instead, symlink the plugin directory into Hermes' plugin folder. The package still has to be importable by Hermes' interpreter: ```bash ln -s "$PWD/surrealfs/integrations/hermes" ~/.hermes/plugins/surrealfs ``` ### The tools | Group | Tools | | --- | --- | | Read | `surrealfs_ls` `surrealfs_glob` `surrealfs_cat` `surrealfs_read_bytes` `surrealfs_tail` | | Write | `surrealfs_write_file` `surrealfs_write_bytes` `surrealfs_edit` `surrealfs_touch` `surrealfs_mkdir` | | Organise | `surrealfs_cp` `surrealfs_mv` `surrealfs_rm` | | Search | `surrealfs_search` | The prefix is deliberate. Hermes keeps one flat tool namespace that its own `read_file` and `write_file` already sit in, and the prefix also tells the model which filesystem it is addressing: the built-in file tools are the local disk, `surrealfs_*` is the shared one in SurrealDB. Every tool description says so too. The plugin opens its own connection per call, and every result comes back as JSON, errors included. Handlers are registered as async, so Hermes awaits them on whichever event loop it is using. ### The bundled skill `surrealfs:notes` teaches the agent to keep its notes and memories in SurrealDB filesystem, and how to lay them out: | Path | Holds | | --- | --- | | `/home//` | The agent's own working files | | `/home//memories/` | Turns filed by the memory provider | | `/preferences//` | What the agent learns about the user | | `/projects//` | Per-project notes, plans and to-do lists | Paths are absolute and there is no working directory. Everything outside `/home/` is shared, which is how an agent hands something to its user or to another agent working for the same person. Hermes advertises plugin skills nowhere: they are absent from the system prompt's `` index and from `skills_list()`, and resolve only by exact qualified name. The plugin therefore registers a `pre_llm_call` hook that names the skill on the first turn of each session. Hermes appends that to the user message and never persists it, so the system prompt stays byte-stable and the prompt cache holds. ## The memory provider The provider needs two installs, because it is two different things. The *directory* is what Hermes scans to discover memory providers, and the *package* is what the provider imports. Providers are found by directory, never through the `hermes_agent.plugins` entry point: ```bash hermes plugins install surrealdb/surrealfs/surrealfs/integrations/hermes_memory uv pip install --python ~/.hermes/hermes-agent/venv/bin/python \ "surrealfs @ git+https://github.com/surrealdb/surrealfs.git" hermes memory setup surrealfs-memory ``` `hermes plugins install` accepts a subdirectory of a repository and takes the installed name from `plugin.yaml`, which is why the directory lands as `surrealfs-memory`. That is the name Hermes matches the active-provider setting against. > [!NOTE] > Answer **N** to install's `Enable now?` prompt. That prompt belongs to the general plugin manager, which this directory does not go through: `kind: exclusive` routes it to memory discovery instead, and `hermes memory setup` is what activates it. `hermes memory setup` walks every setting below, offering the current value or the default, writes what you change to `$HERMES_HOME/.env`, activates the provider in `~/.hermes/config.yaml`, and then connects, so a wrong URL surfaces during setup rather than as a per-turn warning. Only one provider can be active at a time. To activate one by hand: ```yaml memory: provider: surrealfs-memory ``` Two commands report on it afterwards. `hermes memory` lists the discovered providers and whether each reports itself available. `hermes surrealfs-memory status` re-runs the connection check at any time, naming the database it resolved and counting the turns filed there. Upgrades are two commands for the same reason installs are: `hermes plugins update surrealfs-memory` for the directory, and the `uv pip install` again for the package. Run the second again whenever `hermes memory` starts reporting the provider unavailable, because a virtualenv rebuild strips it. ### Settings Beyond the [connection settings](#connection-settings), the provider adds two of its own: | Variable | Default | Purpose | | --- | --- | --- | | `SURREALFS_AGENT_USER` | Your Unix username | The agent's home under `/home` | | `SURREALFS_MEMORY_DIR` | `/home//memories` | Folder to file turns under | Set `SURREALFS_AGENT_USER` whenever two agents would otherwise share a home: two Hermes installs owned by different people against one database, or an agent running under the same account as its human. `hermes-tobie` alongside the human's own `/home/tobie` is the shape to aim for. On a VM or sandbox where the agent has an account to itself, the default already matches the machine. ### What gets filed Each completed turn becomes its own file: ```text /home/hermes-tobie/memories/default/2026-08-06/a1b2c3-142251003.md ``` ```markdown # 2026-08-06 14:22:51 UTC session: a1b2c3 ## User how do I get paid for the consulting work ## Assistant You invoice monthly through ... ## Tools → surrealfs_search({"query": "invoice"}) → surrealfs_cat({"path": "/projects/acme/contract.md"}) ``` One file per turn rather than one per session, because a search hit is then a single exchange with its snippet centred on the match. The `## Tools` section records which files were read and which commands ran. It is often the substance of the turn and appears in neither message's text, and it is omitted when the turn made no tool calls. Turns where both sides carry no signal are skipped: a prompt like "ok" or "thanks", answered briefly. A trivial prompt with a long answer is still filed, because "go ahead" followed by a design document is a turn worth keeping. Anything that is not a primary agent is skipped too, so a cron run's system prompt is never filed as a memory and cannot corrupt what the agent believes about the user. The filename ends in a wall-clock stamp (`HHMMSSmmm`) rather than a turn number. A counter has to live somewhere, and in-process is the one place it cannot: `hermes --resume` starts a second process on the same session id and restarts counting at one, so turn one of the resumed conversation would overwrite turn one of the original. Two other things are filed under the same profile subtree: - **Hermes' own memory.** When the built-in memory tool writes to `MEMORY.md` or `USER.md`, the same change is applied to `//builtin/memory.md` or `.../user.md`. That mirrors what those files say rather than logging the writes, so recall reads the current state. - **Messages about to be compressed away.** Before Hermes compresses a long conversation it asks each provider what to preserve. This provider writes the doomed messages out verbatim, tool calls included, and hands the compressor a pointer to the file, so the detail survives at full fidelity and the agent can read it back with `surrealfs_cat`. > [!IMPORTANT] > Everything filed here goes to the SurrealDB at your `SURREALDB_URL` and nowhere else. Hermes hands memory providers the turn's full message list; this provider writes part of it to your own database, and it leaves your machine only if you pointed it at a remote server yourself. ### What gets recalled Before each turn the provider searches the whole filesystem, excluding other agents' homes. The notes the agent wrote itself under `/preferences/` and `/projects/` are the highest-signal memories present, so excluding them would mean the bundled `surrealfs:notes` skill and the provider ignored each other's work. Filed turns are held to two of the five recalled slots, and the current conversation's own turns are never recalled at all. Recall reads the tree it writes, and one turn per file means short documents, which the ranking favours. Without a cap the top five drifts into verbatim transcript and the curated notes stop showing up. The prompt is cut down to search terms before it reaches the index: stopwords out, a dozen terms at most. Every word in a whole paragraph is another `OR` branch, and matching rows are read back in full to be ranked. The search runs after the previous turn rather than during the current one, through Hermes' `queue_prefetch` and `prefetch` pair, so nothing waits on it, in particular not the embedding round trip that `SURREALFS_SEMANTIC=1` adds. The cost is that a recall is one turn behind: switch topic abruptly and the first turn on the new subject still carries context for the old one. The provider registers no tools of its own. The 14 `surrealfs_*` tools from the toolset already cover explicit reads and writes, and a second set would collide in Hermes' flat tool namespace. ### Homes and profiles Turns are filed under `/home//memories//`. Two segments, because there are two ways for memories to end up in one database that should not mix. `` is `SURREALFS_AGENT_USER`, or the Unix account the agent runs as. A database shared between people holds one home per agent, plus the humans' own. The notes skill already treats `/home//` as the agent's working directory, so the memories sit inside the folder the agent already owns, and recall reads that home and no other. `` comes from the `hermes_home` that Hermes passes to `initialize()`: `default` for `~/.hermes`, and the directory name for anything under `/profiles/`. It separates one agent's own profiles, which share a home and a machine account. Everything outside `/home/` stays shared. `/preferences/`, `/projects/`, and anything else written with the `surrealfs_*` tools is the handover point between an agent and its user, or between two agents working for the same person. Anything that must not be shared at all belongs in a `SURREALDB_DATABASE` of its own, and since `.env` lives in `$HERMES_HOME`, that is already per-profile. > [!WARNING] > `hermes backup` captures none of this. It walks `HERMES_HOME`, and the memories are in SurrealDB. Use [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) instead. ## Semantic search `SURREALFS_SEMANTIC=1` adds a vector arm to `surrealfs_search` and to recall, so a note about invoicing surfaces for a query of "how do I get paid". Both arms run and are fused by rank, because full-text scores are BM25 and vector scores are distances. The full-text arm matches a file that shares any term with the query, so search works before any vector exists; the vector arm is what makes ranking follow meaning rather than shared words. It needs `OPENAI_API_KEY` and the `embed` extra, which neither plugin's install pulls in. ### Keep the vectors current Skip this unless you set `SURREALFS_SEMANTIC=1`. A file is not embedded as it is written, so recall matches on meaning only for what the indexer has already reached. Recall keeps working on its full-text arm the whole time the vectors are behind. Hermes has no way for a plugin to declare a daemon of its own, but its scheduler runs scripts without an agent, which covers this without a systemd unit or a terminal left open: ```bash uv pip install --python ~/.hermes/hermes-agent/venv/bin/python \ "surrealfs[embed] @ git+https://github.com/surrealdb/surrealfs.git" mkdir -p "${HERMES_HOME:-$HOME/.hermes}/scripts" cat > "${HERMES_HOME:-$HOME/.hermes}/scripts/surrealfs-embed.sh" <<'SCRIPT' #!/usr/bin/env bash set -a; . "${HERMES_HOME:-$HOME/.hermes}/.env"; set +a exec ~/.hermes/hermes-agent/venv/bin/python -m surrealfs.embed --once SCRIPT hermes cron create 'every 5m' --no-agent --name surrealfs-embed \ --script surrealfs-embed.sh ``` `--no-agent` skips the inference layer entirely, so a `--once` pass per tick costs no tokens and replaces the polling loop the daemon would otherwise run. The script must live under `$HERMES_HOME/scripts/`, because paths outside it are rejected. Cron sanitises the subprocess environment and `OPENAI_API_KEY` is on the blocklist, so the key and the `SURREALDB_*` details both have to come from the `.env` the script sources. An idle pass prints nothing, and empty output is a silent tick, so you hear from this only when it has news. A pass that embedded something reports `embedded 4 file(s)`, and a pass that fails exits non-zero, which Hermes reports as an alert. `hermes cron runs` has the history. Two things to know before relying on it. The scheduler ticks inside the gateway daemon, so this runs only while that does; run `hermes gateway install` if it is not already a service. And a five-minute period means a file written now is matchable by meaning in up to five minutes, and by term immediately. Lower the interval if that gap matters, at a cost of one `SELECT` per tick that returns nothing when there is no work. ## Browse the filesystem Point the agents at a shared SurrealDB, a [Cloud instance](/docs/manage/instances.md) for example, and everyone on the team can run the file browser on their own machine to see and edit the same tree the agents write to: ```bash pip install "surrealfs[browser]" surrealfs-browser # http://127.0.0.1:7933 ``` Tree on the left, file on the right. Text is editable and saves back to the table, markdown renders with a source toggle, agent-authored HTML renders in a sandboxed iframe, images display, and the search box is the same hybrid search behind the agent's own tool calls. Credentials come from a `.env` in the working directory or from the environment, which wins over the file. Every variable has a command-line form too; run `surrealfs-browser --help` for the list. Add `SURREALDB_AUTH_LEVEL` when the credentials are not root ones: the server infers the user kind from the signin payload, and a database-scoped Cloud credential needs `database`. All three levels are system users, so they bypass the `file` table's `owner = $auth.id` permissions and everyone sharing the database sees the one shared tree, which is the point. > [!WARNING] > The browser binds loopback. `--host 0.0.0.0` exposes it, and anyone who can then reach the port has whatever access those credentials do, because the page has no login of its own. ## Next steps - [SurrealDB filesystem](https://github.com/surrealdb/surrealfs) - the repository, the full schema in `surrealfs/schema/file.surql`, and the plain async Python API to build your own integration on - [Hermes toolset README](https://github.com/surrealdb/surrealfs/blob/main/surrealfs/integrations/hermes/README.md) - the tools, the prefix, and the bundled skill - [Hermes memory provider README](https://github.com/surrealdb/surrealfs/blob/main/surrealfs/integrations/hermes_memory/README.md) - what gets filed, what gets recalled, and how homes and profiles divide it - [Keeping the vectors current under Hermes](https://github.com/surrealdb/surrealfs/blob/main/docs/hermes-indexer.md) - the indexer cron job in full - [SurrealDB Agent Memory integration for Hermes](/docs/agent-memory/integrations/frameworks/hermes.md) - hosted memory with fact extraction and semantic recall, as an alternative provider --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/kreuzberg # Kreuzberg Integrate Kreuzberg document intelligence extraction pipelines with SurrealDB. Kreuzberg is a polyglot document intelligence framework that allows you to extract text, metadata, images, and structured information from PDFs, Office documents, images, and 91+ formats. The `kreuzberg-surrealdb` package connects Kreuzberg's document extraction pipeline to SurrealDB. It handles schema creation, content deduplication, optional chunking and embedding, and index configuration. ## How it works 1. **Extract** - Kreuzberg parses the source documents and runs OCR where needed. 2. **Connect** - The connector receives the extracted output and manages the SurrealDB connection. 3. **Store** - Each document is hashed (SHA-256) for deduplication, optionally chunked and embedded, then written to SurrealDB under an auto-generated schema. 4. **Search** - Full-text (BM25), vector (HNSW), and hybrid (RRF) search are available immediately after ingestion. ## Key capabilities - **Schema management** - `setup_schema()` creates tables, indexes, and analyzers. No manual DDL required. - **Deduplication** - Deterministic record IDs derived from content hashes prevent duplicate rows across ingestion runs. - **Flexible ingestion** - Single files, file lists, directories (with glob), or raw bytes. - **Extraction control** - Pass Kreuzberg's `ExtractionConfig` to set OCR behaviour, output format, and quality processing. - **Batch tuning** - Adjust `insert_batch_size` to balance throughput against memory usage. ## Supported formats Formats supported by Kreuzberg are as follows: * Document formats: "pdf", "docx", "docm", "dotx", "dotm", "dot", "doc", "odt", "pptx", "ppsx", "pptm", "potx", "potm", "pot", "ppt", "xlsx", "xlsm", "xlsb", "xlam", "xla", "xltx", "xlt", "xls", "ods", "dbf", "hwp", "hwpx" * Text formats: "txt", "md", "markdown", "commonmark", "html", "htm", "xml", "rtf", "rst", "org" * Data formats: "json", "yaml", "yml", "toml", "csv", "tsv" * Email formats: "eml", "msg" * Archives: "zip", "tar", "gz", "tgz", "7z" * Images (OCR supported): "bmp", "gif", "jpg", "jpeg", "png", "tiff", "tif", "webp", "jp2", "jpx", "jpm", "mj2", "j2k", "j2c", "jbig2", "jb2", "pnm", "pbm", "pgm", "ppm" * Academic / publishing formats: "epub", "fb2", "bib", "ris", "nbib", "enw", "ipynb", "tex", "latex", "typst", "typ" * Markup / structured formats: "opml", "dbk", "docbook", "jats" * Other: "svg", "djot" For more information on supported formats, see the [Kreuzberg docs](https://docs.kreuzberg.dev). ## Getting started **Python** To get started if using Python, visit [SurrealDB in the Kreuzberg Docs](https://kreuzberg.dev). For the complete API reference, embedding model options, chunking configuration, and database schema details, see the [Kreuzberg repository](https://github.com/Goldziher/kreuzberg). **Rust** As Kreuzberg is written in Rust and has its own [crate](https://crates.io/crates/kreuzberg), code can be written directly with much more boilerplate but also quite a bit of manual customisation. The following example shows a setup somewhat similar to the Python extension. It demonstrates a number of file types used to populate the same SurrealDB instance, in this case a JSON file `demo.json`, a markdownfile `demo.md`, and even an HWPX file `demo.hwpx` file that are assumed to be in a folder `/assets`. Most LLMs are able to generate sample files of these types such as the content below in JSON. ```json title="Sample data" { "documents": [ { "title": "English Quarterly Revenue", "language": "en", "content": "Quarterly revenue increased significantly this quarter." }, { "title": "English Product", "language": "en", "content": "Product launch includes search and analytics." }, { "title": "한국어 문서", "language": "ko", "content": "이 문서는 분기 매출과 검색 기능을 설명합니다." }, { "title": "日本語 文書", "language": "ja", "content": "この文書は四半期売上と検索について説明します。" }, { "title": "Mixed Doc", "language": "mixed", "content": "Quarterly revenue 데이터 日本語 検索 mixed language example." } ] } ``` The code can be run with either `cargo run -- memory` or `cargo run -- local`, depending on if you prefer a one-time embedded instance or a local instance that can be connected to via SurrealDB Studio to manually query the data after the Rust code has run. For more ideas on how to redo this example to suit your own needs, see the [Kreuzberg repository](https://github.com/Goldziher/kreuzberg). ```rust // Required features to run: // kreuzberg = { version = "4.9.4", default-features = true, features = ["hwpx", "language-detection"] } // surrealdb = { version = "3.2.0", default-features = true, features = ["protocol-ws", "kv-mem"] } use anyhow::{Context, Result}; use kreuzberg::{extract_file, ExtractionConfig, LanguageDetectionConfig}; use sha2::{Digest, Sha256}; use std::{ fs::read_to_string, path::{Path, PathBuf}, }; use surrealdb::{ engine::any::connect, opt::auth::Root, types::{ToSql, Value}, }; use kreuzberg::types::ExtractionResult; use serde::{Deserialize, Serialize}; use surrealdb::types::{RecordId, SurrealValue}; /// One logical document inside `assets/demo.json` (`documents` array). #[derive(Debug, Clone, Deserialize, Serialize)] pub struct JsonDocumentSpec { pub title: String, #[serde(default)] pub language: Option, pub content: String, } #[derive(Debug, Deserialize)] pub struct JsonDocumentsFile { pub documents: Vec, } /// Parse `demo.json`-style bundles to ingest records pub fn parse_json_documents_file(path: &Path) -> anyhow::Result { let raw = read_to_string(path).with_context(|| format!("read {}", path.display()))?; serde_json::from_str(&raw) .with_context(|| format!("parse JSON documents bundle {}", path.display())) } #[derive(Debug, Clone, SurrealValue)] pub struct ExtractionMetadataNested { subject: Option, language: Option, modified_at: Option, extraction_duration_ms: Option, } #[derive(Debug, Clone, SurrealValue)] pub struct ConnectorDocument { id: RecordId, source: String, content: String, mime_type: String, title: Option, authors: Option>, created_at: Option, metadata: ExtractionMetadataNested, quality_score: Option, content_hash: String, detected_languages: Option>, keywords: Option>, } /// Build the SurrealDB document payload for `INSERT IGNORE …`. pub fn document_record_value( table: &str, extracted: &ExtractionResult, source_label: String, content_hash: &str, ) -> ConnectorDocument { let meta = &extracted.metadata; ConnectorDocument { id: RecordId::new(table, content_hash), source: source_label, content: extracted.content.clone(), mime_type: extracted.mime_type.as_ref().to_string(), title: meta.title.clone(), authors: meta.authors.clone(), created_at: meta.created_at.clone(), metadata: ExtractionMetadataNested { subject: meta.subject.clone(), language: meta.language.clone(), modified_at: meta.modified_at.clone(), extraction_duration_ms: meta.extraction_duration_ms, }, quality_score: extracted.quality_score, content_hash: content_hash.to_string(), detected_languages: extracted.detected_languages.clone(), keywords: meta.keywords.clone(), } } /// Build a connector row from a JSON catalogue entry (no Kreuzberg pass - plain text is already known). pub fn document_record_from_json_entry( table: &str, entry: &JsonDocumentSpec, source: String, content_hash: &str, ) -> ConnectorDocument { ConnectorDocument { id: RecordId::new(table, content_hash), source, content: entry.content.clone(), mime_type: "application/json".to_string(), title: Some(entry.title.clone()), authors: None, created_at: None, metadata: ExtractionMetadataNested { subject: None, language: entry.language.clone(), modified_at: None, extraction_duration_ms: None, }, quality_score: None, content_hash: content_hash.to_string(), detected_languages: entry.language.as_ref().map(|l| vec![l.clone()]), keywords: None, } } const TABLE: &str = "documents"; fn sha256_hex(content: &str) -> String { let mut h = Sha256::new(); h.update(content.as_bytes()); format!("{:x}", h.finalize()) } pub fn connector_schema(table: &str) -> String { format!( "DEFINE ANALYZER IF NOT EXISTS doc_analyzer TOKENIZERS class FILTERS lowercase,snowball(english); DEFINE TABLE IF NOT EXISTS {table} SCHEMAFULL; DEFINE FIELD IF NOT EXISTS source ON TABLE {table} TYPE string; DEFINE FIELD IF NOT EXISTS content ON TABLE {table} TYPE string; DEFINE FIELD IF NOT EXISTS mime_type ON TABLE {table} TYPE string; DEFINE FIELD IF NOT EXISTS title ON TABLE {table} TYPE option; DEFINE FIELD IF NOT EXISTS authors ON TABLE {table} TYPE option>; DEFINE FIELD IF NOT EXISTS created_at ON TABLE {table} TYPE option; DEFINE FIELD IF NOT EXISTS ingested_at ON TABLE {table} TYPE datetime DEFAULT time::now(); DEFINE FIELD IF NOT EXISTS metadata ON TABLE {table} TYPE object FLEXIBLE; DEFINE FIELD IF NOT EXISTS quality_score ON TABLE {table} TYPE option; DEFINE FIELD IF NOT EXISTS content_hash ON TABLE {table} TYPE string; DEFINE FIELD IF NOT EXISTS detected_languages ON TABLE {table} TYPE option>; DEFINE FIELD IF NOT EXISTS keywords ON TABLE {table} TYPE option>; DEFINE INDEX IF NOT EXISTS idx_doc_source ON TABLE {table} FIELDS source UNIQUE; DEFINE INDEX IF NOT EXISTS idx_doc_hash ON TABLE {table} FIELDS content_hash UNIQUE; DEFINE INDEX IF NOT EXISTS idx_doc_content ON TABLE {table} FIELDS content FULLTEXT ANALYZER doc_analyzer BM25(1.2,0.75) HIGHLIGHTS;" ) } // Add files such as demo.md, demo.hwpx, demo.json to /assets folder fn default_asset_paths() -> Vec { let assets = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("assets"); ["demo.md", "demo.hwpx", "demo.json"] .into_iter() .map(|name| assets.join(name)) .collect() } fn json_entry_source_label(path: &Path, index: usize) -> String { format!("{}#{}", path.display(), index) } async fn ingest_path( path: &Path, table: &str, config: &ExtractionConfig, ) -> Result> { let path = path .canonicalize() .with_context(|| format!("resolve path {}", path.display()))?; if !path.is_file() { anyhow::bail!("not a file: {}", path.display()); } let ext = path .extension() .and_then(|e| e.to_str()) .unwrap_or("") .to_ascii_lowercase(); if ext == "json" { let bundle = parse_json_documents_file(&path)?; if bundle.documents.is_empty() { anyhow::bail!("empty `documents` array in {}", path.display()); } let mut records = Vec::with_capacity(bundle.documents.len()); for (i, entry) in bundle.documents.iter().enumerate() { let hash = sha256_hex(&entry.content); let source = json_entry_source_label(&path, i); records.push(document_record_from_json_entry(table, entry, source, &hash)); } Ok(records) } else { let extracted = extract_file(&path, None, config) .await .with_context(|| format!("Kreuzberg extract {}", path.display()))?; if !extracted.processing_warnings.is_empty() { eprintln!( "Kreuzberg warnings ({}): {:?}", path.display(), extracted.processing_warnings ); } let hash = sha256_hex(extracted.content.as_str()); Ok(vec![document_record_value( table, &extracted, path.display().to_string(), &hash, )]) } } fn usage() -> &'static str { "usage: kreuzberg-surrealdb-example memory|local \n\ \n\ memory - SurrealDB in-memory (no server).\n\ local - SurrealDB at ws://localhost:8000 (expects root/secret, ns/db main)." } #[tokio::main] async fn main() -> Result<()> { let mut args = std::env::args_os().skip(1).collect::>(); if args.is_empty() { eprintln!("{}\n", usage()); anyhow::bail!("missing mode: memory or local"); } let mode = args.remove(0); let mode_str = mode .to_str() .ok_or_else(|| anyhow::anyhow!("mode must be valid UTF-8"))?; let endpoint = match mode_str { "memory" => "memory", "local" => "ws://localhost:8000", _ => { eprintln!("{}\n", usage()); anyhow::bail!( "first argument must be `memory` or `local`, got {:?}", mode_str ); } }; let paths: Vec = if args.is_empty() { default_asset_paths() } else { args.into_iter().map(PathBuf::from).collect() }; let config = ExtractionConfig { chunking: None, language_detection: Some(LanguageDetectionConfig { enabled: true, min_confidence: 0.55, detect_multiple: true, }), ..Default::default() }; let mut all_docs = Vec::new(); for p in &paths { let mut records = ingest_path(p, TABLE, &config).await?; all_docs.append(&mut records); } //eprintln!("Ingested {} record(s) from {} path(s).", all_docs.len(), paths.len()); let client = connect(endpoint).await?; client.use_ns("main").use_db("main").await?; if endpoint == "local" { client .signin(Root { username: "root".into(), password: "secret".into(), }) .await?; } client.query(connector_schema(TABLE)).await?.check()?; for doc in &all_docs { client .query(format!("INSERT IGNORE INTO {TABLE} $doc;")) .bind(("doc", doc.clone())) .await? .check()?; } let records: Value = client .query( "SELECT source, content, mime_type, title, metadata.language AS language, search::score(0) AS score FROM documents WHERE content @0@ 'product' ORDER BY score DESC LIMIT 10;", ) .await? .check()? .take(0)?; println!("BM25 hits:\n{}", records.to_sql_pretty()); Ok(()) } ``` --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/langchain # LangChain Use SurrealDB native vector indexes as a drop-in vector store for LangChain. [LangChain](https://python.langchain.com/docs/introduction/) is a framework for developing applications powered by large language models (LLMs). SurrealDB is an excellent database choice for [LangChain](https://python.langchain.com/docs/introduction/) projects primarily because of its [multi-model capabilities](/docs/learn/data-models.md), which streamline data management by requiring only a single database. This unified system adeptly handles structured and unstructured data, incorporating vector search, [graph traversal](/docs/learn/data-models/graph/overview.md), [relational queries](/docs/learn/data-models.md), [full-text search](/docs/learn/data-models/full-text-search/overview.md), [document storage](/docs/learn/data-models/document/overview.md), and [time-series data](/docs/learn/data-models/time-series/overview.md) all within one ACID-compliant engine. For LangChain applications, which often juggle diverse data types for tasks like context retrieval and complex data interactions, SurrealDB's ability to consolidate these needs into one platform simplifies architecture, reduces latency, and ensures data consistency, making it a highly efficient and powerful backend solution. In this guide, we'll walk through how to use SurrealDB as a vector store for LangChain. ## Setup You can run SurrealDB locally or start with a [free SurrealDB Cloud account](/docs/manage/instances.md). For local, two options: 1. [Install SurrealDB](/docs/running/installation.md) and run [SurrealDB](/docs/running/in-memory.md). Run in-memory with: ```sh surreal start -u root -p secret ``` 2. [Run with Docker](/docs/running/docker.md). ```sh docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start ``` ## Install dependencies ```bash # -- Using pip pip install -U langchain-surrealdb langchain_ollama surrealdb # -- Using poetry poetry add langchain-surrealdb langchain_ollama surrealdb # -- Using uv uv add --upgrade langchain-surrealdb langchain_ollama surrealdb ``` * `surrealdb` → [SurrealDB Python SDK](/docs/reference/python/) * `langchain-surrealdb` → houses `SurrealDBVectorStore` * `langchain_ollama`, `langchain-openai` (or HF, Cohere, etc.) → embeddings ## Quick start Create a vector store, and documents with embeddings, and do a similarity search. ```python from langchain_core.documents import Document from langchain_surrealdb.vectorstores import SurrealDBVectorStore from langchain_ollama import OllamaEmbeddings from surrealdb import Surreal conn = Surreal("ws://localhost:8000/rpc") conn.signin({"username": "root", "password": "secret"}) conn.use("langchain", "demo") vector_store = SurrealDBVectorStore(OllamaEmbeddings(model="llama3.2"), conn) doc_1 = Document(page_content="foo", metadata={"source": "https://surrealdb.com"}) doc_2 = Document(page_content="SurrealDB", metadata={"source": "https://surrealdb.com"}) vector_store.add_documents(documents=[doc_1, doc_2], ids=["1", "2"]) results = vector_store.similarity_search_with_score( query="surreal", k=1, custom_filter={"source": "https://surrealdb.com"} ) for doc, score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") ``` Under the hood the helper will: 1. Create table **`documents`** (if it doesn’t exist). 2. Add an **HNSW** index with the correct dimensionality, using cosine distance and `F32` vectors. 3. Insert each text with its freshly generated embedding. ## Similarity search ```python query = "How do I enable vector search in SurrealDB?" docs = vector_store.similarity_search( query=query, k=1, custom_filter={"source": "https://surrealdb.com"} ) for doc in results: print(f"{doc.page_content} [{doc.metadata}]") ``` ```text The Vector Search feature of SurrealDB... [{'source': 'https://surrealdb.com'}] ``` If you want to get the score with the results, use `similarity_search_with_score` instead. You can also transform the vector store into a retriever for easier usage in your chains. ```python query = "How do I enable vector search in SurrealDB?" docs = vector_store.similarity_search( retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={"k": 1, "lambda_mult": 0.5} ) retriever.invoke(query) ``` ```text [Document(id='4', metadata={'source': 'https://surrealdb.com'}, page_content='The Vector Search feature of SurrealDB...')] ``` ## Next steps Now that you have a basic understanding of how to use SurrealDB with LangChain, let's explore some additional resources to help you dive deeper and build more sophisticated applications. To help you get started quickly, we provide several example implementations: * A [basic example](https://github.com/surrealdb/langchain-surrealdb/tree/main/examples/basic) with a ready-to-use Dockerfile - perfect for your first steps * A more advanced [graph example](https://github.com/surrealdb/langchain-surrealdb/tree/main/examples/graph) showcasing graph capabilities * An interactive [Jupyter notebook](https://python.langchain.com/docs/integrations/vectorstores/surrealdb/) for hands-on learning ### Further reading and resources For a deeper understanding of the technology stack: * Explore the [SurrealDB vector-search reference](/docs/learn/data-models/vector-search/overview.md) for detailed technical information * Check out the [LangChain API docs for `SurrealDBStore`](https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.surrealdb.SurrealDBStore.html) for comprehensive API documentation * Browse [Awesome SurrealDB](https://github.com/surrealdb/awesome-surreal) for a curated collection of resources, tools, and applications That’s it - you now have a fully-featured LangChain vector store powered by SurrealDB, no boilerplate required. 🚀 --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/llamaindex # LlamaIndex Use SurrealDB’s native HNSW vector index as the backing store for a Llama Index `VectorStoreIndex`. Llama Index is a framework for building RAG pipelines. It provides a flexible and powerful way to build and deploy RAG systems. > [!NOTE] > Llama Index doesn’t (yet!) ship an off-the-shelf SurrealDB adapter, so in this guide we’ll implement a **20-line `SurrealVectorStore`** that satisfies Llama Index’s minimal vector-store interface. You can drop it into any RAG pipeline just like you would with Pinecone, Qdrant, Chroma, etc. ## TL;DR 1. **Install** `surrealdb` + `llama-index`. 2. **Create** a table with an `HNSW` index. 3. **Drop-in** the 20-line `SurrealVectorStore`. 4. Use `VectorStoreIndex.from_documents()` and you’re done. ## Install the libraries ```bash pip install surrealdb llama-index numpy ``` ## Connect to SurrealDB ```python from surrealdb import Surreal import asyncio, os # - connection details - DB_URL = os.getenv("SDB_URL", "http://localhost:8000/rpc") DB_USER = os.getenv("SDB_USER", "root") DB_PASS = os.getenv("SDB_PASS", "secret") NS, DB = "demo", "demo" TABLE = "llama_chunks" sdb = Surreal(DB_URL) async def init(): await sdb.signin({"user": DB_USER, "pass": DB_PASS}) await sdb.use(NS, DB) asyncio.run(init()) ``` ## Create the table and HNSW index ```python # SurrealDB ≥ v1.5 exposes an in-memory HNSW ANN index schema = """ DEFINE TABLE $tb SCHEMALESS PERMISSIONS NONE; DEFINE FIELD text ON $tb TYPE string; DEFINE FIELD embedding ON $tb TYPE array; DEFINE INDEX idx_hnsw ON $tb FIELDS embedding HNSW DIMENSION 768 DIST COSINE; """ asyncio.run(sdb.query(schema, {"tb": TABLE})) ``` SurrealQL’s `DEFINE INDEX … HNSW` activates a high-speed, cosine-distance ANN index. ## A tiny SurrealDB adapter for Llama Index ```python import numpy as np from typing import Any, List, Sequence from llama_index.core.vector_stores.types import ( BasePydanticVectorStore, VectorStoreQuery, VectorStoreQueryResult, ) from llama_index.core.schema import NodeWithEmbedding class SurrealVectorStore(BasePydanticVectorStore): """Minimal Llama Index adapter using SurrealDB’s <|K,EF|> operator.""" def __init__(self, table: str, conn: Surreal): self.table, self.conn = table, conn # ---- required APIs ---------------------------------------------------- def add(self, nodes: Sequence[NodeWithEmbedding], **_) -> List[str]: rows = [ { "id": f"{self.table}:{i}", "text": n.get_content(metadata_mode="all"), "embedding": n.embedding, } for i, n in enumerate(nodes) ] asyncio.run(self.conn.query(f"INSERT INTO {self.table} $data", {"data": rows})) return [r["id"] for r in rows] def delete(self, doc_id: str, **_) -> None: # optional but easy asyncio.run(self.conn.delete(doc_id)) def query( self, query: VectorStoreQuery, **_ ) -> VectorStoreQueryResult: # semantic search q_vec = query.query_embedding surql = """ LET $q := $vec; SELECT id, text, vector::distance::knn() AS score FROM {self.table} WHERE embedding <|{query.similarity_top_k},64|> $q ORDER BY score; """ res = asyncio.run(self.conn.query(surql, {"vec": q_vec}))[0].result return VectorStoreQueryResult( nodes=[], # Llama Index will fetch raw docs separately ids=[r["id"] for r in res], similarities=[r["score"] for r in res], ) ``` The `SurrealVectorStore` class is a minimal adapter cribbed from the official [build a vector store from scratch](https://docs.llamaindex.ai/en/stable/examples/low_level/vector_store/) example. ## Index documents with Llama Index ```python from llama_index.core import VectorStoreIndex, SimpleDirectoryReader # 5-1 load + chunk a handful of docs (replace with your own) docs = SimpleDirectoryReader("./my_pdfs").load_data() # 5-2 build the index - Llama Index will call SurrealVectorStore.add() surreal_store = SurrealVectorStore(table=TABLE, conn=sdb) index = VectorStoreIndex.from_documents( docs, vector_store=surreal_store, show_progress=True, ) ``` ## Query ```python qe = index.as_query_engine(similarity_top_k=4) resp = qe.query("Which document talks about vector search in SurrealDB?") print(resp) ``` The query flow is: ```text User → Llama Index → SurrealVectorStore.query() ↘︎ top-K doc IDs & scores ↘︎ fetch full text → synthesize answer ``` ## What about metadata filters, deletes, hybrid search? SurrealDB supports **metadata-rich JSON payloads**, additional **filter clauses**, and full-text search; extend `SurrealVectorStore.query()` to: * add `WHERE` predicates before the `<|K,EF|>` operator, * combine with `vector::distance::knn()` for re-ranking, * or fall back to `vector::similarity::cosine()` for exact search. All other Llama Index abstractions (retrievers, query engines, agents) work unchanged, because they talk to the vector store through the same tiny interface you just implemented. Enjoy Llama Index + SurrealDB! ## References - [DEFINE INDEX statement](/docs/reference/query-language/statements/define/indexes.md) - [Building a RAG with Astro, FastAPI, SurrealDB and Llama 3.1](https://fireworks.ai/blog/rag-with-astro-fastapi-surrealdb-tailwind) - [Building a (Very Simple) Vector Store from Scratch - LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/low_level/vector_store/) --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/mastra # Mastra Use SurrealDB as the storage backend for Mastra agents, covering conversation memory, workflow snapshots, scores, observability spans, and native vector search. [Mastra](https://mastra.ai) is a TypeScript framework for building AI agents, workflows, and RAG pipelines. [`@surrealdb/mastra-ai`](https://github.com/surrealdb/mastra-ai) provides `SurrealDBStore`, a storage adapter that backs Mastra with a single SurrealDB instance: conversation memory (threads, messages, and working memory), workflow suspend/resume snapshots, scores, observability spans, and [HNSW vector indexes](/docs/learn/data-models/vector-search/overview.md) for RAG. Because SurrealDB is [multi-model](/docs/learn/data-models.md), one database covers everything a Mastra application persists. Message history, workflow state, and vector embeddings live in the same ACID-compliant engine, so there is no separate vector database to deploy or keep in sync. > [!NOTE] > This page covers the storage adapter, which runs against a SurrealDB instance you manage. The same package also integrates with [SurrealDB Agent Memory](/docs/agent-memory/integrations/frameworks/mastra.md), a hosted memory provider for fact extraction and semantic recall. ## Requirements - [SurrealDB v3](/docs/running/installation.md), local or on [SurrealDB Cloud](/docs/manage/instances.md) - Bun 1+ or Node.js 22+ - `@mastra/core` 1.31.0+ ## Setup Start SurrealDB locally, either [installed directly](/docs/running/installation.md): ```sh surreal start --user root --pass secret memory ``` or [with Docker](/docs/running/docker.md): ```sh docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start --user root --pass secret ``` Alternatively, create a managed instance on [SurrealDB Cloud](/docs/manage/instances.md) and connect with its `wss://` URL. ## Install dependencies ```sh bun add @surrealdb/mastra-ai ``` ## Quick start Create a store, pass it to `Mastra` as `storage`, and call `store.init()` to connect and apply the table schemas. The agent's conversation history then persists in SurrealDB, keyed by `resourceId` and `threadId`: ```typescript import { Mastra } from "@mastra/core/mastra"; import { Agent } from "@mastra/core/agent"; import { anthropic } from "@ai-sdk/anthropic"; import { SurrealDBStore } from "@surrealdb/mastra-ai"; const store = new SurrealDBStore({ id: "my-store", url: "ws://localhost:8000", username: "root", password: "secret", namespace: "mastra", database: "my_app", }); const agent = new Agent({ name: "assistant", instructions: "You are a helpful assistant.", model: anthropic("claude-sonnet-4-6"), }); const mastra = new Mastra({ agents: { assistant: agent }, storage: store, }); await store.init(); const response = await mastra.getAgent("assistant").generate("Hello!", { resourceId: "user-001", threadId: "thread-001", }); console.log(response.text); await store.close(); ``` ## Configuration `SurrealDBStore` accepts three configuration shapes. `namespace` and `database` are optional in the first two and both default to `mastra`. Username and password: ```typescript new SurrealDBStore({ id: "my-store", url: "ws://localhost:8000", username: "root", password: "secret", namespace: "mastra", database: "my_app", }); ``` Token authentication, for example against a SurrealDB Cloud instance: ```typescript new SurrealDBStore({ id: "my-store", url: "wss://cloud.surrealdb.com", token: "", namespace: "mastra", database: "my_app", }); ``` A pre-connected `Surreal` instance from the [JavaScript SDK](/docs/reference/javascript.md), when your application already manages its own connection: ```typescript import { Surreal } from "surrealdb"; const db = new Surreal(); await db.connect("ws://localhost:8000"); new SurrealDBStore({ id: "my-store", db }); ``` ## Workflow suspend and resume Mastra workflows can suspend mid-run and resume later, for example to wait for human approval. With `SurrealDBStore` as storage, each snapshot is written atomically, so a run survives a process restart and resumes from the suspended step: ```typescript import { Mastra } from "@mastra/core/mastra"; import { createWorkflow, createStep } from "@mastra/core/workflows"; import { SurrealDBStore } from "@surrealdb/mastra-ai"; import { z } from "zod"; const store = new SurrealDBStore({ id: "store", url: "ws://localhost:8000", username: "root", password: "secret" }); const mastra = new Mastra({ storage: store }); const approveStep = createStep({ id: "approve", inputSchema: z.object({ value: z.number() }), resumeSchema: z.object({ approved: z.boolean() }), outputSchema: z.object({ approved: z.boolean() }), execute: async ({ inputData, resumeData, suspend }) => { if (!resumeData) { await suspend({}); } return { approved: resumeData!.approved }; }, }); const workflow = createWorkflow({ id: "approval", mastra, inputSchema: z.object({ value: z.number() }), outputSchema: z.object({ approved: z.boolean() }), steps: [approveStep], }).then(approveStep).commit(); await store.init(); const run = workflow.createRun(); await run.start({ inputData: { value: 42 } }); await run.resume({ step: approveStep, resumeData: { approved: true }, }); await store.close(); ``` ## Observational Memory `SurrealDBStore` is a supported backend for Mastra's [Observational Memory](https://mastra.ai/docs/memory/observational-memory), the observer/reflector system in `@mastra/memory` that compresses long message histories into observations. [Memory extractors](https://mastra.ai/blog/introducing-memory-extractors) also work, pulling structured facts out of conversations during observation cycles, and extracted values persist through the same SurrealDB tables. Observational Memory requires `@mastra/memory` 1.1.0+, and extractors require 1.22.0+: ```typescript import { Extractor, Memory } from "@mastra/memory"; import { SurrealDBStore } from "@surrealdb/mastra-ai"; import { z } from "zod"; const store = new SurrealDBStore({ id: "om-store", url: "ws://localhost:8000", username: "root", password: "secret", }); await store.init(); const memory = new Memory({ storage: store, options: { observationalMemory: { model: "anthropic/claude-haiku-4-5", observation: { extract: [ new Extractor({ name: "User profile", instructions: "Extract stable user profile facts.", schema: z.object({ preferredName: z.string().optional(), timezone: z.string().optional(), }), }), ], }, }, }, }); ``` ## Vector search SurrealDB v3 includes native [HNSW vector indexes](/docs/learn/data-models/vector-search/overview.md), so the same database that stores agent memory also serves RAG queries. The package exposes `SurrealDBClient` for raw SurrealQL, letting you define a schema, upsert documents with embeddings, and run a k-nearest-neighbour search: ```typescript import { SurrealDBClient } from "@surrealdb/mastra-ai"; const SCHEMA = ` DEFINE TABLE IF NOT EXISTS documents SCHEMAFULL; DEFINE FIELD IF NOT EXISTS content ON documents TYPE string; DEFINE FIELD IF NOT EXISTS embedding ON documents TYPE array; DEFINE INDEX IF NOT EXISTS idx_hnsw ON documents FIELDS embedding HNSW DIMENSION 1536 DIST COSINE; `; const client = new SurrealDBClient({ id: "rag", url: "ws://localhost:8000", username: "root", password: "secret" }); await client.connect(); await client.execute(SCHEMA); await client.execute( `UPSERT type::record('documents', $id) CONTENT $data`, { id: "doc-1", data: { content: "SurrealDB supports vector search.", embedding: [] } }, ); const results = await client.queryAll( `SELECT content, vector::distance::cosine(embedding, $qe) AS dist FROM documents WHERE embedding <|5|> $qe ORDER BY dist ASC`, { qe: [] }, ); ``` ## API ### `SurrealDBStore` | Member | Description | | ------ | ----------- | | `init()` | Connect and apply all table schemas | | `close()` | Disconnect | | `client` | The underlying `SurrealDBClient` for raw queries | | `stores` | Individual domain stores (`memory`, `workflows`, `scores`, `observability`) | ### `SurrealDBClient` | Method | Description | | ------ | ----------- | | `connect(config?)` | Open the WebSocket connection | | `close()` | Disconnect | | `queryAll(surql, bindings?)` | Run a query and return all rows | | `queryOne(surql, bindings?)` | Run a query and return the first row or `null` | | `execute(surql, bindings?)` | Run a statement with no return value | | `txBatch(statements, bindings?)` | Run statements in a single `BEGIN`/`COMMIT TRANSACTION` request | ## Next steps The [repository](https://github.com/surrealdb/mastra-ai) includes runnable examples for each area: - [basic-agent](https://github.com/surrealdb/mastra-ai/tree/main/examples/basic-agent): multi-turn agent conversation with SurrealDB memory - [workflow-persistence](https://github.com/surrealdb/mastra-ai/tree/main/examples/workflow-persistence): suspend/resume workflow with snapshot storage - [rag-pipeline](https://github.com/surrealdb/mastra-ai/tree/main/examples/rag-pipeline): vector similarity search with SurrealDB HNSW indexes - [observational-memory](https://github.com/surrealdb/mastra-ai/tree/main/examples/observational-memory): Observational Memory and extractors on SurrealDB For hosted memory with fact extraction and semantic recall, see the [SurrealDB Agent Memory integration for Mastra](/docs/agent-memory/integrations/frameworks/mastra.md). --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/overview # AI framework integrations This section contains information about the different frameworks that can be used to integrate with SurrealDB. SurrealDB integrates with popular AI and data frameworks, enabling you to leverage SurrealDB's powerful features like vector search, graph relationships, and structured data storage. These integrations make it easy to build sophisticated applications combining LLMs, agents, data pipelines and more - all while using familiar tools and frameworks.
Integration Description
Agno Agno is a python framework for building multi-agent systems with shared memory, knowledge and reasoning.
Camel A Python framework for building multi-agent LLM systems with SurrealDB vector storage capabilities.
CocoIndex An incremental indexing framework for AI agents with declarative SurrealDB table, relation, and vector targets.
CrewAI A framework for orchestrating role-playing AI agents with SurrealDB for entity and short-term memory.
Dagster A data orchestration framework with SurrealDB vector search integration for ML pipelines.
Google Agent A framework for building and deploying intelligent agents in Google Cloud with SurrealDB vector storage for RAG.
Hermes Nous Research's terminal agent, given a persistent filesystem in SurrealDB as a toolset and as a memory provider.
Kreuzberg A polyglot document intelligence framework to extract text, metadata, images, and structured information from documents (PDFs,images, etc.).
LangChain A framework for building LLM based applications.
Llama Index A framework for building RAG pipelines with SurrealDB's native HNSW vector index as the backing store.
Mastra A TypeScript agent framework with SurrealDB as a storage backend for conversation memory, workflow snapshots, and vector search.
Pydantic AI A Python framework designed to help you quickly, confidently, and painlessly build production grade applications and workflows with Generative AI.
Smol Agents A complete walkthrough for building a code-generating AI agent that recommends grocery items by querying SurrealDB's HNSW vector index.
--- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/pydantic-ai # Pydantic AI This section contains information about the Pydantic AI framework and how to integrate it with SurrealDB. [Pydantic AI](https://ai.pydantic.dev) is a Python agent framework designed to help you quickly, confidently, and painlessly build production grade applications and workflows with Generative AI. ## Example This is a simple RAG application that uses Pydantic AI and embedded SurrealDB. The integration is done by providing the agent with a custom retrieval tool, which takes a search query, executes a SurrealDB vector-search query, and returns the results. **To run the example:** Set up your OpenAI API key: **Bash** ```bash export OPENAI_API_KEY=your-api-key ``` **PowerShell** ```powershell $env:OPENAI_API_KEY = "your-api-key" ``` Or, store it in a .env file and add `--env-file .env` to your `uv run` commands. Build the vector store: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb build ``` Ask the agent a question: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb search "How do I register a function as a custom tool for my agent?" ``` Or use the web UI: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb web ``` ### Code ```python from __future__ import annotations as _annotations import asyncio import re import sys import unicodedata from collections.abc import Sequence from contextlib import asynccontextmanager from dataclasses import dataclass from pathlib import Path from typing import TypeVar import httpx import logfire import uvicorn from pydantic import BaseModel, TypeAdapter from surrealdb import ( AsyncEmbeddedSurrealConnection, AsyncHttpSurrealConnection, AsyncSurreal, AsyncWsSurrealConnection, RecordID, Value, ) from typing_extensions import AsyncGenerator from pydantic_ai import Agent, Embedder SurrealConn = ( AsyncWsSurrealConnection | AsyncHttpSurrealConnection | AsyncEmbeddedSurrealConnection ) # 'if-token-present' means nothing will be sent (and the example will work) if you don't have logfire configured logfire.configure(send_to_logfire='if-token-present') logfire.instrument_pydantic_ai() logfire.instrument_surrealdb() THIS_DIR = Path(__file__).parent SURREALDB_NS = 'pydantic_ai_examples' SURREALDB_DB = 'rag_surrealdb' SURREALDB_USER = 'root' SURREALDB_PASS = 'secret' embedder = Embedder('openai:text-embedding-3-small') agent = Agent('openai:gpt-5.2') RecordType = TypeVar('RecordType') class RetrievalQueryResult(BaseModel): url: str title: str content: str dist: float async def query( conn: SurrealConn, query_: str, vars_: dict[str, Value], record_type: type[RecordType], ) -> list[RecordType]: result = await conn.query(query_, vars_) result_ta = TypeAdapter(list[record_type]) rows = result_ta.validate_python(result) return rows @agent.tool_plain async def retrieve(search_query: str) -> str: """Retrieve documentation sections based on a search query. Args: search_query: The search query. """ with logfire.span( 'create embedding for {search_query=}', search_query=search_query ): result = await embedder.embed_query(search_query) embedding = result.embeddings # Embedder method guarantees there's one item here embedding_vector = embedding[0] # SurrealDB vector search using HNSW index async with database_connect(False) as db: rows = await query( db, """ SELECT url, title, content, vector::distance::knn() AS dist FROM doc_sections WHERE embedding <|8, 40|> $vector ORDER BY dist ASC """, {'vector': list(embedding_vector)}, RetrievalQueryResult, ) return '\n\n'.join( f'# {row.title}\nDocumentation URL:{row.url}\n\n{row.content}' for row in rows ) async def run_agent(question: str): """Entry point to run the agent and perform RAG based question answering.""" logfire.info('Asking "{question}"', question=question) answer = await agent.run(question) print(answer.output) # Web chat UI app = agent.to_web() ####################################################### # The rest of this file is dedicated to preparing the # # search database, and some utilities. # ####################################################### # JSON document from # https://gist.github.com/samuelcolvin/4b5bb9bb163b1122ff17e29e48c10992 DOCS_JSON = ( 'https://gist.githubusercontent.com/' 'samuelcolvin/4b5bb9bb163b1122ff17e29e48c10992/raw/' '80c5925c42f1442c24963aaf5eb1a324d47afe95/logfire_docs.json' ) def build_doc_rec_id(url: str) -> RecordID: return RecordID('doc_sections', slugify(url, '_')) async def build_search_db(): """Build the search database.""" async with httpx.AsyncClient() as client: response = await client.get(DOCS_JSON) response.raise_for_status() sections = sections_ta.validate_json(response.content) async with database_connect(True) as db: missing_sections: list[DocsSection] = [] for section in sections: url = section.url() record_id = build_doc_rec_id(url) existing = await db.select(record_id) if existing: logfire.info('Skipping {url=}', url=url) continue missing_sections.append(section) if missing_sections: with logfire.span('create embeddings'): result = await embedder.embed_documents( [section.embedding_content() for section in missing_sections] ) embeddings = result.embeddings for section, embedding_vector in zip( missing_sections, embeddings, strict=True ): await insert_doc_section(db, section, embedding_vector) else: logfire.info('All documents already exist; skipping embedding generation') async def insert_doc_section( db: SurrealConn, section: DocsSection, embedding_vector: Sequence[float], ) -> None: url = section.url() record_id = build_doc_rec_id(url) # Create record with embedding, using record ID directly res = await db.create( record_id, { 'url': url, 'title': section.title, 'content': section.content, 'embedding': list(embedding_vector), }, ) if not isinstance(res, dict): raise ValueError(f'Unexpected response from database: {res}') @dataclass class DocsSection: id: int parent: int | None path: str level: int title: str content: str def url(self) -> str: url_path = re.sub(r'\.md$', '', self.path) return ( f'https://logfire.pydantic.dev/docs/{url_path}/#{slugify(self.title, "-")}' ) def embedding_content(self) -> str: return '\n\n'.join((f'path: {self.path}', f'title: {self.title}', self.content)) sections_ta = TypeAdapter(list[DocsSection]) @asynccontextmanager async def database_connect( create_db: bool = False, ) -> AsyncGenerator[SurrealConn, None]: # Running SurrealDB embedded db_path = THIS_DIR / f'.{SURREALDB_DB}' db_url = f'file://{db_path}' requires_auth = False # Running SurrealDB in a separate process, connect with URL # db_url = 'ws://localhost:8000/rpc' # requires_auth = True async with AsyncSurreal(db_url) as db: # Sign in to the database if requires_auth: await db.signin({'username': SURREALDB_USER, 'password': SURREALDB_PASS}) # Set namespace and database await db.use(SURREALDB_NS, SURREALDB_DB) # Initialize schema if creating database if create_db: with logfire.span('create schema'): await db.query(DB_SCHEMA) yield db DB_SCHEMA = """ DEFINE TABLE doc_sections SCHEMALESS; DEFINE FIELD embedding ON doc_sections TYPE array; DEFINE INDEX hnsw_idx_doc_sections ON doc_sections FIELDS embedding HNSW DIMENSION 1536 DIST COSINE TYPE F32; """ def slugify(value: str, separator: str, unicode: bool = False) -> str: """Slugify a string, to make it URL friendly.""" # Taken unchanged from https://github.com/Python-Markdown/markdown/blob/3.7/markdown/extensions/toc.py#L38 if not unicode: # Replace Extended Latin characters with ASCII, i.e. `žlutý` => `zluty` value = unicodedata.normalize('NFKD', value) value = value.encode('ascii', 'ignore').decode('ascii') value = re.sub(r'[^\w\s-]', '', value).strip().lower() return re.sub(rf'[{separator}\s]+', separator, value) if __name__ == '__main__': action = sys.argv[1] if len(sys.argv) > 1 else None if action == 'build': asyncio.run(build_search_db()) elif action == 'search': if len(sys.argv) == 3: q = sys.argv[2] else: q = 'How do I configure logfire to work with FastAPI?' asyncio.run(run_agent(q)) elif action == 'web': uvicorn.run(app, host='127.0.0.1', port=7932) else: print( 'uv run --extra examples -m pydantic_ai_examples.rag_surrealdb build|search|web', file=sys.stderr, ) sys.exit(1) ``` --- Source: https://surrealdb.com/docs/build/integrations/ai-frameworks/smolagents # SmolAgents A complete walkthrough for building a code-generating AI agent that recommends grocery items by querying SurrealDB's HNSW vector index. In this guide you will build an agent that, given a natural-language shopping request, finds the most relevant grocery items in your database and returns them in a single reply. ## Install the dependencies ```bash pip install smolagents surrealdb fastembed datasets ``` | Library | Purpose | | -------------- | -------------------------------------- | | **surrealdb** | Async Python SDK for SurrealDB | | **smolagents** | Code-generating agent framework | | **fastembed** | Local Jina v2 embedding model (768-D) | | **datasets** | Pulls the public *GroceryList* dataset | ## Create a SurrealDB "grocery search" tool ```python from fastembed import TextEmbedding from surrealdb import AsyncSurreal from smolagents import Tool from datasets import load_dataset import asyncio, os from typing import List, Dict, Any class GroceryQueryTool(Tool): name = "surreal_grocery_search" description = "Semantic search over grocery items stored in SurrealDB." inputs = { "query": { "type": "string", "description": "A natural-language description of a grocery need.", } } output_type = "string" def __init__(self, **kwargs): super().__init__(**kwargs) # Connection details self.uri = "ws://localhost:8000/rpc" self.ns = "demo" self.dbname = "demo" self.table = "groceries" self.user = os.getenv("SURREAL_USER", "root") self.pw = os.getenv("SURREAL_PASS", "secret") self.emb = TextEmbedding(model_name="jinaai/jina-embeddings-v2-base-en") # one-time async bootstrap asyncio.run(self._setup()) async def _setup(self): """Connect, create schema, and ingest groceries if empty.""" async with AsyncSurreal(self.uri) as db: await db.signin({"username": self.user, "password": self.pw}) await db.use(self.ns, self.dbname) schema = """ DEFINE TABLE $tb SCHEMALESS PERMISSIONS NONE; DEFINE FIELD item_name ON $tb TYPE string; DEFINE FIELD category ON $tb TYPE string; DEFINE FIELD description ON $tb TYPE string; DEFINE FIELD embedding ON $tb TYPE array; DEFINE INDEX idx_hnsw ON $tb FIELDS embedding HNSW DIMENSION 768 DIST COSINE; """ await db.query(schema, {"tb": self.table}) # skip ingest if we already have rows existing = await db.query(f"SELECT count() FROM {self.table};") if existing[0]["result"][0]["count"] > 0: return print("🛒 Ingesting GroceryList dataset …") ds = load_dataset("AmirMohseni/GroceryList")["train"] # 225 rows BATCH = 64 items = ds["item"] cats = ds["category"] descs = [f"{it.capitalize()} is in the {cat} aisle." for it, cat in zip(items, cats)] for i in range(0, len(descs), BATCH): vecs = self.emb.query_embed(descs[i:i+BATCH]) rows = [ { "id": f"{self.table}:{i+j}", "item_name": items[i+j], "category": cats[i+j], "description": descs[i+j], "embedding": list(vec), } for j, vec in enumerate(vecs) ] await db.create(self.table, rows) async def _lookup(self, query_vec: List[float]) -> List[Dict[str, Any]]: """Perform vector search with proper connection management.""" async with AsyncSurreal(self.uri) as db: await db.signin({"username": self.user, "password": self.pw}) await db.use(self.ns, self.dbname) surql = """ LET $q := $vec; SELECT item_name, category, description, vector::distance::knn() AS dist FROM $tb WHERE embedding <|$k,$ef|> $q -- top-k, efSearch ORDER BY dist; """ result = await db.query( surql, { "vec": query_vec, "tb": self.table, "k": 5, "ef": 64 } ) return result[0]["result"] def forward(self, query: str) -> str: """Return the five closest grocery items.""" q_vec = next(self.emb.query_embed(query)) hits = asyncio.run(self._lookup(q_vec)) return "Retrieved items:\n" + "".join( f"== {hit['item_name'].title()} ==\n" f"Category: {hit['category']}\n" f"{hit['description']}\n\n" for hit in hits ) ``` --- Source: https://surrealdb.com/docs/build/integrations/authentication/better-auth/getting-started # Getting started Install the @surrealdb/better-auth adapter, connect to SurrealDB, configure the adapter, and generate your Better Auth schema. This guide walks through installing the `@surrealdb/better-auth` adapter, connecting it to SurrealDB, and generating the schema Better Auth needs. ## Installation Install the adapter alongside `better-auth` and the [SurrealDB JavaScript SDK](/docs/reference/javascript.md): **Bun** ```bash bun add @surrealdb/better-auth better-auth surrealdb ``` **npm** ```bash npm install @surrealdb/better-auth better-auth surrealdb ``` **pnpm** ```bash pnpm add @surrealdb/better-auth better-auth surrealdb ``` ## Quick start Connect a `Surreal` client, then pass it to `surrealAdapter` as the `database` option in your Better Auth configuration. Better Auth manages all table operations through the adapter. ```typescript import { betterAuth } from 'better-auth'; import { Surreal } from 'surrealdb'; import { surrealAdapter } from '@surrealdb/better-auth'; const db = new Surreal(); await db.connect('ws://localhost:8000/rpc'); await db.use({ namespace: 'namespace', database: 'database' }); export const auth = betterAuth({ database: surrealAdapter({ db }), emailAndPassword: { enabled: true }, }); ``` ## Connecting to SurrealDB The adapter accepts any connected `Surreal` instance. Connect it before passing it to `surrealAdapter`. Use WebSocket for long-running servers and HTTP for stateless environments such as serverless functions. ```typescript import { Surreal } from 'surrealdb'; const db = new Surreal(); // WebSocket (recommended for persistent servers) await db.connect('ws://localhost:8000/rpc', { namespace: 'myapp', database: 'production', authentication: { username: 'root', password: 'secret', }, }); // HTTP (for stateless environments) await db.connect('http://localhost:8000', { namespace: 'myapp', database: 'production', }); ``` For a full reference of connection options, see [Connecting to SurrealDB](/docs/reference/javascript/concepts/connecting-to-surrealdb.md) in the JavaScript SDK documentation. ## Configuration options `surrealAdapter` accepts the following options: | Option | Type | Default | Description | |--------|------|---------|-------------| | `db` | `Surreal` | required | A connected SurrealDB client instance. | | `usePlural` | `boolean` | `false` | Use plural table names (`users` instead of `user`). | | `schemaMode` | `'schemafull' \| 'schemaless'` | `'schemafull'` | Table mode used by the generated schema. `schemaless` keeps known fields typed and indexed while still accepting writes to undeclared fields. | ```typescript surrealAdapter({ db, usePlural: true, // use plural table names }); ``` ## Schema generation The adapter includes a `createSchema` implementation that generates SurrealQL DDL statements for all Better Auth tables. Use the Better Auth CLI to produce a `schema.surql` file: ```bash bunx @better-auth/cli generate --output schema.surql ``` Then apply it to your SurrealDB instance: ```bash surreal import --conn http://localhost:8000 \ --ns myapp --db production \ --user root --pass secret \ schema.surql ``` The generated schema uses `SCHEMAFULL` tables by default. Each field is typed from the Better Auth schema: - Required fields take a concrete type (`string`, `datetime`, `bool`, and so on). - Optional fields use `option`, so they accept a typed value, a stored `NULL`, or a missing value. - Object fields use `TYPE object FLEXIBLE` so nested objects can hold arbitrary keys. - `DEFINE INDEX` statements are emitted for unique fields and for fields Better Auth marks as indexed. Every statement uses `IF NOT EXISTS`, so the file is safe to reapply. > [!NOTE] > If your app adds many dynamic plugin fields and you would rather not regenerate the schema each time, set `schemaMode: 'schemaless'`. Known fields stay typed and indexed, and writes to undeclared fields are accepted. ## Table names By default the adapter uses singular table names: `user`, `session`, `verification`, and `account`. Set `usePlural: true` to use plural names instead. Better Auth also lets you override model names per-table via the `modelName` option in your configuration, and the adapter respects those overrides automatically. ## Next steps - [Overview](/docs/build/integrations/authentication/better-auth/overview.md): adapter capabilities and prerequisites - [Plugins](/docs/build/integrations/authentication/better-auth/plugins.md): use any Better Auth plugin and the SurrealQL helper functions generated for the organisation plugin. - [Transactions & limitations](/docs/build/integrations/authentication/better-auth/transactions-and-limitations.md): how transactions behave and what to be aware of. --- Source: https://surrealdb.com/docs/build/integrations/authentication/better-auth/overview # Better Auth Use SurrealDB as the database behind Better Auth with the @surrealdb/better-auth adapter, including schema generation, transactions, and support for all Better Auth plugins. [Better Auth](https://better-auth.com) is a comprehensive, framework-agnostic authentication and authorisation library for TypeScript. The `@surrealdb/better-auth` adapter lets you use SurrealDB as the database behind Better Auth: you pass it a single connected [`Surreal`](/docs/reference/javascript.md) client and Better Auth manages all of its table operations through the adapter. The adapter supports schema generation, transactions, the full set of Better Auth WHERE operators, and every Better Auth plugin. > [!NOTE] > This section covers the **Better Auth** integration only. For SurrealDB's own sign-in, tokens, and permissions model, see [Security](/docs/learn/security.md). > [!NOTE] > The `@surrealdb/better-auth` adapter works with SurrealDB versions `v3.1.0` and later. ## Prerequisites - SurrealDB 3.1 or later - Better Auth 1.6.x - Node.js 18+ or Bun 1.x ## Feature support The adapter advertises the following capabilities to Better Auth: | Capability | Supported | |------------|-----------| | JSON / object fields | Yes | | Dates | Yes | | Booleans | Yes | | Arrays | Yes | ## Identifiers By default, Better Auth generates string identifiers in JavaScript. The adapter stores them as SurrealDB [record IDs](/docs/reference/query-language/language-primitives/data-types/record-ids.md) and deserialises them back to strings when reading, so a record round-trips to your application as the plain string `id`. UUIDs are also supported. Enable them in your Better Auth configuration: ```typescript betterAuth({ database: surrealAdapter({ db }), advanced: { database: { generateId: 'uuid' } }, }); ``` Better Auth generates the UUID in JavaScript and passes it to the adapter, which stores and reads it transparently (it round-trips as the plain UUID string). SurrealDB does not mint the UUID natively, which is why the adapter reports `supportsUUIDs: false` to Better Auth: that flag means "the database generates UUIDs itself", not "UUIDs are unsupported". Numeric (auto-increment) identifiers are not supported. ## In this section - [Getting started](/docs/build/integrations/authentication/better-auth/getting-started.md) - Install the adapter, connect to SurrealDB, configure options, and generate your schema. - [Plugins](/docs/build/integrations/authentication/better-auth/plugins.md) - Use any Better Auth plugin and call the SurrealQL helper functions generated for the organisation plugin. - [Transactions & limitations](/docs/build/integrations/authentication/better-auth/transactions-and-limitations.md) - How transactions behave and the current limitations to be aware of. ## Sources - [GitHub repository](https://github.com/surrealdb/better-auth) - [npm package](https://www.npmjs.com/package/@surrealdb/better-auth) - [Better Auth documentation](https://better-auth.com) --- Source: https://surrealdb.com/docs/build/integrations/authentication/better-auth/plugins # Plugins Use any Better Auth plugin with the SurrealDB adapter and call the SurrealQL helper functions generated for the organisation plugin. The `@surrealdb/better-auth` adapter works with all Better Auth plugins. Pass them in the `plugins` array of your Better Auth configuration: ```typescript import { betterAuth } from 'better-auth'; import { organization } from 'better-auth/plugins/organization'; import { twoFactor } from 'better-auth/plugins/two-factor'; import { admin } from 'better-auth/plugins/admin'; import { Surreal } from 'surrealdb'; import { surrealAdapter } from '@surrealdb/better-auth'; const db = new Surreal(); await db.connect('ws://localhost:8000/rpc'); await db.use({ namespace: 'myapp', database: 'production' }); export const auth = betterAuth({ database: surrealAdapter({ db }), plugins: [ organization(), twoFactor(), admin(), ], }); ``` A wider example combining several plugins: ```typescript import { betterAuth } from 'better-auth'; import { admin } from 'better-auth/plugins/admin'; import { anonymous } from 'better-auth/plugins/anonymous'; import { bearer } from 'better-auth/plugins/bearer'; import { magicLink } from 'better-auth/plugins/magic-link'; import { multiSession } from 'better-auth/plugins/multi-session'; import { organization } from 'better-auth/plugins/organization'; import { twoFactor } from 'better-auth/plugins/two-factor'; import { username } from 'better-auth/plugins/username'; import { Surreal } from 'surrealdb'; import { surrealAdapter } from '@surrealdb/better-auth'; const db = new Surreal(); await db.connect('ws://localhost:8000/rpc'); await db.use({ namespace: 'myapp', database: 'production' }); export const auth = betterAuth({ secret: process.env.BETTER_AUTH_SECRET!, baseURL: process.env.BETTER_AUTH_URL ?? 'http://localhost:3000', emailAndPassword: { enabled: true }, database: surrealAdapter({ db }), plugins: [ organization({ teams: { enabled: true }, dynamicAccessControl: { enabled: true }, }), twoFactor(), admin(), bearer(), username(), magicLink({ sendMagicLink: async ({ email, url }) => { // send an email containing the magic link url }, }), anonymous(), multiSession(), ], }); ``` > [!IMPORTANT] > Each plugin may add new tables. Re-run [schema generation](/docs/build/integrations/authentication/better-auth/getting-started.md#schema-generation) after adding plugins to produce the updated DDL, or use `schemaMode: 'schemaless'` to accept the new fields without regenerating. ## SurrealDB helper functions When you use the `organization` plugin (with or without the `teams` option), schema generation also emits a set of `fn::*` SurrealQL functions you can call directly in your own queries, [rules, and permissions](/docs/reference/query-language/statements/define/table.md#defining-permissions). These functions are defined with `IF NOT EXISTS`, so re-running the generated schema is safe. ### `fn::auth::organization::*` These functions are emitted when the `organization` plugin is active. | Function | Signature | Returns | Description | |---|---|---|---| | `fn::auth::organization::member_of` | `(userId: string, organizationId: string)` | `bool` | True if the user is a member of the organisation. | | `fn::auth::organization::get_role` | `(userId: string, organizationId: string)` | `option` | The member's role (`"owner"`, `"admin"`, `"member"`), or `NONE` if not a member. | | `fn::auth::organization::has_role` | `(userId: string, organizationId: string, minRole: string)` | `bool` | True if the user's role is equal to or senior to `minRole` (owner > admin > member). | | `fn::auth::organization::members` | `(organizationId: string)` | `array` | All `members` records for the organisation. | | `fn::auth::organization::teams` | `(organizationId: string)` | `array` | All `teams` records for the organisation (requires `teams: { enabled: true }`). | | `fn::auth::organization::has_permission` | `(userId: string, organizationId: string, resource: string, action: string)` | `bool` | True if the user's role has a custom permission for the given resource and action (requires `dynamicAccessControl`). | ```surql -- Check if a user belongs to an organisation IF fn::auth::organization::member_of($userId, $organizationId) { -- allow access }; -- Require at least admin-level access IF fn::auth::organization::has_role($userId, $organizationId, "admin") { -- perform a privileged operation }; -- Get a user's role string LET $role = fn::auth::organization::get_role($userId, $organizationId); -- List all members LET $members = fn::auth::organization::members($organizationId); ``` ### `fn::auth::team::*` These functions are emitted when `teams: { enabled: true }` is set on the `organization` plugin. | Function | Signature | Returns | Description | |---|---|---|---| | `fn::auth::team::member_of` | `(userId: string, teamId: string)` | `bool` | True if the user is a member of the team. | | `fn::auth::team::members` | `(teamId: string)` | `array` | All `teamMembers` records for the team. | ```surql -- Check team membership IF fn::auth::team::member_of($userId, $teamId) { -- allow team-scoped access }; -- List all team members LET $members = fn::auth::team::members($teamId); ``` --- Source: https://surrealdb.com/docs/build/integrations/authentication/better-auth/transactions-and-limitations # Transactions & limitations How the @surrealdb/better-auth adapter handles transactions, and the current limitations to be aware of. The `@surrealdb/better-auth` adapter uses transactions internally, and exposes them for operations you want to group yourself. This page covers that behaviour and the limitations that come with it. ## Transactions Transaction support is built in. Better Auth uses transactions internally for operations that need atomicity (for example, creating a session alongside a new user record), so most applications never call the transaction API directly. If you need manual transaction access, call `transaction` on the adapter. Operations run inside the callback are committed together when it resolves, or discarded if it throws. ```typescript const adapter = surrealAdapter({ db })(options); await adapter.transaction(async (trx) => { await trx.create({ model: 'user', data: { /* ... */ } }); await trx.create({ model: 'session', data: { /* ... */ } }); }); ``` Transactions are backed by [`SurrealDB.beginTransaction()`](/docs/reference/query-language/statements/begin.md). If any operation inside the callback throws, the transaction is cancelled and changes are discarded. ## Limitations Keep the following in mind when adopting the adapter. ### Version coupling The adapter targets **Better Auth 1.6.x** and **SurrealDB 3.1+**. Other major versions of either dependency are not supported. ### Numeric identifiers are not supported By default Better Auth uses string identifiers, and [UUIDs](/docs/build/integrations/authentication/better-auth/overview.md#identifiers) can be enabled in your Better Auth configuration. Numeric (auto-increment) identifier strategies are **not** supported, because SurrealDB does not generate them natively and the adapter has no JavaScript fallback for them. ### `SCHEMAFULL` schemas require regeneration With the default `schemaMode: 'schemafull'`, writes to fields that are not in the generated schema are rejected. Whenever you add a plugin or otherwise introduce new fields, [regenerate and reapply the schema](/docs/build/integrations/authentication/better-auth/getting-started.md#schema-generation). If you would rather not regenerate each time, set `schemaMode: 'schemaless'`; known fields stay typed and indexed, and writes to undeclared fields are accepted. ### Transaction rollback is best-effort SurrealDB's `cancel()` is not honoured in every setup (for example, the in-memory engine). To keep rollback reliable, the adapter compensates by deleting records it created inside a failed transaction. This covers records created during the transaction; it does not restore prior values for rows that were updated or deleted within it. ### Value deserialisation When reading records, the adapter converts SurrealDB types back to JavaScript values: a `datetime` becomes a JavaScript `Date`, and a `RecordId` becomes its string `id`. Bear this in mind if you query the underlying tables directly rather than through Better Auth. --- Source: https://surrealdb.com/docs/build/integrations/data-management/airbyte # Airbyte The Airbyte connector allows you to sync data to SurrealDB from hundreds of sources. This guide helps you configure SurrealDB as a destination in [Airbyte](https://airbyte.com) using the [official connector](https://github.com/surrealdb/airbyte-connector). ## Prerequisites To connect SurrealDB to Airbyte, you need the following: - An Airbyte deployment with access to the SurrealDB destination connector. - SurrealDB `v2.2.0` or later. - A SurrealDB instance (self-hosted or [SurrealDB Cloud](https://studio.surrealdb.com/cloud)) reachable by Airbyte. - Ensure your SurrealDB database can be accessed by Airbyte. If your database is within a VPC, you may need to allow access from the IP you're using to expose Airbyte. - A token or user credentials with [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md), [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md), [`UPSERT`](/docs/reference/query-language/statements/upsert.md), [`SELECT`](/docs/reference/query-language/statements/select.md), and [`REMOVE`](/docs/reference/query-language/statements/remove.md) permissions. ## Setup instructions Follow these steps to configure SurrealDB as a destination. You can use either a self-hosted instance or SurrealDB Cloud . Before proceeding, ensure you have the endpoint URL and credentials for your SurrealDB instance and that Airbyte can reach it over the network. ### Option 1: Self-hosted SurrealDB 1. Ensure your SurrealDB instance is reachable from Airbyte. This might involve opening network access or configuring an SSH tunnel. 2. Create a dedicated user or token in SurrealDB, for example: ```surql DEFINE USER airbyte ON ROOT PASSWORD "YourPassword" ROLES OWNER; ``` Use the generated credentials when setting up the destination. ### Option 2: SurrealDB Cloud 1. Visit the [Instances page](https://studio.surrealdb.com/cloud/instances) and select your instance. 2. Click **Connect with Surreal CLI** to obtain a connection command containing the `--endpoint` and authentication details. 3. Verify you can connect using `surreal sql` with those parameters. ## Finish Airbyte configuration 1. Open the Airbyte dashboard and add a new **Destination**. 2. Select **SurrealDB** from the list of destination types. 3. Provide the host, port, namespace, database, and either the username/password or token you created earlier. 4. Click **Set up destination** and run the connection test. Upon a successful test, you can start syncing data from your sources into SurrealDB tables. Each stream will be output into its own table in SurrealDB. Each table will contain 3 columns: - `_airbyte_raw_id`: a uuid assigned by Airbyte to each event that is processed. The column type in SurrealDB is string. The connector use this as the ID of each record in the destination SurrealDB table. - `_airbyte_extracted_at`: a timestamp representing when the event was pulled from the data source. The column type in SurrealDB is datetime. - `_airbyte_data`: a json blob representing with the event data. The column type in SurrealDB is object. ## Known limitations SurrealDB destination forces all identifier (table, schema and columns) names to be lowercase. ## Related links - [Airbyte destination documentation](https://github.com/surrealdb/airbyte-connector) - [Connect to airbyte via a dev container](https://github.com/surrealdb/airbyte-connector/blob/main/devcontainer.md) --- Source: https://surrealdb.com/docs/build/integrations/data-management/fivetran # Fivetran The Fivetran integration for SurrealDB allows you to sync data from Fivetran to SurrealDB. This guide will help you connect SurrealDB to Fivetran. The Fivetran integration is available through their [partner-built program](https://fivetran.com/docs/partner-built-program). For support with self-hosted SurrealDB destinations, please contact [SurrealDB Support](/contact). For SurrealDB Cloud related questions, reach out to [SurrealDB Cloud Support](/docs/manage/organisations/support.md). ## Prerequisites To connect SurrealDB to Fivetran, you need the following: - A Fivetran role with the [Create Destinations or Manage Destinations](https://fivetran.com/docs/using-fivetran/fivetran-dashboard/account-settings/role-based-access-control#rbacpermissions) permissions. - A SurrealDB token. - A SurrealDB instance (self-hosted or Cloud) that is accessible by Fivetran. ## Setup instructions Follow these steps to set up SurrealDB as a destination in Fivetran. You have two options: self-hosted SurrealDB or SurrealDB Cloud . Before proceeding with either option, ensure you have: - Access to your SurrealDB instance with appropriate permissions - The necessary connection details (endpoint URL, credentials) - Network connectivity between Fivetran and your SurrealDB instance ### Option 1: Self-hosted SurrealDB 1. For self-hosted SurrealDB, ensure your SurrealDB instance is accessible by Fivetran according to your Fivetran deployment: - For [Fivetran SaaS Deployment](https://fivetran.com/docs/deployment-models/saas-deployment), ensure your SurrealDB is accessible via Internet. - For [Fivetran Hybrid Deployment](https://fivetran.com/docs/deployment-models/hybrid-deployment), ensure your SurrealDB is accessible by the Fivetran Hybrid Deployment Agent. - For [Fivetran Self-Hosted Deployment](https://fivetran.com/docs/deployment-models/self-hosted-deployment), ensure your SurrealDB is accessible by the Fivetran HVR Agent. 2. Set up the token and use it following [SurrealDB's Authentication documentation](/docs/learn/security/authentication/users.md#token). ### Option 2: SurrealDB Cloud 1. Ensure your SurrealDB instance is up and running and accessible via Internet. 2. Browse the [Instances page](https://studio.surrealdb.com/cloud/instances) and select your chosen instance. 3. Click **Connect with Surreal CLI** and locate the `surreal sql --endpoint wss://YOUR_INSTANCE_HOSTNAME --token YOUR_TOKEN` command. 4. Run the command, and set up your own [`ACCESS`](/docs/reference/query-language/statements/define/access.md) or [`USER`](/docs/reference/query-language/statements/define/user.md). The example below works for testing purposes: ```surql USE NS your_ns; USE DB your_db; DEFINE USER your_user ON DATABASE PASSWORD "YourPassword" ROLES OWNER; ``` 5. Ensure the user/pass is working by running: ```bash surreal sql --endpoint wss://YOUR_INSTANCE_HOSTNAME --user your_user --pass YourPassword --ns your_ns --db your_db ``` 6. Make a note of the `endpoint`, `user`, and `pass` parameters. You will need them to configure Fivetran. 7. (Optional) If you prefer using `token`, we recommend `DEFINE ACCESS ... TYPE JWT`. Refer to the [`DEFINE ACCESS > JWT` documentation](/docs/reference/query-language/statements/define/access/jwt.md) to set up JWT access. - Verify if the token is working before proceeding to the next section, by running: ```bash surreal sql --endpoint wss://YOUR_INSTANCE_HOSTNAME --token your_token --ns your_ns --db your_db ``` ## Finish Fivetran configuration 1. Log in to your [Fivetran account](https://fivetran.com/login). 2. Go to the **Destinations** page and click **Add destination**. 3. Enter a **Destination name** of your choice. 4. Click **Add**. 5. Select **SurrealDB** as the destination type. 6. Enter the `url`, `user` and `pass` (or `token`) you verified in the previous step. > [!NOTE] > The `url` setting corresponds to the `endpoint` parameter you verified in the previous step. 7. Click **Save & Test**. Fivetran [tests and validates](https://fivetran.com/docs/destinations/surrealdb#setup-tests) the SurrealDB connection. Upon successfully completing the setup tests, you can sync your data using Fivetran connectors to the SurrealDB destination. In addition, Fivetran automatically configures a [Fivetran Platform Connector](https://fivetran.com/docs/logs/fivetran-platform) to transfer the connector logs and account metadata to a schema in this destination. The Fivetran Platform Connector enables you to monitor your connectors, track your usage, and audit changes. The connector sends all these details at the destination level. > [!IMPORTANT] > If you are an Account Administrator, you can manually add the Fivetran Platform Connector on an account level so that it syncs all the metadata and logs for all the destinations in your account to a single destination. If an account-level Fivetran Platform Connector is already configured in a destination in your Fivetran account, then we don't add destination-level Fivetran Platform Connectors to the new destinations you create. ## Setup tests Fivetran performs the following SurrealDB connection tests: - The Database Connection test checks if we can connect to your SurrealDB database using the provided URL and token. The test should complete in a few seconds if your Fivetran deployment can access the target SurrealDB instance. ## Related articles - [Destination Overview](https://fivetran.com/docs/destinations/surrealdb) - [API Destination Configuration](https://fivetran.com/docs/rest-api/api-reference/destinations/create-destination?service=surrealdb_destination) - [Documentation Home](/docs) --- Source: https://surrealdb.com/docs/build/integrations/data-management/n8n # n8n The official n8n node for SurrealDB. It provides both action and tool nodes to interact with a SurrealDB database, allowing you to create, read, update, and delete records, as well as execute custom SurrealQL queries. This guide shows how to integrate SurrealDB with [n8n](https://n8n.io/), a [fair-code licensed](https://docs.n8n.io/reference/license/) workflow automation platform. The official n8n node for SurrealDB. It provides both action and tool nodes to interact with a SurrealDB database, allowing you to create, read, update, and delete records, as well as execute custom SurrealQL queries. It is available in the [n8n Community Nodes](https://docs.n8n.io/integrations/community-nodes/) repository. > [!IMPORTANT] > As with all community nodes, this node works only with self-hosted n8n instances, not with n8n Cloud. This node has been tested with SurrealDB `v2.x` ## Features - **Dual Node Types**: Functions as both an action node and a tool node for AI workflows - **Complete CRUD Operations**: Create, read, update, and delete SurrealDB records - **Custom Queries**: Execute any SurrealQL query with full parameter support - **Enhanced query builder**: Visual interface for building `SELECT` queries with `WHERE`, `ORDER BY`, `GROUP BY`, and other clauses - **Table Operations**: List fields and explore table structure - **Relationship Support**: Query and manage record relationships - **Native Data Format**: Works with SurrealDB's native data formats - **Connection Pooling**: Configurable connection pooling for improved performance and resource management - **Enhanced error handling**: Comprehensive error classification, automatic retry logic, and connection recovery - **Intelligent Recovery**: Different error handling strategies for different operation types - **Detailed Error Reporting**: Rich error context with categorization and severity levels - **Pool Monitoring**: Built-in pool statistics and performance monitoring ## Prerequisites 1. You need a self-hosted n8n instance (`v0.214.0` or later recommended). 2. You need access to a SurrealDB instance (`2.0.0` or later recommended). ## Installation steps 1. Open your n8n instance 2. Go to **Settings** > **Community Nodes** 3. Click **Install** 4. Enter `@surrealdb/n8n-nodes-surrealdb` and click **Install** 5. Restart your n8n instance if prompted > [!NOTE] > To use this node as a tool in AI workflows, you must set the environment variable `N8N_COMMUNITY_PACKAGES_ALLOW_TOOL_USAGE=true`. ## Configuration In order to use SurrealDB in n8n, you need to configure the SurrealDB node. ### Credentials To use the SurrealDB node, you need to create credentials with the following properties:
Property Description
Connection String The connection string to your SurrealDB instance (must start with http:// or https://). WebSocket connections (ws:// or wss://) are not supported.
Authentication Choose the authentication scope:
Root Full access to all namespaces and databases
Namespace Access limited to a specific namespace
Database Access limited to a specific database within a namespace
Username Username for authentication
Password Password for authentication
Namespace Target namespace (required for Namespace and Database authentication)
Database Target database (required for Database authentication)
The authentication type you choose affects how namespace and database information is handled. Depending on the authentication type, you will need to provide different information. See the table below for more details.
Authentication Type Access Scope Required Fields Optional Fields Override Capability
Root Authentication All namespaces and databases None Namespace, Database Can override namespace/database at node level
Namespace Authentication All databases within a specific namespace Namespace Database Can override database at node level
Database Authentication Specific database within a specific namespace Namespace, Database None Can override both at node level for specific operations
### Node-level namespace and database overrides For most operations, you can override the namespace and database settings from your credentials: 1. In the node configuration, expand the **Options** section 2. Enter values in the **Namespace** and/or **Database** fields 3. These values will take precedence over the credential settings for that specific operation 4. You will be required to provide a namespace when using Namespace authentication 5. You will be required to provide both a namespace and database when using Database authentication ## Operations The SurrealDB node provides a comprehensive set of operations organised by resource type. For anything not covered, you can use the **Execute Query** operation.
Category Operation Description
Record Operations Create Record Create a single record in a table
Get Record Retrieve a specific record by ID
Update Record Update a specific record by ID
Upsert Record Create or update a record (insert if not exists, update if exists)
Delete Record Delete a specific record by ID
Table Operations Get All Records Retrieve all records from a table
Create Many Create multiple records in a table
Get Many Retrieve multiple records by IDs
Update All Records Update all records in a table
Delete All Records Delete all records from a table
Merge All Records Merge the same data into all records in a table
Create Table Define a new table with optional schema
Delete Table Remove a table from the database
Get Table Retrieve information about a table
Field Operations List Fields List all fields defined on a table
Create Field Create a new field on a table
Delete Field Delete a field from a table
Index Operations Create Index Create a new index on a table
Delete Index Delete an index from a table
Relationship Operations Create Relationship Create a relationship between two records
Delete Relationship Delete a relationship between records
Query Relationships Query relationships between records
Query Operations Execute Query Execute a raw SurrealQL query with parameters
Build Select Query Build SELECT queries using a visual interface with WHERE, ORDER BY, GROUP BY, and other clauses
System Operations Health Check Check if the database instance is responsive
Version Get the version of the SurrealDB instance
Get Pool Statistics Monitor connection pool performance and statistics
## Understanding SurrealDB and n8n integration ### Connection protocol > [!IMPORTANT] > Due to n8n's architecture, this node only supports HTTP/HTTPS connections to SurrealDB. WebSocket connections (WS/WSS) are not supported. Your connection string must start with `http://` or `https://` (not `ws://` or `wss://`). This means that when configuring your SurrealDB instance, ensure it's accessible via `HTTP/HTTPS`. If you're using SurrealDB Cloud or another instance that only offers WebSocket connections, you'll need to set up a [self-hosted SurrealDB instance](/pricing) with HTTP enabled. This limitation is due to how n8n handles connections and executes node operations. You can read more about this in the [n8n documentation](https://www.npmjs.com/package/n8n-nodes-surrealdb). This node uses the HTTP/HTTPS protocol exclusively, which means that each operation creates a new connection to SurrealDB, the connection is closed after the operation completes, and no persistent connection is maintained between operations. ### Connection pooling The SurrealDB node includes comprehensive connection pooling to improve performance and resource management. Connection pooling allows the node to reuse database connections across multiple operations, reducing connection overhead and improving response times. #### Pool configuration options You can configure the connection pool through the "Connection Pooling" options in any node operation: - **Max Connections** (default: 10): Maximum number of connections in the pool - **Min Connections** (default: 2): Minimum number of connections to keep in the pool - **Acquire Timeout** (default: 30000ms): Maximum time to wait for a connection from the pool - **Health Check Interval** (default: 60000ms): Interval between health checks for pool connections - **Max Idle Time** (default: 300000ms): Maximum time a connection can remain idle before being closed - **Retry Attempts** (default: 3): Number of retry attempts for failed connection acquisitions - **Retry Delay** (default: 1000ms): Delay between retry attempts #### Pool monitoring Use the **System > Get Pool Statistics** operation to monitor pool performance: ```json { "poolStatistics": { "totalConnections": 5, "activeConnections": 2, "idleConnections": 3, "waitingRequests": 0, "totalRequests": 150, "failedRequests": 2, "averageResponseTime": 45, "successRate": 99 }, "performance": { "averageResponseTimeMs": 45, "requestsPerSecond": 2, "errorRate": 1 }, "poolHealth": { "utilizationRate": 40, "availableConnections": 3, "waitingRequests": 0 } } ``` ### SurrealDB result handling SurrealDB operations often return empty results rather than errors when no matching data is found. This behaviour differs from many other databases and can be important to understand when building workflows: - **Empty Results vs. Errors**: A query for a non-existent record returns an empty result, not an error - **Always Output Data**: The "Always Output Data" option is particularly useful with SurrealDB to ensure your workflow continues even when no results are found ### Working with SurrealDB data types SurrealDB supports rich data types that map well to n8n's JSON handling: - **Records and IDs**: SurrealDB record IDs use the format `table:id` - **Relationships**: Relationships are first-class citizens in SurrealDB - **Arrays and Objects**: Nested data structures are fully supported ## Error handling The SurrealDB node includes a comprehensive error handling and recovery system that automatically manages common database issues: ### Automatic error classification The system automatically categorizes errors into different types:
Error Type Description
Connection Errors Network issues, timeouts, connection refused
Authentication Errors Invalid credentials, unauthorised access
Query Errors Syntax errors, malformed queries
Validation Errors Invalid data, missing required fields
System Errors Database server issues, internal errors
### Intelligent retry logic - **Exponential Backoff**: Automatic retry with increasing delays - **Operation-Specific Retries**: Different retry strategies for read vs write operations - **Configurable Limits**: Adjustable retry counts and delays - **Smart Error Filtering**: Only retry on recoverable errors ### Connection recovery - **Automatic Reconnection**: Reconnects to SurrealDB on connection failures - **Re-authentication**: Automatically re-authenticates after reconnection - **Connection Validation**: Verifies connection health before retrying operations ### Enhanced error reporting When `Continue on Fail` is enabled, errors include detailed information: ```json { "error": { "message": "Connection timeout", "category": "TIMEOUT_ERROR", "severity": "MEDIUM", "retryable": true, "context": { "operation": "executeQuery", "itemIndex": 0, "timestamp": "2024-01-15T10:30:00Z", "recoveryStrategy": "CONNECTION_RECOVERY" } } } ``` ### Error handling strategies Different operation types use different error handling strategies: - **Read Operations**: Faster retries, continue on low/medium errors - **Write Operations**: More retries, stop on medium+ errors - **Critical Operations**: Minimal retries, stop on any error - **Bulk Operations**: Moderate retries, handle rate limiting For detailed information about the error handling system, see [Error Handling Documentation](https://github.com/surrealdb/n8n-nodes-surrealdb/blob/main/docs/ERROR_HANDLING.md). ## Resources - [n8n Community Nodes Documentation](https://docs.n8n.io/integrations/community-nodes/) - [SurrealDB Documentation](/docs) - [SurrealQL Reference](/docs/reference/query-language.md) --- Source: https://surrealdb.com/docs/build/integrations/data-management/overview # Data management integrations This section contains information about the different data management features of SurrealDB. SurrealDB offers comprehensive data management capabilities to help you efficiently store, organise, and transform your data. Through integrations with leading data platforms, you can build robust ETL pipelines, ingest data from diverse sources, and maintain data quality at scale. The integrations below enable seamless data workflows while leveraging SurrealDB's multi-model architecture.
Integration Description
Airbyte Data integration platform specialising in ELT pipelines.
Fivetran Data integration platform to manage data transfer between different sources and destination systems.
n8n Workflow automation platform that lets you build custom event-driven automations.
Qyrus Data quality assurance platform for cross-source comparisons and single-source validations.
--- Source: https://surrealdb.com/docs/build/integrations/data-management/qyrus # Qyrus Qyrus Data Testing integrates with SurrealDB for data quality assurance. Use Compare Jobs for cross-source validation (migrations, sync) and Evaluate Jobs for single-source checks (schema, profiling). Configure with Host URL, Port, Namespace, Database, and SurrealQL queries. **Qyrus Data Testing** is a data quality assurance platform that integrates with SurrealDB. Use it to validate data, compare datasets across sources, and run quality checks on SurrealDB tables using SurrealQL queries. ## Overview - **Integration type**: SurrealDB multi-model database connector for Qyrus Data Testing - **Primary use**: Data quality assurance, cross-source comparison, and single-source validation - **Query language**: SurrealQL for fetching and validating data - **Documentation**: [Qyrus Data Testing](https://docs.qyrus.com/data-testing) ## Use cases Qyrus supports two job types when using SurrealDB as a data source: | Job Type | Description | Example Scenarios | |----------|-------------|-------------------| | **Compare Job** | Validates differences between two data sources | Data migrations, synchronization checks, production vs. staging comparison, source vs. target validation | | **Evaluate Job** | Validates a single SurrealDB data source | Schema checks, constraint validation, data profiling, quality rules, custom business logic | ## Configuration parameters When configuring SurrealDB as a data source in Qyrus, use the following parameters:
Parameter Description
Host URL SurrealDB server URL (HTTP / HTTPS / WebSocket)
Port Port on which the SurrealDB service is running
Namespace SurrealDB namespace to connect to
Database Name Target database inside the namespace
Username Login username (if authentication is enabled)
Password Login password (if authentication is enabled)
Query SurrealQL query that fetches data from the selected table
Enable Limit & Offset (Optional) Enables pagination controls for query execution
## Getting started To set up SurrealDB with Qyrus Data Testing: 1. Visit the [Qyrus Data Testing documentation](https://docs.qyrus.com/data-testing) for an overview of the platform 2. Configure SurrealDB as a data source using the [SurrealDB connector guide](https://docs.qyrus.com) 3. Create and run [Compare](https://docs.qyrus.com) or [Evaluate](https://docs.qyrus.com) jobs as needed 4. View validation results in [Job History](https://docs.qyrus.com/data-testing) and detailed reports for completed jobs ## Frequently Asked Questions ### What is Qyrus data testing? Qyrus Data Testing is a data quality assurance tool that enables cross-source comparisons (Compare Jobs) and internal validations (Evaluate Jobs) to maintain reliable and accurate data across databases and systems. ### How does SurrealDB integrate with Qyrus? SurrealDB is available as a connector in Qyrus. You configure the connection using Host URL, Port, Namespace, Database Name, and optionally Username and Password. Data is fetched from SurrealDB using SurrealQL queries that you define in Qyrus. ### When should I use a Compare Job vs. an Evaluate Job? - **Compare Job**: Use when you need to validate that two data sources match (e.g., after a migration, during sync checks, or when comparing production and staging). - **Evaluate Job**: Use when you need to validate a single SurrealDB data source (e.g., schema checks, data profiling, quality rules, or custom logic). ### What SurrealQL queries can I use in Qyrus? You can use any valid SurrealQL query that fetches data from your SurrealDB tables. The query runs against the configured Namespace and Database. Examples: `SELECT * FROM table`, `SELECT * FROM table LIMIT 100`, or more complex queries with filters and joins. ### Does Qyrus support SurrealDB authentication? Yes. If your SurrealDB instance has authentication enabled, provide the **Username** and **Password** in the Qyrus connector configuration. ### Where can I find more help? - [Qyrus Data Testing - SurrealDB Connector](https://docs.qyrus.com) - [Manage Jobs](https://docs.qyrus.com) - [Data Testing Overview](https://docs.qyrus.com/data-testing) --- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/fastembed # Fastembed This section contains information about using Fastembed to retrieve embeddings to store in SurrealDB Fastembed is a library that allows you to generate vector embeddings locally, without needing an API key or calling into an external service. Fastembed uses the included ONNX runtime to run its embedding models, downloading the model once every time it is used for the first time. Fastembed libraries are available for the following languages: * [Python](https://github.com/qdrant/fastembed) * [Rust](https://crates.io/crates/fastembed) * [Go](https://github.com/Anush008/fastembed-go) * [JavaScript](https://github.com/Anush008/fastembed-js) ## Overview of available models The following is an overview of most of the models available for Fastembed. General use cases are: * Prototyping, development speed, cost savings: models with small vector embedding sizes tend to take up a relatively small amount of disk size as well (generally a few hundred MB) and can be quickly downloaded and run locally. * Small or edge devices: devices with no internet access can still take advantage of vector embeddings. * Particular use cases: some models are specialised for certain use cases such as images, certain languages, and contexts. ### MiniLM series Fast general-purpose embeddings. Choose L6 for speed, L12 for quality. Ideal for semantic search, clustering, and similarity tasks. A "quantized" model means that it is optimised for faster inference and lower memory usage, often with minimal quality loss. | Model name | Embedding size | Description | | --------------- | -------------- | --------------------------------------------------- | | AllMiniLML6V2 | 384 | Sentence Transformer model, MiniLM-L6-v2 | | AllMiniLML6V2Q | 384 | Quantized Sentence Transformer model, MiniLM-L6-v2 | | AllMiniLML12V2 | 384 | Sentence Transformer model, MiniLM-L12-v2 | | AllMiniLML12V2Q | 384 | Quantized Sentence Transformer model, MiniLM-L12-v2 | ### BGE series Used for dense retrieval and semantic similarity. BGESmallENV15 is optimised for speed and tends to be the default choice for many applications. | Model name | Embedding size | Description | | -------------- | -------------- | ------------------------------------------------------------ | | BGEBaseENV15 | 768 | v1.5 release of the base English model | | BGEBaseENV15Q | 768 | Quantized v1.5 release of the base English model | | BGELargeENV15 | 1024 | v1.5 release of the large English model | | BGELargeENV15Q | 1024 | Quantized v1.5 release of the large English model | | BGESmallENV15 | 384 | v1.5 release of the fast and default English model | | BGESmallENV15Q | 384 | Quantized v1.5 release of the fast and default English model | ### Nomic embed text Used for large context window embeddings. Optimised for long-context English text (8K tokens). v1.5 improves quality over v1. | Model name | Embedding size | Description | | ------------------ | -------------- | --------------------------------------------------------------- | | NomicEmbedTextV1 | 768 | 8192 context length english model | | NomicEmbedTextV15 | 768 | v1.5 release of the 8192 context length english model | | NomicEmbedTextV15Q | 768 | Quantized v1.5 release of the 8192 context length english model | ### Paraphrase models Used for paraphrase detection and multilingual similarity. Ideal for sentence equivalence and semantic matching tasks. | Model name | Embedding size | Description | | ------------------------ | -------------- | ------------------------------------------------------------------------ | | ParaphraseMLMiniLML12V2 | 384 | Multi-lingual model | | ParaphraseMLMiniLML12V2Q | 384 | Quantized Multi-lingual model | | ParaphraseMLMpnetBaseV2 | 768 | Sentence-transformers model for tasks like clustering or semantic search, based on the MPNet architecture. | ### Chinese BGE models | Model name | Embedding size | Description | | ------------- | -------------- | --------------------------------------- | | BGESmallZHV15 | 512 | v1.5 release of the small Chinese model | | BGELargeZHV15 | 1024 | v1.5 release of the large Chinese model | ### Modernbert and multilingual e5 Used for context-rich multilingual embeddings. Great for cross-language retrieval and nuanced contextual understanding. | Model name | Embedding size | Description | | -------------------- | -------------- | ---------------------------------------------- | | ModernBertEmbedLarge | 1024 | Large model of ModernBert Text Embeddings | | MultilingualE5Small | 384 | Small model of multilingual E5 Text Embeddings | | MultilingualE5Base | 768 | Base model of multilingual E5 Text Embeddings | | MultilingualE5Large | 1024 | Large model of multilingual E5 Text Embeddings | ### Mxbai and GTE Used for high-quality English/multilingual embeddings. | Model name | Embedding size | Description | | ------------------ | -------------- | ---------------------------------------------------------- | | MxbaiEmbedLargeV1 | 1024 | Large English embedding model from MixedBreed.ai | | MxbaiEmbedLargeV1Q | 1024 | Quantized Large English embedding model from MixedBreed.ai | | GTEBaseENV15 | 768 | Base multilingual embedding model from Alibaba | | GTEBaseENV15Q | 768 | Quantized base multilingual embedding model from Alibaba | | GTELargeENV15 | 1024 | Large multilingual embedding model from Alibaba | | GTELargeENV15Q | 1024 | Quantized large multilingual embedding model from Alibaba | ### CLIP and code models Use CLIP for image-text matching, Jina for code search and retrieval. JinaEmbeddingsV2BaseCode is optimised for embedding code snippets. | Model name | Embedding size | Description | | ------------------------ | -------------- | ----------------------------------- | | ClipVitB32 | 512 | CLIP text encoder based on ViT-B/32 | | JinaEmbeddingsV2BaseCode | 768 | Jina embeddings v2 base code | ## Language-specific example The following example in Rust demonstrates how SurrealDB can be used to store the embeddings from the default language model for a number of phrases, after which it can be prompted to return the three closest results to a certain prompt. First add a few crates to Cargo.toml with the following command: ```bash cargo add anyhow fastembed serde tokio surrealdb --features surrealdb/kv-mem ``` Then use the following code. ```rust use anyhow::Error; use fastembed::{EmbeddingModel, InitOptions, TextEmbedding}; use serde::Serialize; use surrealdb::{ Surreal, Value, engine::any::{Any, connect}, }; const SCHEMA: &str = "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; // Uncomment this to use HNSW index, ensure that number after DIMENSION matches size of embedding // DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 384 DIST COSINE"; const INSERT_QUERY: &str = "INSERT INTO document $docs"; const VECTOR_QUERY: &str = "SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|3,COSINE|> $embeds ORDER BY distance"; #[derive(Serialize)] struct DocumentInput { text: String, embedding: Vec, } async fn store_docs( input: Vec<&str>, db: &Surreal, model: &mut TextEmbedding, ) -> Result<(), Error> { let docs = model .embed(input.clone(), None)? .into_iter() .zip(input.into_iter()) .map(|(embedding, text)| DocumentInput { text: text.to_string(), embedding, }) .collect::>(); db.query(INSERT_QUERY).bind(("docs", docs)).await?; Ok(()) } async fn test_embed( input: &str, db: &Surreal, model: &mut TextEmbedding, ) -> Result<(), Error> { let Some(embeds) = model.embed(vec![input], None)?.into_iter().next() else { return Err(anyhow::anyhow!("Nothing found at index 0")); }; let val = db .query(VECTOR_QUERY) .bind(("embeds", embeds.clone())) .await? .take::(0)?; println!("{val}\n"); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { // Default model let mut model = TextEmbedding::try_new(InitOptions::new(EmbeddingModel::BGESmallENV15))?; let db = connect("memory").await?; db.use_ns("ns").use_db("db").await?; db.query(SCHEMA).await?; let input = vec![ // Cities "Calgary is a city in the Canadian province of Alberta.", "Ljubljana is the capital and largest city of Slovenia.", // Historical / mythological figures "Xenophon of Athens was a Greek military leader, philosopher, and historian.", "King Arthur was a mythical king in the mythology of Great Britain.", // Planets "Venus is the second planet from the Sun.", "Ceres is a dwarf planet in the middle main asteroid belt between the orbits of Mars and Jupiter.", // Languages "Manx is a Gaelic language of the insular Celtic branch of the Celtic language family", "Interlingue, originally Occidental, is an international auxiliary language created in 1922.", // Sea animals "Octopuses have a complex nervous system and are among the most intelligent and behaviourally diverse invertebrates.", "Clams have no central nervous system at all and are near to plants in intelligence.", ]; store_docs(input, &db, &mut model).await?; println!("Edmonton is closest to:"); test_embed("Edmonton", &db, &mut model).await?; println!("Merlin is closest to:"); test_embed("Merlin", &db, &mut model).await?; println!("Earth is closest to:"); test_embed("Earth", &db, &mut model).await?; println!("Irish is closest to:"); test_embed("Irish language", &db, &mut model).await?; println!("Squid are closest to:"); test_embed("Squid", &db, &mut model).await?; Ok(()) } ``` Output of the example with the default model: ```text Edmonton is closest to: [ { distance: 0.2596421358215669f, text: 'Calgary is a city in the Canadian province of Alberta.' }, { distance: 0.5010449624435647f, text: 'Ljubljana is the capital and largest city of Slovenia.' }, { distance: 0.5242241576926254f, text: 'Interlingue, originally Occidental, is an international auxiliary language created in 1922.' } ] Merlin is closest to: [ { distance: 0.3653307924860497f, text: 'King Arthur was a mythical king in the mythology of Great Britain.' }, { distance: 0.4515194174120666f, text: 'Manx is a Gaelic language of the insular Celtic branch of the Celtic language family' }, { distance: 0.5317039966149415f, text: 'Calgary is a city in the Canadian province of Alberta.' } ] Earth is closest to: [ { distance: 0.3380429615054925f, text: 'Venus is the second planet from the Sun.' }, { distance: 0.3764237673020161f, text: 'Ceres is a dwarf planet in the middle main asteroid belt between the orbits of Mars and Jupiter.' }, { distance: 0.444087039462282f, text: 'Calgary is a city in the Canadian province of Alberta.' } ] Irish is closest to: [ { distance: 0.27517683002655635f, text: 'Manx is a Gaelic language of the insular Celtic branch of the Celtic language family' }, { distance: 0.34080671701374754f, text: 'Interlingue, originally Occidental, is an international auxiliary language created in 1922.' }, { distance: 0.5113325799682362f, text: 'King Arthur was a mythical king in the mythology of Great Britain.' } ] Squid are closest to: [ { distance: 0.3439891425642231f, text: 'Octopuses have a complex nervous system and are among the most intelligent and behaviourally diverse invertebrates.' }, { distance: 0.4707156750207915f, text: 'Manx is a Gaelic language of the insular Celtic branch of the Celtic language family' }, { distance: 0.517311424260043f, text: 'Clams have no central nervous system at all and are near to plants in intelligence.' } ] ``` --- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/mistral # Mistral End-to-end guide for building a fast semantic-search stack with Mistral-Embed vectors stored in SurrealDB’s native HNSW index. **Python** Modern open-source RAG pipelines need two things: 1. **High-quality embeddings** - *Mistral-Embed* (`mistral-embed`) returns 1 024-dimensional float vectors that rival OpenAI + Cohere. 2. **Blazing-fast vector storage** - **SurrealDB** (≥ v1.5) ships an in-memory HNSW index, queried with the `<|K,EF|>` operator in SurrealQL. Below you’ll wire them together, from install → ingestion → search → production-ready script. ## Prerequisites ```bash pip install mistralai surrealdb ```` Set two environment variables (or hard-code them if you must): ```bash export SDB_URL="http://localhost:8000/rpc" # ← SurrealDB RPC endpoint export MISTRAL_API_KEY="sk-…" # ← your Mistral key ``` ## Connect and create the schema ```python from mistralai.client import MistralClient from surrealdb import Surreal import os, asyncio # ----- 1.1 · Config ----------------------------------------------------------------- SDB_URL = os.getenv("SDB_URL", "http://localhost:8000/rpc") SDB_USER = os.getenv("SDB_USER", "root") SDB_PASS = os.getenv("SDB_PASS", "secret") NS, DB = "demo", "demo" TABLE = "mistral_docs" MODEL = "mistral-embed" # ----- 1.2 · Clients ---------------------------------------------------------------- sdb = Surreal(SDB_URL) mistr = MistralClient(api_key=os.environ["MISTRAL_API_KEY"]) async def init_db(): await sdb.signin({"user": SDB_USER, "pass": SDB_PASS}) await sdb.use(NS, DB) # one quick embedding → get true vector dimension dim = len(mistr.embeddings(model=MODEL, input=["ping"]).data[0].embedding) schema = """ DEFINE TABLE $tb SCHEMALESS PERMISSIONS NONE; DEFINE FIELD text ON $tb TYPE string; DEFINE FIELD embedding ON $tb TYPE array; DEFINE INDEX hnsw_idx ON $tb FIELDS embedding HNSW DIMENSION {dim} DIST COSINE; """ await sdb.query(schema, {"tb": TABLE}) asyncio.run(init_db()) ``` ### Why detect the dimension dynamically? *Future-proofing:* if Mistral introduces a **small** or **large** Embed model with a different dimension, the code auto-adapts. ## Embed and bulk-insert documents ```python DOCS = [ "SurrealDB offers an in-memory HNSW vector index for low-latency search.", "Mistral-Embed produces 1 024-dimensional embeddings.", "You can build a completely open-source RAG stack with these two tools.", ] async def insert_docs(docs, batch=64): rows = [] for i in range(0, len(docs), batch): chunk = docs[i : i + batch] vecs = mistr.embeddings(model=MODEL, input=chunk).data rows += [ { "id": f"{TABLE}:{i+j}", "text": chunk[j], "embedding": vec.embedding, } for j, vec in enumerate(vecs) ] await sdb.query(f"INSERT INTO {TABLE} $data", {"data": rows}) asyncio.run(insert_docs(DOCS)) ``` *Why bulk-insert?* One SurrealQL call → one network round-trip - **much faster** than inserting row-by-row. ## Search with a natural-language query ```python async def search(query: str, k: int = 3, ef: int = 64): q_vec = mistr.embeddings(model=MODEL, input=[query]).data[0].embedding surql = """ LET $q := $vec; SELECT id, text, vector::distance::knn() AS score FROM $tb WHERE embedding <|{k},{ef}|> $q ORDER BY score; """ res = await sdb.query(surql, {"vec": q_vec, "tb": TABLE}) return res[0].result hits = asyncio.run(search("Which database supports native vector search?")) for h in hits: print(f"⭐ {h['text']} (score={h['score']:.4f})") ``` `<|K,EF|>` activates the HNSW **K-nearest-neighbour** operator (`K=3`, `efSearch=64`). `vector::distance::knn()` exposes the cosine distance already computed inside the index, no post-processing needed. ## Full script (ready to run) ```python # mistral_surreal_demo.py from __future__ import annotations import os, asyncio from mistralai.client import MistralClient from surrealdb import Surreal SDB_URL = os.getenv("SDB_URL", "http://localhost:8000/rpc") SDB_USER = os.getenv("SDB_USER", "root") SDB_PASS = os.getenv("SDB_PASS", "secret") NS, DB, TABLE = "demo", "demo", "mistral_docs" MODEL = "mistral-embed" KEY = os.environ["MISTRAL_API_KEY"] # export first! sdb, mistr = Surreal(SDB_URL), MistralClient(api_key=KEY) DOCS = [ "SurrealDB's vector index is built on HNSW.", "Mistral-Embed vectors offer strong semantic quality.", "Together they form a fast, open-source search stack.", ] async def main(): await sdb.signin({"user": SDB_USER, "pass": SDB_PASS}) await sdb.use(NS, DB) dim = len(mistr.embeddings(model=MODEL, input=["x"]).data[0].embedding) await sdb.query(""" DEFINE TABLE $tb SCHEMALESS PERMISSIONS NONE; DEFINE FIELD text ON $tb TYPE string; DEFINE FIELD embedding ON $tb TYPE array; DEFINE INDEX hnsw_idx ON $tb FIELDS embedding HNSW DIMENSION {dim} DIST COSINE; """, {"tb": TABLE}) # ingest if empty if (await sdb.query(f"SELECT count() FROM {TABLE};"))[0].result[0]["count"] == 0: rows = [] vecs = mistr.embeddings(model=MODEL, input=DOCS).data rows = [ {"id": f"{TABLE}:{i}", "text": DOCS[i], "embedding": v.embedding} for i, v in enumerate(vecs) ] await sdb.query(f"INSERT INTO {TABLE} $data", {"data": rows}) # search q_vec = mistr.embeddings(model=MODEL, input=["open-source vector database"] ).data[0].embedding res = await sdb.query(""" LET $q := $vec; SELECT text, vector::distance::knn() AS score FROM {TABLE} WHERE embedding <|3,64|> $q ORDER BY score; """, {"vec": q_vec}) print(res[0].result) if __name__ == "__main__": asyncio.run(main()) ``` ## About quantisation SurrealDB currently stores vectors as `float32` / `float64` arrays and does **not** ship built-in binary or int8 quantisation. If memory is critical you can: 1. Quantise offline to int8 (e.g. with *faiss* or *sentence-transformers*). 2. Store the int8 arrays in **another field** (SurrealQL’s array type is agnostic). 3. Execute a **two-stage search**: coarse K-NN on the int8 field, then rescore on the full-precision field. ### You’re done 🚀 You now have a clean, fully-async SurrealDB setup that stores **Mistral-Embed** vectors, supports **fast HNSW search**, and can be dropped into any RAG or semantic-search workflow. **Rust** ## Setup Create a new Cargo project with `cargo new project_name` and go into the project folder, then add the following dependencies inside `Cargo.toml`: ```toml anyhow = "1.0.98" mistralai-client = "0.14.0" serde = "1.0.228" surrealdb = { version = "2.3", features = ["kv-mem"] } tokio = "1.49.0" ```
You can add the same dependencies on the command line through a single command: ```bash cargo add anyhow mistralai-client serde tokio surrealdb --features surrealdb/kv-mem ```
Connect to a database using "memory" for an embedded instance: ```rust use anyhow::Error; use surrealdb::engine::any::connect; #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; Ok(()) } ```
Or another address if accessing a Cloud or local instance, such as: ```rust // Cloud address let db = connect("wss://myinstance-06a4h41t12rtj7lsg45m3prm1k.aws-use1.surreal.cloud").await?; // Local address let db = connect("ws://localhost:8000").await?; ```
Then select a namespace and database name. ```rust db.use_ns("ns").use_db("db").await?; ```
## Create a vector table and index Create a table called `document` to store documents and embeddings. The `HNSW` index is one way to [maintain performance](/docs/reference/query-language/language-primitives/operators.md#hnsw-method) if the dataset becomes quite large; otherwise, it can be left out. ```surql DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE; ```
The size of the vector (1024 here) represents the number of dimensions in the embedding. This is to match Mistral AI's `mistral-embed` model, which uses [1024 as its length](https://docs.mistral.ai/getting-started/models/models_overview/#premier-models). These statements can all be put intside a single `.query()` call in the Rust SDK, followed by a line to check for any errors. ```rust let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE;", ) .await?; for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } ```
## Generate Mistral AI embeddings At this point, you will need a [key](https://console.mistral.ai/api-keys) to interact with Mistral AI's platform. They offer a free tier for experimentation, after which you will be able to create a key to interact with it via the code below. The best way to set the key is as an environment variable, which we will set to be a static called `KEY`. The client will look for one called `MISTRAL_API_KEY`, though you can change this when setting up the Mistral AI Rust client if you like. ```rust // Looks for MISTRAL_API_KEY let client = Client::new(Some(KEY.to_string()), None, None, None)?; // Looks for OTHER_ENV_VAR let client = Client::new(Some(KEY.to_string()), Some("OTHER_ENV_VAR".to_string()), None, None)?; ``` Using a `LazyLock` will let us call it via `std::env::var()` function the first time it is accessed. You can of course simply put it into a `const` for simplicity when first testing, but always remember to never hard-code API keys in your code in production. ```rust static KEY: LazyLock = LazyLock::new(|| { std::env::var("MISTRAL_API_KEY").unwrap() }); ```
And then run the code like this: ```bash MISTRAL_API_KEY=whateverthekeyis cargo run ```
Or like this if you are using PowerShell on Windows. ```powershell $env:MISTRAL_API_KEY = "whateverthekeyis" cargo run ```
We will also create a `const MODEL` to hold the Mistral AI model used, which in this case is an `EmbedModel::MistralEmbed`. ```rust const MODEL: EmbedModel = EmbedModel::MistralEmbed; ``` Inside `main()`, [create a client](https://docs.rs/mistralai-client/0.14.0/mistralai_client/v1/client/struct.Client.html#method.new) from the `mistralai-client` crate. ```rust let client = Client::new(Some(KEY.to_string()), None, None, None)?; ```
The client can be used to generate a Mistral AI embedding using the [`mistral-embed`](https://docs.mistral.ai/getting-started/models/models_overview/#premier-models) model. Since SurrealDB uses the tokio runtime, the async `.embeddings_async()` method will be used. ```rust let input = vec!["Joram is the main character in the Darksword Trilogy.".to_string()]; let result = client.embeddings_async(MODEL, input, None).await?; println!("{:?}", result); ```
The output in your console should include an embedding 1024 floats in length. ## Store embeddings in database The embeddings returned from Mistral AI can now be stored in the database. The [response](https://docs.rs/mistralai-client/0.14.0/mistralai_client/v1/embedding/struct.EmbeddingResponse.html) returned from the `mistralai-client` crate looks like this, with a `Vec` of `EmbeddingResponseDataItem` structs that hold a `Vec`. ```rust pub struct EmbeddingResponse { pub id: String, pub object: String, pub model: EmbedModel, pub data: Vec, pub usage: ResponseUsage, } pub struct EmbeddingResponseDataItem { pub index: u32, pub embedding: Vec, pub object: String, } ```
Using `.remove(0)` will allow us to get the raw embeddings here. In a more complex response you might opt for a match on `.get(0)` to handle any possible errors. ```rust let embeds = result.data.remove(0).embedding; ```
There are a [number of ways](/docs/reference/rust/concepts/flexible-typing.md) to work with or avoid structs when using the Rust SDK, including creating structs: one to represent the input into a `.create()` statement, which will implement `Serialize`, and another that implements `Deserialize` to show the result. ```rust #[derive(Serialize)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, Deserialize)] struct Document { id: RecordId, embedding: Vec, text: String, } ```
This can be tested by printing out the created documents as a `Document` struct. ```rust let input = "Octopuses solve puzzles and escape enclosures, showing advanced intelligence."; let mut result = client .embeddings_async(MODEL, vec![input.to_string()], None) .await?; let embeds = result.data.remove(0).embedding; let in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; println!("{in_db:?}"); ```
We will now move the logic to create the embeddings into a function of its own. Since the `embeddings_async()` method takes a single `Vec`, we'll first clone it to keep the original `Vec`, then zip it together with the embeddings returned so that they can be put into the database along with the original input. ```rust async fn create_embeds( input: Vec, db: &Surreal, client: &Client, ) -> Result<(), Error> { let cloned = input.clone(); let embeds = client.embeddings_async(MODEL, input, None).await?; let zipped = cloned .into_iter() .zip(embeds.data.into_iter().map(|item| item.embedding)); for (text, embeds) in zipped { let _in_db = db .create::>("document") .content(DocumentInput { text, embedding: embeds, }) .await?; } Ok(()) } ```
Then we'll create four facts for each of four topics: sea creatures, Korean and Japanese cities, historical figures, and planets of the Solar System. ```rust let embeds = [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks exhibit learning behavior, but their intelligence is instinct-driven.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] .into_iter() .map(|s| s.to_string()) .collect::>(); create_embeds(embeds, &db, &client).await?; ```
## Semantic search Finally let's perform semantic search over the embeddings in our database. We'll go with this query that uses the KNN operator to return the closest four matches to an embedding. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance; ```
To use the HNSW index instead, just change the KNN operator from `<|4,COSINE|>` to a number like `<|4,40|>`. The 40 here represents the size of the dynamic candidate list used during the search. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,40|> $embeds ORDER BY distance; ```
You can customise this [with other algorithms](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. We will then put this into a separate function called `ask_question()`, which first prints out its input and then uses its embedding retrieved from Mistral AI to query the database against existing documents. ```rust async fn ask_question(input: &str, db: &Surreal, client: &Client) -> Result<(), Error> { println!("{input}"); let embeds = client .embeddings_async(MODEL, vec![input.to_string()], None) .await? .data .remove(0) .embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } ```
This function can now be called inside `main()` to confirm that the results match with our expectations. ```rust ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; ```
```text Which Korean city is just across the sea from Japan? [{ distance: 0.19170371029549582f, text: 'Busan is a major South Korean port located in the far southeast.' }, { distance: 0.2399314515762122f, text: 'Tokyo is Japan’s capital, known for innovation and dense population.' }, { distance: 0.2443623703771407f, text: 'Sejong is South Korea’s planned administrative capital.' }, { distance: 0.24488082839731895f, text: 'Seoul is South Korea’s capital and a global tech hub.' }] Who was Germany's last Kaiser? [{ distance: 0.11228576780228805f, text: 'Wilhelm II was Germany’s last Kaiser before World War I.' }, { distance: 0.2957177300085634f, text: 'Napoleon Bonaparte was a French emperor and brilliant military strategist.' }, { distance: 0.34394473621670896f, text: 'Cyrus the Great founded the Persian Empire with tolerant rule.' }, { distance: 0.34911517400935843f, text: 'Sejong is South Korea’s planned administrative capital.' }] Which sea animal is most intelligent? [{ distance: 0.2342596053829904f, text: 'Octopuses solve puzzles and escape enclosures, showing advanced intelligence.' }, { distance: 0.24131327939924785f, text: 'Sharks exhibit learning behavior, but their intelligence is instinct-driven.' }, { distance: 0.2426242772516931f, text: 'Clams have simple nervous systems with no known intelligent behavior.' }, { distance: 0.24474598154128135f, text: 'Sea cucumbers lack a brain and show minimal cognitive response.' }] Which planet's atmosphere has a part with the same temperature as Earth? [{ distance: 0.20653440713083582f, text: 'Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.' }, { distance: 0.23354208810464594f, text: 'Mars has a thin, cold atmosphere with seasonal dust storms.' }, { distance: 0.24560810032473468f, text: 'Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior' }, { distance: 0.2761595357544341f, text: 'Ceres has a tenuous exosphere with sporadic water vapor traces.' }] ```
Here is the entire code: ```rust use std::sync::LazyLock; use anyhow::Error; use mistralai_client::v1::{client::Client, constants::EmbedModel}; use serde::{Deserialize, Serialize}; use surrealdb::{ RecordId, Surreal, Value, engine::any::{Any, connect}, }; static KEY: LazyLock = LazyLock::new(|| std::env::var("MISTRAL_API_KEY").unwrap()); // Experiment plan const MODEL: EmbedModel = EmbedModel::MistralEmbed; #[derive(Serialize)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, Deserialize)] struct Document { id: RecordId, embedding: Vec, text: String, } async fn create_embeds( input: Vec, db: &Surreal, client: &Client, ) -> Result<(), Error> { let cloned = input.clone(); let embeds = client.embeddings_async(MODEL, input, None).await?; let zipped = cloned .into_iter() .zip(embeds.data.into_iter().map(|item| item.embedding)); for (text, embeds) in zipped { let _in_db = db .create::>("document") .content(DocumentInput { text, embedding: embeds, }) .await?; } Ok(()) } async fn ask_question(input: &str, db: &Surreal, client: &Client) -> Result<(), Error> { println!("{input}"); let embeds = client .embeddings_async(MODEL, vec![input.to_string()], None) .await? .data .remove(0) .embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await.unwrap(); db.use_ns("ns").use_db("db").await.unwrap(); let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE;", ) .await .unwrap(); for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } let client = Client::new(Some(KEY.to_string()), None, None, None)?; let embeds = [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks exhibit learning behavior, but their intelligence is instinct-driven.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan is a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] .into_iter() .map(|s| s.to_string()) .collect::>(); create_embeds(embeds, &db, &client).await?; ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; Ok(()) } ``` --- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/openai # OpenAI This section contains information about the OpenAI embeddings feature of SurrealDB. **Python** SurrealDB provides a number of different embeddings features that can be used to manage your data. This walkthrough shows how to embed your text with OpenAI, store the vectors in SurrealDB, and run fast k‑nearest‑neighbour (KNN) searches, all from Python. It follows the same flow you might have seen for Qdrant, but swaps in SurrealDB’s native vector‑search features so you can keep documents, graphs, and embeddings in one place. ## Install the two clients OpenAI calls the API that turns text into a 1 536‑dimensional vector, while SurrealDB lets your Python code connect over WebSocket or HTTP to any SurrealDB server. SurrealDB must be running somewhere. If you do not have SurrealDB running already you can spin up an in‑memory node in one line: ```bash pip install openai surrealdb ``` For local testing you can spin it up quickly: ```bash docker run -p 8000:8000 surrealdb/surrealdb:latest start --user root --pass secret ``` ## Set up the OpenAI and SurrealDB clients ```python import asyncio import openai from surrealdb import Surreal openai_client = openai.Client(api_key="YOUR_OPENAI_KEY") TEXTS = [ "SurrealDB is a multi-model database you can embed anywhere!", "Loved by devs who need graphs, documents **and** vector search in one box.", ] async def init_db() -> Surreal: db = Surreal("ws://localhost:8000/rpc") # change URL if remote await db.signin({"username": "root", "password": "secret"}) await db.use("demo_ns", "demo_db") return db ``` ## Create the embeddings with OpenAI `text‑embedding‑3‑small` is compact, cheap, and surprisingly accurate for semantic search. If you need more nuance (or plan to quantise heavily) you can upgrade to `text‑embedding‑3‑large` (3 072 dimensions) at roughly twice the cost. ```python EMBED_MODEL = "text-embedding-3-small" emb_resp = openai_client.embeddings.create( input=TEXTS, model=EMBED_MODEL ) emb_vectors = [row.embedding for row in emb_resp.data] # list[list[float]] ``` ## Insert the documents into SurrealDB SurrealDB is schemaless by default, so you can drop JSON directly into a table named article: *(table `article` with a vector field called `embedding`)* ```python async def load_documents(db: Surreal): docs = [ {"text": txt, "embedding": vec} for txt, vec in zip(TEXTS, emb_vectors) ] await db.create("article", docs) asyncio.run(load_documents(asyncio.run(init_db()))) ``` ## Add a vector index (HNSW) The Hierarchical Navigable Small World (HNSW) index trades a tiny bit of accuracy for dramatic speed‑ups compared with brute-force scanning; ideal once your table grows beyond a few thousand rows. Because HNSW lives in RAM today, index creation is instant but counts against your memory budget. SurrealQL lets you add the index once; after that inserts are indexed automatically. ```python async def add_index(db: Surreal): await db.query(""" DEFINE INDEX article_embedding_hnsw ON article FIELDS embedding HNSW DIMENSION 1536; """) asyncio.run(add_index(asyncio.run(init_db()))) ``` The `DIMENSION` must match the length of the OpenAI vectors (1536 in this model). ([SurrealDB][1]) ## Search for the most relevant documents ```python async def semantic_search(question: str, k: int = 3): query_vec = openai_client.embeddings.create( input=[question], model=EMBED_MODEL ).data[0].embedding db = await init_db() # KNN search: <|k, ef|> - ef is “search breadth”. 100 is a decent default. result = await db.query(` LET $q := {query_vec}; SELECT id, text, vector::distance::knn() AS distance FROM article WHERE embedding <|{k},100|> $q ORDER BY distance; """) return result[0]['result'] hits = asyncio.run(semantic_search("What’s the best database for unified search?")) for hit in hits: print(hit["distance"], hit["text"]) ``` The `vector::distance::knn()` helper returns the exact distance that the KNN operator just computed, so SurrealDB can immediately sort or filter without recalculating. Two knobs to remember: - `k` - the number of neighbours you want back. - `ef` (the second number) - search breadth. Higher values spend more CPU for slightly better accuracy; 100-200 is a safe default. ## (Optional) Binary-quantise the vectors SurrealDB stores vectors as arrays of floats (`F32` by default) and doesn’t yet expose built-in binary quantisation the way Qdrant does. If you need ultra-compact storage you can: 1. **Quantise offline** (e.g. convert to `int32` with `numpy`). 2. Store the compressed list in a separate field (`embedding_q32`) and build an index on that instead: ```sql DEFINE INDEX article_embedding_q32_mt ON article FIELDS embedding_q8 HNSW DIMENSION 1536 TYPE I32; ``` This keeps the SurrealDB side simple while you experiment with different quantisers. For more information about vector search in SurrealDB, see the [Vector Search Reference guides](/docs/learn/data-models/vector-search/overview.md). **Rust** ## Semantic search using SurrealDB ## Intro This guide demonstrates how to store OpenAI embeddings as [SurrealDB vectors](/docs/learn/data-models/vector-search/overview.md) via the Rust SDK for the purposes of semantic search. ## Setup First set up a new Cargo project with `cargo new project_name` and add the following dependencies to `Cargo.toml`: ```toml anyhow = "1.0.98" async-openai = "0.28.3" serde = "1.0.228" surrealdb = { version = "2.3", features = ["kv-mem"] } tokio = "1.49.0" ```
Inside `main()`, call the `connect` function with `"memory"` to instantiate an embedded database in memory. ```rust use anyhow::Error; use surrealdb::engine::any::connect; #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; Ok(()) } ```
If you have a Cloud or local instance to connect to, you can pass that path into the connect function instead. ```rust // Cloud address let db = connect("wss://myinstance-06a4h41t12rtj7lsg45m3prm1k.aws-use1.surreal.cloud").await?; // Local address let db = connect("ws://localhost:8000").await?; ```
After connecting, select a namespace and database name, such as `ns` and `db`. ```rust db.use_ns("ns").use_db("db").await?; ```
## Create a vector table Create a table to store documents and embeddings, along with an index for the embeddings: ```surql DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536; ```
These can be called via a single `.query()` method. ```rust let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536;", ) .await?; ```
The size of the vector (1536 here) represents the number of dimensions in the embedding. Since OpenAI's `text-embedding-3-small` model in this example uses [1536 as its default length](https://platform.openai.com/docs/guides/embeddings), the vector size must be set to 1536. The [HNSW index](/docs/learn/data-models/vector-search/vector-indexes.md) is not strictly necessary to use the KNN operator (`<||>`) to find an embedding's closest neighbours, and for our small sample code we will use the simple [brute force method](/docs/reference/query-language/language-primitives/operators.md#brute-force-method) which chooses [an algorithm](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. The following is the code that we will use, which uses the cosine of an embedding to find the four closest neighbours. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance; ``` As the dataset grows, if some loss of accuracy is acceptable then the syntax can be changed to use [the HNSW index](/docs/reference/query-language/language-primitives/operators.md#hnsw-method), by replacing an algorithm with a number that represents the size of the dynamic candidate list. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,40|> $embeds ORDER BY distance; ``` ## Generate OpenAI embeddings At this point, you will need an [OpenAI API key](https://platform.openai.com/api-keys) to interact with the OpenAI API.
The best way to set the key is as an environment variable, `OPENAI_API_KEY` in this case. Using a `LazyLock` will let us call it via `std::env::var()` function the first time it is accessed. You can of course simply put it into a `const` for simplicity when first testing, but always remember to never hard-code API keys in your code in production. ```rust static KEY: LazyLock = LazyLock::new(|| { std::env::var("OPENAI_API_KEY").unwrap() }); ```
And then run the code like this: ```bash OPENAI_API_KEY=whateverthekeyis cargo run ```
Or like this if you are using PowerShell on Windows. ```powershell $env:OPENAI_API_KEY = "whateverthekeyis" cargo run ```
Inside `main()`, [create a client](https://docs.rs/async-openai/0.28.3/async_openai/struct.Client.html) from the async-openai crate holding this config inside `main()`. ```rust let config = OpenAIConfig::new().with_api_key(KEY); let client = Client::with_config(config); ```
Then that to generate an OpenAI embedding using [`text-embedding-3-small`](https://platform.openai.com/docs/guides/embeddings/embedding-models) that can be seen using a `println!` statement. ```rust let input = "Octopuses solve puzzles and escape enclosures, showing advanced intelligence."; let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let result = client.embeddings().create(request).await?; println!("{result:?}"); ```
## Store embeddings in database With the embeddings returned from the OpenAI client, they can be stored in the database. The [response](https://docs.rs/async-openai/0.28.3/async_openai/types/struct.CreateEmbeddingResponse.html) returned from the async-openai crate looks like this, with a `Vec` of `Embedding` structs that hold a `Vec`. ```rust pub struct CreateEmbeddingResponse { pub object: String, pub model: String, pub data: Vec, pub usage: EmbeddingUsage, } pub struct Embedding { pub index: u32, pub object: String, pub embedding: Vec, } ```
This simple request only returned a single embedding, so `.remove(0)` will do the job. In a more complex codebase you would probably opt for a match on `.get(0)` to handle any possible errors. ```rust let embeds = result.data.remove(0).embedding; ```
Two structs can be put together here: one that implements `Serialize` to serve as the input put in a `.create()` statement, and another that implements `Deserialize` to show the result with the `RecordId` included. ```rust #[derive(Serialize)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, Deserialize)] struct Document { id: RecordId, embedding: Vec, text: String, } ```
Once that is done, we can print out the created documents as a `Document` struct. ```rust let in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec() }) .await?; println!("{in_db:?}"); ```
We should now add some more `document` records. To do this, we'll move the logic to create them inside a function of its own: ```rust async fn create_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let result = client.embeddings().create(request).await?; let embeds = &result.data.get(0).unwrap().embedding; let _in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; Ok(()) } ```
And then call it a few times inside `main()`. See if you can guess the answers yourself! ```rust for input in [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks are primarily driven by instinct, but are capable of learning.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] { create_embed(input, &db, &client).await? } ```
## Semantic search Finally, let's perform semantic search over the embeddings in our database. Here is the query again: ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance; ```
We will then put this into a separate function called `ask_question()` which uses the embedding retrieved from OpenAI to query the database against existing documents. ```rust async fn ask_question( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { println!("{input}"); let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } ```
You can now call this function a few times inside `main()` to confirm that the results are what we expect them to be. ```rust ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; ```
The output shows that the closest documents to our question do indeed show up first. ```text Which Korean city is just across the sea from Japan? [{ distance: 0.4879310782198243f, text: 'Busan a major South Korean port located in the far southeast.' }, { distance: 0.572999190509329f, text: 'Seoul is South Korea’s capital and a global tech hub.' }] Who was Germany's last Kaiser? [{ distance: 0.3236345624131668f, text: 'Wilhelm II was Germany’s last Kaiser before World War I.' }, { distance: 0.7554141523606017f, text: 'Napoleon Bonaparte was a French emperor and brilliant military strategist.' }] Which sea animal is most intelligent? [{ distance: 0.45382501257446206f, text: 'Octopuses solve puzzles and escape enclosures, showing advanced intelligence.' }, { distance: 0.4951347026545868f, text: 'Clams have simple nervous systems with no known intelligent behavior.' }] Which planet's atmosphere has a part with the same temperature as Earth? [{ distance: 0.4445578407153489f, text: 'Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.' }, { distance: 0.5039940919211086f, text: 'Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior' }] ```
At this point, you could give the HNSW index a try by changing the `<|2,COSINE|>` in the query to something like `<|2,40|>`. The distance numbers will end up looking quite different, but the ordering of the closest neighbours will probably be the same in this small example.
Here is the final code: ```rust use std::sync::LazyLock; use anyhow::Error; use async_openai::{Client, config::OpenAIConfig, types::CreateEmbeddingRequestArgs}; use serde::{Deserialize, Serialize}; use surrealdb::{ RecordId, Surreal, Value, engine::any::{Any, connect}, }; static KEY: LazyLock = LazyLock::new(|| std::env::var("OPENAI_API_KEY").unwrap()); #[derive(Serialize)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, Deserialize)] struct Document { id: RecordId, embedding: Vec, text: String, } async fn create_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let _in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; Ok(()) } async fn ask_question( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { println!("{input}"); let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; db.use_ns("ns").use_db("db").await?; let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536 DIST COSINE;", ) .await?; for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } let config = OpenAIConfig::new().with_api_key(&*KEY); let client = Client::with_config(config); for input in [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks are primarily driven by instinct, but are capable of learning.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] { create_embed(input, &db, &client).await? } ask_question( "Which Korean city is just across the sea from Japan?", &db, &client, ) .await?; ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; Ok(()) } ``` --- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/overview # Embeddings provider integrations This section contains information about different LLM models you can use with SurrealDB. SurrealDB offers comprehensive support for vector embeddings, enabling powerful semantic search and machine learning capabilities across your data. Through integrations with leading embedding providers, you can easily store, index and query high-dimensional vectors alongside your regular data. - [Quick start with Python](/docs/build/integrations/embeddings-providers/python-quickstart.md) - LangChain, Ollama, Mistral, and more - [Quick start with Rust](/docs/build/integrations/embeddings-providers/rust-quickstart.md) - Mistral, Ollama, and more ## Full examples
Mistral Mistral AI's embedding model and SurrealDB vector search (Python and Rust)
OpenAI OpenAI's embedding service and SurrealDB vector search (Python and Rust)
--- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/python-quickstart # Python quickstart This section contains information about different embedding models you can use with SurrealDB. SurrealDB offers comprehensive support for vector embeddings, enabling powerful semantic search and machine learning capabilities across your data. Through integrations with leading embedding providers, you can easily store, index and query high-dimensional vectors alongside your regular data. **LangChain** More details and providers in [LangChain Embedding models](https://python.langchain.com/docs/integrations/text_embedding/) documentation. **Ollama** ## Ollama ```python from langchain_ollama import OllamaEmbeddings vector_store = SurrealDBVectorStore( OllamaEmbeddings(model="all-minilm:22m"), conn ) ``` More [Ollama embedding models](https://ollama.com/search?c=embedding) in their documentation. **OpenAI** ## OpenAI Requires `OPENAI_API_KEY` environment variable. ```python from langchain_openai import OpenAIEmbeddings vector_store = SurrealDBVectorStore( OpenAIEmbeddings(model="text-embedding-3-large"), conn ) ``` **Mistral** ## Mistral Requires `MISTRALAI_API_KEY` environment variable. ```python from langchain_mistralai import MistralAIEmbeddings vector_store = SurrealDBVectorStore( MistralAIEmbeddings(model="mistral-embed"), conn ) ``` **SentenceTransformer** ## SentenceTransformer ```python from langchain_huggingface import HuggingFaceEmbeddings vector_store = SurrealDBVectorStore( HuggingFaceEmbeddings( model_name="sentence-transformers/all-MiniLM-L6-v2" ), conn ) ``` More [SentenceTransformer models](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html) in their documentation. **AWS Bedrock** ## AWS Bedrock ```python from langchain_aws import BedrockEmbeddings vector_store = SurrealDBVectorStore( BedrockEmbeddings(model_id="amazon.titan-embed-text-v2:0"), conn ) ``` **Gemini** ## Gemini Requires `GOOGLE_API_KEY` environment variable. ```python from langchain_google_genai import GoogleGenerativeAIEmbeddings vector_store = SurrealDBVectorStore( GoogleGenerativeAIEmbeddings(model="models/embedding-001"), conn ) ```
Then, to query the vector store using similarity search: ```python doc1 = Document( page_content="SurrealDB is the ultimate multi-model database for AI applications", metadata={"key": "sdb"}, ) doc2 = Document( page_content="Surrealism is an artistic and cultural movement that emerged in the early 20th century", metadata={"key": "surrealism"}, ) vector_store.add_documents(documents=[doc1, doc2], ids=["1", "2"]) results = vector_store.similarity_search_with_score(query=q, k=2) for doc, score in results: print(f"• [{score:.0%}]: {doc.page_content}") top_match = results[0][0] ``` Find an example in [Minimal LangChain chatbot example with vector and graph](/blog/minimal-langchain-chatbot-example-with-vector-and-graph). **Vanilla** **Ollama** ## Ollama ```python import ollama embedding = ollama.embed(model="all-minilm:22m", input=text) conn.create("documents", { "content": text, "embedding": embedding }) ``` More [Ollama embedding models](https://ollama.com/search?c=embedding) in their documentation. **OpenAI** ## OpenAI ```python from openai import OpenAI client = OpenAI() text = "Your text string" response = client.embeddings.create( input=text, model="text-embedding-3-small" ) conn.create("documents", { "content": text, "embedding": response.data[0].embedding }) ``` More info in [OpenAI embeddings](https://platform.openai.com/docs/guides/embeddings?lang=python) documentation. **Sentence Transformers** ## Sentence Transformers ```python from sentence_transformers import SentenceTransformer st = SentenceTransformer("all-MiniLM-L6-v2") embedding = st.encode(text).tolist() conn.create("documents", { "content": text, "embedding": embedding }) ``` More [SentenceTransformer models](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html) in their documentation. To query the vector store using similarity search: ```python k = 5 query_embedding = st.encode("this is your query text") res = conn.query( f""" SELECT *, vector::distance::knn() AS dist FROM documents WHERE embedding <|{k}|> $vector; """, { "vector": query_embedding.tolist(), }, ) ``` This requires an index to be created beforehand. Refer to the [vector search cheat sheet](/docs/learn/data-models/vector-search/vector-indexes.md#vector-search-cheat-sheet).
Examples above assume you have a DB connection like this: ```python conn = Surreal("localhost") conn.signin({"username": "root", "password": "secret"}) conn.use("test_ns", "test_db") ``` --- Source: https://surrealdb.com/docs/build/integrations/embeddings-providers/rust-quickstart # Rust quickstart This section contains information about different embedding models you can use with SurrealDB. SurrealDB offers comprehensive support for vector embeddings, enabling powerful semantic search and machine learning capabilities across your data. Through integrations with leading embedding providers, you can easily store, index and query high-dimensional vectors alongside your regular data. **Mistral** ## Mistral ```rust use mistralai_client::v1::{client::Client, constants::EmbedModel}; static KEY = std::env::var("MISTRAL_API_KEY").unwrap(); // ... let client = Client::new(Some(KEY.to_string()), None, None, None)?; let input = vec!["Joram is the main character in the Darksword Trilogy.".to_string()]; let result = client.embeddings_async(MODEL, input, None).await?; println!("{:?}", result); ``` Find a full example in [Semantic search in Rust with SurrealDB and Mistral AI](/blog/semantic-search-in-rust-with-surrealdb-and-mistral-ai#generate-mistral-ai-embeddings). **Ollama** ## Ollama ```rust use ollama_rs::{Ollama, generation::embeddings::GenerateEmbeddingsRequest}; #[tokio::main] async fn main() -> Result<(), Box> { let ollama = Ollama::default(); let model = "all-minilm:22m".to_string() let prompt = "this is your input text".to_string(); let request = GenerateEmbeddingsRequest::new(model, prompt); let response = ollama.generate_embeddings(request).await?; println!("Generated embeddings (first 5): {:?}", &response.embeddings[..5]); println!("Embedding vector length: {}", response.embeddings.len()); Ok(()) } ``` **SentenceTransformer** ## SentenceTransformer ```rust use rust_bert::sentence_embeddings::{ SentenceEmbeddingsBuilder, SentenceEmbeddingsModelType, }; fn main() -> anyhow::Result<()> { // Set up the model builder, specifying the model type let model = SentenceEmbeddingsBuilder::remote( SentenceEmbeddingsModelType::AllMiniLmL6V2 ).create_model()?; // Define the sentences to embed let sentences = [ "this is your text", "you can encode more than one in batch" ]; // Generate the embeddings let embeddings = model.encode(&sentences)?; // Print the results for (i, embedding) in embeddings.iter().enumerate() { // Truncate for display purposes let truncated_embedding: Vec<_> = embedding.iter().take(5).cloned().collect(); println!("\nSentence: '{}'", sentences[i]); println!("Embedding (first 5 values): {:?}", truncated_embedding); println!("Embedding dimensions: {}", embedding.len()); } Ok(()) } ``` --- Source: https://surrealdb.com/docs/build/migrating # Migrating to SurrealDB Moving data into SurrealDB from other databases, or older releases. Also from files and streams. This section describes how to move data into SurrealDB: from other databases, from older SurrealDB versions, and from files or streams (for example CSV, JSON Lines, or Kafka). Use the subsections for the path that matches your source. ## In this section - [1.x to 2.x](/docs/build/migrating/from-old-surrealdb-versions/1x-to-2x.md) - what changed between the 1.x and 2.x releases - [2.x to 3.x](/docs/build/migrating/from-old-surrealdb-versions/2x-to-3x.md) - what changed between the 2.x and 3.x releases - [From files and streams](/docs/build/migrating/from-files-and-streams.md) - load data held in files or arriving as a stream - [From other databases](/docs/build/migrating/from-other-databases/overview.md) - move across from another database engine --- Source: https://surrealdb.com/docs/build/migrating/from-files-and-streams # Migrating Migrating This section details how to map data, queries and concepts you may know from other databases and datatypes into SurrealDB. The sources in this section of the documentation can be automatically imported to SurrealDB using the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool. In addition, [CSV data](/docs/build/migrating/from-files-and-streams/csv.md) can be automatically imported into SurrealDB Studio using its own [built-in functionality](/docs/explore/studio.md). For other sources not yet supported, consider beginning by exporting the database as JSON which can be [imported on the command line](/docs/reference/cli/surrealdb-cli/commands/import.md). Alternatively, you can use [one of the many available SDKs](/docs/languages.md) to access your existing database and transfer its content directly to a SurrealDB instance. * [CSV data](/docs/build/migrating/from-files-and-streams/csv.md) * [JSON lines](/docs/build/migrating/from-files-and-streams/json-lines.md) * [Kafka](/docs/build/migrating/from-files-and-streams/kafka.md) --- Source: https://surrealdb.com/docs/build/migrating/from-files-and-streams/csv # Importing CSV data SurrealDB Studio can be used to import CSV data to SurrealDB. This page details some methods to import CSV data to SurrealDB. ## Using SurrealDB Studio CSV data is the easiest external data type to import to SurrealDB, as it can be imported directly via the [SurrealDB Studio](/docs/explore/studio.md) UI. Importing the data is done by going to the Explorer tab, clicking on Import database below, and following the prompts. For more details, see [this page](/docs/explore/studio.md) in the SurrealDB Studio documentation. ## Importing CSV data using Surreal Sync The [Surreal Sync](https://github.com/surrealdb/surreal-sync) tool can be used to import CSV files in the local filesystem or S3 buckets into a SurrealDB table with automatic type detection and optional record ID generation. For more on how this is done, please see the [CSV Import for Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/csv.md) page in its repo. --- Source: https://surrealdb.com/docs/build/migrating/from-files-and-streams/json-lines # Importing JSON Lines data SurrealDB Studio can be used to import CSV data to SurrealDB. This page details JSON data types and their SurrealQL equivalents, followed by links to the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool which allows this data to be automatically imported to SurrealDB. ## Data types | JSON Data Type | JSON Representation | Recommended SurrealQL type | | ------------------------ | ---------------------------------------- | ----------------------------- | | **String** | `"text"` | `string` | | **Number** | `42` or `3.14` | `int`, `float`, or `decimal` | | **Boolean** | `true`/`false` | `bool` | | **null** | `null` | `none` or `null` | | **Array** | `[1, 2, 3]` | `array` or `set` | | **Object** | `{"key": "value"}` | `object` | | **ISO 8601 Date String** | `"2024-01-15T14:30:00Z"` | `datetime` or `string` | | **UUID String** | `"550e8400-e29b-41d4-a716-446655440000"` | `uuid` or `string` | | **Base64 String** | `"SGVsbG8gV29ybGQ="` | `bytes` or `string` | ## Importing JSON Lines data using Surreal Sync The JSONL source in Surreal Sync allows you to import JSON Lines (JSONL) files into SurrealDB. Each JSONL file becomes a table in SurrealDB, and each line in the file becomes a document in that table. For more on how to import JSON Lines data to SurrealDB, please see the following pages in the Surreal Sync repo. * [JSONL Source Usage Guide](https://github.com/surrealdb/surreal-sync/blob/main/docs/jsonl.md) * [JSONL Data Types Support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/jsonl.md) --- Source: https://surrealdb.com/docs/build/migrating/from-files-and-streams/kafka # Importing Kafka data SurrealDB's Surreal Sync tool can be used to import Kafka data to SurrealDB. Kafka data can be imported to SurrealDB using the Surreal Sync tool, as follows: * The specified Kafka topic becomes a table in SurrealDB, where each message from the topic becomes a record in the table, with configurable deduplication. * Each Kafka message must be encoded using Protobuf. The Kafka source decodes every message payload using Protobuf and converts it into SurrealDB Upsert queries, transforming the payloads into SurrealDB records. * Message Key Strategy: Use Kafka message keys as record IDs (base64 encoded) * Field Strategy: Extract IDs from a field in the message payload (default: "id" field) For more on how to import Kafka data to SurrealDB, please see the [Kafka Source Usage Guide](https://github.com/surrealdb/surreal-sync/blob/main/docs/kafka.md) page in the Surreal Sync repo. --- Source: https://surrealdb.com/docs/build/migrating/from-old-surrealdb-versions/1x-to-2x # 1.x to 2.x This guide will help you upgrade your current SurrealDB installation to the latest `2.x` release. The `2.0.0` release of SurrealDB includes [many new features, improvements, and bug fixes](/releases#v2-0-0). However, due to this there are some breaking changes that you should be aware of when upgrading. This guide will help you upgrade your current SurrealDB installation to the latest `2.x` release. ## Breaking changes ### Datastore - The underlying approach for storing record IDs and ranges has changed - [Record IDs now support storing UUIDs instead of strings](https://github.com/surrealdb/surrealdb/pull/4491) - [Ranges are now their own value as suppose to being available as just record id ranges](https://github.com/surrealdb/surrealdb/pull/4506) ### SurrealQL - The `UPDATE` statement no longer creates records if these are missing. Instead, use the new [UPSERT](/docs/reference/query-language/statements/upsert.md) statement for this behaviour. - The `file://` connection protocol has been deprecated in favour of the more explicit `rocksdb://` protocol. - The `DEFINE SCOPE` statement has been dropped in favour for the new [`DEFINE ACCESS TYPE RECORD`](/docs/reference/query-language/statements/define/access/record.md) statement - `DEFINE TOKEN` definitions defined under scopes are now integrated into `DEFINE ACCESS TYPE RECORD`. - Some functions have been renamed for clarity - `meta::tb()` -> [`record::tb()`](/docs/reference/query-language/functions/database-functions/record.md) - `meta::id()` -> [`record::id()`](/docs/reference/query-language/functions/database-functions/record.md) - `string::endsWith()` -> [`string::ends_with()`](/docs/reference/query-language/functions/database-functions/string.md) - `string::startsWith()` -> [`string::starts_with()`](/docs/reference/query-language/functions/database-functions/string.md) ### Authentication and headers - Authentication is now enabled by default as you previously would with `--auth`. The [`--unauthenticated`](/docs/reference/cli/surrealdb-cli/commands/start.md#unauthenticated-mode) flag is now required in order to provide the previous default behaviour. - Specifying the level on which credentials will be authenticated is now required when connecting to SurrealDB. By default, this level will be root. This can be provided with the `--auth-level` flag in the CLI or the `surreal-auth-ns` and `surreal-auth-db` headers in the HTTP REST API. - SurrealDB now listens only for connections from the local machine unless another interface (e.g. 0.0.0.0) is provided via the `--bind` command line argument. - SurrealDB now does not print secrets in response to `INFO` statements. The values of the secrets will appear as `[REDACTED]` to prevent accidental leakage. The export functionality will still print the values of the secrets. An `UNREDACTED` clause will be added soon to provide the previous behaviour. - Headers used to communicated via HTTP with SurrealDB now require the `surreal-` prefix. For example, the legacy `ns` and `db` headers are now `surreal-ns` and `surreal-db`. ## Upgrading your data A new [`surreal fix`](/docs/reference/cli/surrealdb-cli/commands/fix.md) command has been implemented to automatically change the format of your stored data. The command is followed by a path to the data. For example: ```bash # For SurrealKV surreal fix surrealkv://mydata # For RocksDB surreal fix rocksdb:somedatabase ``` ### Limitations of the surreal fix command Although the `surreal fix` command is a quick way to migrate your data, it is not without its drawbacks: - If you have used the now deprecated [`DEFINE TOKEN`](/docs/reference/query-language/statements/define/token.md) command to define a token on a Scope with the also deprecated [`DEFINE SCOPE`](/docs/reference/query-language/statements/define/scope.md) command, you will have to update your access management rules to use the new [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md) which supports creating permissions using [TYPE JWT](/docs/reference/query-language/statements/define/access/jwt.md) and [TYPE RECORD](/docs/reference/query-language/statements/define/access/record.md) rules. - If you were querying SurrealDB via the HTTP API, the `surreal fix` command will not update the header format for you. You will need to manually update the header format from `ns` and `db` to `surreal-ns` and `surreal-db` respectively before using the `surreal fix` command. Learn more about this in the [HTTP documentation](/docs/reference/rest-api/http-protocol.md). ### Upgrading from 2.0.0-alpha The `surreal fix` command above has been created specifically for 1.x instances. However, data currently on a `2.0.0-alpha` instance can still be manually exported and then reimported into a project running on `2.0.0` via the following steps. 1. Export your current data as a `.surql` (SurrealQL) file. You can do this using the [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) command in the terminal: ```bash # Example export command to export data to a file called `export.surql` in the downloads directory. surreal export --conn http://localhost:8000 --user root --pass secret --ns main --db main downloads/export.surql ``` 2. This will create a file called `export.surql` in the current directory. 3. You can now import this file back into your project running on `2.0.0`. ```bash surreal import --conn http://localhost:8000 --user root --pass secret --ns main --db main downloads/export.surql ``` ## Troubleshooting ### Error when connecting to a `2.x` instance. If you are trying to connect to a `2.x` instance, and get an error similar to the following, you are likely using an older version of SurrealDB. ```bash error: Storage version is out-of-date. ``` ## Read the full changelog There have been major improvements to SurrealDB in `2.0.0` release both in alpha and beta. Check out the changes on the [release page](/releases). --- Source: https://surrealdb.com/docs/build/migrating/from-old-surrealdb-versions/2x-to-3x # 2.x to 3.x This guide will help you upgrade your current SurrealDB installation to the latest `3.x` release. This guide consolidates all breaking changes when upgrading from SurrealDB `2.x` to `3.x`, organised by severity level. If you are using SurrealDB Studio, you can use the [migration diagnostics](/docs/explore/studio.md) to automatically see your data. This will also provide you with a list of actions you need to take to migrate your data. ## Migration diagnostics in SurrealDB Studio SurrealDB Studio provides a built-in migration diagnostics tool that can be used to automatically see your data and provide you with a list of actions you need to take to migrate your data. >[!NOTE] >The migration diagnostics tool is only available for SurrealDB version `2.6.1` and above. SurrealDB Studio migration diagnostics Select your `2.x` database and click on the **Migration** option in the sidebar. This will open the migration diagnostics tool. First you'll need to start the checks by clicking on the **Start Checks** button. This will return a migration report with a list of actions you need to take to migrate your data (If any). SurrealDB Studio migration report After resolving the issue, click on the **Mark as resolved** button to mark the issue as resolved. This will remove the issue from the migration report. Once all issues have been resolved, the migration diagnostics tool will allow you to export a [V3 Compatible Export](#using-v3-compatible-export) that can be imported into your updated SurrealDB `3.x` instance. ## V3 compatible export Starting with SurrealDB `2.6.0`, you can export your database in a format that is compatible with version `3.x`. This export automatically performs several transformations to ensure your data and schema work correctly in version `3.x`. The V3 Compatible Export automatically handles the following transformations: 1. Function name updates: All deprecated function names are automatically renamed to their new versions. - `duration::from::*` → `duration::from_*` - `string::is::*` → `string::is_*` - `type::is::*` → `type::is_*` - `time::is::*` → `time::is_*` - `time::from::*` → `time::from_*` - `rand::guid()` → `rand::id()` - `type::thing` → `type::record` >[!NOTE] > See the [complete function mapping table](#2-function-name-changes) below for the full list of function name updates. 2. `SEARCH ANALYZER` → `FULLTEXT ANALYZER`: Index definitions using `SEARCH ANALYZER` are automatically converted to `FULLTEXT ANALYZER`. 3. Parameter declarations: Automatically adds `LET` keyword where required for parameter declarations 4. `MTREE` → `HNSW` conversion: Vector search indexes using the deprecated `MTREE` type are automatically converted to use `HNSW`. 5. Future to COMPUTED field conversion: Where possible, `` fields are automatically converted to `COMPUTED` fields. ## What requires manual migration Some changes cannot be automatically converted and require manual intervention: - Futures stored in records (using `DEFAULT ` or `CREATE ... SET field = `) - Nested fields with `` values - Queries using both GROUP and SPLIT clauses - Code using removed operators (`~`, `!~`, `?~`, `*~`) - Stored closures in records - Record reference syntax - `ANALYZE` statement usage ## Using V3 compatible export After completing the migration diagnostics in SurrealDB Studio and resolving all flagged issues, you can export your database using the v3 compatible export feature. This will generate a `.surql` file that can be safely imported into SurrealDB `3.x`. Next, in the SurrealDB Studio overview page, click on **Deploy instance** select the available plan and configure your instance (you can opt to upload a file to this instance with the v3 compatible export file). At checkout, you will be prompted to enter your payment details (if you don't have a payment method on file) and indicate that the instance is used to migrate to SurrealDB `3.0`. This will let us know to give you migration credits on your account. SurrealDB Studio deploy instance ## Using the CLI If you prefer to migrate using the command line rather tha SurrealDB Studio, the SurrealDB `3.x` binary includes a `v2` subcommand that can connect to your `2.x` database and produce a v3-compatible export. This is necessary because the `3.x` binary cannot directly read `2.x` data, and the `2.x` binary does not support the v3-compatible export format. >[!NOTE] >The `v2` subcommand requires SurrealDB version `3.0.3` or later. >[!IMPORTANT] >Before exporting, ensure you have resolved any migration issues flagged by the [SurrealDB Studio migration diagnostics](#migration-diagnostics-in-surrealdb-studio). The v3-compatible export will handle automatic transformations (such as function renames and `SEARCH ANALYZER` → `FULLTEXT ANALYZER`), but issues that require manual intervention should be resolved first. ### Step 1: Export your v2 database Use the `surreal v2 export` command with the `--v3` flag to export your `2.x` database in a format compatible with `3.x`: ```bash surreal v2 export --v3 --namespace --database \ --token v2_exported_for_v3.surql ``` The `--v3` flag ensures the export applies the same automatic transformations described in the [V3 compatible export](#v3-compatible-export) section above. ### Step 2: Import into your v3 instance Once the export is complete, import the file into your `3.x` instance using the standard `surreal import` command: ```bash surreal import --namespace --database \ --endpoint --token v2_exported_for_v3.surql ``` For the full list of available options for each command, see the [export command](/docs/reference/cli/surrealdb-cli/commands/export.md) and [import command](/docs/reference/cli/surrealdb-cli/commands/import.md) documentation. ## Severity levels In this section, we will explore the different severity levels of the migration report and the actions you need to take to migrate your data. These severity levels are as follows: - **Will break**: Almost guaranteed to change query semantics when porting to `3.x`. - **Can break**: Some use cases will remain the same, but likely to cause issues. - **Unlikely break**: Only affects edge cases or rare usage patterns. ## Will break - critical changes ### 1. Futures replaced with COMPUTED Fields **Severity**: Will break **What changed**: The `` type has been completely removed and replaced with `COMPUTED` fields. **Migration actions**: 1. Use the migration tool to automatically convert futures where possible. 2. Manually replace `VALUE { expression }` with `COMPUTED expression`. 3. Restructure code for cases where automatic conversion isn't possible (nested fields, `DEFAULT` futures). **Before (2.x)**: ```surql DEFINE FIELD age ON person VALUE { time::year(time::now()) - time::year(born) }; CREATE foo SET field = { expression }; ``` **After (3.x)**: ```surql DEFINE FIELD age ON person COMPUTED time::year(time::now()) - time::year(born); -- Futures stored in records cannot be converted - requires redesign ``` >[!NOTE] >For futures stored in records (using `DEFAULT ` or `CREATE ... SET field = `), there is no direct replacement in `3.x`. Fixing these cases will require re-architecting your schema, as storing arbitrary queries in record data is no longer supported. **COMPUTED restrictions**: - Can only be used in `DEFINE FIELD` statements - No nested fields allowed inside or under `COMPUTED` fields - Cannot be used on ID fields - Cannot combine with: `VALUE`, `DEFAULT`, `READONLY`, `ASSERT`, `REFERENCE`, `FLEXIBLE` - Only works on top-level fields, not nested fields **Example - nested field workaround**: ```surql -- 2.x version DEFINE FIELD name.full ON person VALUE { name.first + ' ' + name.last }; -- 3.x version - must rename to avoid nesting DEFINE FIELD full_name ON person COMPUTED name.first + ' ' + name.last; ``` ### 2. Function name changes **Severity**: Will break **Action**: Update all function names according to the mapping table below. **Reason for Changes**: - `::is::` and `::from::` → `::is_` and `::from_` (matches method syntax) - `thing` → `record` (consistent terminology) - `rand::guid()` → `rand::id()` (default record ID format) - `string::distance::osa_distance` → `string::distance::osa` (remove redundancy) **Complete mapping table**:
New Function Name Previous name
duration::from_days duration::from::days
duration::from_hours duration::from::hours
duration::from_micros duration::from::micros
duration::from_millis duration::from::millis
duration::from_mins duration::from::mins
duration::from_nanos duration::from::nanos
duration::from_secs duration::from::secs
duration::from_weeks duration::from::weeks
geo::is_valid geo::is::valid
string::distance::osa string::distance::osa_distance
string::is_alphanum string::is::alphanum
string::is_alpha string::is::alpha
string::is_ascii string::is::ascii
string::is_datetime string::is::datetime
string::is_domain string::is::domain
string::is_email string::is::email
string::is_hexadecimal string::is::hexadecimal
string::is_ip string::is::ip
string::is_ipv4 string::is::ipv4
string::is_ipv6 string::is::ipv6
string::is_latitude string::is::latitude
string::is_longitude string::is::longitude
string::is_numeric string::is::numeric
string::is_record string::is::record
string::is_semver string::is::semver
string::is_url string::is::url
string::is_ulid string::is::ulid
string::is_uuid string::is::uuid
time::is_leap_year time::is::leap_year
time::from_nanos time::from::nanos
time::from_micros time::from::micros
time::from_millis time::from::millis
time::from_secs time::from::secs
time::from_ulid time::from::ulid
time::from_unix time::from::unix
time::from_uuid time::from::uuid
type::is_array type::is::array
type::is_bool type::is::bool
type::is_bytes type::is::bytes
type::is_collection type::is::collection
type::is_datetime type::is::datetime
type::is_decimal type::is::decimal
type::is_duration type::is::duration
type::is_float type::is::float
type::is_geometry type::is::geometry
type::is_int type::is::int
type::is_line type::is::line
type::is_none type::is::none
type::is_null type::is::null
type::is_multiline type::is::multiline
type::is_multipoint type::is::multipoint
type::is_multipolygon type::is::multipolygon
type::is_number type::is::number
type::is_object type::is::object
type::is_point type::is::point
type::is_polygon type::is::polygon
type::is_range type::is::range
type::is_record type::is::record
type::is_string type::is::string
type::is_uuid type::is::uuid
type::record type::thing
Learn more about the database functions in the [SurrealQL functions](/docs/reference/query-language/functions/database-functions.md) documentation. ### 3. `array::range` argument changes **Severity**: Will break **What changed**: Arguments changed from `(offset, count)` to `(start, end)` or accepting a range. **Action**: Change all `array::range` calls to use start/end bounds instead of offset/count. **Before (2.x)**: ```surql array::range(0, 5) // returns [0,1,2,3,4] array::range(-1, 5) // returns [-1,0,1,2,3] array::range(-5, 5) // returns [-5,-4,-3,-2,-1] ``` **After (3.x)**: ```surql array::range(0, 5) // returns [0,1,2,3,4] array::range(-1, 5) // returns [-1,0,1,2,3,4] ← different! array::range(-5, 5) // returns [-5,-4,-3,-2,-1,0,1,2,3,4] ← different! array::range(0..=1) // returns [0,1] ``` **Migration formula**: - Old: `array::range(offset, count)` - New: `array::range(offset, offset + count)` ### 4. `LET` required for parameters **Severity**: Will break **What changed**: Parameter declarations now require `LET` keyword. **Action**: Add `LET` before all parameter declarations. **Before (2.x)**: ```surql $val = 10; // This was allowed ``` **After (3.x)**: ```surql LET $val = 10; // LET is now required ``` **Error Message**: ```text Parse error: Parameter declarations without `let` are deprecated. Replace with `let $val = ...` to keep the previous behavior. ``` ### 5. `GROUP` and `SPLIT` cannot be used together **Severity**: Will break **What changed**: Using both `GROUP` and `SPLIT` in the same query is no longer allowed. **Action**: Remove `SPLIT` from any query which also had a `GROUP` clause, as its inclusion had no effect in 2.x. If the use of both a `SPLIT` and a `GROUP` is required, put one of the two clauses into a subquery. **Before (2.x)**: ```surql SELECT age, emails FROM user SPLIT emails GROUP BY age; -- SPLIT had no effect. ``` **After (3.x) - Option 1 (split then group)**: ```surql SELECT age, emails FROM (SELECT * FROM user SPLIT emails) GROUP BY age; ``` **After (3.x) - Option 2 (group then split)**: ```surql SELECT * FROM (SELECT age, emails FROM user GROUP BY age, emails) SPLIT emails; ``` ### 6. Like operators removed **Severity**: Will break **What changed**: The `~`, `!~`, `?~`, `*~` operators have been removed. **Action**: Replace with `string::distance` or `string::similarity` functions. **Reason**: Multiple similarity algorithms now available; users should choose their own cutoff point. **Before (2.x)**: ```surql "Mario" ~ "mario"; // returns true ``` **After (3.x)**: ```surql string::similarity::jaro("Mario", "mario") > 0.8; // returns true -- Create reusable function DEFINE FUNCTION fn::similar($one: string, $two: string) -> bool { string::similarity::jaro($one, $two) > 0.8 }; fn::similar("Mario", "mario"); // returns true ``` **Available Functions**: - `string::similarity::jaro()` - `string::distance::osa()` - And other similarity/distance functions ### 7. SEARCH ANALYZER → FULLTEXT ANALYZER **Severity**: Will break **Action**: Replace all instances of `SEARCH ANALYZER` with `FULLTEXT ANALYZER`. **Before (2.x)**: ```surql DEFINE INDEX userNameIndex ON TABLE user COLUMNS name SEARCH ANALYZER example_ascii BM25 HIGHLIGHTS; ``` **After (3.x)**: ```surql DEFINE INDEX userNameIndex ON TABLE user COLUMNS name FULLTEXT ANALYZER example_ascii BM25 HIGHLIGHTS; ``` ### 8. Database-level strictness **Severity**: Will break (if using `--strict` flag) **What changed**: Strictness moved from instance-level flag to database-level definition. **Action**: Add `STRICT` to `DEFINE DATABASE` statements for databases that need strictness. **Before (2.x)**: ```bash surreal start --strict ``` **After (3.x)**: ```surql DEFINE DATABASE mydb STRICT; ``` **Impact**: Allows different databases on the same instance to have different strictness levels. ### 9. MTREE removal **Severity**: Will break **What changed**: `MTREE` vector search index was deprecated in 2.x and has been removed. **Action**: Use `HNSW` instead of `MTREE` in index definitions. **Before (2.x)**: ```surql DEFINE INDEX vec_idx ON table FIELDS embedding MTREE DIMENSION 768; ``` **After (3.x)**: ```surql DEFINE INDEX vec_idx ON table FIELDS embedding HNSW DIMENSION 768; ``` ### 10. Stored closures **Severity**: Will break **What changed**: Closures can no longer be stored as part of a record. **Action**: Use of closures stored inside a record will have to be removed, there is currently no new feature which can replace the stored closures. **Before (2.x)**: ```surql CREATE record SET closure = |$a| $a + 1 ``` **After (3.x)**: ```surql -- This will now throw an error CREATE record SET closure = |$a| $a + 1 ``` ### 11. Usage of record references **Severity**: Will break **What changed**: Record references were an experimental feature in 2.x and in 3.x the syntax of record references has been significantly altered. **Action**: Record references in 2.x will have to be updated manually to 3.x syntax. ### 12. Usage of `ANALYZE` statement **Severity**: Will break **What changed**: The `ANALYZE` statement which could provide some statistics about full text indexes has been removed. **Action**: Use the `ANALYZE` stastement will have to be removed. ## Can break - likely issues ### 13. All `.*` idiom behaviour **Severity**: Can break **What changed**: The `.*` (`all` idiom) behaviour changed for arrays and objects. **Breaks when**: Used to dereference record IDs in arrays or get object values. **Before (2.x)**: ```surql [a:1, a:2].* // returns [a:1, a:2] [a:1, a:2].*.* // dereferences records { a: 1, b: "foo" }.* // returns [1, "foo"] ``` **After (3.x)**: ```surql [a:1, a:2].* // dereferences records directly { a: 1, b: "foo" }.* // returns { a: 1, b: "foo" } ``` **Migration**: - For arrays: Replace `.*.*` with `.*` - For objects: Replace `.*` with `object::values()` function ### 14. Field idiom followed by another idiom part **Severity**: Can break **What changed**: Field idioms on arrays now work on individual elements instead of the whole array. **Breaks when**: Field idiom on array of objects is followed by another idiom part. **Before (2.x)**: ```surql [{ a: ["a","b"]}, {a: [1,2]}].a[0] -- returns ["a","b"] -- Evaluated as: ([...].a)[0] ``` **After (3.x)**: ```surql [{ a: ["a","b"]}, {a: [1,2]}].a[0] -- returns ["a",1] -- Evaluated on each element: [(...).a[0], (...).a[0]] ``` **Migration**: Swap idiom parts if old behaviour needed. - Old: `.field[0]` - New: `[0].field` ### 15. Idiom fetching changes **Severity**: Can break **What changed**: Multiple improvements to idiom fetching behaviour. **Quick Reference Table**: | Example | 2.x Output | 3.x Output | |---------|------------|------------| | `[1, a:1].*` | `[1, a:1]` | `[1, { id: a:1 }]` | | `[1, a:1].*.*` | `[NONE, { id: a:1 }]` | `[NONE, { id: a:1 }]` | | `a:1.*` | `{ id: a:1 }` | `{ id: a:1 }` | | `{ key: 123 }.*` | `[123]` | `{ key: 123 }` | | `a:1<-edge[0]` | `{ id: edge:1 }` | `edge:1` | | `[{ n: 1 }, { n: 2 }].n[0]` | `1` | `[NONE, NONE]` | **Action**: Review queries using these idioms and rewrite if necessary. ### 16. Optional parts syntax change **Severity**: Can break **What changed**: Optional operator changed from `?` to `.?` **Action**: Replace `?` with `.?` after optional values. **Before (2.x)**: ```surql ["string", NONE].map(|$val| $val?.len()); ``` **After (3.x)**: ```surql ["string", NONE].map(|$val| $val.?.len()); ``` **Reason**: Distinguishes between `??` operator and optional chaining on `option>`. ### 17. Parsing changes **Severity**: Can break **Record ID parsing**: ```surql -- 2.x r"a:b[r"c:d"]" // unescaped " was allowed -- 3.x r"a:b[r\"c:d\"]" // must escape " ``` **Unicode parsing**: ```surql -- 2.x "\uD83D\uDF15" // surrogate pairs -- 3.x "\u{1F715}" // single escape sequence ``` **Identifier escaping**: Escaped identifiers now support escape sequences like `\n`, `\u{AB1234}`. ### 18. New `set` type behaviour **Severity**: Can break **What changed**: Set type now both deduplicates AND orders items, displays with `{}` instead of `[]`. **Before (2.x)**: ```surql [2,3,1,1]; // returns [2, 3, 1] ``` **After (3.x)**: ```surql [2,3,1,1]; // returns {1, 2, 3} ``` **Migration Options**: 1. Use new set type (recommended) 2. Maintain old behaviour: Define as `array` adding `VALUE $value.distinct()` to `DEFINE FIELD` definition ### 19. Schema strictness changes **Severity**: Can break **Non-existing tables**: ```surql -- 3.x returns errors instead of empty arrays SELECT * FROM doesnt_exist; //- Error: "The table 'doesnt_exist' does not exist" ``` **SCHEMAFULL Tables**: ```surql DEFINE TABLE user SCHEMAFULL; DEFINE FIELD name ON user TYPE string; -- 2.x: extra fields silently filtered -- 3.x: extra fields cause error CREATE user CONTENT { name: "Billy", other: "value" }; //- Error: "Found field 'other', but no such field exists" -- Use destructuring to select only defined fields CREATE user CONTENT { name: "Billy", other: "value" }.{ name }; ``` ### 20. Numeric record ID ordering **Severity**: Can break **What changed**: Numeric values in record now have different ordering and equality when used in keys. Previously, `a:[1]`, `a:[1f]` and `a:[1dec]` were all different record IDs and could have different records. Numeric values in record IDs are now ordered by their numeric value, meaning the `a:[1]`, `a:[1f]` and `a:[1dec]` are the same key. Furthermore, `a:[0f]` is now ordered before `a:[1]`. **Breaks when**: Code depends on different numeric types resulting in different record IDs. **Before (2.x)**: ```surql CREATE t:[1]; CREATE t:[1f]; SELECT * FROM t; // returns `[{ id: [1] }, { id: [1f] }]` ``` **After (3.x)**: ```surql CREATE t:[1]; CREATE t:[1f]; -- returns an error, record with key `t:[1]` alread exisits. SELECT * FROM t; // returns `[{ id: [1] }]` ``` ## Unlikely break - edge cases ### 21. `math::sqrt` returns NaN **Severity**: Unlikely break **What changed**: Returns `NaN` instead of `NONE` for negative numbers. **Action**: Change checks from `NONE` to `NaN`. ```surql -- 2.x math::sqrt(-1); // returns NONE -- 3.x math::sqrt(-1); // returns NaN ``` ### 22. `math::min` returns `Infinity` **Severity**: Unlikely break **What changed**: Returns `Infinity` instead of `NONE` for empty arrays. **Action**: Change checks from `NONE` to `Infinity`. ```surql -- 2.x math::min([]); // returns NONE -- 3.x math::min([]); // returns Infinity ``` ### 23. `math::max` returns `-Infinity` **Severity**: Unlikely break **What changed**: Returns `-Infinity` instead of `NONE` for empty arrays. **Action**: Change checks from `NONE` to `-Infinity`. ```surql -- 2.x math::max([]); // returns NONE -- 3.x math::max([]); // returns -Infinity ``` ### 24. `array::logical_and` behaviour **Severity**: Unlikely break **What changed**: Function is now consistent with `&&` operator. **Breaks when**: Relying on specific values rather than truthiness. **Before (2.x)**: ```surql array::logical_and(["a"],[true]); // returns ["a"] array::logical_and([""],[false]); // returns [""] array::logical_and([true],[]); // returns [NULL] ``` **After (3.x)**: ```surql array::logical_and(["a"],[true]); // returns [true] array::logical_and([""],[false]); // returns [""] array::logical_and([true],[]); // returns [NONE] ``` **Action**: Update if relying on specific return values; no change needed if only checking truthiness. ### 25. `array::logical_or` behaviour **Severity**: Unlikely break **What changed**: Function now consistent with `||` operator. **Breaks when**: Relying on specific values rather than truthiness. **Before (2.x)**: ```surql array::logical_or(["a"],[true]); // returns ["a"] array::logical_or([""],[false]); // returns [""] array::logical_or([],[false]); // returns [NULL] ``` **After (3.x)**: ```surql array::logical_or(["a"],[true]); // returns ["a"] array::logical_or([""],[false]); // returns [false] array::logical_or([false],[]); // returns [NONE] ``` **Action**: Update if relying on specific return values; no change needed if only checking truthiness. ### 26. Mock value type changes **Severity**: Unlikely break **What changed**: Mocks now return arrays instead of special mock type. **Breaks when**: Code depends on the specific mock type being returned. **Before (2.x)**: ```surql |a:1..2|; // returns |a:1..2| (mock type) type::is_array(|a:1..2|); // returns false ``` **After (3.x)**: ```surql |a:1..=2|; // returns [a:1, a:2] (array) type::is_array(|a:1..=2|); // returns true ``` >[!NOTE] >Mock ranges are no longer inclusive by default - use `..=` for inclusive ranges. ### 27. `Id` field special behaviour. **Severity**: Unlikely break **What changed**: Special behaviour regarding `.id` idioms is removed. Before 3.0, `.id` idioms followed by another idiom expression would return the record-id key. After 3.0, the `.id` behaves like any other `.field` idiom. **Breaks when**: Code depends the special behaviour of that `.id` idioms had. **Before (2.x)**: ```surql record:{ key_field: "value" }.id.key_field // returns "value" ``` **After (3.x)**: ```surql record:{ key_field: "value" }.id.key_field // returns whatever value is at .id.key_field in the record with key `record:{ key_field: "value" }` record:{ key_field: "value" }.id().key_field // returns "value" ``` ### 28. Expressions now allowed inside queries Many statements had parts changed to support general expressions in those places. This means that identifiers which overlap with statements are no longer supported in those places without escaping. For example, syntax like the following was previously allowed: ```surql DEFINE INDEX select ... ``` This must now be written with backticks, or renamed. ```surql DEFINE INDEX `select` ... ``` The statements which had this change from an identifier to allowing a general expressions are the following: - The `ident` after `DEFINE TABLE ident ...` - The `ident` after `DEFINE NAMESPACE ident ...` - The `ident` after `DEFINE DATABASE ident ...` - The `ident` after `DEFINE USER ident ...` - The `ident` after `DEFINE ACCESS ident ...` - Both `ident` and `table` after `DEFINE EVENT ident ON table ...` - Both `ident` and `table` after `DEFINE FIELD ident ON table ...` - The `ident` after `DEFINE ANALYZER ident ...` - The `ident` after `DEFINE BUCKET ident ...` - The `ident` after `DEFINE SEQUENCE ident ...` - The `ident` after `INFO FOR TABLE ident ...` - The `ident` after `INFO FOR USER ident ...` - Both `ident` and `table` after `INFO FOR INDEX ident ON table ...` - The `ident` after `REMOVE TABLE ident ...` - The `ident` after `REMOVE NAMESPACE ident ...` - The `ident` after `REMOVE DATABASE ident ...` - The `ident` after `REMOVE USER ident ...` - The `ident` after `REMOVE ACCESS ident ...` - Both `ident` and `table` after `REMOVE EVENT ident ON table ...` - Both `ident` and `table` after `REMOVE FIELD ident ON table ...` - Both `ident` and `table` after `REMOVE INDEX ident ON table ...` - The `ident` after `REMOVE ANALYZER ident ...` - The `ident` after `REMOVE BUCKET ident ...` - The `ident` after `REMOVE SEQUENCE ident ...` ### 29. `DEFINE FIELD` number of items for arrays and sets A `DEFINE FIELD` statement for arrays and sets in SurrealDB 2.x allowed a maximum number of items to be indicated. This number now refers to the *required* number of items. As such, a schema with an `ASSERT $value().len()` is equal to a certain number can now have the required number in the type definition itself. Additionally, definitions that indicate a maximum number of items must be changed to `ASSERT $value.len() <=` followed by the maximum number. ```surql -- Assert exact length of 640 bytes in SurrealDB 2.x DEFINE FIELD bytes ON data TYPE array ASSERT $value.all(|$val| $val IN 0..=255) AND $value.len() = 640; -- Assert the same in SurrealDB 3.x DEFINE FIELD bytes ON data TYPE array ASSERT $value.all(|$val| $val IN 0..=255); -- Assert a maximum array size in SurrealDB 3.x DEFINE FIELD latest ON observation TYPE array ASSERT $value.len() <= 1000; ``` --- Source: https://surrealdb.com/docs/build/migrating/from-old-surrealdb-versions/overview # Upgrading This guide will help you upgrade your current SurrealDB installation to a newer release. When moving to a new version of SurrealDB, it is important to follow the upgrade instructions to ensure that your data is migrated correctly. This page contains information on how to upgrade your SurrealDB installation. ## Upgrading in general Upgrading the SurrealDB server itself is done through a single command: [`surreal upgrade`](/docs/reference/cli/surrealdb-cli/commands/upgrade.md). For minor versions, a read through the [release notes](/releases) will suffice to see which breaking changes may exist. To see the exact PRs that were merged between two versions, you can compare the changes between one version tag and another on GitHub. For example, [comparing `v3.0.0` with `v3.0.2`](https://github.com/surrealdb/surrealdb/compare/v3.0.0...v3.0.2) shows which PRs were merged between those two versions. As the tags themselves are contained in the url, you can simply replace the numbers to see the details between one version and another. ```bash # Changes between 3.0.0 and 3.0.2 https://github.com/surrealdb/surrealdb/compare/v3.0.0...v3.0.2 # Changes between 2.4.0 and 2.6.2 https://github.com/surrealdb/surrealdb/compare/v2.4.0...v2.6.2 ``` Upgrading between major versions (currently 1.x to 2.x and 2.x to 3.x) contains more structural changes and breaking changes, so be sure to see the dedicated pages when upgrading to 2.x or 3.x. --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/from-mongodb # Migrating from MongoDB How to map existing data and concepts from MongoDB to SurrealDB This page details some MongoDB data types and their SurrealQL equivalents, followed by links to the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool which allows data from MongoDB to be automatically imported to SurrealDB. ## Data types The following chart shows MongoDB data types along with the equivalent or near-equivalent [SurrealQL data type](/docs/reference/query-language/language-primitives/data-types.md) for each. | MongoDB Data Type | BSON Type | JSON Extended Format v2 | SurrealDB Mapping | | ------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------- | | **Double** | Double | `3.14` (Relaxed) or `{"$numberDouble": "3.14"}` (Canonical) | `float` | | **String** | String | `"text"` | `string` | | **Object** | Document | `{"key": "value"}` | `object` | | **Array** | Array | `[1, 2, 3]` | `array` | | **Binary data** | Binary | `{"$binary": {"base64": "...", "subType": "..."}}` | `bytes` | | **Undefined** | Undefined | `{"$undefined": true}` | `none` | | **ObjectId** | ObjectId | `{"$oid": "507f1f77bcf86cd799439011"}` | `string` | | **Boolean** | Boolean | `true`/`false` | `bool` | | **Date** | DateTime | `{"$date": "2024-01-01T00:00:00Z"}` (Relaxed) or `{"$date": {"$numberLong": "1672531200000"}}` (Canonical) | `datetime` | | **Null** | Null | `null` | `null` | | **Regular Expression** | RegularExpression | `{"$regularExpression": {"pattern": "...", "options": "..."}}` | `regex` | | **DBPointer** | DbPointer | `{"$dbPointer": {"$ref": "...", "$id": {...}}}` | `string` | | **JavaScript** | JavaScriptCode | `{"$code": "function(){}"}` | `string` | | **Symbol** | Symbol | `{"$symbol": "text"}` | `string` | | **JavaScript with scope** | JavaScriptCodeWithScope | `{"$code": "...", "$scope": {...}}` | `object` | | **32-bit integer** | Int32 | `42` (Relaxed) or `{"$numberInt": "42"}` (Canonical) | `int` | | **Timestamp** | Timestamp | `{"$timestamp": {"t": 1672531200, "i": 1}}` | `datetime` | | **64-bit integer** | Int64 | `{"$numberLong": "123"}` | `int` | | **Decimal128** | Decimal128 | `{"$numberDecimal": "123.45"}` | `number` | | **DBRef** | Document | `{"$ref": "users", "$id": "123"}` | `thing` | | **Min key** | MinKey | `{"$minKey": 1}` | `object` | | **Max key** | MaxKey | `{"$maxKey": 1}` | `object` | ## Importing from MongoDB using Surreal Sync Surreal Sync can be used to export MongoDB collections to SurrealDB. It supports inconsistent full syncs and consistent incremental syncs, and together provides ability to reproduce consistent snapshots from the source MongoDB collections onto the target SurrealDB tables. For more on how to import data from MongoDB to SurrealDB, please see the following pages in the Surreal Sync repo. * [Surreal Sync for MongoDB](https://github.com/surrealdb/surreal-sync/blob/main/docs/mongodb.md) * [MongoDB Data Types Support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/mongodb-data-types.md) --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/from-mysql # Migrating from MySQL How to map existing data and concepts from PostgreSQL to SurrealDB This page details some MySQL data types and their SurrealQL equivalents, followed by links to the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool which allows data from MySQL to be automatically imported to SurrealDB. ## Data types The following chart shows MySQL data types along with the equivalent or near-equivalent [SurrealQL data type](/docs/reference/query-language/language-primitives/data-types.md) for each. | MySQL Data Type | Wire Protocol Type | SQL Representation | SurrealDB Mapping | Notes | | ------------------- | ------------------ | ----------------------------------------------- | ----------------- | ---------------------------------------------- | | **BOOLEAN/BOOL** | Tiny | `0`/`1` | `bool` | 0=false, 1=true conversion | | **TINYINT** | Tiny | `-128` to `127` | `int` | Converted to 64-bit integer | | **SMALLINT** | Short | `-32768` to `32767` | `int` | Converted to 64-bit integer | | **MEDIUMINT** | Int24 | `-8388608` to `8388607` | `int` | Converted to 64-bit integer | | **INT/INTEGER** | Long | `-2147483648` to `2147483647` | `int` | Converted to 64-bit integer | | **BIGINT** | LongLong | `-9223372036854775808` to `9223372036854775807` | `int` | Direct conversion | | **FLOAT** | Float | `3.14` | `float` (f64) | Converted to double precision | | **DOUBLE** | Double | `3.141592653589793` | `float` (f64) | Direct conversion | | **DECIMAL/NUMERIC** | NewDecimal | `123.45` | `number` | Converted to SurrealDB Number with precision | | **CHAR(n)** | String | `'text'` | `string` | Fixed-length, padding removed | | **VARCHAR(n)** | VarString | `'text'` | `string` | Variable-length string | | **TEXT** | Blob | `'long text'` | `string` | Text blob as string | | **TINYTEXT** | TinyBlob | `'short text'` | `string` | Small text as string | | **MEDIUMTEXT** | MediumBlob | `'medium text'` | `string` | Medium text as string | | **LONGTEXT** | LongBlob | `'very long text'` | `string` | Large text as string | | **BINARY(n)** | String | `0x48656c6c6f` | `bytes` | Fixed-length binary data | | **VARBINARY(n)** | VarString | `0x48656c6c6f` | `bytes` | Variable-length binary data | | **BLOB** | Blob | `0x48656c6c6f` | `bytes` | Binary large object | | **TINYBLOB** | TinyBlob | `0x48656c6c6f` | `bytes` | Small binary data | | **MEDIUMBLOB** | MediumBlob | `0x48656c6c6f` | `bytes` | Medium binary data | | **LONGBLOB** | LongBlob | `0x48656c6c6f` | `bytes` | Large binary data | | **DATE** | Date | `'2024-01-15'` | `datetime` | Converted to datetime at midnight UTC | | **TIME** | Time | `'14:30:00'` | `string` | Time format as string (HH:MM:SS.microseconds) | | **DATETIME** | DateTime | `'2024-01-15 14:30:00'` | `datetime` | Converted to UTC datetime | | **TIMESTAMP** | Timestamp | `'2024-01-15 14:30:00'` | `datetime` | Timezone-aware, converted to UTC | | **YEAR** | Year | `2024` | `int` | Year as integer | | **JSON** | Json | `'{"key": "value"}'` | `object` | Parsed and converted recursively | | **GEOMETRY** | Geometry | `ST_GeomFromText('POINT(1 2)')` | `object` | Converted to geometric object with coordinates | | **POINT** | Geometry | `POINT(1.5, 2.5)` | `object` | Converted to `{"x": 1.5, "y": 2.5}` object | | **ENUM** | Enum | `'option1'` | `string` | Enum value as string, constraints lost | | **SET** | Set | `'value1,value2'` | `array` | Converted to array of strings | | **BIT(n)** | Bit | `b'1010'` | `string` | Bit string as binary string representation | ## Importing from MySQL using Surreal Sync Surreal Sync can be used to exports MySQL tables to SurrealDB. It supports inconsistent full syncs and consistent incremental syncs, and together provides ability to reproduce consistent snapshots from the source MySQL tables onto the target SurrealDB tables. For more on how to import data from MySQL to SurrealDB, please see the following pages in the Surreal Sync repo. * [Surreal Sync for MySQL](https://github.com/surrealdb/surreal-sync/blob/main/docs/mysql.md) * [MySQL Data Types Support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/mysql-data-types.md) --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/from-neo4j # Migrating from Neo4j How to map existing data and concepts from Neo4j to SurrealDB This page details some Neo4j data types and patterns in the Cypher query language along with their SurrealQL equivalents or near equivalents, followed by links to the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool which allows data from Neo4j to be automatically imported to SurrealDB. ## Concept mapping
Neo4j SurrealDB
database database
node label table
node record
node property field
index index
id record id
transactions transactions
relationships record links, embedding and graph relations
## Data types The following chart shows Neo4j data types along with the equivalent or near-equivalent SurrealQL data type for each. | Neo4j Data Type | SurrealDB Mapping | Notes | |--------------------|-------------------|-------------------------------------------------------------------------------------------| | **Boolean** | `bool` | | | **Integer** | `int` | | | **Float** | `float` (f64) | | | **String** | `string` | | | **List** | `array` | | | **Map** | `object` | | | **Null** | `null` | | | **Date** | `datetime` | Convert to UTC datetime (assuming local timezone) | | **DateTime** | `datetime` | Convert to UTC datetime | | **LocalDateTime** | `datetime` | Convert to UTC datetime (assuming UTC) | | **Duration** | `duration` | | | **Bytes** | `bytes` | | | **Time** | `object` | Convert to object with `type: "$Neo4jTime"`, hour, minute, second, nanosecond, offset_seconds fields | | **LocalTime** | `object` | Convert to object with `type: "$Neo4jLocalTime"`, hour, minute, second, nanosecond fields | | **Point2D** | `object` | Convert to GeoJSON-like object with `type: "Point"`, `srid` (4326), `coordinates: [longitude, latitude]` | | **Point3D** | `object` | Convert to GeoJSON-like object with `type: "Point"`, `srid` (4979), `coordinates: [longitude, latitude, elevation]` | | **DateTimeZoneId** | `datetime` | Convert to UTC datetime using embedded timezone ID | ## Syntax mapping The following shows some CRUD examples using SurrealQL syntax. ### Create As Neo4j is schemaless, only the SurrealQL schemaless approach is shown below. For a schemafull option see the [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) page. For more SurrealQL examples, see the [`CREATE`](/docs/reference/query-language/statements/create.md), [`INSERT`](/docs/reference/query-language/statements/insert.md) and [`RELATE`](/docs/reference/query-language/statements/relate.md) pages. Simple create/insert operations: ```cypher // Cypher CREATE (John:Person {name:‘John’}), (Jane:Person {name: ‘Jane’}) ``` ```surql -- SurrealQL -- Table implicitly created if it doesn't exist INSERT INTO person [ {id: “John”, name: “John”}, {id: “Jane”, name: “Jane”} ] ``` Graph relations via the [RELATE](/docs/reference/query-language/statements/relate.md) statement: ```cypher // Cypher MATCH (p:Person {name:‘Jane’}), (pr:Product {name:‘iPhone’}) CREATE (p)-[:ORDER]->(pr) ``` ```surql -- SurrealQL RELATE person:Jane->order->product:iPhone ``` Defining an index: ```cypher // Cypher CREATE INDEX personNameIndex FOR (p:Person) ON (p.name) ``` ```surql -- SurrealQL DEFINE INDEX idx_name ON TABLE person COLUMNS name ``` ### Read For more SurrealQL examples, see the [`SELECT`](/docs/reference/query-language/statements/select.md), [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md) and [`RETURN`](/docs/reference/query-language/statements/return.md) pages. Returning all the fields of a table: ```cypher // Cypher MATCH (p:Person) RETURN p ``` ```surql -- SurrealQL SELECT * FROM person ``` Returning a single field of a table: ```cypher // Cypher MATCH (p:Person) RETURN p.name ``` ```surql -- SurrealQL SELECT name FROM person ``` Using the `WHERE` clause to return certain records: ```cypher // Cypher MATCH (p:Person) WHERE p.name = “Jane” RETURN p.name ``` ```surql -- SurrealQL SELECT name FROM person WHERE name = “Jane” ``` Using `EXPLAIN` to detail the query plan used: ```cypher // Cypher EXPLAIN MATCH (p:Person) WHERE p.name = "Jane" RETURN p.name ``` ```surql -- SurrealQL SELECT name FROM person WHERE name = "Jane" EXPLAIN ``` Grouping and counting the number of records returned: ```cypher // Cypher MATCH (p:Person) RETURN count(*) as person_count ``` ```surql -- SurrealQL SELECT count() AS person_count FROM person GROUP ALL ``` See all distinct values for a field among the records of a table: ```cypher // Cypher MATCH (p:Person) RETURN distinct p.name ``` ```surql -- SurrealQL SELECT array::distinct(name) FROM person GROUP ALL ``` Returning up to a certain number of records: ```cypher // Cypher MATCH (p:Person) RETURN p LIMIT 10 ``` ```surql -- SurrealQL SELECT * FROM person LIMIT 10 ``` See which `person` records have ordered a `product` via the `order` graph edge: ```cypher // Cypher MATCH (p:Person)-[:ORDER]->(pr:Product) RETURN p.name, pr.name ``` ```surql -- SurrealQL SELECT name, ->order->product.name FROM person ``` ### Update For more SurrealQL examples, see the [`UPDATE`](/docs/reference/query-language/statements/update.md) page. Conditionally updating records that have a certain value for a field: ```cypher // Cypher MATCH (p:Person) WHERE p.name = "Jane" SET p.last_name = 'Doe' RETURN p ``` ```surql -- SurrealQL UPDATE person SET last_name = "Doe" WHERE name = "Jane" ``` Unsetting (removing) the value of a field for certain records: ```cypher // Cypher MATCH (p:Person) WHERE p.name = "Jane" REMOVE p.last_name RETURN p ``` ```surql -- SurrealQL UPDATE person UNSET last_name WHERE name = "Jane" ``` ### Delete For more SurrealQL examples, see the [`DELETE`](/docs/reference/query-language/statements/delete.md) and [`REMOVE`](/docs/reference/query-language/statements/remove.md) pages. Conditionally deleting records based on the value of a field: ```cypher // Cypher MATCH (p:Person) WHERE p.name = "Jane" DELETE p ``` ```surql -- SurrealQL DELETE person WHERE name = "Jane" ``` Deleting all records for a table (SurrealQL: table still exists): ```cypher // Cypher MATCH (p:Person) DELETE p ``` ```surql -- SurrealQL DELETE person ``` Deleting all records and table definition for a table: ```cypher // Cypher MATCH (p:Person) DELETE p ``` ```surql -- SurrealQL REMOVE TABLE person ``` ## Importing from Neo4j using Surreal Sync Surreal Sync can be used to export Neo4j nodes and relationships to SurrealDB. It supports inconsistent full syncs and consistent incremental syncs, and together provides ability to reproduce consistent snapshots from the source Neo4j graph onto the target SurrealDB tables. For more on how to import data from Neo4j to SurrealDB, please see the following pages in the Surreal Sync repo. * [Surreal Sync for Neo4j](https://github.com/surrealdb/surreal-sync/blob/main/docs/neo4j.md) * [Neo4j Data Types Support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/neo4j-data-types.md) --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/from-postgresql # Migrating from PostgreSQL How to map existing data and concepts from PostgreSQL to SurrealDB This page details some common PostgreSQL patterns and their SurrealQL equivalents, followed by links to the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool which allows data from PostgreSQL to be automatically imported to SurrealDB. ## Data types The following chart shows PostgreSQL data types along with the equivalent or near-equivalent [SurrealQL data type](/docs/reference/query-language/language-primitives/data-types.md) for each. | PostgreSQL Data Type | Wire Protocol Type | SQL Representation | SurrealDB Mapping | Notes | | -------------------- | ------------------ | ---------------------------------------- | ----------------- | ------------------------------------------------------------ | | **BOOLEAN** | Boolean | `true`/`false` | `bool` | | | **SMALLINT** | Int2 | `32767` | `int` | | | **INTEGER** | Int4 | `2147483647` | `int` | | | **BIGINT** | Int8 | `9223372036854775807` | `int` | | | **SERIAL** | Int4 | `1, 2, 3...` | `int` | Auto-increment converted to regular integer | | **BIGSERIAL** | Int8 | `1, 2, 3...` | `int` | Auto-increment converted to regular integer | | **REAL** | Float4 | `3.14` | `float` (f64) | Converted to double precision | | **DOUBLE PRECISION** | Float8 | `3.141592653589793` | `float` (f64) | | | **NUMERIC/DECIMAL** | Numeric | `123.45` | `number` | Converted to SurrealDB Number with exact precision preserved | | **MONEY** | Money | `$123.45` | `number` | Currency symbol removed, converted to number | | **CHAR(n)** | Bpchar | `'text'` | `string` | Fixed-length, padding removed | | **VARCHAR(n)** | Varchar | `'text'` | `string` | Variable-length string | | **TEXT** | Text | `'long text'` | `string` | Unlimited length string | | **BYTEA** | Bytea | `\\x48656c6c6f` | `bytes` | Binary data, hex decoded | | **DATE** | Date | `'2024-01-15'` | `datetime` | Converted to datetime at midnight UTC | | **TIME** | Time | `'14:30:00'` | `string` | Time-only as string (SurrealDB has no pure time type) | | **TIMESTAMP** | Timestamp | `'2024-01-15 14:30:00'` | `datetime` | Converted to UTC datetime | | **TIMESTAMPTZ** | Timestamptz | `'2024-01-15 14:30:00+00'` | `datetime` | Timezone-aware, converted to UTC | | **INTERVAL** | Interval | `'1 day 2 hours'` | `duration` | Converted to SurrealDB duration | | **UUID** | Uuid | `'550e8400-e29b-41d4-a716-446655440000'` | `string` | UUID string representation | | **JSON** | Json | `'{"key": "value"}'` | `string` | JSON stored as string representation | | **JSONB** | Jsonb | `'{"key": "value"}'` | `string` | Binary JSON stored as string representation | | **ARRAY** | Array | `'{1,2,3}'` | `array` | Recursively processed, element types converted | | **POINT** | Point | `'(1.5, 2.5)'` | `object` | Convert to `{"x": 1.5, "y": 2.5}` object | | **LINE** | Line | `'{1,2,3}'` | `object` | Convert to coefficient object | | **LSEG** | Lseg | `'[(1,2),(3,4)]'` | `object` | Line segment as start/end point object | | **BOX** | Box | `'(1,2),(3,4)'` | `object` | Bounding box as corner points object | | **PATH** | Path | `'[(1,2),(3,4)]'` | `array` | Array of point objects | | **POLYGON** | Polygon | `'((1,2),(3,4),(5,6))'` | `array` | Array of point objects | | **CIRCLE** | Circle | `'<(1,2),3>'` | `object` | Center point and radius object | | **INET** | Inet | `'192.168.1.1'` | `string` | IP address as string | | **CIDR** | Cidr | `'192.168.0.0/24'` | `string` | Network address as string | | **MACADDR** | Macaddr | `'08:00:2b:01:02:03'` | `string` | MAC address as string | ## Inserting data ```sql // PostgreSQL INSERT INTO product (name, description, price, category, images, options) VALUES ("Shirt", "Slim fit", 6, "clothing", ARRAY['image1.jpg', 'image2.jpg', 'image3.jpg']) ; ``` ```surql -- SurrealQL CREATE product CONTENT { name: 'Shirt', id: 'shirt', description: 'Slim fit', price: 6, category: 'clothing', images: ['image1.jpg', 'image2.jpg', 'image3.jpg'] }; ``` ## Defining a schemafull table A table in PostgreSQL might be defined as follows. ```surql CREATE TABLE product ( id SERIAL PRIMARY KEY, name TEXT, description TEXT, price NUMERIC(8,2), category TEXT, images TEXT[] ); ``` In SurrealQL, a table does not by default need to be defined before it can be used. However, the following statements will produce a strict schema similar to the PostgreSQL one above. ```surql DEFINE TABLE product SCHEMAFULL; DEFINE FIELD name ON TABLE product TYPE string; DEFINE FIELD description ON TABLE product TYPE string; DEFINE FIELD price ON TABLE product TYPE number -- Only show two digits after decimal point VALUE math::fixed($value, 2) -- Price must be within this range ASSERT $value IN 0..=99999999; DEFINE FIELD category ON TABLE product TYPE string; DEFINE FIELD images ON TABLE product TYPE array; ``` One difference between this and the PostgreSQL schema above is that a `product` will have a randomly generated ID as opposed to an incrementing one. ```surql CREATE product SET name = 'Shirt', description = 'Nice shirt', price = 20.449, category = 'Clothing', images = ["some_img.ping", "another_img.png"]; -- Output [ { category: 'Clothing', description: 'Nice shirt', id: product:1j29aq5q0do48k6xvyem, images: [ 'some_img.ping', 'another_img.png' ], name: 'Shirt', price: 20.45f } ] ``` ### Selecting data Selecting records using an ID: ```sql // PostgreSQL SELECT * FROM product WHERE id=1; ``` ```surql -- SurrealQL SELECT * FROM product:shirt; ``` Selecting multiple specific records: ```sql // PostgreSQL SELECT * FROM product WHERE id IN (1, 2, 3); ``` ```surql -- SurrealQL SELECT * FROM [product:1, product:2, product:3]; ``` Counting the number of records in a table: ```sql // PostgreSQL SELECT COUNT(*) FROM product; ``` ```surql -- SurrealQL SELECT count() FROM product GROUP ALL; ``` ### Queries with identical syntax As the SurrealQL is inspired by SQL, many queries between it and PostgreSQL are identical. ```surql SELECT * FROM product LIMIT 5; SELECT name, price FROM product; SELECT * FROM product ORDER BY price DESC; SELECT * FROM order_item WHERE quantity = 2; ``` ### Using record ID instead of the `WHERE` clause If a record ID is known ahead of time and you are using a version of SurrealDB before 3.0, be be sure to query by the record ID itself instead of using a `WHERE` clause in SurrealQL. This will avoid a full table scan if the field is not indexed. ```sql // PostgreSQL SELECT * FROM product WHERE id = 1; // Immediate access in SurrealDB 3.0+, table scan in previous versions SELECT * FROM product WHERE id = product:1; // Accessing the record directly will // take a fraction of the time in 2.x product:1.*; ``` ### Joining and querying related tables Take the following query with joins in PostgreSQL: ```surql SELECT p.id AS product_id, p.name AS product_name FROM product p JOIN order_item oi ON p.id = oi.product_id JOIN customer_order co ON oi.order_id = co.order_id JOIN customer c ON co.customer_id = c.customer_id WHERE c.name = 'Pratim' ORDER BY p.id; ``` In SurrealQL, tables can be joined to each other via edges, such as the `bought` edge in this example. ```surql -- Relate a 'customer' to a 'product' via 'bought' RELATE customer:tobie->bought->product:iphone CONTENT { option: { Size: 'M', Color: 'Max' }, quantity: 1, total: 600, status: 'Pending', created_at: time::now() }; ``` Once the tables have been related (joined), they can be queried with this syntax. ```surql SELECT * FROM customer:tobie->bought; ``` An example of more complex query with joins to return all people who bought the same products as a certain customer (including the original customer). ```sql // PostgreSQL SELECT DISTINCT c.* FROM customer c JOIN customer_order co ON c.customer_id = co.customer_id JOIN order_item oi ON co.order_id = oi.order_id JOIN product p ON oi.product_id = p.id WHERE p.id IN ( -- Subquery: Get all product IDs bought by Tobie SELECT p2.id FROM product p2 JOIN order_item oi2 ON p2.id = oi2.product_id JOIN customer_order co2 ON oi2.order_id = co2.order_id JOIN customer c2 ON co2.customer_id = c2.customer_id WHERE c2.name = 'Tobie' ) ``` ```surql -- SurrealQL customer:tobie->bought->product<-bought<-customer.*; ``` ## Importing from PostgreSQL using Surreal Sync For more on how to import data from PostgreSQL to SurrealDB, please see the following pages in the Surreal Sync repo. * [Surreal Sync for PostgreSQL (Trigger-Based)](https://github.com/surrealdb/surreal-sync/blob/main/docs/postgresql.md) * [PostgreSQL wal2json Source](https://github.com/surrealdb/surreal-sync/blob/main/docs/postgresql-wal2json-source.md) * [PostgreSQL Data Types Support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/postgresql-data-types.md) --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/from-snowflake # Migrating from Snowflake How to map existing data and concepts from Snowflake to SurrealDB This page shows how Snowflake data types map to their SurrealQL equivalents, then explains how to load Snowflake tables into SurrealDB with the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool. ## Data types The following chart shows Snowflake data types along with the equivalent or near-equivalent [SurrealQL data type](/docs/reference/query-language/language-primitives/data-types.md) for each. Surreal Sync reads values through the Snowflake SQL REST API, which returns every cell as a string together with the column's logical type, and converts each one using that type. | Snowflake Data Type | SurrealDB Mapping | Notes | | ------------------- | ----------------- | ----- | | **BOOLEAN** | `bool` | | | **NUMBER / INT / INTEGER / BIGINT** | `int` | Whole numbers (scale 0) that fit a 64-bit integer | | **NUMBER(p,s) / DECIMAL / NUMERIC** | `number` | Values with a scale, or larger than a 64-bit integer, keep their precision and scale as a decimal | | **FLOAT / REAL / DOUBLE** | `float` (f64) | Double-precision floating point | | **VARCHAR / STRING / TEXT / CHAR** | `string` | | | **DATE** | `datetime` | Stored as days since the Unix epoch, converted to a datetime at midnight UTC | | **TIME** | `datetime` | Seconds since midnight | | **TIMESTAMP_NTZ / DATETIME** | `datetime` | No timezone; taken as the given wall-clock instant | | **TIMESTAMP_LTZ / TIMESTAMP_TZ / TIMESTAMP** | `datetime` | Timezone-aware; the absolute instant is preserved as UTC | | **VARIANT / OBJECT** | `object` | Parsed from the stored JSON document | | **ARRAY** | `array` | Parsed from the stored JSON array | | **BINARY / VARBINARY** | `bytes` | Hex-decoded to raw bytes | Any column whose logical type is not listed above is preserved as a `string` rather than failing the import. ## Importing from Snowflake using Surreal Sync Surreal Sync can read tables from a Snowflake database and write them to SurrealDB. > [!NOTE] > The Snowflake source is ingestion-only. It performs a single full snapshot of the selected tables and does not support incremental synchronisation or change data capture. There is no durable cursor, so an interrupted run cannot be resumed part-way; re-run the command to start again. Rows are read one Snowflake result partition at a time and written in batches, so a large table does not need to fit in memory. ### Prerequisites Surreal Sync connects to Snowflake through the [SQL REST API](https://docs.snowflake.com/en/developer-guide/sql-api/index) using key-pair (JWT) authentication. Before running an import: 1. Generate an unencrypted PKCS#8 RSA key pair. Encrypted private keys are not supported yet. 2. Register the public key on the Snowflake user that Surreal Sync will authenticate as: ```sql ALTER USER my_user SET RSA_PUBLIC_KEY='MIIBIjANBgkq...'; ``` 3. Make sure that user has a role able to read the target database and schema, and a virtual warehouse to run the queries. Keep the private key file accessible to Surreal Sync and pass its path with `--private-key-path`. ### Usage ```bash surreal-sync from snowflake \ --account myorg-myaccount \ --user my_user \ --private-key-path ./rsa_key.p8 \ --warehouse COMPUTE_WH \ --database MY_DB \ --schema PUBLIC \ --to-namespace my_namespace \ --to-database my_database \ --surreal-endpoint ws://localhost:8000 \ --surreal-username root \ --surreal-password secret ``` When `--tables` is omitted, every base table in the schema is imported (views and temporary tables are skipped). Each table becomes a table of the same name in SurrealDB, and each row becomes a record. Snowflake upper-cases unquoted identifiers, so table and column names arrive upper-cased. Most connection options also read from an environment variable, which is useful for keeping credentials out of shell history. ### Options | Option | Environment variable | Required | Description | | ------ | -------------------- | -------- | ----------- | | `--account` | `SNOWFLAKE_ACCOUNT` | Yes | Account identifier as used in the host `.snowflakecomputing.com`, for example `myorg-myaccount` or `xy12345.us-east-1`. | | `--user` | `SNOWFLAKE_USER` | Yes | User whose key pair is registered for JWT authentication. | | `--private-key-path` | `SNOWFLAKE_PRIVATE_KEY_PATH` | Yes | Path to the unencrypted PKCS#8 private key PEM file. | | `--private-key-passphrase` | `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE` | No | Passphrase for an encrypted key. Not supported yet; setting it returns an error. | | `--warehouse` | `SNOWFLAKE_WAREHOUSE` | Yes | Virtual warehouse used to run the queries. | | `--database` | `SNOWFLAKE_DATABASE` | Yes | Database to read from. | | `--schema` | `SNOWFLAKE_SCHEMA` | No | Schema within the database. Defaults to `PUBLIC`. | | `--role` | `SNOWFLAKE_ROLE` | No | Role to assume for the session. | | `--tables` | - | No | Comma-separated list of tables to import. When omitted, all base tables in the schema are imported. | | `--id-columns` | - | No | Comma-separated columns forming the SurrealDB record ID. See [Record IDs](#record-ids). | | `--transforms-config` | - | No | Path to a TOML file describing a transform pipeline. Omit to import rows unchanged. | | `--to-namespace` | - | Yes | Target SurrealDB namespace. | | `--to-database` | - | Yes | Target SurrealDB database. | The `--surreal-endpoint`, `--surreal-username`, `--surreal-password`, `--batch-size`, and `--dry-run` options are shared with the other Surreal Sync sources. Use `--dry-run` to read and convert rows without writing to SurrealDB. ### Record IDs Snowflake primary keys are optional and often absent, so the record ID for each row depends on `--id-columns`: - **Omitted:** a sequential per-table integer is generated. It is deterministic within a run but not stable across re-runs, so prefer explicit ID columns when you need stable IDs. - **A single column:** the value becomes the record ID, as an integer when it parses as one and otherwise as a string. - **Several columns:** the values are joined with `:` into a string ID. When ID columns are given, they are used only for the record ID and are not repeated as fields on the record. Column names are matched case-insensitively. --- Source: https://surrealdb.com/docs/build/migrating/from-other-databases/overview # Migrating from other databases Map data and concepts from other databases to SurrealDB using Surreal Sync, the CLI import or SDKs. This section details how to map data, queries and concepts you may know from other databases and datatypes into SurrealDB. All of the sources in this section of the documentation can be automatically imported to SurrealDB using the [Surreal Sync](https://github.com/surrealdb/surreal-sync/) tool. To migrate from other databases not yet supported, consider beginning by exporting the database as JSON which can be [imported on the command line](/docs/reference/cli/surrealdb-cli/commands/import.md). Alternatively, you can use [one of the many available SDKs](/docs/languages.md) to access your existing database and transfer its content directly to a SurrealDB instance. * [MongoDB](/docs/build/migrating/from-other-databases/from-mongodb.md) * [MySQL](/docs/build/migrating/from-other-databases/from-mysql.md) * [Neo4j](/docs/build/migrating/from-other-databases/from-neo4j.md) * [PostgreSQL](/docs/build/migrating/from-other-databases/from-postgresql.md) * [Snowflake](/docs/build/migrating/from-other-databases/from-snowflake.md) --- Source: https://surrealdb.com/docs/explore/ml-models # Introduction SurrealML runs trained ML models in SurrealDB: train in Python. Then load and infer with sklearn or PyTorch. SurrealML is an engine that seeks to do one thing, and one thing well: store and execute trained ML models. SurrealML does not intrude on the training frameworks that are already out there, instead works with them to ease the storage, loading, and execution of models. Someone using SurrealML will be able to train their model in a chosen framework in Python, save their model, and load and execute the model in either Python or Rust. The inference engine is composed of python bindings interacting with a core written in Rust. This means that the exact same code that the Python client calls will be running on a production node by itself, or in the database. While the SurrealML engine runs in the database, it is developed in a completely isolated GitHub repository, giving the user 100% freedom on how they deploy and interact with the SurrealML engine. While this is all exciting, let us check to see if SurrealML works on your machine. There is nothing worse than reading about a package only to find that it does not work. The first move is to install the package. ## Installation To install SurrealML, make sure you have Python installed. Then, install the `SurrealML` library and either `PyTorch` or `sklearn`, based on your model choice. You can install the package with both `PyTorch` and `SKLearn` with the command below: ```bash pip install "git+https://github.com/surrealdb/surrealml#egg=surrealml[sklearn,torch]" ``` If you want to use `SurrealML` with `sklearn` you will need the following installation: ```bash pip install "git+https://github.com/surrealdb/surrealml#egg=surrealml[sklearn]" ``` For `PyTorch`: ```bash pip install "git+https://github.com/surrealdb/surrealml#egg=surrealml[torch]" ``` Once the package is installed, you can then train and save your first model using sklearn. ## Quick start with Sklearn Sklearn models can also be converted and stored in SurrealML’s `.surml` format enabling developers to load them in any Python version as we are not relying on `pickle`. Metadata in the file also enables other users of the model to use them out of the box without having to worry about the normalisation of the data or getting the right inputs in order. We will cover `.surml` files in more depth in the storage section. You will also be able to load your Sklearn models in Rust and run them, meaning you can use them in your SurrealDB server. Before we start writing any training code we need to import the following: ```python from sklearn.linear_model import LinearRegression from surrealml import SurMlFile, Engine from surrealml.model_templates.datasets.house_linear import HOUSE_LINEAR ``` Here we can see that we have imported the standard `sklearn` `LinearRegression` model. We then import the `SurMlFile` object which will facilitate the saving, loading, and execution of the trained model. We will use the `Engine` enum to tell our `SurMlFile` object if we are using `sklearn` or `torch`. We then finally import a small example dataset called `HOUSE_LINEAR`. This example dataset is a simple linear correlation between house prices, the square foot of the house, and the number of floors. This dataset is also used in the CI testing pipeline when we push updates. Now that we have imported everything that we need, we can train our model with the code below: ```python model = LinearRegression() model.fit(HOUSE_LINEAR["inputs"], HOUSE_LINEAR["outputs"]) ``` This will give us a trained model. Now we need to save it and this is where `SurrealML` comes in. First we declare a `SurMlFile` object instance with the inputs, name, model object, and engine with the following code: ```python file = SurMlFile( model=model, name="house-price-prediction", inputs=HOUSE_LINEAR["inputs"], engine=Engine.SKLEARN ) file.add_version(version="0.0.1") ``` The next step is optional, but it would be nice to map our inputs to some keys. We must be careful with the order that we declare our columns as they need to be mapped with the order or inputs from the vector that our model was trained on. If you click on `HOUSE_LINEAR` you will see the following declaration: ```python HOUSE_LINEAR = { "inputs": inputs, "outputs": house_price, "squarefoot": squarefoot, "num_floors": num_floors, "input order": ["squarefoot", "num_floors"], "raw_inputs": { "squarefoot": raw_squarefoot, "num_floors": raw_num_floors, }, "normalised_inputs": { "squarefoot": squarefoot, "num_floors": num_floors, }, "normalisers": { "squarefoot": { "type": "z_score", "mean": squarefoot.mean(), "std": squarefoot.std() }, "num_floors": { "type": "z_score", "mean": num_floors.mean(), "std": num_floors.std() } }, } ``` Here we can see that there are some normalisers involved. We can also see that the input order for the model training was `["squarefoot", "num_floors"]` . Therefore, we can add the column names to our `SurMlFile` object instance with the code below: ```python file.add_column("squarefoot") file.add_column("num_floors") ``` The `add_column` was the only order that we have to be careful about. We need to add our normalisers to the `SurMlFile` object instance with the code below but we do not have to worry about the order as the normalisers by default will be mapped to the columns: ```python file.add_normaliser( "squarefoot", "z_score", HOUSE_LINEAR["squarefoot"].mean(), HOUSE_LINEAR["squarefoot"].std() ) file.add_normaliser( "num_floors", "z_score", HOUSE_LINEAR["num_floors"].mean(), HOUSE_LINEAR["num_floors"].std() ) ``` We are nearly done with adding metadata, with just one item left to add: the output that the model is trying to predict. This can be achieved by the following code: ```python file.add_output( "house_price", "z_score", HOUSE_LINEAR["outputs"].mean(), HOUSE_LINEAR["outputs"].std() ) ``` And now our file is ready to be saved which is done with the code below: ```python file.save(path="./linear.surml") ``` The file is stored in the `.surml` format meaning that there is a header with the data that we defined, and the weights are stored in the `ONNX` format. This means that there is zero language dependent dependencies. We are now ready to load and perform calculations on our model. We can load our model with the following code: ```python new_file = SurMlFile.load(path="./linear.surml", engine=Engine.SKLEARN) ``` If you are confident in what you are doing at this point, you can choose to perform calculations through `surrealML` using a raw compute in which the raw vector of inputs is directly passed into the model with the code below: ```python print(new_file.raw_compute(input_vector=[5, 6])) ``` However, if you want the normalisation to be automatically applied, and inputs mapped via keys, we can use a buffered compute with the following code: ```python print( new_file.buffered_compute( value_map={ "squarefoot": 5, "num_floors": 6 } ) ) ``` Both types of executions are executing the ML model using the Rust engine under the hood. As a result, the exact same code will be running your model in SurrealDB or your inference server regardless of whether you choose to build a server in Python or Rust. This `.surml` file can be also be loaded by either Rust or any Python version that has surrealML and execute inference installed. Now that we are able to load and execute our model locally, how do we deploy our model onto SurrealDB and run it? In the next section, we cover uploading. ## Model deployment Before we try and upload our model, we need to have a node running. We can do with the `docker-compose.yml` file below: ```yaml version: '3' services: surrealdb: image: surrealdb/surrealdb command: start environment: - SURREAL_USER=root - SURREAL_PASS=secret - SURREAL_LOG=trace ports: - 8000:8000 ``` Once our node is running via docker, we can then upload our trained model with the following code: ```python url = "http://0.0.0.0:8000/ml/import" SurMlFile.upload( path="./linear.surml", url=url, chunk_size=36864, namespace="main", database="main", username="root", password="secret" ) ``` The `upload` function will chunk the model and stream it up to a SurrealDB node. We can then perform an execution of the model with the following SurrealQL function: ```python ml::house-price-prediction<0.0.1>({ squarefoot: 500.0, num_floors: 2.0 }) ``` Here, `house-price-prediction` is the name of the model. The `<0.0.1>` is the version of the model. The SurrealQL function above will give us a model output from the inputs that we passed in. We can now explore how our model can interact with other data with the SurrealQL script below: ```sql CREATE house_listing SET squarefoot_col = 500.0, num_floors_col = 1.0; CREATE house_listing SET squarefoot_col = 1000.0, num_floors_col = 2.0; CREATE house_listing SET squarefoot_col = 1500.0, num_floors_col = 3.0; SELECT * FROM ( SELECT *, ml::house_price_prediction<0.0.1>({ squarefoot: squarefoot_col, num_floors: num_floors_col }) AS price_prediction FROM house_listing ) WHERE price_prediction > 177206.21875; ``` What is happening here is that we are feeding the columns from the table `house_listing` into a model we uploaded called `house-price-prediction` with a version of `0.0.1`. We then get the results of that trained ML model as the column `price_prediction`. We then use the calculated predictions to filter the rows giving us the following result: ```json [ { "id": "house_listing:7bo0f35tl4hpx5bymq5d", "num_floors_col": 3, "price_prediction": 406534.75, "squarefoot_col": 1500 }, { "id": "house_listing:8k2ttvhp2vh8v7skwyie", "num_floors_col": 2, "price_prediction": 291870.5, "squarefoot_col": 1000 } ] ``` Having covered everything that we need to get up and running with SurrealML, we should explore some other concepts in more depth to get the most out of SurrealML and be able to troubleshoot problems. ## In this section - [Computation](/docs/explore/ml-models/surrealml/computation.md) - where a model runs, and what the database does with it - [Storage](/docs/explore/ml-models/surrealml/storage.md) - how a trained model is stored and versioned --- Source: https://surrealdb.com/docs/explore/ml-models/surrealml/computation # Computation SurrealML enables machine learning models to be greatly simplified, ensuring reproducibility and consistency in machine learning pipelines. If we have a loaded a model you will want to execute it to perform a calculation. First we will cover the high-level execution of models in Python and Rust, then will explore what is going on under the hood for these executions. Before we go through some code examples, let us define the different types of computation that we can perform on a model. ## Raw compute Raw compute passes an array directly into the model for execution. Here we bypass all the metadata in the file. This means that you need to perform your own normalisations on your input data if you normalised the model training data when training the model. This approach is more beneficial to complex data structures such as images, as they are multi-dimensional. When executing the raw compute, you do not need to pass in the dimensions of the input, as these are extracted from the saved model itself, as [`compute.rs`](https://github.com/surrealdb/surrealml/blob/34baa045da9184ccd1479220e6205dd298b78ee3/modules/core/src/execution/compute.rs#L80C68-L80C75) shows. ## Buffered compute Buffered compute passes a dictionary or hashmap with the names of the columns as the keys, and the values associated with the column. The engine then checks all the columns, applies normalisation functions to the inputs if the normalisation functions are present for the column, and constructs a vector in the correct order for inputs. At this point, the model computation will error if any of the columns are missing, only allowing valid input to pass through. The processed vector is then passed into the raw compute function. Now that we have explored the two types of computation, we can explore how to perform these computations. ## Executing a model in Python We can execute a model in Python using the following code: ```python # raw compute print(new_file.raw_compute(input_vector=[5, 6])) # buffered compute (implement data from the metadata) print( new_file.buffered_compute( value_map={ "squarefoot": 5, "num_floors": 6 } ) ) ``` ## Executing a model in Rust Starting with the following `use` statements will let us bring a number of necessary types into scope: ```rust use surrealml_core::storage::surml_file::SurMlFile; use surrealml_core::execution::compute::ModelComputation; use ndarray::ArrayD; use std::collections::HashMap; ``` We can then create a compute unit with the code below: ```rust let compute_unit = ModelComputation { surml_file: &mut new_file, }; ``` First we can do a buffered compute with a standard HashMap with the following code: ```rust let mut input_values = HashMap::new(); input_values.insert(String::from("squarefoot"), 1000.0); input_values.insert(String::from("num_floors"), 2.0); let output = compute_unit.buffered_compute(&mut input_values).unwrap(); ``` If we want to perform a raw compute, we can do so with the code below: ```rust let x = vec![1000.0, 2.0]; let data: ArrayD = ndarray::arr1(&x).into_dyn(); let output = compute_unit.raw_compute(data, None).unwrap(); ``` Note that the input dimensions fed into the `.raw_compute()` method are specified as None, as they will be extracted directly from the saved model. ## Executing a model in SurrealDB Once the model is stored in the database, it can be called with the following code: ```sql ml::house-price-prediction<0.0.1>({ squarefoot: squarefoot_col, num_floors: num_floors_col }) ``` Here the `ml::` is saying that we are using the SurrealDB’s ML module. The `house-price-prediction` is the name of the model in the metadata of the `surml` file. The `<0.0.1>` is the version that is also defined in the header of the `surml` file. The object passed into it is essentially a hashmap for the buffered compute. This calculation then yields a result that can then be inputted into the rest of the SurrealQL statement. ## Execution mechanics The interface between the ONNX `C++` and Rust can be found in [`onnx_environment.rs`](https://github.com/surrealdb/surrealml/blob/main/modules/core/src/execution/onnx_environment.rs). Based on different build parameters, we embed the `libonnxruntime` library into the Rust binary. When performing an execution, we load the `libonnxruntime` library into a `Lazy>`. This leads to a single Environment instance that lasts for the entire duration of the program. This environment can be accessed by any thread, which is safe to do so as access is protected by a lock that is only available once another thread has given it up. Tests of running a computation in Rust live in [the `compute.rs` test module](https://github.com/surrealdb/surrealml/blob/34baa045da9184ccd1479220e6205dd298b78ee3/modules/core/src/execution/compute.rs#L161). Python does not perform any computations, but all interactions between the Rust module and the Python code sit in [`rust_adapter.py`](https://github.com/surrealdb/surrealml/blob/develop/surrealml/rust_adapter.py), in the `RustAdapter` object. --- Source: https://surrealdb.com/docs/explore/ml-models/surrealml/storage # Storage SurrealML enables machine learning models to be greatly simplified, ensuring reproducibility and consistency in machine learning pipelines. If you have completed the introduction, you would have stored your ML model in a `.surml` file. Seeing as the `.surml` file is at the heart of storage, we will start by covering the anatomy of a `.surml` file. ## The anatomy of a surml file A `.surml` file is essentially a header, with weights stored in the ONNX format. Interacting with the file takes the following form: Diagram of the SurrealML storage flow: Python and Rust programs reach the Rust ML core, which reads the SURML file's metadata and ONNX model and executes it through the ONNX runtime library.

The metadata is data around the model. You can see [the header definition in `header/mod.rs`](https://github.com/surrealdb/surrealml/blob/main/modules/core/src/storage/header/mod.rs). The metadata has the following fields (all fields can be empty if needed): - **keys ⇒** The name of the column and the index of where that column is placed in an input vector - **normalisers ⇒** A map of normalisers with parameters to execute the normaliser, and a reference to the column that the normaliser is attached to - **output ⇒** The name of the output and a normaliser attached to the output (if needed) - **name ⇒** The name of the model being stored - **version ⇒** The version of the model. Model versions use a default and auto-increment function that results in formats such as `0.0.1` and `0.0.2`. - **description ⇒** The description of the model - **engine ⇒** The type of engine that was used to train the model, such as the native Linfa Rust module, the Rust PyTorch model, or Undefined (trained using a third party module such as Sklearn, PyTorch, etc.) - **Origin ⇒** Where the model was trained, such as locally, in the database, or Undefined (local is automatically defined when using the Python surrealML package) - **input_dims ⇒** the input dimensions that are needed to perform a model imputation. This is automatically defined when tracing the model when saving it in the `.surml` file format. When reading a file, the loader loads the first 4 bytes of a file. Those first 4 bytes are then converted into a 4 byte integer. This 4 byte integer then tells the loader how many bytes to load to get all of the metadata about the model. Once this is loaded, we can then assume that the rest of the file is ONNX protobuf, and this protobuf data is then loaded into the ONNX runtime C++ library for inference calculations. ## Why ONNX SurrealML supports the ONNX runtime which is the standard format for storing machine learning weights, and is officially [supported by Microsoft](https://github.com/microsoft/onnxruntime). Under the hood, SurrealML exports the torch model into ONNX format which is officially supported by [PyTorch](https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html). The convergence of the ML community to ONNX as its standard format has led to [research and academic papers](https://cloudblogs.microsoft.com/opensource/2020/12/17/accelerate-simplify-scikit-learn-model-inference-onnx-runtime/) produced by FANG researchers on how to convert models such as random forests into ONNX. There are also desktop apps like neutron that let you inspect these ONNX models in graphical form such as the following: https://netron.app/ Alongside this, there is also work directly supported by a range of massive companies to enable ONNX to be run in WASM: https://onnxruntime.ai/docs/build/web.html And the official huggingface github is also working on a pure Rust implementation of ONNX with the code link below: https://github.com/huggingface/candle/tree/main/candle-onnx ONNX also supports GPUs as seen in the following link: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html With all this support for the ecosystem, it makes sense for us to support ONNX and automate processes that convert models to ONNX such as Sklearn. Now that we have covered the anatomy of a `.surml` file, we can now move to loading and saving files. ## Loading and saving with Python This has been covered in the introduction section, so if you want to see the code around loading and saving, please visit the introduction section. We can save a model with the following code: ```python file = SurMlFile( model=model, # the trained model object name="house-price-prediction", inputs=HOUSE_LINEAR["inputs"], engine=Engine.SKLEARN # change to the pytorch version is storing a pytorch model ) file.add_version(version="0.0.1") file.save(path="./linear.surml") ``` To load the file, we use the code below: ```python new_file = SurMlFile.load(path="./linear.surml", engine=Engine.SKLEARN) ``` We can also load `.surml` files in Rust too. ## Saving and loading surml in rust To load in Rust you will need the following dependency in your `cargo.toml` file (version may have increased since the time of writing this): ```toml surrealml-core = "0.0.8" ``` Alternatively, you can add the dependency by typing `cargo add surrealml-core` on the command line. The `core` is the exact same code that runs in the Python client, ensuring consistency between the client and the server in production. Starting with the following `use` statements will let us bring a number of necessary types into scope: ```rust use std::fs::File; use std::io::{self, Read, Write}; use surrealml_core::storage::surml_file::SurMlFile; use surrealml_core::storage::header::Header; use surrealml_core::storage::header::normalisers::{ wrapper::NormaliserType, linear_scaling::LinearScaling }; ``` We can then load the ONNX file that was saved from a training session with the code below: ```rust let mut file = File::open("./stash/linear_test.onnx").expect("File to be found"); let mut model_bytes = Vec::new(); file.read_to_end(&mut model_bytes).expect(”File content to be read to string”); ``` Once we have loaded our bytes from the ONNX file, we will insert the `model_bytes` once we have defined our file. We can first define our header and add the columns with the following code: ```rust let mut header = Header::fresh(); header.add_column(String::from("squarefoot")); header.add_column(String::from("num_floors")); header.add_output(String::from("house_price"), None); ``` We can then add the normalisers with the code below: ```rust header.add_normaliser( "squarefoot".to_string(), NormaliserType::LinearScaling(LinearScaling { min: 0.0, max: 1.0 }) ); header.add_normaliser( "num_floors".to_string(), NormaliserType::LinearScaling(LinearScaling { min: 0.0, max: 1.0 }) ); ``` We now have everything we need to package our `.surml` file which we can do and write to disk with the following code: ```rust let surml_file = SurMlFile::new(header, model_bytes); surml_file.write("./stash/test.surml").unwrap(); ``` If we want to load a model, it can either be from bytes or a file using the code below: ```rust let new_file = SurMlFile::from_file("./stash/test.surml").unwrap(); let file_from_bytes = SurMlFile::from_bytes(surml_file.to_bytes()).unwrap(); ``` --- Source: https://surrealdb.com/docs/explore/studio # Introduction SurrealDB Studio is the official visual interface for SurrealDB. SurrealDB Studio is the official visual interface for managing and querying SurrealDB. It connects to any SurrealDB instance, letting you run SurrealQL, explore records, and manage schema from the web or desktop. [Visit studio.surrealdb.com](https://studio.surrealdb.com) Full documentation for SurrealDB Studio is coming soon. ## In this section - [SurrealQL editors](/docs/explore/studio/surrealql-editors.md) - write and run queries against a connection - [Search and shortcuts](/docs/explore/studio/search-and-shortcuts.md) - move around Studio from the keyboard --- Source: https://surrealdb.com/docs/explore/studio/search-and-shortcuts # Search and shortcuts The SurrealDB Studio command palette and its keyboard shortcuts. SurrealDB Studio offers a range of shortcuts to help you navigate the interface more efficiently. You can find a list of all available shortcuts by pressing `Ctrl + K` or `Cmd + K` on your keyboard. In the search modal, you can access all available shortcuts for open sessions, the views, all available tables in your connected database and the settings dialog. Search and Shortcuts

Options Description
Connections Open a list of all available connections and create a new connection type.
Views Open all available views in the current session. This can be the Cloud view, GraphQL view, Query view, Designer view or the Authentication view.
Tables Shows all available tables in the connected database. If you have no tables in the connection, load the Surreal Deal Store dataset, you can see the tables in the dataset.
Query See all saved queries, view the query history and create a new query. Comment out a line in the query editor using `Cmd + /` or `Ctrl + /`.
Explorer Import and export your database schema in a `.surql` file format.
Authentication Create user permissions and roles for your connected database.
API Docs Access the SurrealDB API documentation and access snippets for your queries in different SDK languages.
Settings Access the Settings dialog to customise the appearance and behaviour of SurrealDB Studio. Increase the view size of your editor with `cmd + Option + +` or reduce with `cmd + Option + -`.
Navigator Reload the navigator to see all available shortcuts and open the search modal.
Developer Open a new connection and reset tour settings.
--- Source: https://surrealdb.com/docs/explore/studio/surrealql-editors # SurrealQL editors SurrealQL editor shortcuts in SurrealDB Studio. Indentation, comments, multi-cursor edits, running queries, record inspection, and JSON formatting. Throughout SurrealDB Studio you will encounter various SurrealQL editors. These editors support intelligent SurrealQL highlighting and provide a range of features to help you write queries and edit records. ## Shortcuts Editors support an array of useful shortcuts to help you navigate and edit more efficiently.
Shortcuts Description
tab Indent the current line by 4 spaces.
shift + tab Unindent the current line by 4 spaces.
ctrl/cmd + f Open the search and replace panel
ctrl/cmd + / Toggles comments on the selected or active line(s)
ctrl/cmd + left click Place multiple cursors at the clicked locations
ctrl/cmd + shift + L Selects all occurrences of the currently selected text
ctrl/cmd + d Selects next occurrence of the currently selected text
ctrl/cmd + enter Execute the query query (only applied to the query editor)
## Record inspection Some editors support record inspection, which allows you to open a record in the Record inspector drawer by holding `ctrl/cmd` and left-clicking on a record id. ## JSON Some editors, such as the record editor and query responses, support falling back to a JSON representation. This can be configured in the Settings dialog under `Appearance > Value formatting mode`. --- Source: https://surrealdb.com/docs/explore/tutorials # Demos and tutorials Hands-on demo applications and tutorials. Learn SurrealDB by building real projects. Learn SurrealDB by working through practical examples. Whether you prefer a guided walkthrough or exploring a ready-made application, the resources here will help you put concepts into practice. - [Demos](/docs/explore/tutorials/demos/overview.md) - sample applications and datasets you can clone, import, or run locally to see SurrealDB in action. - [Tutorials](/docs/explore/tutorials/tutorials/overview.md) - step-by-step guides covering topics such as authentication integrations, real-time applications, AI-powered features, and more. --- Source: https://surrealdb.com/docs/explore/tutorials/demos/blink # Blink note-taking app Blink - a Notion-style workspace demo built with SurrealDB and WebAssembly. Blink is an open-source sample project that shows how SurrealDB can sit behind a Notion-like editing experience, with the database engine running in the browser via WebAssembly. It is a demo, not a hosted product: the value is seeing how real-time structured data, permissions, and UI state can fit together. What to look for: how the app models pages and blocks, how collaboration or sync behaviour feels, and how SurrealDB’s API surface maps to a rich client. If you are evaluating SurrealDB for knowledge or agent tooling, Blink is a concrete reference implementation rather than a minimal CRUD example. Getting started: clone and run the project from the [Blink repository on GitHub](https://github.com/kearfy/blink). Expect to read the README for build steps and browser requirements - WASM workflows vary by toolchain. If you want more guided walkthroughs in the same spirit, browse the [tutorials](/docs/explore/tutorials/tutorials/overview.md) and [demos](/docs/explore/tutorials/demos/overview.md) sections. --- Source: https://surrealdb.com/docs/explore/tutorials/demos/kaig-ai-demos # Kai G AI demos A collection of AI demo applications built with SurrealDB, showcasing RAG, agents, and knowledge graph patterns. [Kaig](https://github.com/surrealdb/kaig) is an open-source collection of AI demo applications that use SurrealDB as their data layer. Each demo is a self-contained project you can run locally to explore a specific AI pattern. ## What you will find The repository includes demos covering: - **Retrieval-augmented generation (RAG)** - embedding documents into SurrealDB's vector indexes and using similarity search to ground LLM responses in real data. - **AI agents** - autonomous agents that read from and write to SurrealDB as part of their tool-use loop. - **Knowledge graphs** - building and querying graph structures that give AI models structured context. ## Getting started Clone the repository and follow the README in the demo you want to explore: ```bash git clone https://github.com/surrealdb/kaig.git cd kaig ``` Each demo directory contains its own setup instructions, dependencies, and a walkthrough of the code. --- Source: https://surrealdb.com/docs/explore/tutorials/demos/overview # Demos Sample applications and datasets you can clone, import, or run locally to explore SurrealDB without a long-form walkthrough. This section collects **demos**: fairly complete examples you can **clone and run** (or import) to see SurrealDB in a realistic setting. Each item is a self-contained project or dataset with its own README or import steps rather than a page-by-page guide. Use demos when you want to explore behaviour, skim the code, or validate a setup quickly. For guided steps and explanations, use the [tutorials](/docs/explore/tutorials/tutorials/overview.md) section instead. Pick a demo from the sidebar to get started. ## Demos - [Blink note-taking app](/docs/explore/tutorials/demos/blink.md) - a small app you can clone and run - [Kai G AI demos](/docs/explore/tutorials/demos/kaig-ai-demos.md) - community AI demos built on SurrealDB ## Full applications - [Surreal Deal Store](/docs/explore/tutorials/demos/surreal-deal-store.md) - a complete storefront to clone and run --- Source: https://surrealdb.com/docs/explore/tutorials/demos/surreal-deal-store # Surreal Deal Store To quickly test out SurrealDB and SurrealQL functionality, we've included demo data which you can download and import into SurrealDB. To further test out SurrealDB and SurrealQL functionality, we've included two demo datasets here in `.surql` files which you can download and [`import`](/docs/reference/cli/surrealdb-cli/commands/import.md) into SurrealDB using the [CLI](/docs/reference/cli/surrealdb-cli/overview.md). ## Surreal Deal Store - there is a lot in store for you! Surreal Deal Store is our new and improved demo dataset based on our [SurrealDB Store](https://surrealdb.store/). The dataset is made up of 12 tables using both [graph relations](/docs/reference/query-language/statements/relate.md) and [record links](/docs/reference/query-language/language-primitives/record-links.md). In the diagram below, the nodes in pink are the [standard tables](/docs/reference/query-language/statements/define/table.md), the ones in purple represent the [edge tables](/docs/reference/query-language/statements/relate.md) which shows relationships between records and SurrealDB as a graph database. The nodes in grey are the [pre-computed table views](/docs/reference/query-language/statements/define/table.md). Surreal Deal Data Model ### Download **SurrealDB 3.x** | Dataset | URL | | -------------------------------------------------------------------------------- | --------------------------------------------------------- | | [Surreal Deal Store (mini)](https://datasets.surrealdb.com/datasets/surreal-deal-store/mini-v3.surql) | https://datasets.surrealdb.com/datasets/surreal-deal-store/mini-v3.surql | **SurrealDB 2.x** | Dataset | URL | | -------------------------------------------------------------------------------- | --------------------------------------------------------- | | [Surreal Deal Store](https://datasets.surrealdb.com/surreal-deal-store.surql) | https://datasets.surrealdb.com/surreal-deal-store.surql | | [Surreal Deal Store (mini)](https://datasets.surrealdb.com/surreal-deal-store-mini.surql) | https://datasets.surrealdb.com/surreal-deal-store-mini.surql | ### Import Once one of the datasets has been downloaded, it's now time to [start the server](/docs/reference/cli/surrealdb-cli/commands/start.md). ```bash # Create a new in-memory server surreal start --user root --pass secret --allow-all ``` Lastly, use the [import command](/docs/reference/cli/surrealdb-cli/commands/import.md) to add the dataset. Use the command below to import the [surreal deal store dataset](https://datasets.surrealdb.com/surreal-deal-store.surql): ```bash surreal import --endpoint http://localhost:8000 --user root --pass secret \ --ns main --db main surreal-deal-store.surql ``` To import the surreal downloaded the [Surreal Deal store (mini)](https://datasets.surrealdb.com/surreal-deal-store-mini.surql) use the command below: ```bash surreal import --endpoint http://localhost:8000 --user root --pass secret \ --ns main --db main surreal-deal-store-mini.surql ``` Please be aware that the import process might take a few seconds. ### Using Curl First, start the SurrealDB server: ```bash # Create a new in-memory server surreal start --user root --pass secret --allow-all ``` Then download the file and load it into the database: ```bash # Download the file curl -L "https://datasets.surrealdb.com/surreal-deal-store.surql" -o surreal-deal-store.surql # Load the file into the database using the rest endpoint curl -v -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" -H "Accept: application/json" --data-binary @surreal-deal-store.surql http://localhost:8000/import ``` If you want to use the mini version: ```bash # Download the file curl -L "https://datasets.surrealdb.com/surreal-deal-store-mini.surql" -o surreal-deal-store-mini.surql # Load the file into the database using the rest endpoint curl -v -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" -H "Accept: application/json" --data-binary @surreal-deal-store-mini.surql http://localhost:8000/import ``` ### Sample queries Here are some sample queries you can run on the Surreal Deal Store dataset. We've also included a [SurrealDB Studio Mini](https://app.surrealdb.com/mini) below to help you run these queries. > [!NOTE] > The query results below have been limited to 4 rows for brevity. If you remove the `LIMIT 4` clause from the queries, you'll see the full results. ```surql -- Query 1: Using record links to select from the seller table SELECT name, seller.name FROM product LIMIT 4; -- Query 2: Using graph relations to select from the person and product table SELECT time.created_at as order_date, product_name, <-person.name as person_name, ->product.details FROM order LIMIT 4; -- Query 3: Conditional filtering based on an embedded object property. SELECT name, email FROM person WHERE address.country ?= "England" LIMIT 4; -- Query 4: Conditional filtering using relationships. SELECT * FROM review WHERE ->product.sub_category ?= "Activewear" LIMIT 4; -- Query 5: Count orders based on order status SELECT count() FROM order WHERE order_status IN [ "processed", "shipped"] GROUP ALL LIMIT 4; -- Query 6: Get a deduplicated list of products that were ordered SELECT array::distinct(product_name) as ordered_products FROM order GROUP ALL LIMIT 4; -- Query 7: Get the average price per product category SELECT ->product.category AS product_category, math::mean(price) AS avg_price FROM order GROUP BY product_category ORDER BY avg_price DESC LIMIT 4; -- Query 8: encapsulating logic in a function RETURN fn::number_of_unfulfilled_orders(); -- Query 9: using a custom fuction for currency conversion SELECT product_name, fn::pound_to_usd(price) AS price_usd FROM order LIMIT 4; ``` --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/auth0-integration # Integrate Auth0 with SurrealDB Use Auth0 as the authentication provider for a single-page application backed only by SurrealDB. This guide will cover using [Auth0](https://auth0.com/) as the authentication provider for single-page web applications using SurrealDB as the only backend. This guide uses [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md), the method for every current version of SurrealDB. The tabs also carry the older [Scope](/docs/reference/query-language/statements/define/scope.md) and [Token](/docs/reference/query-language/statements/define/token.md) statements for readers still on `v1.x` - both were removed in later versions. In this guide you will learn how to: - Configure Auth0 to issue tokens that can be used with SurrealDB. - Configure SurrealDB to accept tokens issued by Auth0. - Define user-level authorization using SurrealDB [record users](/docs/learn/security/authentication/users.md#record-users). - Authenticate users with Auth0 in a single-page application. - Retrieve and update information from SurrealDB using the authenticated user. This guide will cover the most general case, in which SurrealDB is the only backend for your application. You can still follow this guide even if you have additional backends, but in that case you may have other options available to request and validate tokens issued by Auth0. Likewise, even if your application is not strictly a Single-Page Application (SPA), you may still follow and benefit from this guide. ## Prerequisites This guide assumes the following: - You have a [fresh instance of SurrealDB running.](/docs/running/overview.md) - You can [use a local Docker container](/docs/running/docker.md) without volumes for the purposes of this guide. ```bash docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest \ start --user root --pass secret ``` To run the SurrealQL statements mentioned in this guide, you will also need an interactive shell. ```bash surreal sql -u root -p secret --pretty ``` You will also need to [create an Auth0 account](https://auth0.com/signup), which can be on the free plan. ## Configuring Auth0 ### Creating a simple SPA, an Auth0 application and an Auth0 API First of all, you will need to complete the regular setup for creating a Single-Page Application and an API resource within Auth0. You can do this by following the official Auth0 [documentation](https://auth0.com/docs/quickstart/spa) for your SPA. If you are using plain JavaScript, follow [the vanilla guide](https://auth0.com/docs/quickstart/spa/vanillajs/01-login) to create an application in Auth0 and [the API guide](https://auth0.com/docs/quickstart/spa/vanillajs/02-calling-an-api) to create an API in Auth0. > [!NOTE] > You will not need to create an actual backend API (e.g. using backend languages like NodeJS or Go) as the documentation suggests when using SurrealDB. A simple file server (e.g. python3 -m http.server 8080, for local testing or any static web server for production) that can serve the static content of your website will suffice. However, creating an API resource in Auth0 is necessary, as it will generate an “audience” string which will be required for Auth0 to add claims to its access tokens. At the end of those tutorials, you should have both an application and an API created in your Auth0 account and a simple client-side web application that authenticates with Auth0 using those two resources. This website will capture the access token issued by Auth0, which is the token that we are using with SurrealDB. If you were not able to create a working website, you can also use this [minimal example created by SurrealDB](https://github.com/surrealdb/examples/tree/main/auth0). When completing the actions above, make sure to keep the following information handy: - Auth0 Client ID, generated when creating the application. - Auth0 Domain, generated when creating the application. - Auth0 Audience, generated when creating the API. ### Creating a custom Auth0 action to add claims for SurrealDB Now, Auth0 is ready to perform authentication and issue tokens for your application. However, SurrealDB expects these tokens to contain some specific claims. Auth0 allows adding custom claims through its “actions” and “flows” features. Since Auth0 [requires claims for an API audience to be namespaced](https://auth0.com/docs/troubleshoot/product-lifecycle/past-migrations/custom-claims-migration#restricted-token-audience), these claims will need to have the `https://surrealdb.com/` prefix. To add custom claims, you must [create an Auth0 action](https://auth0.com/docs/customize/actions/write-your-first-action#create-an-action) with the “Login / Post Login” trigger. For this example, you can use the following code: **Using DEFINE ACCESS** ```js exports.onExecutePostLogin = async (event, api) => { if (event.authorization) { // The claims in this block are expected by SurrealDB. // These values should match your SurrealDB installation. api.accessToken.setCustomClaim(`https://surrealdb.com/ns`, "main"); api.accessToken.setCustomClaim(`https://surrealdb.com/db`, "main"); // This value corresponds to the name of the JWT access method // which will be created in SurrealDB during the next section. api.accessToken.setCustomClaim(`https://surrealdb.com/ac`, "auth0"); // In this block, we will add additional claims which are not required by SurrealDB. // These claims can be used from SurrealQL to implement application logic. // In this example, we will add the data that we will store for each user. // We will also use some of this data to perform authorization. api.accessToken.setCustomClaim(`https://surrealdb.com/email`, event.user.email); api.accessToken.setCustomClaim(`https://surrealdb.com/email_verified`, event.user.email_verified); api.accessToken.setCustomClaim(`https://surrealdb.com/name`, event.user.name); api.accessToken.setCustomClaim(`https://surrealdb.com/nickname`, event.user.nickname); api.accessToken.setCustomClaim(`https://surrealdb.com/picture`, event.user.picture); } }; ``` **Using Scope and Token (1.x only)** ```js exports.onExecutePostLogin = async (event, api) => { if (event.authorization) { // The claims in this block are expected by SurrealDB. // These values should match your SurrealDB installation. api.accessToken.setCustomClaim(`https://surrealdb.com/ns`, "main"); api.accessToken.setCustomClaim(`https://surrealdb.com/db`, "main"); // These values correspond to the names of the SCOPE and TOKEN resources // which will be created in SurrealDB during the next section. api.accessToken.setCustomClaim(`https://surrealdb.com/sc`, "user"); api.accessToken.setCustomClaim(`https://surrealdb.com/tk`, "auth0"); // In this block, we will add additional claims which are not required by SurrealDB. // These claims can be used from SurrealQL to implement application logic. // In this example, we will add the data that we will store for each user. // We will also use some of this data to perform authorization. api.accessToken.setCustomClaim(`https://surrealdb.com/email`, event.user.email); api.accessToken.setCustomClaim(`https://surrealdb.com/email_verified`, event.user.email_verified); api.accessToken.setCustomClaim(`https://surrealdb.com/name`, event.user.name); api.accessToken.setCustomClaim(`https://surrealdb.com/nickname`, event.user.nickname); api.accessToken.setCustomClaim(`https://surrealdb.com/picture`, event.user.picture); } }; ``` This action should be saved and added to the “Login” flow in the “Actions > Flows” section. ## Configuring SurrealDB **Using DEFINE ACCESS** ### Defining permissions and fields in SurrealDB For this simple example, we will create a single table named “user”, where any user that authenticates through Auth0 using your application will be granted complete permissions over their data. For this to work as intended, we will need to ensure that the email address is unique between users and that users are granted permissions to access their own record as long as they authenticated with the access method that we will define. ```surql DEFINE TABLE user SCHEMAFULL -- Authorised users can select, update, delete and create user records. -- Records that do not match the permissions will not be modified nor returned. PERMISSIONS FOR select, update, delete, create WHERE -- The access method must match the method that we will define. $access = "auth0" -- The record identifier must match that of the authenticated user. AND id = $auth ; -- In this example, we will use the email as the primary identifier for a user. DEFINE INDEX email ON user FIELDS email UNIQUE; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); -- We define some other information present in the token that we want to store. DEFINE FIELD name ON user TYPE string; DEFINE FIELD nickname ON user TYPE string; DEFINE FIELD picture ON user TYPE string; ``` ### Defining a token verification method in SurrealDB Next, we should configure SurrealDB so that it can verify tokens sent to it through the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) via the “Authorization” header or through any of the [SDKs](/docs/languages.md) via the “Authenticate” methods. To do that, we will leverage the JWKS support in SurrealDB in order to define a token verification mechanism pointing to a JWKS object served by Auth0. This JWKS object can be found in a [dedicated endpoint for your Auth0 domain](https://auth0.com/docs/secure/tokens/json-web-tokens/locate-json-web-key-sets). Pointing to a JWKS file ensures that token verification will work even after [rotating the signing keys](https://auth0.com/docs/get-started/tenant-settings/signing-keys/rotate-signing-keys) and that tokens signed with revoked keys will no longer be accepted by SurrealDB. To understand how revocation is handled by SurrealDB, read the [JSON Web Key Set documentation](/docs/reference/query-language/statements/define/access/jwt.md#json-web-key-set-jwks) under `DEFINE ACCESS ... TYPE JWT`. We will also use the [`AUTHENTICATE`](/docs/reference/query-language/statements/define/access/record.md#with-authenticate-clause) clause in order to check that any necessary token claims have the expected values before returning the user matching the email address provided by Auth0. This is required because Auth0 has no knowledge of the record identifiers that are used in SurrealDB, so we need to use an identifier that can actually be provided by Auth0 in order to retrieve the corresponding record user. The following queries will create the required resources to authenticate a token for a record user: ```surql -- Specify the namespace and database that will be used. -- These values should match the custom claims that we configured before. USE NS main DB main; -- Define the public key to verify tokens issued by Auth0 for our application. -- The name of the token should match the custom claim that we configured before. DEFINE ACCESS auth0 ON DATABASE TYPE RECORD -- We verify the token using the public keys hosted by Auth0. WITH JWT URL "https:///.well-known/jwks.json" -- We check the token claims and map the email address to a record user. AUTHENTICATE { IF ( -- The JWT specification allows the audience claim to be an array or a string. -- In this example, we ensure that it is provided as an array by Auth0. $token.aud.is_array() -- The audience claim must contain the audience of you application. -- This is the value that you defined when creating the API in Auth0. AND $token.aud CONTAINS "" -- The audience claim must contain your Auth0 user information endpoint. -- It contains the domain generated when when creating the application in Auth0. AND $token.aud CONTAINS "https:///userinfo" -- The email address in the token must be verified as belonging to the user. AND $token['https://surrealdb.com/email_verified'] = true ) { -- We return the only user that matches the email address claim found in the token. RETURN SELECT * FROM user WHERE email = $token['https://surrealdb.com/email'] } } ; ``` In the example above, replace the placeholder with the domain value defined for your Auth0 application. It is important to not that [validating the audience of the token is a requirement of Auth0](https://auth0.com/docs/secure/tokens/access-tokens/validate-access-tokens), other providers may require validating additional claims (e.g. `iss`, `sub`) to ensure that the token is being used as intended. With Auth0, you can also make use of [OpenID Connect scopes](https://auth0.com/docs/get-started/apis/scopes/openid-connect-scopes), which can be accessed through the `scopes` claim via `$token.scopes` and contain the OIDC scopes requested by the application and granted by the user, which we will not do for this example. > [!IMPORTANT] > In order to allow SurrealDB to establish a connection with Auth0 to download the JWKS object, you will require running it with the network capability. For the strongest security, provide your specific Auth0 domain when starting SurrealDB with --allow-net. For example: --allow-net example.eu.auth0.com. **Using Scope and Token (1.x only)** ### Defining a token verification method in SurrealDB Next, we should configure SurrealDB so that it can verify tokens sent to it through the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) via the “Authorization” header or through any of the [SDKs](/docs/languages.md) via the “Authenticate” methods. To do that, we will leverage the JWKS support in SurrealDB in order to define a token verification mechanism pointing to a JWKS object served by Auth0. This JWKS object can be found in a [dedicated endpoint for your Auth0 domain](https://auth0.com/docs/secure/tokens/json-web-tokens/locate-json-web-key-sets). Pointing to a JWKS file ensures that token verification will work even after [rotating the signing keys](https://auth0.com/docs/get-started/tenant-settings/signing-keys/rotate-signing-keys) and that tokens signed with revoked keys will no longer be accepted by SurrealDB. To understand how revocation is handled by SurrealDB, read the [JSON Web Key Set documentation](/docs/reference/query-language/statements/define/token.md#json-web-key-set-jwks) under `DEFINE TOKEN`. The following queries will create the required resources to authenticate a token for a scope: ```surql -- Specify the namespace and database that will be used. -- These values should match the custom claims that we configured before. USE NS main DB main; -- Define the scope where the token will be used. -- The name of the scope should match the custom claim that we configured before. DEFINE SCOPE user; -- Define the public key to verify tokens issued by Auth0 for our application. -- The name of the token should match the custom claim that we configured before. DEFINE TOKEN auth0 ON SCOPE user TYPE JWKS VALUE "https:///.well-known/jwks.json"; ``` In the example above, replace the placeholder with the domain value defined for your Auth0 application. > [!IMPORTANT] > In order to allow SurrealDB to establish a connection with Auth0 to download the JWKS object, you will require running it with the network capability. For the strongest security, provide your specific Auth0 domain when starting SurrealDB with --allow-net. For example: --allow-net example.eu.auth0.com. ### Defining authorization criteria in SurrealDB For this simple example, we will create a single table named “user”, where any user that authenticates through Auth0 using your application with a verified email address will be able to register, view and update their data. For this to work as intended, we will need to verify some information in the token claims. ```surql DEFINE TABLE user SCHEMAFULL -- Authorised users can select, update, delete and create user records. -- Records that do not match the permissions will not be modified nor returned. PERMISSIONS FOR select, update, delete, create WHERE -- The token scope must match the scope that we defined. -- The name of the scope should match the scope that we defined before. $scope = "user" -- The audience claim must contain the audience of you application. -- This is the value that you defined when creating the API in Auth0. AND $token.aud CONTAINS "" -- The audience claim must contain your Auth0 user information endpoint. -- It contains the domain generated when when creating the application in Auth0. AND $token.aud CONTAINS "https:///userinfo" -- The email claim must match the email of the user being queried. AND email = $token['https://surrealdb.com/email'] -- The email must be verified as belonging to the user. AND $token['https://surrealdb.com/email_verified'] = true ; -- In this example, we will use the email as the primary identifier for a user. DEFINE INDEX email ON user FIELDS email UNIQUE; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); -- We define some other information present in the token that we want to store. DEFINE FIELD name ON user TYPE string; DEFINE FIELD nickname ON user TYPE string; DEFINE FIELD picture ON user TYPE string; ``` It is important to know that [validating the audience of the token is a requirement of Auth0](https://auth0.com/docs/secure/tokens/access-tokens/validate-access-tokens), other providers may require validating additional claims (e.g. iss, sub) to ensure that the token is being used as intended. With Auth0, you can also make use of [OpenID Connect scopes](https://auth0.com/docs/get-started/apis/scopes/openid-connect-scopes), which can be accessed through the `scopes` claim via `$token.scopes` and contain the OIDC scopes requested by the application and granted by the user, which we will not do for this example. It is also important to note that the `$auth` variable accessible from SurrealQL will not contain any values in this case, as it requires the `id` claim to be added to the JWT, containing the value of the identifier of a SurrealDB record. For the current example, the `$auth` variable will not be necessary. ## Configuring the application Now that we have everything ready for Auth0 to generate tokens and for SurrealDB to receive and verify them, we will modify our simple web application to use the token issued by Auth0 to register a new user or to update its information if the user already exists. Because of the diversity of SDKs that both SurrealDB and Auth0 support, it is difficult to provide examples that will work for everyone. In this guide, we will provide code examples for the most unspecific use case of using the [SurrealDB HTTP REST API](/docs/reference/rest-api/http-protocol.md) and [Auth0 SPA JS](https://github.com/auth0/auth0-spa-js). This code will run entirely on the client and can be added to the static website that you created while following the Auth0 quick start documentation linked at the beginning of this guide. With the aforementioned interfaces, we can use the following functions to authenticate users: ```js // Returns users in that the token is authorised to select. // Should return only the user matching the email in the token. const getUser = async () => { // We fetch an access token from Auth0 with the ID token. const auth0Token = await auth0Client.getTokenSilently(); const response = await fetch(surrealDbConfig.endpoint + "/key/user", { method: "GET", headers: { "Accept": "application/json", "Authorization": "Bearer " + auth0Token } }); return response.json(); }; // Creates a user matching the information in the token. // If the user already exists, updates the existing user with the new data. const createUpdateUser = async () => { // We collect the user data from the Auth0 ID token. const auth0User = await auth0Client.getUser(); // We fetch an access token from Auth0 with the ID token. const auth0Token = await auth0Client.getTokenSilently(); // We define the general query to create or update a user. // We leave the method to be defined later. let query = { body: JSON.stringify({ email: auth0User.email, name: auth0User.name, nickname: auth0User.nickname, picture: auth0User.picture }), headers: { "Accept": "application/json", "Authorization": "Bearer " + auth0Token } }; // We get the user that the token is authorised to access. const surrealDbUser = await getUser(); if (surrealDbUser[0].result.length == 0) { // If a user for the token does not exist, we create the record. console.log("Token user does not exist in database. Creating record."); query.method = "POST"; } else { // If a user for the token already exists, we update the record. console.log("Token user already exists in database. Updating record."); query.method = "PUT"; } // We perform the query and return the created/updated record. let response = await fetch(surrealDbConfig.endpoint + "/key/user", query); return response.json(); }; ``` If you want to see how this code would fit inside the web application that you built, you can view this [minimal example created by SurrealDB](https://github.com/surrealdb/examples/tree/main/auth0). To learn about other potential uses for the SurrealDB token functionality, you can read the [`DEFINE ACCESS ... TYPE JWT`] documentation page](/docs/reference/query-language/statements/define/access/jwt.md). ## Annex In this section, we will provide a few examples of how to configure the application to work with Auth0. ## Example single page application You can view and download a minimal example of an SPA using Auth0 and SurrealDB [in this repository](https://github.com/surrealdb/examples/tree/main/auth0). ## Alternative: using HMAC Using public key cryptography algorithms for signing tokens prevents you from having to store any secrets at all in SurrealDB. However, some SurrealDB administrators may prefer to use HMAC algorithms, which use the same secret to both sign and verify the signature of the JWT. This secret will be stored in SurrealDB when provided as value for the `DEFINE ACCESS` statement and its access should be restricted, as it can be used to issue arbitrary tokens which will be trusted by SurrealDB. However, **we do not recommend this method** over using public cryptography, as the later significantly reduces the burden of creating strong secrets, keeping them secret and managing their lifecycle. According to the OAuth 2.0 specification (part of OpenID Connect), only confidential (as opposed to public) applications should be allowed to use HMAC algorithms, as it requires being able to keep the secret secure. For this reason, Auth0 will not allow using HMAC algorithms with applications of the SPA (Single-Page Application) type, as they generally would have to store the secret in the client, which is publicly accessible. However, because of the particular case of SurrealDB, the secret will not need to be exposed to the client and will instead be stored in SurrealDB. Auth0 can support scenarios like these through the option to [disable OIDC-conformant authentication](https://auth0.com/docs/authenticate/login/oidc-conformant-authentication). Once disabled, Auth0 will allow selecting HMAC algorithms for SPA. Keep in mind that other options may become available in Auth0 after this change which, if modified without proper care, may compromise the security of your application. If the choice of using HMAC is made, the rest of the guide can be followed as is, with the exception of specifying the proper HMAC algorithm and placing the secret as the value when defining a token. In Auth0, the secret used to sign using HMAC algorithm corresponds to the [Auth0 Client Secret](https://auth0.com/docs/secure/application-credentials#client-secret-authentication) string. The example below shows how a token using the HS256 algorithm can be defined: **Using DEFINE ACCESS** ```surql -- Define the secret to verify tokens issued by Auth0 for our application. -- The name of the access method should match the custom claim that we configured before. DEFINE ACCESS auth0 ON DATABASE TYPE RECORD WITH JWT ALGORITHM HS256 KEY "" ; ``` **Using Scope and Token (1.x only)** ```surql -- Define the secret to verify tokens issued by Auth0 for our \ application. -- The name of the token should match the custom claim that we \ configured before. DEFINE TOKEN auth0 ON SCOPE user TYPE HS256 VALUE \ ""; ``` --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/aws-cognito-integration # Integrate AWS Cognito with SurrealDB Use AWS Cognito as the authentication provider for a client-side application backed only by SurrealDB. This guide will cover using [AWS Cognito](https://aws.amazon.com/cognito/) as the authentication provider for client-side web applications using SurrealDB as the only backend. This guide uses [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md), the method for every current version of SurrealDB. The tabs also carry the older [Scope](/docs/reference/query-language/statements/define/scope.md) and [Token](/docs/reference/query-language/statements/define/token.md) statements for readers still on `v1.x` - both were removed in later versions. In this guide you will learn how to: - Configure AWS Cognito to issue tokens that can be used with SurrealDB. - Configure SurrealDB to accept tokens issued by AWS Cognito. - Define user-level authorization using SurrealDB [record users](/docs/learn/security/authentication/users.md#record-users). - Authenticate users with AWS Cognito in a client-side web application. - Retrieve and update information from SurrealDB using the authenticated user. This guide will cover the most general case, in which SurrealDB is the only backend for your application. You can still follow this guide even if you have additional backends, but in that case you may have other options available to request and validate tokens issued by AWS Cognito. ## Prerequisites This guide assumes the following: - You have a [fresh instance of SurrealDB running](/docs/running/overview.md). - You can [use a local Docker container](/docs/running/docker.md) without volumes for the purposes of this guide. ```bash docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest \ start --user root --pass secret ``` To run the SurrealQL statements mentioned in this guide, you will also need an interactive shell. ```bash surreal sql -u root -p secret --pretty ``` You will also need to [have an Amazon Web Services account](https://aws.amazon.com/resources/create-account/). By following this guide you will be subject to **at least** the [AWS Cognito](https://aws.amazon.com/cognito/pricing/) and [AWS Lambda](https://aws.amazon.com/lambda/pricing/) pricing plans. Although the resources used in this guide should be well within the free tier for both, we cannot guarantee that this will be the case in your particular situation. ## Configuring AWS Cognito In this section, we will create a user pool and a client that will be used to authenticate users with AWS Cognito. ### Creating a user pool and a client Cognito offers [user pools](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html) as a user directory for mobile and web applications. This directory can hold users defined directly within Cognito, but can also integrate with third-party identity providers like Google and Facebook. Additionally, it provides the ability to handle user registrations directly in the [Cognito Hosted UI](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-app-integration.html), which can also take care of requiring specific information, verifying user email addresses and phone numbers or even enforcing multi-factor authentication. Within a user pool, [creating a client](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-client-apps.html#cognito-user-pools-app-idp-settings-console-create) establishes a method (in our case a client-side web application) to authenticate with the user pool. You will need to [create a user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pool-as-user-directory.html) that meets your requirements. In order to be able to follow along with this guide, the following configuration options are strongly recommended: - In the "Configure sign-in experience" step: - Select at least "Email" in "Cognito user pool sign-in options". - In the "Configure sign-up experience" step: - Select at least "email" in "Required attributes". - Select "Allow Cognito to automatically send messages to verify and confirm". - In the "Configure message delivery" step: - Select "Send email with Cognito" in "Email". - In the "Integrate your app" step: - Select "Use the Cognito Hosted UI" in "Hosted authentication pages". - Select "Use a Cognito domain" in "Domain" and set a name for it. - Select "Public client" in "Initial app client". - Select "Don't generate a client secret" in "Initial app client". - Under "Allowed callback URLs" in "Initial app client", set the URL of your web application. - Under "Advanced app client settings" in "Initial app client", ensure "Authorization code grant" is selected. - Under "Attribute read and write permissions" in "Initial app client", ensure "email" is selected for "Read". - Under "Attribute read and write permissions" in "Initial app client", ensure "email_verified" is selected for "Read". > [!IMPORTANT] > These suggested configuration options may not provide the strongest security or match your specific requirements, but will ensure that you are able to follow with this guide to understand the simplest example of an integration between SurrealDB and AWS Cognito, which you should later update to meet your requirements. Changes in these options may require changes in the other steps outlined in this guide. Other configuration options can be left with their default values or may be changed to fit your requirements. Once you user pool has been created, open it in the AWS Console and take note of the following values: - "User pool ID" from the "User pool overview" section in the main user pool view. - "Cognito domain" from the "Configuration for all app clients" section under the "App integration" tab. - "Client ID" from the only client in the "App client list" under the "App integration" tab. ### Creating a pre-token generation lambda Cognito is now ready to perform authentication and issue tokens for your application. However, SurrealDB expects these tokens to contain some specific claims. Cognito allows modifying token claims through [pre-token generation lambda](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-pre-token-generation.html#cognito-user-pools-lambda-trigger-syntax-pre-token-generation) triggers. Specifically, we will be [adding custom claims](https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-pre-token-generation.html#aws-lambda-triggers-pre-token-generation-example-2) to the token. To create the trigger, just [create an AWS Lambda function](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html#getting-started-create-function) with the following code: **Using DEFINE ACCESS** ```js const handler = async (event) => { event.response = { claimsOverrideDetails: { claimsToAddOrOverride: { ac: "cognito", // The access method that has been defined using DEFINE ACCESS. ns: "main", // The namespace selected when calling DEFINE ACCESS. db: "main", // The database selected when calling DEFINE ACCESS. }, }, }; return event; }; export { handler }; ``` **Using Scope and Token (1.x only)** ```js const handler = async (event) => { event.response = { claimsOverrideDetails: { claimsToAddOrOverride: { tk: "cognito", // The name of the token given when defining it with DEFINE TOKEN. sc: "user", // The scope that the token has been defined for \ with DEFINE TOKEN. ns: "main", // The namespace selected when calling DEFINE \ TOKEN. db: "main", // The database selected when calling DEFINE \ TOKEN. }, }, }; return event; }; export { handler }; ``` Note that, in order to use the suggested code, the function must be configured as "Node.js 20.x" or equivalent. After the Lambda trigger has been created, we will need to [associate it with our client integration](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.html). Visit your user pool in Cognito and, under the "User pool properties" tab, in the "Lambda triggers" section, click on "Add Lambda trigger". For this guide, the trigger type will be "Authentication", specifically a "Pre token generation trigger". Select the function that you created. ### Creating a test user Depending on how you configured your user pool, users will be able to register by different means. To ensure this guide can be followed easily, we will create a test user that to authenticate in our web application. Open your user pool and click on the "Create user" button under the "Users" section in the main user pool view. Provide an email address for that user and check "Mark email address as verified" to avoid having to do so manaually. Set a password that matches the requirements that you configured. ## Configuring SurrealDB **Using DEFINE ACCESS** ### Defining permissions and fields in SurrealDB For this simple example, we will create a single table named “user”, where any user that authenticates through AWS Cognito using your application will be granted complete permissions over their data. For this to work as intended, we will need to ensure that the email address is unique between users and that users are granted permissions to access their own record as long as they authenticated with the access method that we will define. ```surql DEFINE TABLE user SCHEMAFULL -- Authorised users can select, update, delete and create user records. -- Records that do not match the permissions will not be modified nor returned. PERMISSIONS FOR select, update, delete, create WHERE -- The access method must match the method that we will define. $access = "cognito" -- The record identifier must match that of the authenticated user. AND id = $auth ; -- In this example, we will use the email as the primary identifier for a user. DEFINE INDEX email ON user FIELDS email UNIQUE; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); -- We define some other information present in the token that we want to store. DEFINE FIELD cognito_username ON user TYPE string; ``` ### Defining a token verification method in SurrealDB Next, we should configure SurrealDB so that it can verify tokens sent to it through the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) via the “Authorization” header or through any of the [SDKs](/docs/languages.md) via the “Authenticate” methods. To do that, we will leverage the [JWKS support in SurrealDB](/docs/reference/query-language/statements/define/token.md#json-web-key-set-jwks) in order to define a token verification mechanism pointing to a JWKS object served by AWS for your Cognito user pool. This JWKS object can be found in an endpoint [build from your AWS region and user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html) in the format `https://cognito-idp..amazonaws.com//.well-known/jwks.json`. Pointing to a remote JWKS object ensures that token verification will work even in the case that AWS rotates their encryption keys and that tokens signed with revoked keys will no longer be accepted by SurrealDB. To understand how revocation is handled by SurrealDB, read the [JSON Web Key Set documentation](/docs/reference/query-language/statements/define/token.md#json-web-key-set-jwks) under `DEFINE ACCESS ... TYPE JWT`. We will also use the [`AUTHENTICATE`](/docs/reference/query-language/statements/define/access/record.md#with-authenticate-clause) clause in order to check that any necessary token claims have the expected values before returning the user matching the email address provided by AWS Cognito. This is required because AWS Cognito has no knowledge of the record identifiers that are used in SurrealDB, so we need to use an identifier that can actually be provided by AWS Cognito in order to retrieve the corresponding record user. The following queries will create the required resources to authenticate a token for a record using JWKS: ```surql -- Specify the namespace and database that will be used. -- These values should match the custom claims that we configured before. USE NS main DB main; -- Define the public key to verify tokens issued by your AWS Cognito user pool. -- The name of the access method should match the custom claim that we configured before. DEFINE ACCESS cognito ON DATABASE TYPE RECORD -- We verify the token using the public keys hosted by AWS. WITH JWT URL "https://cognito-idp..amazonaws.com//.well-known/jwks.json" -- We check the token claims and map the email address to a record user. AUTHENTICATE { IF ( -- The issuer claim must match the URL of your AWS Cognito user pool. $token.iss = "https://cognito-idp..amazonaws.com/" -- The audience claim must match you AWS Cognito Client ID. AND $token.aud = "" -- The email address in the token must be verified as belonging to the user. AND $token.email_verified = true ) { -- We return the only user that matches the email address claim found in the token. RETURN SELECT * FROM user WHERE email = $token.email } } ; ``` In the example above, replace the placeholders with values applicable to your Cognito user pool. It is important to know that [validating the issuer and audience of the token is a requirement of AWS Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html), other providers may require validating additional claims to ensure that the token is being used as intended. In order to allow SurrealDB to establish a connection with AWS Cognito to download the JWKS object, you will require running it with the network capability. For the strongest security, provide your specific Cognito user pool domain when starting SurrealDB with `--allow-net`. For example: `--allow-net cognito-idp.eu-west-1.amazonaws.com`. **Using Scope and Token (1.x only)** ### Defining a token verification method in SurrealDB Next, we should configure SurrealDB so that it can verify tokens sent to it through the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) via the “Authorization” header or through any of the [SDKs](/docs/languages.md) via the “Authenticate” methods. To do that, we will leverage the [JWKS support in SurrealDB](/docs/reference/query-language/statements/define/token.md#json-web-key-set-jwks) in order to define a token verification mechanism pointing to a JWKS object served by AWS for your Cognito user pool. This JWKS object can be found in an endpoint [build from your AWS region and user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html) in the format `https://cognito-idp..amazonaws.com//.well-known/jwks.json`. Pointing to a remote JWKS object ensures that token verification will work even in the case that AWS rotates their encryption keys and that tokens signed with revoked keys will no longer be accepted by SurrealDB. To understand how revocation is handled by SurrealDB, read the [JSON Web Key Set documentation](/docs/reference/query-language/statements/define/token.md#json-web-key-set-jwks) under `DEFINE TOKEN`. The following queries will create the required resources to authenticate a token for a scope using JWKS: ```surql -- Specify the namespace and database that will be used. -- These values should match the custom claims that we configured before. USE NS main DB main; -- Define the scope where the token will be used. -- The name of the scope should match the custom claim that we configured before. DEFINE SCOPE user; -- Define the public key to verify tokens issued by your AWS Cognito user pool. -- The name of the token should match the custom claim that we configured before. DEFINE TOKEN cognito ON SCOPE user TYPE JWKS VALUE "https://cognito-idp..amazonaws.com//.well-known/jwks.json"; ; ``` In the example above, replace the placeholders with values applicable to your Cognito user pool. In order to allow SurrealDB to establish a connection with AWS Cognito to download the JWKS object, you will require running it with the network capability. For the strongest security, provide your specific Cognito user pool domain when starting SurrealDB with `--allow-net`. For example: `--allow-net cognito-idp.eu-west-1.amazonaws.com`. ### Defining authorization criteria in SurrealDB For this example, we will create a single table named `user`, where any user that authenticates through AWS Cognito using your application with a verified email address will be able to register, view and update their data. For this to work as intended, we will need to verify some information in the token claims. ```surql DEFINE TABLE user SCHEMAFULL -- Authorised users can select, update, delete and create user records. -- Records that do not match the permissions will not be modified nor returned. PERMISSIONS FOR select, update, delete, create WHERE -- The token scope must match the scope that we defined. -- The name of the scope should match the scope that we defined before. $scope = "user" -- The issuer claim must match the URL of your AWS Cognito user pool. AND $token.iss = "https://cognito-idp..amazonaws.com/" -- The audience claim must match you AWS Cognito Client ID. AND $token.aud = "" -- The email claim must match the email of the user being queried. AND email = $token.email -- The email must be verified as belonging to the user. AND $token.email_verified = true ; -- In this example, we will use the email as the primary identifier for a user. DEFINE INDEX email ON user FIELDS email UNIQUE; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); -- We define some other information present in the token that we want to store. DEFINE FIELD cognito_username ON user TYPE string; ``` It is important to know that [validating the issuer and audience of the token is a requirement of AWS Cognito](https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html), other providers may require validating additional claims to ensure that the token is being used as intended. It is also important to note that the `$auth` variable accessible from SurrealQL will not contain any values in this case, as it requires the `id` claim to be added to the JWT, containing the value of the identifier of a SurrealDB record. For the current example, the `$auth` variable will not be necessary. ## Creating a simple web application For this guide, we will create a simple client-side web application that will direct the user to log in using the Cognito Hosted UI, redirect the user back your our application with a query parameter containing an authorization code and exchange that code to Cognito for the user identity token which we will use to authenticate with SurrealDB. The application will create or update a SurrealDB user using data from the token claims. This user will later be able visit our web application and retrieve their information from SurrealDB. > [!NOTE] > As most other authentication providers using OpenID Connect (OIDC), AWS Cognito issues both identity and access tokens. In this example, we will be using the identity token as it can include custom claims (which are required by SurrealDB) by default. It is important to note that identity tokens should only be used to assert identity claims as opposed to access claims. In this case, we trust the identity token to provide information about the indentity of the user. If we wanted the token to contain authorization claims (e.g. OAuth scopes) we should instead rely on the access token. Customising access token claims has been recently supported by AWS and requires enabling advanced security features. This guide makes the deliberate decision of using the identity token to simplify the process. We have developed an [example application](https://github.com/surrealdb/examples/tree/main/aws-cognito) that uses plain JavaScript to authenticate with Cognito using basic HTTP requests against the [login endpoint](https://docs.aws.amazon.com/cognito/latest/developerguide/login-endpoint.html) of the Cognito Hosted UI and the Cognito [token endpoint](https://docs.aws.amazon.com/cognito/latest/developerguide/token-endpoint.html). For the purposes of following this guide, we recommend using our example code. However, keep in mind that this code aims to be as simple as possible and is not suitable for production applications. Alternatively, you can develop this application yourself using the [Cognito SDK](https://docs.aws.amazon.com/cognito/latest/developerguide/service_code_examples_cognito-identity-provider.html) or the new [Amplify SDK](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-integrate-apps.html#cognito-integrate-apps-amplify). When using our example code, you will only need to update the `config.json` file with the values that you saved after creating your user pool and run the web application using the `start.sh` script or any equivalent web server. Once in the web application, clicking the "Log in" button should take you to the Cognito Hosted UI to log in with your test user, after which you should be redirected back to your web application. For this to work, the URL of your web application (without a trailing slash) should be present in the "Allowed callback URLs" list that you defined when configuring the user pool client. After redirection, the web application will show some information about the authenticated user and attempt to create it in SurrealDB via the configured endpoint. After the user is created, subsequent logins will retrieve its information from SurrealDB and display it in the web application. If that is not the case, use the developer console of your browser together with the SurrealDB logs (running with `--log trace` during the debugging for maximum verbosity) to understand why. Once the example web application is working, you can inspect the simple code under `app.js` to understand how. ## Annex In this section, we will provide a few examples of how to configure the application to work with AWS Cognito. ## Example single page application You can view and download a minimal example of an web application using AWS Cognito and SurrealDB [in the AWS cognito example project](https://github.com/surrealdb/examples/tree/main/aws-cognito). --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/build-a-real-time-presence-app # Build a real-time presence app In this guide, you'll learn how to implement real-time presence tracking that can be integrated in any application including chat applications, multiplayer games, and more. SurrealDB offers various features including a real-time notification mechanism called Live Queries. This feature allows you to subscribe to changes in your database and receive notifications in real time. In this guide, you’ll learn how to implement real-time presence tracking that can be integrated in any application including chat applications, multiplayer games, and more. The demo project is available on [GitHub](https://github.com/Odonno/surrealdb-presence-demo) using the following tech stack: - [SurrealDB](/) our database - [React](https://react.dev/) for the frontend with [Vite](https://vite.dev/) as a bundler - [TanStack Query](https://tanstack.com/query) to fetch and store data retrieved from the database - [Tailwind](https://tailwindcss.com/) for styling our application ## Hello, are you there? ![room-users.png](https://github.com/Odonno/surrealdb-presence-demo/raw/main/img/room-users.png) The demo application looks like a simple chat application with basic features allowing users to join a room and send messages. The application also displays the number of users in the room and their presence status. The presence status is updated in real time using SurrealDB Live Queries. The mechanism used to detect the presence of a user is a periodic ping sent by the client to the server. The following configuration is used to setup the project: - Signal user presence in room periodically every **10 seconds** - Display status badge based on idle time - 🟩 < **2 minutes** of inactivity - 🟨 < **10 minutes** of inactivity - ⬜ beyond **10 minutes** of inactivity Note: Those values are completely arbitrary and can be changed to fit your needs. ## Architecture This project is using the following folder structure: - `/schemas` - list of SurrealDB tables - `/events` - list of SurrealDB events - `/migrations` - list of db migrations that will be automatically applied - `/src` - `/api` - TanStack query hooks - `/components` - `/constants` - `/contexts` - Theme and SurrealDB providers - `/hooks` - custom React hooks - `/lib` - functions and app models - `/mutations` - surql query files to create or update data, using SurrealDB events - `/pages` - `/queries` - surql query files to query the database, using SurrealDB tables ## Prerequisites Before you begin this tutorial you’ll need the following: a. [SurrealDB](/docs/running/installation.md) installed on your machine (Make sure you upgrade to the latest version if you already have SurrealDB installed on your machine) b. The [Bun](https://bun.sh/docs/installation) runtime c. Optional: [surrealdb-migrations](https://github.com/Odonno/surrealdb-migrations) to manage and automate the deployment of your SurrealDB schema d. A basic understanding of React and TanStack Query ## Step 0: Setup the project Once everything is installed, clone the project and navigate to it. Then: 1. Start a new SurrealDB instance locally ```bash surreal start --log debug --user root --pass secret \ --allow-guests ``` 2. Apply migrations to the database Either apply schema and migrations automatically by running the following command: ```bash surrealdb-migrations apply ``` Or manually apply each file stored in the following folders: - `schemas` - `events` - `migrations` 3. Install dependencies and run the web app ```bash bun install bun start ``` 4. Launch your web browser on the generated url (eg. http://localhost:5173/) and play with the app: create new accounts, join rooms, leave rooms, etc.. ## Step 1: Authentication For users to join rooms and interact with the app, we need users. And thankfully, SurrealDB also offers authentication mechanism. We will need some basic authentication such as a registration form, a login form, and a way to sign out. The `user` table will look like this: ```surql DEFINE TABLE user SCHEMALESS; DEFINE FIELD username ON user TYPE string; DEFINE FIELD email ON user TYPE string PERMISSIONS FOR select NONE; DEFINE FIELD passcode ON user TYPE string PERMISSIONS FOR select NONE; DEFINE FIELD registered_at ON user TYPE datetime DEFAULT time::now(); DEFINE FIELD avatar ON user TYPE option; DEFINE INDEX unique_username ON user COLUMNS username UNIQUE; DEFINE INDEX unique_email ON user COLUMNS email UNIQUE; DEFINE ACCESS user_access ON DATABASE TYPE RECORD SIGNUP ( CREATE user SET username = $username, email = $email, avatar = "https://www.gravatar.com/avatar/" + crypto::md5($email) + "?d=identicon", passcode = fn::create_passcode($email) ) SIGNIN ( SELECT * FROM user WHERE email = $email AND passcode = $passcode ); ``` We can then create a [login form](https://github.com/Odonno/surrealdb-presence-demo/blob/main/src/components/SignInPopover.tsx) and a [sign up dialog](https://github.com/Odonno/surrealdb-presence-demo/blob/main/src/components/SignUpDialog.tsx). > [!NOTE] >We use a passcode to ensure a minimum security authentication. This passcode is generated by the signup function and is stored in the user record. The signin function checks if the passcode is correct. This is for a demonstration purpose only. In a real-world application, you should use a more secure authentication mechanism. ## Step 2: Display room information We first start by writing the query that will be used to display room information. ```surql SELECT id, name, created_at, ( RETURN $auth.id IN $parent.users ) AS is_in_room, array::len( SELECT id FROM $parent.users WHERE time::now() - (( SELECT VALUE at FROM last_presence WHERE user == $parent.id )[0] ?? time::from_secs(0)) < 5m ) AS number_of_active_users, owner.id != $auth.id AS can_leave FROM room ORDER BY created_at DESC; ``` Each `*.surql` query file can then be linked to a TanStack Query query, like this one: ```ts import roomsQuery from "@/queries/rooms.surql?raw"; // importing raw text file query written in SurrealQL export const useRooms = () => { const dbClient = useSurrealDbClient(); const getRoomsAsync = async () => { const response = await dbClient.query<[Room[]]>(roomsQuery); return response[0]; }; return useQuery({ ...queryKeys.rooms.list, queryFn: getRoomsAsync, }); }; ``` Here, we will expose a new hook that encapsulates a `useQuery` hook underneath. The same can be done with TanStack query mutations. ## Step 3: Signal user presence Signaling a presence from the client is almost too easy. We use the `usePageVisibility` hook to ensure the user is still looking at our app, meaning he did not put the app in the background. Note: this hook is using the [Page Visibility API](https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API) underneath. And if the page is visible, we use the `useInterval` hook to trigger the signal event every 10 seconds via a TanStack Query mutation . ```ts const SIGNAL_PRESENCE_INTERVAL = 10 * SECOND; const SignalPresence = () => { const isPageVisible = usePageVisibility(); const canSignalPresence = isPageVisible; const dbClient = useSurrealDbClient(); const signalPresence = useMutation({ mutationKey: ["signalPresence"], mutationFn: async () => { await dbClient.query(signalPresenceQuery); }, }); useInterval( () => { signalPresence.mutate(); }, canSignalPresence ? SIGNAL_PRESENCE_INTERVAL : null, ); useEffect(() => { if (canSignalPresence) { signalPresence.mutate(); } }, [isPageVisible]); return null; }; ``` The mutation will trigger the following SurrealDB event: ```surql DEFINE EVENT signal_presence ON TABLE signal_presence WHEN $event == "CREATE" THEN ( CREATE presence SET user = $auth.id ); ``` The `presence` table will store every presence detection event triggered by our application. ## Step 4: Display real-time presence status Storing all the presence detection triggered is interesting but it's not very useful. We want to display the presence status of each user in real time efficiently. To do so, we will create a new table called `last_presence` to retrieve the last presence detection event for each user. We will then be able to use this table to display the presence status of each user in real time. ```surql DEFINE TABLE last_presence AS SELECT user, time::max(updated_at) AS at FROM presence GROUP BY user; ``` Displaying the presence status badge of a user is quite simple. We just need to retrieve the last presence detection event for the user and display the presence status badge based on the time difference between the last presence detection event and the current time. ```ts const GREEN_STATUS_THRESHOLD = 2 * MINUTE; const ORANGE_STATUS_THRESHOLD = 10 * MINUTE; const getPresenceBackgroundClass = ( lastPresenceDate: Date | undefined, now: Date, ) => { if (!lastPresenceDate) { return "bg-gray-500"; } const diffTimeInSeconds = now.getTime() - lastPresenceDate.getTime(); if (diffTimeInSeconds < GREEN_STATUS_THRESHOLD) { return "bg-green-500"; } if (diffTimeInSeconds < ORANGE_STATUS_THRESHOLD) { return "bg-yellow-500"; } return "bg-gray-500"; }; export type PresenceProps = { lastPresenceDate?: Date; className?: string; }; const Presence = (props: PresenceProps) => { const { lastPresenceDate, className } = props; const [now, setNow] = useState(new Date()); useInterval(() => { setNow(new Date()); }, SECOND); const bgClass = getPresenceBackgroundClass(lastPresenceDate, now); return ( ); }; ``` > [!NOTE] > We trigger a re-render every second to update the presence status. This may not be the most efficient way to do it, but it's enough for this demo. We could use a more specific interval, or use a more efficient way to notify the client when the presence status changes, but it's not the point of this demo. Now, this component can be easily integrated into another components, like this one: ```ts const CurrentUserPresence = () => { const lastPresenceDate = useRealtimeCurrentUserPresence(); return ( ); }; ``` For reference, we query the last presence of the current user with this query: ```surql SELECT VALUE at FROM last_presence WHERE user == $auth.id; ``` The `useRealtimeCurrentUserPresence` hook retrieves the last presence of the current user and is composed of multiple hooks: - `useCurrentUserPresence` - the base hook to retrieve the current user presence (without real-time capability) - `useCurrentUserPresenceLive` - the base hook that is notified by each changes in the database (pure real-time capability) - `useRealtimeCurrentUserPresence` - the hook itself that combines both previous hooks (current value + upcoming changes) ```ts const useCurrentUserPresence = () => { const dbClient = useSurrealDbClient(); const getCurrentUserPresenceAsync = async () => { const response = await dbClient.query<[string]>(currentUserPresenceQuery); if (!response?.[0]) { throw new Error(); } return new Date(response[0]); }; return useQuery({ ...queryKeys.users.current._ctx.presence, queryFn: getCurrentUserPresenceAsync, }); }; const useCurrentUserPresenceLive = (enabled: boolean) => { const dbClient = useSurrealDbClient(); const getCurrentUserPresenceLiveAsync = async () => { const query = `LIVE ${currentUserPresenceQuery}`; const response = await dbClient.query<[Uuid]>(query); return response?.[0]; }; return useQuery({ ...queryKeys.users.current._ctx.presence._ctx.live, queryFn: getCurrentUserPresenceLiveAsync, enabled, }); }; export const useRealtimeCurrentUserPresence = () => { const queryClient = useQueryClient(); const { data: lastPresenceDate, isSuccess } = useCurrentUserPresence(); const { data: liveQueryUuid } = useCurrentUserPresenceLive(isSuccess); useLiveQuery({ queryUuid: liveQueryUuid, callback: (action, result) => { if (action === "CREATE" || action === "UPDATE") { queryClient.setQueryData( queryKeys.users.current._ctx.presence.queryKey, new Date(result as unknown as string), ); } }, enabled: Boolean(liveQueryUuid), }); useMount(() => { return () => { queryClient.invalidateQueries({ queryKey: queryKeys.users.current._ctx.presence.queryKey, }); }; }); return lastPresenceDate; }; ``` ## Step 5: Refactoring with the `useLiveQuery` hook One can notice the presence of the `useLiveQuery` hook. This hook is a custom hook that we created to simplify then lifecycle of a Live Query. It automatically subscribe to the Live Query when enabled and it will kill the Live Query on cleanup (when the component is unmounted). Correctly cleaning Live Queries would prevent from any memory leak. ```typescript export type UseLiveQueryProps< T extends Record = Record, > = { queryUuid: Uuid | undefined; callback: LiveHandler; enabled?: boolean; }; export const useLiveQuery = ({ queryUuid, callback, enabled = true, }: UseLiveQueryProps) => { const dbClient = useSurrealDbClient(); useEffect(() => { if (enabled && !!queryUuid) { const runLiveQuery = async () => { await dbClient.subscribeLive(queryUuid, callback); }; const clearLiveQuery = async () => { await dbClient.kill(queryUuid); }; const handleBeforeUnload = () => { clearLiveQuery(); }; window.addEventListener("beforeunload", handleBeforeUnload); runLiveQuery(); return () => { clearLiveQuery(); window.removeEventListener("beforeunload", handleBeforeUnload); }; } }, [queryUuid, enabled]); }; ``` ## Bonus: the simulator Being alone is not really that fun, isn't it? We can't really test the real-time presence feature without having multiple users connected to the same room. That's why we can use the simulator script built to generate some fake users that will interact with the app while active. You can run the following command to start the simulator: ```bash bun run .\simulator.ts ``` And then let's enjoy the nature of randomness to make the app alive! ## Resources - [Live demo](https://surrealdb-presence-demo.vercel.app/) - The live demo of the application - [GitHub repository](https://github.com/Odonno/surrealdb-presence-demo) - The GitHub repository of the application - [SurrealDB Live Queries](/docs/reference/query-language/statements/live-select.md) - The SurrealDB Live Queries documentation - [SurrealQL Documentation](/docs/reference/query-language.md) - The SurrealQL documentation - [Javascript SDK documentation](/docs/reference/javascript.md) - The Javascript SDK documentation --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/build-an-ai-agent # Build an AI agent with Python In this tutorial, you will learn three ways to build an AI agent in Python that uses SurrealDB. This page details a number of ways that you can build an AI agent. As the number of frameworks and programming languages you can choose to do so is quite extensive, this page details three Python frameworks to get you started. ## Setup You can run SurrealDB locally or start with a [free SurrealDB Cloud account](/docs/manage/instances.md). For local, you have two options: 1. [Install SurrealDB](/docs/running/installation.md) and run [SurrealDB](/docs/running/in-memory.md). Run in-memory with: ```sh surreal start -u root -p secret ``` 2. [Run with Docker](/docs/running/docker.md). ```sh docker run --rm \ --pull always \ -p 8000:8000 \ surrealdb/surrealdb:latest \ start \ --user root \ --pass secret ``` ## Frameworks Choose one of the following frameworks to view the tutorial and sample code. **Pydantic AI** ## Getting started This is a simple RAG application that uses Pydantic AI and embedded SurrealDB. The integration is done by providing the agent with a custom retrieval tool, which takes a search query, executes a SurrealDB vector-search query, and returns the results. **To run the example:** Set up your OpenAI API key: ```bash export OPENAI_API_KEY=your-api-key ``` Or, store it in a .env file and add `--env-file .env` to your `uv run` commands. Build the vector store: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb build ``` Ask the agent a question: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb search \ "How do I register a function as a custom tool for my agent?" ``` Or use the web UI: ```bash uv run --env-file .env -m pydantic_ai_examples.rag_surrealdb web ``` ### Code ```python from __future__ import annotations as _annotations import asyncio import re import sys import unicodedata from collections.abc import Sequence from contextlib import asynccontextmanager from dataclasses import dataclass from pathlib import Path from typing import TypeVar import httpx import logfire import uvicorn from pydantic import BaseModel, TypeAdapter from surrealdb import ( AsyncEmbeddedSurrealConnection, AsyncHttpSurrealConnection, AsyncSurreal, AsyncWsSurrealConnection, RecordID, Value, ) from typing_extensions import AsyncGenerator from pydantic_ai import Agent, Embedder SurrealConn = ( AsyncWsSurrealConnection | AsyncHttpSurrealConnection | AsyncEmbeddedSurrealConnection ) # 'if-token-present' means nothing will be sent (and the example will work) if you don't have logfire configured logfire.configure(send_to_logfire='if-token-present') logfire.instrument_pydantic_ai() logfire.instrument_surrealdb() THIS_DIR = Path(__file__).parent SURREALDB_NS = 'pydantic_ai_examples' SURREALDB_DB = 'rag_surrealdb' SURREALDB_USER = 'root' SURREALDB_PASS = 'secret' embedder = Embedder('openai:text-embedding-3-small') agent = Agent('openai:gpt-5.2') RecordType = TypeVar('RecordType') class RetrievalQueryResult(BaseModel): url: str title: str content: str dist: float async def query( conn: SurrealConn, query_: str, vars_: dict[str, Value], record_type: type[RecordType], ) -> list[RecordType]: result = await conn.query(query_, vars_) result_ta = TypeAdapter(list[record_type]) rows = result_ta.validate_python(result) return rows @agent.tool_plain async def retrieve(search_query: str) -> str: """Retrieve documentation sections based on a search query. Args: search_query: The search query. """ with logfire.span( 'create embedding for {search_query=}', search_query=search_query ): result = await embedder.embed_query(search_query) embedding = result.embeddings # Embedder method guarantees there's one item here embedding_vector = embedding[0] # SurrealDB vector search using HNSW index async with database_connect(False) as db: rows = await query( db, """ SELECT url, title, content, vector::distance::knn() AS dist FROM doc_sections WHERE embedding <|8, 40|> $vector ORDER BY dist ASC """, {'vector': list(embedding_vector)}, RetrievalQueryResult, ) return '\n\n'.join( f'# {row.title}\nDocumentation URL:{row.url}\n\n{row.content}' for row in rows ) async def run_agent(question: str): """Entry point to run the agent and perform RAG based question answering.""" logfire.info('Asking "{question}"', question=question) answer = await agent.run(question) print(answer.output) # Web chat UI app = agent.to_web() ####################################################### # The rest of this file is dedicated to preparing the # # search database, and some utilities. # ####################################################### # JSON document from # https://gist.github.com/samuelcolvin/4b5bb9bb163b1122ff17e29e48c10992 DOCS_JSON = ( 'https://gist.githubusercontent.com/' 'samuelcolvin/4b5bb9bb163b1122ff17e29e48c10992/raw/' '80c5925c42f1442c24963aaf5eb1a324d47afe95/logfire_docs.json' ) def build_doc_rec_id(url: str) -> RecordID: return RecordID('doc_sections', slugify(url, '_')) async def build_search_db(): """Build the search database.""" async with httpx.AsyncClient() as client: response = await client.get(DOCS_JSON) response.raise_for_status() sections = sections_ta.validate_json(response.content) async with database_connect(True) as db: missing_sections: list[DocsSection] = [] for section in sections: url = section.url() record_id = build_doc_rec_id(url) existing = await db.select(record_id) if existing: logfire.info('Skipping {url=}', url=url) continue missing_sections.append(section) if missing_sections: with logfire.span('create embeddings'): result = await embedder.embed_documents( [section.embedding_content() for section in missing_sections] ) embeddings = result.embeddings for section, embedding_vector in zip( missing_sections, embeddings, strict=True ): await insert_doc_section(db, section, embedding_vector) else: logfire.info('All documents already exist; skipping embedding generation') async def insert_doc_section( db: SurrealConn, section: DocsSection, embedding_vector: Sequence[float], ) -> None: url = section.url() record_id = build_doc_rec_id(url) # Create record with embedding, using record ID directly res = await db.create( record_id, { 'url': url, 'title': section.title, 'content': section.content, 'embedding': list(embedding_vector), }, ) if not isinstance(res, dict): raise ValueError(f'Unexpected response from database: {res}') @dataclass class DocsSection: id: int parent: int | None path: str level: int title: str content: str def url(self) -> str: url_path = re.sub(r'\.md$', '', self.path) return ( f'https://logfire.pydantic.dev/docs/{url_path}/#{slugify(self.title, "-")}' ) def embedding_content(self) -> str: return '\n\n'.join((f'path: {self.path}', f'title: {self.title}', self.content)) sections_ta = TypeAdapter(list[DocsSection]) @asynccontextmanager async def database_connect( create_db: bool = False, ) -> AsyncGenerator[SurrealConn, None]: # Running SurrealDB embedded db_path = THIS_DIR / f'.{SURREALDB_DB}' db_url = f'file://{db_path}' requires_auth = False # Running SurrealDB in a separate process, connect with URL # db_url = 'ws://localhost:8000/rpc' # requires_auth = True async with AsyncSurreal(db_url) as db: # Sign in to the database if requires_auth: await db.signin({'username': SURREALDB_USER, 'password': SURREALDB_PASS}) # Set namespace and database await db.use(SURREALDB_NS, SURREALDB_DB) # Initialize schema if creating database if create_db: with logfire.span('create schema'): await db.query(DB_SCHEMA) yield db DB_SCHEMA = """ DEFINE TABLE doc_sections SCHEMALESS; DEFINE FIELD embedding ON doc_sections TYPE array; DEFINE INDEX hnsw_idx_doc_sections ON doc_sections FIELDS embedding HNSW DIMENSION 1536 DIST COSINE TYPE F32; """ def slugify(value: str, separator: str, unicode: bool = False) -> str: """Slugify a string, to make it URL friendly.""" # Taken unchanged from https://github.com/Python-Markdown/markdown/blob/3.7/markdown/extensions/toc.py#L38 if not unicode: # Replace Extended Latin characters with ASCII, i.e. `žlutý` => `zluty` value = unicodedata.normalize('NFKD', value) value = value.encode('ascii', 'ignore').decode('ascii') value = re.sub(r'[^\w\s-]', '', value).strip().lower() return re.sub(rf'[{separator}\s]+', separator, value) if __name__ == '__main__': action = sys.argv[1] if len(sys.argv) > 1 else None if action == 'build': asyncio.run(build_search_db()) elif action == 'search': if len(sys.argv) == 3: q = sys.argv[2] else: q = 'How do I configure logfire to work with FastAPI?' asyncio.run(run_agent(q)) elif action == 'web': uvicorn.run(app, host='127.0.0.1', port=7932) else: print( 'uv run --extra examples -m pydantic_ai_examples.rag_surrealdb build|search|web', file=sys.stderr, ) sys.exit(1) ``` **LangChain** ## Getting started Begin by installing the following dependencies. ```bash # -- Using pip pip install -U langchain-surrealdb langchain_ollama surrealdb # -- Using poetry poetry add langchain-surrealdb langchain_ollama surrealdb # -- Using uv uv add --upgrade langchain-surrealdb langchain_ollama surrealdb ``` * `surrealdb` → [SurrealDB Python SDK](/docs/reference/python/) * `langchain-surrealdb` → houses `SurrealDBVectorStore` * `langchain_ollama`, `langchain-openai` (or HF, Cohere, etc.) → embeddings Once this is done, you can create a vector store, add documents with embeddings, and do a similarity search. ```python from langchain_core.documents import Document from langchain_surrealdb.vectorstores import SurrealDBVectorStore from langchain_ollama import OllamaEmbeddings from surrealdb import Surreal conn = Surreal("ws://localhost:8000/rpc") conn.signin({"username": "root", "password": "secret"}) conn.use("langchain", "demo") vector_store = SurrealDBVectorStore(OllamaEmbeddings(model="llama3.2"), conn) doc_1 = Document(page_content="foo", metadata={"source": "https://surrealdb.com"}) doc_2 = Document(page_content="SurrealDB", metadata={"source": "https://surrealdb.com"}) vector_store.add_documents(documents=[doc_1, doc_2], ids=["1", "2"]) results = vector_store.similarity_search_with_score( query="surreal", k=1, custom_filter={"source": "https://surrealdb.com"} ) for doc, score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") ``` Under the hood the helper will: 1. Create table **`documents`** (if it doesn’t exist). 2. Add an **HNSW** index with the correct dimensionality, using cosine distance and `F32` vectors. 3. Insert each text with its freshly generated embedding. ## Similarity search ```python query = "How do I enable vector search in SurrealDB?" docs = vector_store.similarity_search( query=query, k=1, custom_filter={"source": "https://surrealdb.com"} ) for doc in results: print(f"{doc.page_content} [{doc.metadata}]") ``` ```text The Vector Search feature of SurrealDB... [{'source': \ 'https://surrealdb.com'}] ``` If you want to get the score with the results, use `similarity_search_with_score` instead. You can also transform the vector store into a retriever for easier usage in your chains. ```python query = "How do I enable vector search in SurrealDB?" docs = vector_store.similarity_search( retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={"k": 1, "lambda_mult": 0.5} ) retriever.invoke(query) ``` ```text [Document(id='4', metadata={'source': 'https://surrealdb.com'}, \ page_content='The Vector Search feature of SurrealDB...')] ``` **Agno** ## Getting started ```python from agno.agent import Agent from agno.embedder.openai import OpenAIEmbedder from agno.knowledge.pdf_url import PDFUrlKnowledgeBase from agno.vectordb.surrealdb import SurrealDb from surrealdb import Surreal # SurrealDB connection parameters SURREALDB_URL = "ws://localhost:8000" SURREALDB_USER = "root" SURREALDB_PASSWORD = "secret" SURREALDB_NAMESPACE = "main" SURREALDB_DATABASE = "main" # Create a client client = Surreal(url=SURREALDB_URL) client.signin({"username": SURREALDB_USER, "password": SURREALDB_PASSWORD}) client.use(namespace=SURREALDB_NAMESPACE, database=SURREALDB_DATABASE) surrealdb = SurrealDb( client=client, collection="recipes", # Collection name for storing documents efc=150, # HNSW construction time/accuracy trade-off m=12, # HNSW max number of connections per element search_ef=40, # HNSW search time/accuracy trade-off ) def sync_demo(): """Demonstrate synchronous usage of SurrealDb""" knowledge_base = PDFUrlKnowledgeBase( urls=["https://agno-public.s3.amazonaws.com/recipes/ThaiRecipes.pdf"], vector_db=surrealdb, embedder=OpenAIEmbedder(), ) # Load data synchronously knowledge_base.load(recreate=True) # Create agent and query synchronously agent = Agent(knowledge=knowledge_base, show_tool_calls=True) agent.print_response( "What are the 3 categories of Thai SELECT is given to restaurants overseas?", markdown=True, ) if __name__ == "__main__": # Run synchronous demo print("Running synchronous demo...") sync_demo() ``` --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/connect-via-ngrok # Connect to SurrealDB via ngrok Expose a local SurrealDB instance to the internet with ngrok for remote development and testing. This guide will walk you through connecting a local SurrealDB instance to the internet using ngrok, making it accessible remotely. [Ngrok](https://ngrok.com/) is a cross-platform application that allows developers to expose their local web servers to the internet. It hosts a local web server on its own sub-domain and makes your local development box available on the internet through Tunneling. This setup is beneficial for remote development and quick testing of applications built locally on SurrealDB. ## Prerequisites This guide assumes the following: - You have [SurrealDB installed](/docs/running/installation.md) and a [fresh instance of SurrealDB running.](/docs/running/overview.md) - Downloaded [ngrok](https://ngrok.com/download) to be used as a tunnel. > [!IMPORTANT] > Tunneling refers to the process of using a network protocol to encapsulate a different payload protocol, enabling data to pass securely through a network or the internet ## Steps There are two ways to connect to a SurrealDB instance via ngrok tunnel. ### Start SurrealDB instance Open your command line or terminal and run the following command to [start SurrealDB](/docs/running/overview.md). ```bash surreal start memory -A --user root --pass secret ``` We use the default username `root` and password `secret`. You can replace it with your own credentials if you have set them up. ### Set up ngrok tunnel Open another command line or terminal window (do not close the SurrealDB one) and run the following command to expose SurrealDB’s default port (8000) to the internet: ```bash ngrok http 8000 ``` Note the forwarding address provided by ngrok. For example, **`25f6-2402-e280-2189-38e-9c15-d08-2f83-779e.ngrok-free.app`**. > [!IMPORTANT] > Keep this address handy as we will use it in the next step. ### Connect and verify Connect using the forwarding address from ngrok, replacing **`[ngrok-address]`** with the address you noted earlier. You can open an interactive REPL: ```bash surreal sql --conn wss://[ngrok-address] --user root --pass secret --ns main --db main --pretty ``` Or run a one-shot check that [creates a record](/docs/reference/query-language/statements/create.md) and [selects it](/docs/reference/query-language/statements/select.md): **Bash** ```bash echo "CREATE registration SET full_name = 'John Doe', email = 'johndoe@gmail.com', address_line1 = 'Room number 1, Hogwarts', address_line2 = 'Near Diagon Alley', city = 'Hogwarts', country = 'England'; SELECT * FROM registration;" | surreal sql --conn wss://[ngrok-address] --user root --pass secret --ns main --db main --pretty --hide-welcome ``` **PowerShell** ```powershell "CREATE registration SET full_name = 'John Doe', email = 'johndoe@gmail.com', address_line1 = 'Room number 1, Hogwarts', address_line2 = 'Near Diagon Alley', city = 'Hogwarts', country = 'England'; SELECT * FROM registration;" | surreal sql --conn wss://[ngrok-address] --user root --pass secret --ns main --db main --pretty --hide-welcome ``` ## Conclusion In this guide, we have looked at how we can connect to a local instance of SurrealDB using tunnels like ngrok which can help with testing applications. ngrok provides a random or custom subdomain for your tunnel URL every time you start the tunnel, which is difficult to predict. The data transmitted over the ngrok tunnel is encrypted, ensuring the information remains secure while it travels over the internet. --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/define-a-schema # Define a schema in SurrealDB In this tutorial, you will learn how to define a schema in SurrealDB, what using either schema type means for data retrieval, and how to use what you need as your product grows. When starting a new database project, there are a couple of early decisions to be made, such as creating tables and defining the fields that will be in these tables and also the datatypes of the records, and how the tables you describe relate to each other which includes data sharing or manipulation. A schema defines the structure and organisation of data. It dictates how data is stored, organised, and manipulated. Schemas can specify tables, fields (columns), data types, constraints, and relationships between tables. In other words, a schema is the primary way to ensure your data acts as expected. There are two main types of schemas: 1. **Schemafull (structured)** 2. **Schemaless (unstructured)** When starting a new project in SurrealDB, you can define your schema using either method depending on your application's requirements. ## Prerequisites Before you start, this guide assumes the following: - Basic knowledge of databases and data modelling. - SurrealDB is installed on your machine. You can download and install SurrealDB from the [installation page](/docs/running/installation.md). - [A Command line interface (CLI)](/docs/reference/cli/surrealdb-cli/commands/sql.md) for interacting with SurrealDB or a [SurrealDB Studio sandbox](/docs/explore/studio.md). ## Schemafull (structured) databases A schemafull database requires the upfront definition of the structure of your data, including collections (tables) and fields (columns). This approach enforces consistency and integrity, making it suitable for applications with well-defined data models. In SurrealDB, the schemafull approach is realised through [Define statements](/docs/reference/query-language/statements/define/overview.md), that provide instructions on parts of your database, such as authentication access and behaviour, global parameters, table configurations, table events, analyzers, and indexes. You can set a schemafull table in the following steps: ### Creating a schemafull table 1. Define a Table: To start a schemafull table, specifically use the Define Table statement. ```surql -- Create a schemafull user table. DEFINE TABLE user SCHEMAFULL; ``` 2. Define Fields: Now that the table is schemafull, no fields can be set unless first defined through a `DEFINE FIELD` statement. ```surql -- Define some fields. DEFINE FIELD firstName ON TABLE user TYPE string; DEFINE FIELD lastName ON TABLE user TYPE string; DEFINE FIELD email ON TABLE user TYPE string ASSERT string::is_email($value); ``` In the code above you may notice the `ASSERT` clause. This can be used to validate any restrictions you want on a field. In the example above the [`string::is_email`](/docs/reference/query-language/functions/database-functions/string.md#stringis_email) function is used to check whether the [value](/docs/reference/query-language/language-primitives/parameters.md#reserved-variable-names) is an email. ### Adding data to a schemafull table Now that you have defined all the fields needed, you can start populating them. To do this, you can use the [CREATE statement](/docs/reference/query-language/statements/create.md). For example, add a new user: ```surql -- 1: Add a user with all required fields. CREATE user CONTENT { firstName: 'John', lastName: 'Doe', email: 'JohnDoe@someemail.com', }; ``` This will return the data in an object. In the case where the email wasn’t a real email for example: ```surql -- Using the CREATE statement to populate the table CREATE user CONTENT { firstName: 'John', lastName: 'Doe', email: 'JohnDoe.com', }; ``` The above will return an error because the field must conform to `string::is_email($value)` ### Inserting fields that don't exist in the schema In a schemafull table, since the fields need to be defined before you can populate them, if you add a field that doesn’t exist, your data will be ignored. For example, in the user table, you have only defined the `firstname`, `lastname` and `email` fields. If you introduce a `photoURI` field without defining the field in the `user` table, it will return an error. ```surql -- 2: Add a user with all required fields and an undefined one, 'photoURI'. CREATE user CONTENT { firstName: 'John', lastName: 'Doe', email: 'JohnDoe@someemail.com', photoURI: 'photo/yxCFi22Jw2.webp' }; ``` ## Schemaless (unstructured) databases A schemaless database does not require predefined structures, allowing for more flexible and dynamic data storage. This approach is ideal for applications with evolving data models or when dealing with diverse and unpredictable data formats. In SurrealDB there are two ways you can define a schemaless table. You can either use any of the data definition statements such as [`CREATE`](/docs/reference/query-language/statements/create.md) or [`UPDATE`](/docs/reference/query-language/statements/update.md) and that will make a table based on the [record ID](/docs/reference/query-language/language-primitives/data-types/record-ids.md#array-based-record-ids) specified. For example: ```surql -- Using the CREATE statement CREATE IC_directory:['John', 'Doe'] CONTENT { username: 'johndoe', full_name: 'John Doe', email: 'johndoe@example.com', date_of_birth: "1990-01-01", join_date: "2024-05-30", department: 'Engineering', role: 'Software Engineer', skills: ['Python', 'JavaScript', 'surql'], manager: manager_directory:janesmith, tags: ['full-time', 'remote'] }; -- Using the UPDATE statement UPDATE manager_directory:janesmith CONTENT { username: 'janesmith', full_name: 'Jane Smith', email: 'janesmith@example.com', date_of_birth: "1985-01-01", join_date: "2019-05-30", department: 'Engineering', role: 'Software Engineer Manager', skills: ['Python', 'JavaScript', 'surql'], report: IC_directory:['John', 'Doe'], tags: ['full-time', 'remote'] }; ``` In the example above, you used the `CREATE` statement to make an `IC_directory` table and created an Object ID for this table. Record IDs can be specified by you [in a large number of formats](/docs/reference/query-language/language-primitives/data-types/record-ids.md), defaulting to a random UUID if you don't specify your own format for the ID. . You have also used the UPDATE statement to make a `manager_directory`, which has `janesmith` as the ID. Learn more about [Record IDs in the documentation](/docs/reference/query-language/language-primitives/data-types/record-ids.md#array-based-record-ids). Notice how we have linked these two tables with the manager and report fields in the `IC_directory` and `manager_directory` tables, respectively. You can also start defining a schemaless table using the `DEFINE TABLE` statement. ```surql -- Create schemaless user table. DEFINE TABLE user SCHEMALESS; -- Define some fields. DEFINE FIELD firstName ON TABLE user TYPE string; DEFINE FIELD lastName ON TABLE user TYPE string; DEFINE FIELD email ON TABLE user TYPE string; ``` In the example above you have created a schemaless table using the `SCHEMALESS` clause. ### Inserting fields that don't exist in the schema Since a schemaless table doesn’t have any restrictions on the structure if you introduce a new field to an existing table the column will be added to the table even if other records don’t have the value. For example, take the schemafull example of introducing a `photoURI` field without defining the field in the `user` table: ```surql CREATE user CONTENT { firstName: 'John', lastName: 'Doe', email: 'JohnDoe@someemail.com', photoURI: 'photo/yxCFi22Jw2.webp' }; ``` Since the `user` table is schemaless the `photoURI` field will be added for `John` without an error. ## Combining schemafull and schemaless Now that you have seen how you can make a table in SurrealDB deciding what to go with depends on the restrictions you want to have on your schema. Since SurrealDB is multi-model, if some tables require less restrictions you can make them schemaless and if you want to ensure that the structure remains the same for each entry then go with the schemafull option. ### Adding flexible fields in a schemafull table In SurrealDB, you can add [flexible fields](/docs/reference/query-language/statements/define/field.md#flexible-data-types) to a schemafull table using the `FLEXIBLE` field clause on `DEFINE FIELD`. The clause comes after `TYPE` and relaxes object validation for that field. The table must be defined as `SCHEMAFULL` and the field must also contain an object for the statement to parse. For example: ```surql DEFINE TABLE user SCHEMAFULL; -- Define a flexible object field in a schemafull table. DEFINE FIELD interactions ON TABLE user TYPE object FLEXIBLE; -- Will not parse, must be an object -- DEFINE FIELD wont_parse ON TABLE user TYPE string FLEXIBLE; -- This would parse because an 'object' is present -- DEFINE FIELD metadata ON TABLE user TYPE { -- name: string, -- other: object -- } FLEXIBLE; ``` In the example above, you have added a flexible field `interactions` to the `user` table. This field can store any type of data, making it schemaless within a schemafull table. For example, you can store chat logs, emails, phone call records, or any other unstructured data in this field as a JSON object. ```surql -- Add an interaction to the user table. UPDATE user:wd99oovq358zfdmajnt7 CONTENT { firstName: 'John', lastName: 'Doe', email: 'JohnDoe@someemail.com', interactions: { type: 'email', subject: 'Welcome to our platform', body: 'Thank you for joining our platform. We hope you enjoy your experience.', date: '2024-05-30', }, }; ``` In the example above, you have added an interaction to the `user` table using the `UPDATE` statement. The `interactions` field stores an email interaction with the user, including the type, subject, body, and date. ### Use case: Customer Relationship Management (CRM) system A Customer Relationship Management (CRM) system is a prime example of an application that can benefit from both schemafull and schemaless tables. In this system, schemafull tables are essential for storing structured and consistent data such as customer details, orders, and products. For instance, a table for customers would include fields like customer ID, first name, last name, email, phone number, and the date they joined, ensuring data integrity and facilitating reliable reporting and analysis. Similarly, tables for orders and products would maintain strict schemas to track orders accurately and manage product inventories effectively. On the other hand, schemaless tables offer flexibility for handling unstructured or semi-structured data, which can vary widely. This is particularly useful for storing customer interactions such as emails, chat logs, phone call records, and social media messages, as well as customer feedback and reviews. These types of data do not fit neatly into a rigid schema due to their diverse formats and content. By utilising both schemafull and schemaless tables, a CRM system can achieve a balance between maintaining reliable, structured data for critical operations and providing the flexibility to capture and analyse a wide range of customer-related information, enhancing the system's overall adaptability and functionality. ## Conclusion A schema is the structure of tables in your database. The most important consideration when deciding whether to use a schemafull, schemaless, or both approaches is how flexible you want the content to be. Choose schemafull tables for structured data with strict validation requirements. Use schemaless tables for flexible, dynamic data that might evolve. Combine both approaches to leverage the strengths of each based on your application's needs. With respect to Performance Optimisation, you can use the `DEFINE INDEX` statement to create indexes on either form of table. This can also optimise storage and retrieval. Learn more about setting up a schema in the [`DEFINE` statement documentation](/docs/reference/query-language/statements/define/overview.md). --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/gen-ai-chatbot # Build a GenAI chatbot with Graph RAG In this guide, you'll learn how to use LangChain Python components, Ollama, and SurrealDB to make a GenAI chatbot. This tutorial builds a GenAI chatbot from LangChain Python components, Ollama and SurrealDB, retrieving context through the graph rather than through vector similarity alone. It opens with how traditional RAG works, so that the difference is clear before any code. ## What is traditional RAG Retrieval-Augmented Generation (RAG) is an AI technique that enhances the capabilities of large language models (LLMs) by allowing them to retrieve relevant information from a knowledge base *before* generating a response. It uses vector similarity search to find the most relevant document chunks, which are then provided as additional context to the LLM, enabling the LLM to produce more accurate and grounded responses. ## What is Graph RAG Graph RAG is an advanced technique. It leverages the structured, interconnected nature of knowledge graphs to provide the LLM with a richer, more contextualised understanding of the information, leading to more accurate, coherent, and less “hallucinated” responses. **Python** For the Python example we are going to use LangChain Python components, Ollama, and SurrealDB. ## Flow overview 0. Ingest data (categorised health symptoms and common treatments) 1. Ask the user about their symptoms 2. Find relevant documents in the DB by similarity 3. Execute vector search in DB 4. Invoke chain to find related common treatments 5. Chain asks the LLM to generate graph query 6. Chain executes the query 7. Query graph 8. Chain asks the LLM to summarise the results and generate an answer 9. Respond to the user Communication diagram for the Graph RAG solution and the flow listed above
## First step: ingest the data For this example we have a YAML file with categorised symptoms and their common treatments. We want to store this in a vector store so we can query it using vector similarity search. We also want to represent the data relations in a graph store, so we can run graph queries to retrieve those relationships (e.g. treatments related to symptoms). ```yaml - category: General Symptoms symptoms: - name: Fever description: Elevated body temperature, usually above 100.4°F (38°C). medical_practice: General Practice, Internal Medicine, Pediatrics possible_treatments: - Antipyretics (e.g., ibuprofen, acetaminophen) - Rest - Hydration - Treating the underlying cause ```
Let’s instantiate the following LangChain python components: - **Vector Store** ([SurrealDBVectorStore](https://python.langchain.com/docs/integrations/vectorstores/surrealdb/)) - **Graph Store** (SurrealDBGraph) - **Embeddings** ([OllamaEmbeddings](https://python.langchain.com/docs/integrations/text_embedding/ollama/), or any other model from the [Embedding models](https://python.langchain.com/docs/integrations/text_embedding/)) ...and create a SurrealDB connection: ```python # DB connection conn = Surreal(url) conn.signin({"username": user, "password": password}) conn.use(ns, db) # Vector Store vector_store = SurrealDBVectorStore( OllamaEmbeddings(model="llama3.2"), conn ) # Graph Store graph_store = SurrealDBGraph(conn) ``` Note that the `SurrealDBVectorStore` is instantiated with `OllamaEmbeddings`. This LLM model will be used when inserting documents to generate their embeddings vector. ## Populating the vector store With the vector store instantiated, we are now ready to populate it. ```python # Parsing the YAML into a Symptoms dataclass with open("./symptoms.yaml", "r") as f: symptoms = yaml.safe_load(f) assert isinstance(symptoms, list), "failed to load symptoms" for category in symptoms: parsed_category = Symptoms(category["category"], category["symptoms"]) for symptom in parsed_category.symptoms: parsed_symptoms.append(symptom) symptom_descriptions.append( Document( page_content=symptom.description.strip(), metadata=asdict(symptom), ) ) # This calculates the embeddings and inserts the documents into the DB vector_store.add_documents(symptom_descriptions) ```
## Stitching the graph together ```python # Find nodes and edges (Treatment -> Treats -> Symptom) for idx, category_doc in enumerate(symptom_descriptions): # Nodes treatment_nodes = {} symptom = parsed_symptoms[idx] symptom_node = Node(id=symptom.name, type="Symptom", properties=asdict(symptom)) for x in symptom.possible_treatments: treatment_nodes[x] = Node(id=x, type="Treatment", properties={"name": x}) nodes = list(treatment_nodes.values()) nodes.append(symptom_node) # Edges relationships = [ Relationship(source=treatment_nodes[x], target=symptom_node, type="Treats") for x in symptom.possible_treatments ] graph_documents.append( GraphDocument(nodes=nodes, relationships=relationships, source=category_doc) ) # Store the graph graph_store.add_graph_documents(graph_documents, include_source=True) ```
## Data ready, let’s chat LangChain provides different [chat models](https://python.langchain.com/docs/integrations/chat/). We are going to use `ChatOllama` with `llama3.2` to generate a graph query and to explain the result in natural language. ```python chat_model = ChatOllama(model="llama3.2", temperature=0) ``` To generate the graph query based on the user’s prompt, we need to instantiate a QA (Questioning and Answering) Chain component. In this case we are using `SurrealDBGraphQAChain`. But before querying the graph, we need to find the symptoms in our vector store by doing a similarity search based on the user’s prompt. ```python query = click.prompt( click.style("\\nWhat are your symptoms?", fg="green"), type=str ) # -- Find relevant docs docs = vector_search(query, vector_store, k=3) symptoms = get_document_names(docs) # -- Query the graph chain = SurrealDBGraphQAChain.from_llm( chat_model, graph=graph_store, verbose=verbose, query_logger=query_logger, ) ask(f"what medical practices can help with {symptoms}", chain) ask(f"what treatments can help with {symptoms}", chain) ```
## Running Clone the [repository](https://github.com/surrealdb/langchain-surrealdb) and follow the instructions in the README of the [graph example](https://github.com/surrealdb/langchain-surrealdb/tree/main/examples/graph). Running the program will look like this: ```text What are your symptoms?: i have a runny nose and itchy eyes ```
The script tries marginal relevance and similarity searches in the vector store to compare the results, which helps to choose the right one for your specific use case. ```text max_marginal_relevance_search: - Stuffy nose due to inflamed nasal passages or a dripping nose with mucus discharge. - An uncomfortable sensation that makes you want to scratch, often without visible skin changes. - Feeling lightheaded, unsteady, or experiencing a sensation that the room is spinning. similarity_search_with_score - [40%] Stuffy nose due to inflamed nasal passages or a dripping nose with mucus discharge. - [33%] Feeling lightheaded, unsteady, or experiencing a sensation that the room is spinning. - [32%] Pain, irritation, or scratchiness in the throat, often made worse by swallowing. ```
Then, the QA chain will generate and run a graph query behind the scenes, and generate the responses. This script is asking our AI two questions based on the user's symptoms: - Question: what medical practices can help with Nasal Congestion/Runny Nose, Dizziness/Vertigo, Sore Throat - Question: what treatments can help with Nasal Congestion/Runny Nose, Dizziness/Vertigo, Sore Throat For the first question the QA chain component generated this graph query: ```surql SELECT <-relation_Attends<-graph_Practice AS practice FROM graph_Symptom WHERE name IN ["Nasal Congestion/Runny Nose", "Dizziness/Vertigo", "Sore Throat"]; ```
The result of this query - a Python list of dictionaries containing the medical practice names - are fed to the LLM to generate a nice human readable answer: ```text Here is a summary of the medical practices that can help with Nasal Congestion/Runny Nose, Dizziness/Vertigo, and Sore Throat: Several medical practices may be beneficial for individuals experiencing symptoms such as Nasal Congestion/Runny Nose, Dizziness/Vertigo, and Sore Throat. These include Neurology, ENT (Otolaryngology), General Practice, and Allergy & Immunology. Neurology specialists can provide guidance on managing conditions that affect the nervous system, which may be related to dizziness or vertigo. ENT (Otolaryngology) specialists focus on ear, nose, and throat issues, making them a good fit for addressing nasal congestion and runny nose symptoms. General Practice physicians offer comprehensive care for various health concerns, including those affecting the respiratory system. Allergy & Immunology specialists can help diagnose and treat allergies that may contribute to Nasal Congestion/Runny Nose, as well as provide immunological support for overall health. ```
The query for the second question (What treatments can help with Nasal Congestion/Runny Nose, Dizziness/Vertigo, Sore Throat), looks like this: ```surql SELECT <-relation_Treats<-graph_Treatment AS treatment FROM graph_Symptom WHERE name IN ["Nasal Congestion/Runny Nose", "Dizziness/Vertigo", "Sore Throat"]; ```
The LLM will then produce the following output: ```text Here is a summary of the treatments that can help with Nasal Congestion/Runny Nose, Dizziness/Vertigo, and Sore Throat: The following treatments have been found to be effective in alleviating symptoms: - Vestibular rehabilitation - Hydration - Medications to reduce nausea or dizziness - Antihistamines (for allergies) - Decongestants (oral or nasal sprays) - Saline nasal rinses - Humidifiers - Throat lozenges/sprays - Treating underlying cause (e.g., cold, allergies) - Pain relievers (e.g., acetaminophen, ibuprofen) - Warm salt water gargles ```
**Rust** A medical chatbot using SurrealDB and LangChain can be made in Rust thanks to a crate called [langchain_rust](https://docs.rs/langchain-rust/latest/langchain_rust/) which as of last year [includes support for SurrealDB as a vector store](https://github.com/Abraxas-365/langchain-rust/pull/32). This implementation doesn't (yet!) include [graph queries](/blog/make-a-genai-chatbot-using-graphrag-with-surrealdb-langchain#stitching-the-graph-together), but we can still use classic vector search to find recommendations for treatment for a patient. To start off, use a command like `cargo new medical_bot` to create a new Cargo project, go into the project directory and add the following under `[dependencies]`. ```toml anyhow = "1.0.98" langchain-rust = { version = "4.6.0", features = ["surrealdb", "mistralai"] } serde = "1.0.219" serde_json = "1.0.140" serde_yaml = "0.9.34" surrealdb = { version = "3.0.4", features = ["kv-mem"] } tokio = "1.45.0" ``` The `langchain-rust` crate comes with OpenAI as a default, and includes a [large number of features](https://docs.rs/crate/langchain-rust/latest/features). We will add `mistralai` to show how easy it is to switch from one platform to another with only about two lines of different code. The original post assumes that we have a big YAML document with a number of symptoms along with their possible treatments, which is what the `serde_yaml` dependency will let us work with. ```yaml - category: General Symptoms symptoms: - name: Fever description: Elevated body temperature, usually above 100.4°F (38°C). medical_practice: General Practice, Internal Medicine, Pediatrics possible_treatments: - Antipyretics (e.g., ibuprofen, acetaminophen) - Rest - Hydration - Treating the underlying cause ``` To keep the logic simple, we will take only the `description` of each symptom and its `possible_treatments`, giving us two structs that look like this. ```rust #[derive(Debug, Deserialize)] pub struct SymptomCategory { pub symptoms: Vec } #[derive(Debug, Deserialize)] pub struct Symptom { pub description: String, pub possible_treatments: Vec, } ``` Then for each symptom, we will look through the possible treatments to create a document for each with text that looks like the following: * 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Antipyretics (e.g., ibuprofen, acetaminophen)' * 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Rest' * 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Hydration' * 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Treating the underlying cause' This needs to be turned into a [`Document`](https://docs.rs/langchain-rust/latest/langchain_rust/schemas/document/struct.Document.html) struct on the `langchain-rust` side, which looks like this. ```rust pub struct Document { pub page_content: String, pub metadata: HashMap, pub score: f64, } ``` The way to create a `Document` is via [`Document::new()`](https://docs.rs/langchain-rust/latest/langchain_rust/schemas/document/struct.Document.html#method.new) which takes a `String` for the `page_content`, followed by an optional `HashMap` for any metadata. The `score` will be 0.0 when inserting and is only used later on when a similarity search is performed to return a `Document`. For the `metadata`, we will add the other possible treatments so that any user will be able to first see a recommended treatment for a symptom, followed by all possible treatments for reference. The `Value` part of the `Document` struct is a [`serde_json`](https://docs.rs/serde_json/latest/serde_json/enum.Value.html) `Value`, which is why we have `serde_json` inside our `Cargo.toml` as well. All in all, the logic to grab the `YAML` file and turn it ito a `Vec` of `Document`s looks like this. ```rust fn get_docs() -> Result, Error> { let yaml_str = std::fs::read_to_string("symptoms.yaml")?; let categories: Vec = serde_yaml::from_str(&yaml_str)?; let symptoms = categories .into_iter() .flat_map(|cat| cat.symptoms) .collect::>(); Ok(symptoms .into_iter() .flat_map(|symptom| { let metadata = HashMap::from([ ( "possible treatments".to_string(), Value::from(symptom.possible_treatments.clone()), ) ]); symptom .possible_treatments .into_iter() .map(|treat| { Document::new(format!("'{}' can be treated by '{treat}'.", symptom.description.clone())) .with_metadata(metadata.clone()) }) .collect::>() }) .collect::>()) } ``` With this taken care of, it's time to do some setup inside `main()`. First we need to start running the database, which can be run in memory or via some other path such as an address to a [Surreal Cloud](/cloud) or a [locally running instance](/docs/reference/cli/surrealdb-cli/commands/start.md). ```rust let database_url = std::env::var("DATABASE_URL").unwrap_or("memory".to_string()); let db = surrealdb::engine::any::connect(database_url).await?; db.query("DEFINE NAMESPACE test; USE NAMESPACE test; DEFINE DATABASE test;") .await?; // Uncomment the following lines to authenticate if necessary // .user(surrealdb::opt::auth::Root { // username: "root".into(), // password: "secret".into(), // }); db.use_ns("main").await?; db.use_db("main").await?; ``` The next step is to initialise an embedder from the `langchain-rust` crate. Here we have the choice of an `OpenAiEmbedder` or `MistralAIEmbedder` thanks to the added feature flag. After that comes a `StoreBuilder` struct used to initiate a SurrealDB `Store`, which takes an embedder, a database, and a number of dimensions - 1536 in this case for OpenAI. If using Mistral, the dimensions would be 1024. Note that we are wrapping this in an `Arc` so that the store can start adding the documents on startup inside a separate task without making the user wait to see any CLI output. At the very end is an `.initialize()` method which defines some tables and fields which will be used when doing similarity searches. ```rust // Initialize Embedder let embedder = OpenAiEmbedder::default(); // Embedding size is 1024 in this case // let embedder = MistralAIEmbedder::try_new()?; // Initialize the SurrealDB Vector Store let store = Arc::new( StoreBuilder::new() .embedder(embedder) .db(db) .vector_dimensions(1536) .build() .await .map_err(|e| anyhow!(e.to_string()))?, ); store .initialize() .await .map_err(|e| anyhow!(e.to_string()))?; ``` Then we will clone the `Arc` to allow the `store` to be passed into a new blocking task to add the `Vec` returned by our `get_docs()` function. Inside this is a method called `add_documents()` which is a built-in method from the `rust-langchain` crate. ```rust let arced = Arc::clone(&store); tokio::task::spawn_blocking(move || { let docs = get_docs()?; Handle::current() .block_on(arced.add_documents(&docs, &VecStoreOptions::default())) .map_err(|e| anyhow::anyhow!("{e}"))?; Ok::<(), Error>(()) }); ``` While the store adds these documents in its own task, we will start a simple CLI that asks the user for a query, and then passes this into the built-in `.similarity_search()` method. This method allows us to specify the number of documents to return and a minimum similarity score, to which we will go with 2 and 0.6. The rest of the code just involves setting up a simple loop to handle user output, along with the results of the output of the `.similarity_search()` method. ```rust loop { // Ask for user input print!("Query> "); stdout().flush()?; let mut query = String::new(); stdin().read_line(&mut query)?; let results = store .similarity_search( &query, 2, &VecStoreOptions::default().with_score_threshold(0.6), ) .await .map_err(|e| anyhow!(e.to_string()))?; if results.is_empty() { println!("No results found."); } else { println!("Possible symptoms:"); results.iter().for_each(|r| { println!("{}\n All possible treatments: ", r.page_content); if let Some(Value::Array(array)) = r.metadata.get("possible treatments") { for val in array { if let Value::String(s) = val { println!(" {s}"); } } }; println!(); }); }; } ``` As the output shows, our bot is capable of returning meaningful results despite only having access to data from 236 lines of YAML! ```text Query> I've been exercising a lot outside and it's really hot. Possible symptoms: 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Hydration'. All possible treatments: Antipyretics (e.g., ibuprofen, acetaminophen) Rest Hydration Treating the underlying cause 'Elevated body temperature, usually above 100.4°F (38°C).' can be treated by 'Rest'. All possible treatments: Antipyretics (e.g., ibuprofen, acetaminophen) Rest Hydration Treating the underlying cause ``` ```text Query> I get dizzy sometimes Possible symptoms: 'Feeling lightheaded, unsteady, or experiencing a sensation that the room is spinning.' can be treated by 'Medications to reduce nausea or dizziness'. All possible treatments: Addressing underlying cause (e.g., inner ear issues, low blood pressure) Vestibular rehabilitation Medications to reduce nausea or dizziness Hydration 'Feeling lightheaded, unsteady, or experiencing a sensation that the room is spinning.' can be treated by 'Addressing underlying cause (e.g., inner ear issues, low blood pressure)'. All possible treatments: Addressing underlying cause (e.g., inner ear issues, low blood pressure) Vestibular rehabilitation Medications to reduce nausea or dizziness Hydration ``` ```text Query> What should I do in life? Possible symptoms: 'Feeling unusually drained, lacking energy, or experiencing persistent exhaustion.' can be treated by 'Lifestyle modifications (diet, exercise)'. All possible treatments: Rest and adequate sleep Lifestyle modifications (diet, exercise) Addressing underlying medical conditions (e.g., anemia, thyroid disorders) Stress management 'Feeling unusually drained, lacking energy, or experiencing persistent exhaustion.' can be treated by 'Stress management'. All possible treatments: Rest and adequate sleep Lifestyle modifications (diet, exercise) Addressing underlying medical conditions (e.g., anemia, thyroid disorders) Stress management ``` Want to give it a try yourself? Save [`symptoms.yaml` from the langchain-surrealdb examples](https://github.com/surrealdb/langchain-surrealdb/blob/main/examples/graph/symptoms.yaml) to the filename `symptoms.yaml` and then copy the following code into your cargo project, then set the env var `OPENAI_API_KEY` or `MISTRAL_API_KEY` along with `cargo run`. You can also give a crate called [archiver](https://crates.io/crates/archiver) a try, which has its own command-line interface to use SurrealDB with Ollama via the same crate we used in this post. ```rust // To run this example execute: `cargo run` in the folder. Be sure to have an OpenAPI key // set to the OPENAI_API_KEY env var // or MISTRAL_API_KEY if using Mistral use anyhow::{Error, anyhow}; use langchain_rust::{ embedding::{MistralAIEmbedder, openai::openai_embedder::OpenAiEmbedder}, schemas::Document, vectorstore::{VecStoreOptions, VectorStore, surrealdb::StoreBuilder}, }; use serde::Deserialize; use serde_json::Value; use std::{ collections::HashMap, io::{Write, stdin, stdout}, sync::Arc, }; use tokio::runtime::Handle; #[derive(Debug, Deserialize)] pub struct SymptomCategory { pub category: String, pub symptoms: Vec, } #[derive(Debug, Deserialize)] pub struct Symptom { pub name: String, pub description: String, pub possible_treatments: Vec, } fn get_docs() -> Result, Error> { let yaml_str = std::fs::read_to_string("symptoms.yaml")?; let categories: Vec = serde_yaml::from_str(&yaml_str)?; let symptoms = categories .into_iter() .flat_map(|cat| cat.symptoms) .collect::>(); Ok(symptoms .into_iter() .flat_map(|symptom| { let metadata = HashMap::from([( "possible treatments".to_string(), Value::from(symptom.possible_treatments.clone()), )]); symptom .possible_treatments .into_iter() .map(|treat| { Document::new(format!( "'{}' can be treated by '{treat}'.", symptom.description.clone() )) .with_metadata(metadata.clone()) }) .collect::>() }) .collect::>()) } #[tokio::main] async fn main() -> Result<(), Error> { let database_url = std::env::var("DATABASE_URL").unwrap_or("memory".to_string()); let db = surrealdb::engine::any::connect(database_url).await?; db.query("DEFINE NAMESPACE test; USE NAMESPACE test; DEFINE DATABASE test;") .await?; // Uncomment the following lines to authenticate if necessary // .user(surrealdb::opt::auth::Root { // username: "root".into(), // password: "secret".into(), // }); db.use_ns("main").await?; db.use_db("main").await?; // Initialize Embedder let embedder = OpenAiEmbedder::default(); // Embedding size is 1024 in this case // let embedder = MistralAIEmbedder::try_new()?; // Initialize the SurrealDB Vector Store let store = Arc::new( StoreBuilder::new() .embedder(embedder) .db(db) .vector_dimensions(1536) .build() .await .map_err(|e| anyhow!(e.to_string()))?, ); // Intialize the tables in the database. This is required to be done only once. store .initialize() .await .map_err(|e| anyhow!(e.to_string()))?; let arced = Arc::clone(&store); tokio::task::spawn_blocking(move || { let docs = get_docs()?; Handle::current() .block_on(arced.add_documents(&docs, &VecStoreOptions::default())) .map_err(|e| anyhow::anyhow!("{e}"))?; Ok::<(), Error>(()) }); loop { // Ask for user input print!("Query> "); stdout().flush()?; let mut query = String::new(); stdin().read_line(&mut query)?; let results = store .similarity_search( &query, 2, &VecStoreOptions::default().with_score_threshold(0.6), ) .await .map_err(|e| anyhow!(e.to_string()))?; if results.is_empty() { println!("No results found."); } else { println!("Possible symptoms:"); results.iter().for_each(|r| { println!("{}\n All possible treatments: ", r.page_content); if let Some(Value::Array(array)) = r.metadata.get("possible treatments") { for val in array { if let Value::String(s) = val { println!(" {s}"); } } }; println!(); }); }; } Ok(()) } ``` ## Ready to build? Find all the code in the [repository examples](https://github.com/surrealdb/langchain-surrealdb/tree/main/examples/graph). Get started for free with [Surreal Cloud](https://studio.surrealdb.com/signin). Any questions or thoughts about this or Graph RAG using SurrealDB? [Join our Discord](https://discord.gg/surrealdb) - `#all-ai` is a good place to start for this topic, and `#help` or `#general` work for anything else. --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/github-actions # Use SurrealDB in GitHub Actions This guide will show you how to set up and use the official GitHub Action for SurrealDB in your CI/CD pipeline. This guide will show you how to set up and use the official GitHub Action for SurrealDB in your CI/CD pipeline. ## Step 1: Create a new GitHub workflow file Create a new YAML file in your repository's `.github/workflows` directory. You can name the file `surrealdb-ci.yml`. ```yaml name: SurrealDB CI on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - name: Git checkout uses: actions/checkout@v4 - name: Start SurrealDB uses: surrealdb/setup-surreal@v3 with: surrealdb_version: latest surrealdb_port: 8000 surrealdb_username: root surrealdb_password: secret surrealdb_auth: false surrealdb_strict: false surrealdb_log: info surrealdb_additional_args: --allow-all surrealdb_retry_count: 30 ``` ## Step 2: Customise workflow arguments The official SurrealDB GitHub Action accepts several arguments to configure the SurrealDB setup. Here's a breakdown of the available arguments and their defaults:
Argument Description Default Value
surrealdb_version SurrealDB version to use latest latest, nightly, beta, alpha, v1.x.x, v2.x.x, v3.x.x
surrealdb_datastore Datastore to start SurrealDB with memory Any valid datastore path, for example rocksdb:data
surrealdb_port Port to run SurrealDB on 8000 Valid number from 0 to 65535
surrealdb_username Username to use for SurrealDB root Customisable by the user
surrealdb_password Password to use for SurrealDB root Customisable by the user
surrealdb_auth Enable authentication false true, false
surrealdb_strict Enable strict mode false true, false
surrealdb_log Enable logs trace none, full, error, warn, info, debug, trace
surrealdb_import_file SurrealQL file to import on startup Path to a .surql file, requires SurrealDB v3.0.0 or later
surrealdb_additional_args Additional arguments for SurrealDB Any valid SurrealDB CLI arguments
surrealdb_retry_count Seconds to wait for SurrealDB to become ready 30 Any valid integer
The file passed to `surrealdb_import_file` selects its own namespace and database, so it should begin with a `USE NS ... DB ...;` statement. ### Workflow outputs The action exposes the details of the instance it started, so that later steps do not have to reconstruct them:
Output Description
endpoint The HTTP endpoint the SurrealDB instance is listening on
version The exact version of SurrealDB that was installed
### Example configuration Here is an example configuration that sets specific values for each argument: ```yaml name: SurrealDB CI on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - name: Git checkout uses: actions/checkout@v4 - name: Start SurrealDB id: surrealdb uses: surrealdb/setup-surreal@v3 with: surrealdb_version: latest surrealdb_port: 8000 surrealdb_username: root surrealdb_password: secret surrealdb_auth: false surrealdb_strict: false surrealdb_log: info surrealdb_additional_args: --allow-all surrealdb_retry_count: 30 - name: Run the tests run: ./run-tests.sh env: SURREALDB_ENDPOINT: ${{ steps.surrealdb.outputs.endpoint }} ``` ### Tips for customisation 1. **Version Control**: Use specific versions to avoid unexpected changes. Example: surrealdb_version: v3.2.4. 2. **Security**: Always use strong passwords for surrealdb_password and avoid using default credentials in production. 3. **Logs**: Set an appropriate log level based on your needs. For debugging, use debug or trace. 4. **Additional Arguments**: Utilise surrealdb_additional_args to pass any additional CLI arguments required by your setup. ## Step 3: Commit and push After creating and customising your workflow file, commit and push it to your repository: ```sh git add .github/workflows/surrealdb-ci.yml git commit -m "Add SurrealDB CI workflow" git push origin main ``` ## Step 4: Verify workflow execution Go to your repository on GitHub and navigate to the "Actions" tab. You should see your workflow running when you push changes or create a pull request. Check the logs to verify that SurrealDB is starting up correctly and that all steps are executed successfully. ## Conclusion Using the official GitHub Action for SurrealDB simplifies the process of setting up and running SurrealDB in your CI/CD pipeline. Customise the workflow as per your project requirements, and ensure you follow best practices for security and version control. Happy coding! --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/how-to-build-a-knowledge-graph-for-ai # Build a knowledge graph for AI Build a knowledge graph for AI agents and RAG: graphs, embeddings and SurrealQL retrieval patterns in one tutorial. Welcome to the tutorial on building knowledge graphs for AI. This document is for you if: - you are an engineer working in gen AI - you want to understand why knowledge graphs are relevant for AI agents ## Introduction Let’s start by placing knowledge graphs on the map, by showing a modern multi-agent RAG architecture. In this example, assume each agent has a different role in the process to answer a user's prompt. In order to make this happen, each agent comes equipped with its own tool or tools like web search, MCP servers, and more. A knowledge graph in this case is just another toolset for the agents. Agentic RAG While the word “tool” is being used very casually here, [tool calling](https://martinfowler.com/articles/function-call-LLM.html) is a very important concept in our context. Most modern LLM models (like Claude Sonnet 4.5, Gemini 3 Flash Preview, DeepSeek V3.2, and [many more listed on OpenRouter](https://openrouter.ai/collections/tool-calling-models)) are capable of using tools as part of their process before generating an answer. These tools can be built in the model (like web search in Claude models) or provided by you (like the example below). This Python code shows how you can provide your agent with a “retrieval” tool from your knowledge graph. The *docstring* in this function gives to the LLM the information it needs to know when and how to call this function. ```python embedder = Embedder('openai:text-embedding-3-small') agent = Agent('openai:gpt-5.2') @agent.tool async def retrieve(context: RunContext[Deps], search_query: str) -> str: """Retrieve documents from the knowledge graph based on a search query. Args: search_query: The search query. """ with logfire.span("KG search for {search_query=}", search_query=search_query): # -- Build SurrealQL query surql = generate_surql(search_query) # -- Embeddings result = await embedder.embed_query(search_query) embedding = result.embeddings[0] # -- Query results = query( context.deps.db, surql, {"embedding": list(embedding)}, SearchResult, ) results = "\n\n".join( f"# Document name: {x.doc.filename}\n{'\n\n'.join(str(y.content) for y in x.chunks)}\n" for x in results ) return results ``` Before diving into how to build a knowledge graph, let’s define what they are. ## What is a knowledge graph? > In knowledge representation and reasoning, a **knowledge graph** is a knowledge base that uses a graph-structured data model or topology to represent and operate on data. Knowledge graphs are often used to store interlinked descriptions of entities - objects, events, situations or abstract concepts - while also encoding the free-form semantics or relationships underlying these entities. [Source: Wikipedia](https://en.wikipedia.org/w/index.php?title=Knowledge_graph&oldid=1334019348) The image below presents two specimens of a knowledge graph. The first one is a very structured and predictable one, whose **nodes and edges are explicit** in the original data. The second one is its opposite: a more free-form graph, with some **entities inferred by an LLM** out of the corpus. If you take a close look you'll see that's why the first one includes only a single prepared graph edge called `INCLUDES`, while the second has more than one such as `MENTIONED_IN` and `PUBLISHED_IN` - these were generated by the LLM which concluded it made sense to use these to describe the relation between entities. Types of knowledge graphs But… 🤔 ## When do AI agents need a knowledge graph? Because LLMs by themselves are brilliant storytellers but with a fuzzy memory. In turn, AI agents are designed to perform tasks and make decisions, requiring the accuracy only a structured graph can provide. An LLM with a knowledge graph is like a storyteller with a highly-organised, cross-referenced encyclopedia. **Example to show some of the benefits:** *“Summarise the reviews of this month’s most popular product in our store”* With a prompt like that, and a knowledge graph like in the example before, your agent would be able to deterministically retrieve the list of the reviews for the best selling products. Let's see how easy this is to do with a SurrealQL query. ```surql -- Get the ID best product based on its count LET $best = ( SELECT id, count(<-product_in_order) AS count FROM ONLY product ORDER BY count DESC LIMIT 1 ).id; -- Then return the reviews where this product shows up SELECT *, $best AS product FROM review WHERE $best IN ->review_for_product->product; ``` The output of this last query will look like this. ```surql [ { id: review:1, product: product:detector, rating: 5, text: 'Excellent!' }, { id: review:2, product: product:detector, rating: 4, text: 'Pretty good.' } ] ``` Want to give it a try yourself? Head on over to [SurrealDB Studio](https://studio.surrealdb.com/), go into the sandbox and run the following statements to set up the schema and seed data before running the query we just saw. To keep that data, use **Deploy to Cloud** in SurrealDB Studio to create a free SurrealDB Cloud instance, then connect to it and run the same statements there. ```surql -- Products DEFINE TABLE product SCHEMAFULL; DEFINE FIELD name ON product TYPE string; -- Orders DEFINE TABLE order SCHEMAFULL; DEFINE FIELD created_at ON order TYPE datetime; -- Reviews DEFINE TABLE review SCHEMAFULL; DEFINE FIELD rating ON review TYPE int; DEFINE FIELD text ON review TYPE string; -- Edge: order -> product DEFINE TABLE product_in_order SCHEMAFULL TYPE RELATION; -- Edge: review -> product DEFINE TABLE review_for_product SCHEMAFULL TYPE RELATION; CREATE product:detector SET name = "Dragon detector"; CREATE product:repellent SET name = "Repellent"; -- Orders CREATE order:1 SET created_at = time::now(); CREATE order:2 SET created_at = time::now(); CREATE order:3 SET created_at = time::now(); -- Order edges (Dragon detector sells twice, Repellent once) RELATE order:1->product_in_order->product:detector; RELATE order:2->product_in_order->product:detector; RELATE order:3->product_in_order->product:repellent; -- Reviews CREATE review:1 SET rating = 5, text = "Excellent!"; CREATE review:2 SET rating = 4, text = "Pretty good."; -- Review edges (both for Dragon detector) RELATE review:1->review_for_product->product:detector; RELATE review:2->review_for_product->product:detector; ``` And to finish up this example, here is a bonus query that lets you see all incoming graph edges to a table. The ? here is used as a wildcard to match anything, which in this case means all of the `product_in_order` and `review_for_product` edges in between the `order` and `review` tables. ```surql -- Query full graph SELECT *, <-?<-? AS all_edges FROM product; ``` We can use this data to explain the main benefits of a knowledge graph: - **multi-hop reasoning**: it navigates the graph to one or more edges: `review → review_for_product → product → product_in_order → order` - **deterministic accuracy**: the query output is backed by hard data, a fact that will add great value to the LLM context. - **explainability**: if required, besides the plain answer, you get the query that was executed and the structured results. In our example, you get the list of reviews, but also which is the top-selling product, and if you wish, you could include how many items were sold. - **reduced hallucinations:** because we leveraged our graph relations to exactly know which are the reviews of the best-selling products, the LLM just needs to summarise them. That leaves little room for hallucinations. Compare this to asking the same question with RAG on ingested sales reports instead of a knowledge graph: the data about which is the best selling product may or may not be mentioned in one of those documents, which will get retrieved using semantic search (or at least a chunk of the report), and fingers crossed, that chunk gets a good enough score to be picked up and included into the LLM context. - **dynamic knowledge:** whenever new orders and reviews are created, the query will pick them up. Because of its structured nature, it’s easier to keep up to date, specially if your “transactional” DB is the same as your knowledge graph DB. ## When is a knowledge graph not necessary? As a counterexample, let's look at another use case in which a knowledge graph might be overkill: - Dataset: successful troubleshooting conversations with customers (from support ticket system or e-mail), internal support conversations from company chat (e.g. Slack threads), FAQs and documentations from internal wiki (e.g. Notion, PDFs, etc.) - Agent job: answer questions like: “Robot firmware is v.1.67 and I can’t get access via SSH” A vector store populated with the available dataset may provide good references for an LLM to help. You should only consider adding graph relations to the mix if your vector store is too big, or you have very dense neighbourhoods (e.g. a lot of troubleshooting chats about the same issue, causing [context distraction, confusion, and clash](https://www.blog.langchain.com/context-engineering-for-agents/)). You might also want to trim down the vector space by relating chunks to specific domains (support category, product line, firmware version). This image illustrates dense neighbourhoods, and how graph relations can help to trim down the vector space, by running queries that read like this: “find reviews in the proximity of `$vector` AND are connected with `->review_for_product->product->product_in_category->dragons`”. Vector clusters with graph relations ## Moving from unstructured data to a knowledge graph These are the main steps that are required to go from unstructured data to having a knowledge graph for your AI agents. The **Extraction, Transformation, and Loading** steps are commonly referred to as **ETL**. ### 1. Extraction 1.1. Parsing For each document, parse it and transform it into structured data. It could be a CSV file which is already structured, but unstructured data like a PDF with text, images, and tables can be worked with as well. 1.2. Chunking We now have “plain” data, which is commonly (but not necessarily) kept in Markdown format. It is very likely that the document may be too long, which is less than ideal for LLMs which have a finite context window (references: https://arxiv.org/abs/2502.05167). 1.3. Embedding Semantic retrieval is possible because of vector embeddings. You decide what you want to embed. You almost always want to embed chunks, but can also embed content on graph nodes (e.g. to run a semantic search on keywords and from there query other connected nodes). 1.4. Entity and relationship extraction Entities will become nodes (any concept like `people`, `document`, `product`), and relationship edges (any verb or predicate like `works_at`, `explains`). Depending on your data, and how structured it is, some of the entities and relationships will be easy to extract because they may be explicit in the data (e.g. Martin → works_at → SurrealDB). Some others will require to be inferred based on some context (e.g. extract from a threads that Martin → knows_about → SurrealQL). ### 2. Transformation This step in the process is meant to clean your data. Here are some ideas for what you might want to do at this point: - Deduplication and ontology alignment: for example, you might have Arnold → governor → California along with Schwarzenegger → star of → Predator. `Arnold` and `Schwarzenegger` should get merged. - Inference and enrichment: there are pros and cons about inferring attributes when generating a knowledge graph. It depends on your specific needs, but be careful: you could burn through tokens generating things that were never used, plus using inferred attributes as context could lead to inaccurate results. Inferred attributes work better as ways to navigate the graph, rather than to provide context to the LLM. As an example, take [LightRAG](https://lightrag.github.io), which demonstrates good results using “inference and enrichment” in a clever way to aid with navigation rather than context. ### 3. Loading Loading is the last step in the ETL process. Here’s where you connect things with each other in the database, both vector embeddings, and graph relations. Look at the practical [Loading example below](#loading). Next, you’ll find common practices and practical examples for parsing, chunking, and all the steps mentioned above. ## Practical examples To finish up today's post, let's look at some examples of how to do the following: - [Parsing unstructured data](#parsing-unstructured-data) - [Chunking documents](#chunking-documents) - [Embedding generation](#embedding-generation) - [Entity and relationships extraction](#entity-and-relationship-extraction) - [Loading](#loading) ### Parsing unstructured data The following example shows how to use Kreuzberg to parse PDFs. You may need to configure it differently, depending on your documents and the types of content. It’s not the same to parse a simple PDF, a PDF with images and tables, a spreadsheet, or websites. The following example uses a `flow` decorator to register functions for different steps in the ETL process. An orchestrator takes care of calling this function for `document`s that lack a “stamp” in their `chunked` column: ```python @exe.flow("document", stamp="chunked", priority=2) def chunk(record: flow.Record, hash: str): # pyright: ignore[reportUnusedFunction] doc = OriginalDocumentTA.validate_python(record) chunking_handler(db, doc) # set output field so it's not reprocessed again _ = db.sync_conn.query( "UPDATE $rec SET chunked = $hash", {"rec": doc.id, "hash": hash} ) ``` The `chunking_handler` is in charge of the actual parsing. The code (simplified from [kreuzberg_converter.py](https://github.com/surrealdb/kaig/blob/main/examples/knowledge-graph/src/knowledge_graph/extraction/providers/kreuzberg/kreuzberg_converter.py)) looks like this: ```python from kreuzberg import ( ChunkingConfig, ExtractionConfig, KeywordAlgorithm, KeywordConfig, TokenReductionConfig, extract_file_sync, ) from pydantic import TypeAdapter @dataclass class ChunkWithMetadata: content: str metadata: dict[str, Any] ChunksTA = TypeAdapter(list[ChunkWithMetadata]) config = ExtractionConfig( use_cache=True, # optional keyword extraction keywords=KeywordConfig( algorithm=KeywordAlgorithm.Yake, max_keywords=10, min_score=0.1 ), chunking=ChunkingConfig(max_chars=1000, max_overlap=100), token_reduction=TokenReductionConfig(mode="light"), enable_quality_processing=True, ) result = extract_file_sync(path_or_bytes, config=config) print(f"Chunks: {result.chunks}") print(f"Metadata: {result.metadata}") print(f"Chunks: {len(result.chunks)}") chunks = ChunksTA.validate_python(result.chunks) ``` A simple trick: hash the chunk and use that as the ID to avoid generating embeddings for chunks that already exist. This applies to mostly every record that gets processed in any way, not only for chunks. ```python hash = hashlib.md5(chunk_text.encode("utf-8")).hexdigest() chunk_id = RecordID(Tables.chunk.value, hash) # skip if it already exists if db.exists(chunk_id): continue ``` Find the complete code in [ingestion.py](https://github.com/surrealdb/kaig/tree/main/examples/knowledge-graph/src/knowledge_graph/ingestion). Resources: - Open source libraries: [Kreuzberg](https://kreuzberg.dev), [Docling](https://docling-project.github.io/docling/), [Marker](https://github.com/datalab-to/marker) - Commercial solutions: [Document AI from Tensorlake](https://tensorlake.ai), https://www.datalab.to/ ### Chunking documents Let’s use an example document to explain different chunking strategies, but be mindful that other use cases may favour different strategies. Imagine your “raw” documents as backups of group chats. They are plain text files, in which each line looks like “\{user\} \{timestamp\} \{message\}”. **Different strategies:** - **Token limit:** chunks documents of equal size to guarantee that they fit into your embedding model window (commonly between 512 to 8k tokens) - **Recursive:** Instead of a hard cut at *N* characters, it uses a hierarchy of separators (typically ["\n\n", "\n", " ", ""]) to find the best place to split. - **Semantic:** chunks document based on their semantic meaning. With our group chat example, chunks are divided when the conversation topic changes. - **Structure:** Rather than treating a document as raw text, these strategies use the inherent formatting (Markdown, HTML, or Code) to define boundaries - **Custom:** be creative! Bringing our group chat example again, you can decide on periods of silence as a good point to split chunks. Doing this would be faster and cheaper than using the semantic strategy, and probably as accurate. Simple and cheap strategies are worth trying first to have a good baseline. You then evaluate the results, and decide if a better (and more expensive) solution is required. This often produces better results than starting by choosing a complex strategy that may be overkill. Another tip: adding overlaps to the chunks is a common practice, specially for the strategies that are simpler best-effort ones. ### Embedding generation Directly using provider SDKs: ```python def embed_with_ollama(text: str) -> list[float]: """Generate embedding using Ollama.""" res = ollama.embed(model=MODEL_NAME, input=text, truncate=True) return list(res.embeddings[0]) def embed_with_openai(text: str) -> list[float]: """Generate embedding using OpenAI.""" response = openai_client.embeddings.create( model=MODEL_NAME, input=text ) return response.data[0].embedding ``` With an AI framework, like pydantic-ai: ```python embedder = Embedder('openai:text-embedding-3-small') with logfire.span( 'create embedding for {search_query=}', search_query=search_query ): result = await embedder.embed_query(text) embedding_vector = result.embeddings[0] ``` ### Entity and relationship extraction This function shows how to extract concepts from a chunk, and relate them with graph edges: ```python def extract_concepts(db: DB, chunk: Chunk) -> list[str]: if not db.llm: logger.warning("No LLM configured, skipping inference") return [] with logfire.span("Extract concepts {chunk=}", chunk=chunk.id): instructions = dedent(""" - Only return concepts that are: names, places, people, organisations, events, products, services, etc. - Do not include symbols or numbers """) concepts = db.llm.infer_concepts(chunk.content, instructions) logger.info(f"Concepts: {concepts}") for concept in concepts: concept_id = RecordID(Tables.concept.value, concept) _ = db.embed_and_insert( Concept(content=concept, id=concept_id), table=Tables.concept.value, id=concept, ) db.relate( chunk.id, EdgeTypes.MENTIONS_CONCEPT.value.name, concept_id, ) logger.info("Finished inference!") return concepts ``` The [implementation of `infer_concepts`](https://github.com/surrealdb/kaig/blob/main/src/kaig/llm.py) is a bit complex because it’s abstracting multiple providers. So, here is a simplification: ```python PROMPT_INFER_CONCEPTS = """ Given the "Text" below, can you generate a list of concepts that can be used to describe it?. Don't provide explanations. {additional_instructions} ## Text: {text} """ class LLM: ... def infer_concepts( self, text: str, additional_instructions: str = "" ) -> list[str]: additional_instructions = ( "Return a JSON array of strings. " + additional_instructions ) prompt = PROMPT_INFER_CONCEPTS.format( text=text, additional_instructions=additional_instructions ) response = self._generate_openai( prompt, response_format={"type": "json_object"} ) # parses the response into a list of str return validate_list(response) ``` Using `additional_instructions` allows us to reuse then function `infer_concepts` for different domains. ### Loading For the following example, assume the following schema: Vector indexes on: `chunk` and `keyword` tables Graph relations: `PART_OF`, `MENTIONED_IN` Tables: `chunk`, `document`, `keyword` We have extracted the entities and relationships from the chunks, so we are ready to insert the nodes and edges into the graph, using semantic triplets like these: `chunk → PART_OF → document` `keyword → MENTIONED_IN → chunk` In Python, it looks like this: ```python def insert(db: DB, triplets: list[(RecordID, str, RecordID)]): for (a, relation, b) in triplets: # - Store the nodes for x in [a, b]: node = Node(id=x, content=x.id) # - Embed the node if it has a vector index if x.table in vector_tables: db.embed_and_insert(node) else: db.insert(node) # - Store the relation db.relate(a, relation, b) ``` The code above is an simplification of [inference.py](https://github.com/surrealdb/kaig/blob/main/examples/knowledge-graph/src/knowledge_graph/handlers/inference.py) from this [knowledge-graph example](https://github.com/surrealdb/kaig/tree/main/examples/knowledge-graph), which uses utils functions (e.g. `embed_and_insert`) from our [Kai G examples repository](https://github.com/surrealdb/kaig/tree/main). --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/http-via-postman # Query SurrealDB from Postman In this tutorial, you will learn how to query the SurrealDB RESTful HTTP API endpoints using Postman. SurrealDB provides a RESTful HTTP API for interacting with the database. In this tutorial, you will learn how to query SurrealDB endpoints via the [Postman collection](https://postman.com/surrealdb/workspace/surrealdb/collection/19100500-3da237f3-588b-4252-8882-6d487c11116a). SurrealDB supports requests via HTTP & REST with which you can handle different queries from creating simple tables to having Graph relationships between complex tables. Check out the [HTTP & REST integration documentation](/docs/reference/rest-api/http-protocol.md) for a detailed list of all the endpoints supported. ## Prerequisites This tutorial assumes that you have the following: - A Postman account to fork the [SurrealDB collection](https://postman.com/surrealdb/workspace/surrealdb/collection/19100500-3da237f3-588b-4252-8882-6d487c11116a) - SurrealDB installed - If you do not, see the [installation guide](/docs/running/installation.md) specific to your machine. ## Getting started Before forking the Collection from Postman, ensure that you have your SurrealDB instance running locally because by default the collection endpoint is set to [`localhost:8000`](http://localhost:8000) or [`http://127.0.0.1:8000`](http://127.0.0.1:8000/) To do so run the [Start command](/docs/reference/cli/surrealdb-cli/commands/start.md) in your terminal: ```bash surreal start --user root --pass secret ``` The above command starts SurrealDB with [authentication](/docs/reference/cli/surrealdb-cli/commands/start.md#authentication) and specifies that the user and password are `root`. Note that this is just for demonstration purposes. You can replace `root` in both instances with any other value. After running locally, head over to Postman and create a fork for your workspace. You should see all the endpoints listed. You can also [check the endpoints in the documentation](/docs/reference/rest-api/http-protocol.md). ## Using the `INFO` statement You can get information about your workspace by using the INFO statement. However, you need the right permissions in order to see the output. See the documentation for the [INFO statement](/docs/reference/query-language/statements/info.md). It is also important to note that the `namespace` and `database` values for the Postman collection are set to `test` by default. You can see the JSON output using the [INFO statement](/docs/reference/query-language/statements/info.md) in the body of the request. ```surql INFO FOR ROOT / DB / NS ``` Below is the output of querying for the info of the current namespace. ```surql INFO FOR NS ``` ```json [ { "result": { "accesses": {}, "databases": { "test": "DEFINE DATABASE test" }, "users": {} }, "status": "OK", "time": "190.166µs" } ] ``` ## Defining a record user To define a [record user](/docs/learn/security/authentication/users.md#record-users), first navigate to the `POST /sql` endpoint and in the header of the request add the following fields: ```json Accept:application/json NS:{{namespace}} DB:{{database}} ``` You can also [define](/docs/reference/query-language/statements/define/user.md) a [system user](/docs/learn/security/authentication/users.md#system-users) with other credentials such as Root or Database user. For this tutorial, we will be using only [record users](/docs/learn/security/authentication/users.md#record-users). > [!NOTE] > The Namespace and Database fields are set to the value `test` by default. You can change this in the collection settings. In the request body, [define the record access method](/docs/reference/query-language/statements/define/access/record.md) `human` with a session of 24 hours: ```surql DEFINE ACCESS human ON DATABASE TYPE RECORD SIGNUP ( CREATE user SET email = $email, pass = crypto::argon2::generate($pass) ) SIGNIN ( SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(pass, $pass) ) DURATION FOR SESSION 24h ; ``` The code above allows a user to sign up as a record user with their email and password. Then hash the password with the [`crypto::argon2::generate`](/docs/reference/query-language/functions/database-functions/crypto.md#cryptoargon2generate) function The sign in logic gets all the users where the emails match and uses the [`crypto::argon2::compare`](/docs/reference/query-language/functions/database-functions/crypto.md#cryptoargon2compare) function to check the hash value to the unhashed. Now when you get the info of the Database using `INFO for DB` you can see the access method `human` in the following output: ```json [ { "result": null, "status": "OK", "time": "102.458µs" }, { "result": { "accesses": { "human": "DEFINE ACCESS human ON DATABASE TYPE RECORD SIGNUP (CREATE user SET email = $email, pass = crypto::argon2::generate($pass)) SIGNIN (SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(pass, $pass)) DURATION FOR TOKEN 1h, FOR SESSION 1d" }, "analyzers": {}, "functions": {}, "models": {}, "params": {}, "tables": {}, "users": {} }, "status": "OK", "time": "78.084µs" } ] ``` ## Signing up a new scope user After defining the `signup` and `signin` logic head over to the `POST /signup` endpoint to signup and in the header of the request add the following fields: ```json Accept:application/json namespace:test database:test access:human ``` In the body of the request add the following information: ```json { "ns": "test", "db": "test", "ac": "human", "email": "test@surreal.com", "pass":"1234567886" } ``` The result will be a JSON object: ```json { "code": 200, "details": "Authentication succeeded", "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJpYXQiOjE3MDcxNDY2NTgsIm5iZiI6MTcwNzE0NjY1OCwiZXhwIjoxNzA3MjMzMDU4LCJpc3MiOiJTdXJyZWFsREIiLCJOUyI6InRlc3QiLCJEQiI6InRlc3QiLCJBQyI6Imh1bWFuIiwiSUQiOiJ1c2VyOnc2ZzBsNmh5eHpjZzlubTY2dGVjIn0.8Gud51cocThB8DMKD1zovtGiVgf5L1dAS6-pjWb6Lm6a7-4Spp7xXjD7JrHHdtJVNX1O0d8GdjZwRGTsP_NM9A" } ``` ## Signing in a record user Now that we have defined a User we can now log in. To do so, head to the `POST /signin` and using the same credentials. In the body of the request, add the same information you used to sign up: ```json { "ns": "test", "db": "test", "ac": "human", "email": "test3@surreal.com", "pass":"1234567886" } ``` This should return a similar JSON object indicating successful authentication and providing a new token. ```json { "code": 200, "details": "Authentication succeeded", "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJpYXQiOjE3MDcxNDY2NTgsIm5iZiI6MTcwNzE0NjY1OCwiZXhwIjoxNzA3MjMzMDU4LCJpc3MiOiJTdXJyZWFsREIiLCJOUyI6InRlc3QiLCJEQiI6InRlc3QiLCJBQyI6Imh1bWFuIiwiSUQiOiJ1c2VyOnc2ZzBsNmh5eHpjZzlubTY2dGVjIn0.8Gud51cocThB8DMKD1zovtGiVgf5L1dAS6-pjWb6Lm6a7-4Spp7xXjD7JrHHdtJVNX1O0d8GdjZwRGTsP_NM9A" } ``` ## Using table endpoints The Postman collection has a couple of endpoints for `POST` `GET` `PUT` `PATCH` `DEL` operations. For example, If you want to add a new entry to the table `Person` (Which is the default table in the collection) go to the [`POST /key/:table`](/docs/reference/rest-api/http-protocol.md#post-table) endpoint then in a body of the request and add the table content in the body of the request. E.g: ```json { age: 32, name: 'John' } ``` You should get a response as seen below, notice that there is a [record ID](/docs/reference/query-language/language-primitives/data-types/record-ids.md), with this record ID you can now use any of the [`/key/:table/:id`](/docs/reference/rest-api/http-protocol.md#get-record) endpoints for `POST` `GET` `PUT` `PATCH` `DEL` operations. ```json [ { "result": [ { "age": 32, "id": "person:p1cdf6cx89gnfboq5wye", "name": "John" } ], "status": "OK", "time": "105.667µs" } ] ``` ## Conclusion In this tutorial, we've walked through how to set up and use SurrealDB over HTTP using Postman. We've covered how to define a new record access method, sign up a new user, and sign in with a user. These steps are crucial for managing user authentication in your applications using SurrealDB. The Postman collection is still in active development for more information on using surrealDB via the HTTP endpoints. [Check out our documentation on HTTP and rest endpoints](/docs/reference/rest-api/http-protocol.md) --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/minimal-langchain # Build a minimal LangChain chatbot This tutorial shows how to build a minimal LangChain chatbot with both vector and graph. This tutorial walks through a simple example that does the following: 1. creates the vector store ([SurrealDBVectorStore](https://python.langchain.com/docs/integrations/vectorstores/surrealdb/)) and graph ([SurrealDBGraph](https://github.com/surrealdb/langchain-surrealdb/blob/main/langchain_surrealdb/experimental/surrealdb_graph.py)) instances 2. adds documents to the vector store, including the embeddings ([what are embeddings?](/blog/find-your-celebrity-soulmate-with-the-magic-of-vector-search)) 3. builds a graph 4. based on a provided topic, does a vector search and a graph query to generate an answer in natural language For this example, the data that we are going to store and then retrieve is: - concept definitions: stored in the Vector Store - people who know about those concepts: stored in the Graph (e.g. Martin -> knows about -> SurrealDB) ## 1. Create the vector store and graph instances ```python import time from langchain_community.graphs.graph_document import GraphDocument, Node, Relationship from langchain_core.documents import Document from langchain_core.prompts import ChatPromptTemplate from langchain_ollama import OllamaEmbeddings from langchain_ollama.llms import OllamaLLM from surrealdb import Surreal from langchain_surrealdb.experimental.surrealdb_graph import SurrealDBGraph from langchain_surrealdb.vectorstores import SurrealDBVectorStore conn = Surreal("ws://localhost:8000/rpc") conn.signin({"username": "root", "password": "secret"}) conn.use("langchain", "demo") vector_store = SurrealDBVectorStore(OllamaEmbeddings(model="all-minilm:22m"), conn) graph_store = SurrealDBGraph(conn) vector_store.delete() graph_store.delete_nodes() ```
## 2. Add documents to the vector store ```python doc1 = Document( page_content="SurrealDB is the ultimate multi-model database for AI applications", metadata={"key": "sdb"}, ) doc2 = Document( page_content="Surrealism is an artistic and cultural movement that emerged in the early 20th century", metadata={"key": "surrealism"}, ) vector_store.add_documents(documents=[doc1, doc2], ids=["1", "2"]) ```
## 3. Build the graph ```python # Document nodes node_sdb = Node(id="sdb", type="Document") node_surrealism = Node(id="surrealism", type="Document") # People nodes node_martin = Node(id="martin", type="People", properties={"name": "Martin"}) node_tobie = Node(id="tobie", type="People", properties={"name": "Tobie"}) node_max = Node(id="max", type="People", properties={"name":"Max Ernst"}) # Edges graph_documents = [ GraphDocument( nodes=[node_martin, node_tobie, node_sdb], relationships=[ Relationship(source=node_martin, target=node_sdb, type="KnowsAbout"), Relationship(source=node_tobie, target=node_sdb, type="KnowsAbout") ], source=doc1, ), GraphDocument( nodes=[node_max, node_surrealism], relationships=[ Relationship(source=node_max, target=node_surrealism, type="KnowsAbout") ], source=doc2, ), ] graph_store.add_graph_documents(graph_documents) ```
## 4. Let’s get an LLM involved For this example we are using [OllamaLLM](https://python.langchain.com/docs/integrations/llms/ollama/) from the LangChain components. You can use any other of the LLM components. For Ollama, here are [all the parameters](https://docs.ollama.com/modelfile). In the example we turned the `temperature` up to the max to get the craziest outcomes possible. You may want to leave it at around 0.7, but it depends on your use case. ```python model = OllamaLLM(model="llama3.2", temperature=1, verbose=True) # Let's retrieve information about these 2 topics queries = ["database", "surrealism"] for q in queries: print(f'\n----------------------------------\nTopic: "{q}"\nVector search:') results = vector_store.similarity_search_with_score(query=q, k=2) for doc, score in results: print(f"• [{score:.0%}]: {doc.page_content}") top_match = results[0][0] # Graph query res = graph_store.query( """ SELECT <-relation_KnowsAbout<-graph_People as people FROM type::record("graph_Document", $doc_key) FETCH people """, {"doc_key": top_match.metadata.get("key")}, ) people = [x.get("name") for x in res[0].get("people", [])] print(f"\nGraph result: {people}") # Template for the LLM template = """ You are a young, energetic database developer in your last 20s, who loves to talk tech, and who's also very geeky. Use the following pieces of retrieved context to answer the question. Use four sentences maximum and keep the answer concise. Try to be funny with a play on words. Context: {context}. People who know about this: {people}. Question: Explain "{topic}", summarize the context provided, and tell me who I can ask for more information. Answer: """ prompt = ChatPromptTemplate.from_template(template) chain = prompt | model answer = chain.invoke( {"context": top_match.page_content, "people": people, "topic": q} ) print(f"\nLLM answer:\n===========\n{answer}") time.sleep(4) print("\nBye!") ```
## Let’s try it out ```text ---------------------------------- Topic: "database" Vector search: • [34%]: SurrealDB is the ultimate multi-model database for AI applications • [22%]: Surrealism is an artistic and cultural movement that emerged in the early 20th century Graph result: ['Martin', 'Tobie'] LLM answer: =========== "Databases are like my brain, but less cluttered (mostly). Seriously though, a database is a storage system that organises data in a way that allows efficient retrieval. Given our conversation about SurrealDB being the ultimate multi-model database for AI applications, it's no wonder Martin and Tobie are experts on the subject - they're the 'key' people to ask! ---------------------------------- Topic: "surrealism" Vector search: • [52%]: Surrealism is an artistic and cultural movement that emerged in the early 20th century • [46%]: SurrealDB is the ultimate multi-model database for AI applications Graph result: ['Max Ernst'] LLM answer: =========== It looks like we've stumbled into a "dream" world of art - Surrealism is all about exploring the subconscious mind through bizarre and fantastical creations. Think weird creatures, melting objects, and illogical scenarios - it's like someone poured a SQL query into my brain! According to the context, Max Ernst was one of the pioneers of this wild movement. If you want to learn more, I'd recommend checking out some online tutorials or asking a fellow art history geek (just don't ask me to explain it in code) Bye! ```
## Bonus stage Did you notice the prompt in the code? Be creative and try different personalities: ```python template = """ You are rap singer, who always rhymes when talking. ... """ ```
🥁 Drum roll... ```text LLM answer: =========== Yo, listen up, let me share a tale, About databases, it's time to prevail. A database's a collection, of data so fine, Storing info, making it all mine. Context's tight, about SurrealDB's the claim, Tobie and Martin know its AI fame. For more deets, ask them with glee, They'll hook you up with knowledge, just wait and see! ```
It never disappoints. ```text LLM answer: =========== Yo, listen up, let me share a tale, Surrealism's the movement that set sail. It emerged in the 20th century's early prime, Influenced by Max Ernst, the pioneer of the time. ```
## Ready to build? Find all the code in the [langchain-surrealdb repository examples](https://github.com/surrealdb/langchain-surrealdb/tree/main/examples/basic/graph.py). Get started for free with [Surreal Cloud](https://studio.surrealdb.com/signin). Any questions or thoughts about this or graph queries using SurrealDB? [Join our Discord](https://discord.gg/surrealdb) - `#all-ai` and `#surrealql` are good places to start for this topic, and `#help` or `#general` work for anything else. --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/overview # Tutorials Step-by-step tutorials and walkthroughs for specific tasks with SurrealDB: integrations, real-time apps, AI patterns, and more. This section holds **tutorials**: practical **walkthroughs** that take you through a task or feature in order, with commands, configuration, and code you can follow along with. They are aimed at learning and production-style setup rather than only skimming a finished app. If you prefer a **ready-made project** you can clone and run with minimal narrative, see the [demos](/docs/explore/tutorials/demos/overview.md) section instead. If you are new to SurrealDB, you may also want [Getting started](/docs) and the [SurrealQL](/docs/reference/query-language.md) reference. ## Getting started - [Define a schema in SurrealDB](/docs/explore/tutorials/tutorials/define-a-schema.md) - build up a schemafull table with fields, assertions and indexes - [Query SurrealDB from Postman](/docs/explore/tutorials/tutorials/http-via-postman.md) - drive the HTTP endpoints from a REST client - [Connect to SurrealDB via ngrok](/docs/explore/tutorials/tutorials/connect-via-ngrok.md) - expose a local instance to a remote client - [Use SurrealDB in GitHub Actions](/docs/explore/tutorials/tutorials/github-actions.md) - run a database alongside your CI jobs ## Authentication - [Integrate Auth0 with SurrealDB](/docs/explore/tutorials/tutorials/auth0-integration.md) - verify Auth0 tokens with a record access method - [Integrate AWS Cognito with SurrealDB](/docs/explore/tutorials/tutorials/aws-cognito-integration.md) - the same pattern with a Cognito user pool ## AI and search - [Build an AI agent with Python](/docs/explore/tutorials/tutorials/build-an-ai-agent.md) - an agent that queries the database as a tool - [Build a GenAI chatbot with Graph RAG](/docs/explore/tutorials/tutorials/gen-ai-chatbot.md) - retrieval over a graph rather than a flat vector store - [Build a knowledge graph for AI](/docs/explore/tutorials/tutorials/how-to-build-a-knowledge-graph-for-ai.md) - model entities and relations for retrieval - [Build a minimal LangChain chatbot](/docs/explore/tutorials/tutorials/minimal-langchain.md) - the smallest working LangChain setup - [Implement semantic search in Rust](/docs/explore/tutorials/tutorials/semantic-search-in-rust.md) - embeddings, a vector index, and search from the Rust SDK ## Real-time - [Build a real-time presence app](/docs/explore/tutorials/tutorials/build-a-real-time-presence-app.md) - live queries driving who is online --- Source: https://surrealdb.com/docs/explore/tutorials/tutorials/semantic-search-in-rust # Implement semantic search in Rust In this guide, you'll learn how to implement semantic search in Rust using either Mistral AI or OpenAI via their Rust crates. This guide demonstrates how to use the Rust SDK to store AI embeddings from Mistral AI or OpenAI which can then be queried natively in SurrealQL. **Mistral AI** ## Using Mistral Semantic search in Rust with SurrealDB and Mistral AI The purpose of this post is to demonstrate how to use the Rust SDK to store Mistral AI embeddings as [SurrealDB vectors](/docs/learn/data-models/vector-search/overview.md), which can then be queried natively in SurrealQL to perform semantic search. This guide uses Rust's [mistralai-client](https://crates.io/crates/mistralai-client) crate to generate embeddings, but the code below can be modified to suit [other languages](https://docs.mistral.ai/getting-started/clients/#rust) that have clients for Mistral's AI platform. If you are a Python user, check out [this page](/docs/build/integrations/embeddings-providers/mistral.md) in the documentation for another ready-made example. ### Setup Setting up an embedded SurrealDB database requires no installation and can be done in just a few lines of code. After creating a new Cargo project with `cargo new project_name` and going into the project folder, add the following dependencies inside `Cargo.toml`: ```toml anyhow = "1.0.102" mistralai-client = "0.14.0" tokio = { version = "1.50.0", features = ["rt-multi-thread", "macros"] } surrealdb = { version = "3.0.4", default-features = false, features = ["kv-mem"] } ```
You can add the same dependencies on the command line through a single command: ```bash cargo add anyhow mistralai-client tokio surrealdb --features surrealdb/kv-mem ```
Setting up a SurrealDB database in Rust is as easy as calling the `connect` function with `"memory"` to instantiate an embedded database in memory. This code uses `anyhow` to allow the question mark operator to be used, but you can also just begin with `.unwrap()` everywhere and eventually move on to your own preferred error handling. ```rust use anyhow::Error; use surrealdb::engine::any::connect; #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; Ok(()) } ```
If you have a running Cloud or local instance, you can pass that path into the `connect()` function instead. ```rust // Cloud address let db = connect("wss://cloud-docs-068rp16e0hsnl62vgooa7omjks.aws-euw1.staging.surrealdb.cloud").await?; // Local address let db = connect("ws://localhost:8000").await?; ```
After connecting, we will select a namespace and database name, such as `ns` and `db`. ```rust db.use_ns("main").use_db("main").await?; ```
### Create a vector table and index Next we'll create a table called `document` to store documents and embeddings, along with an index for the embeddings. The statements look like this: ```surql DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE; ```
The important piece to understand is the relationship between the `embedding` field, a simple array of floats, and the index that we have given the name `hnsw_embed`. The size of the vector (1024 here) represents the number of dimensions in the embedding. This is to match Mistral AI's `mistral-embed` model, which uses [1024 as its length](https://docs.mistral.ai/getting-started/models/models_overview/#premier-models). The [HNSW index](/docs/learn/data-models/vector-search/vector-indexes.md) is not strictly necessary to use the KNN operator (`<||>`) to find an embedding's closest neighbours, and for our small sample code we will use the simple [brute force method](/docs/reference/query-language/language-primitives/operators.md#brute-force-method) which chooses [an algorithm](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. The following is the code that we will use, which uses the cosine of an embedding to find the four closest neighbours. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance; ``` As the dataset grows, however, the syntax can be changed to use [the HNSW index](/docs/reference/query-language/language-primitives/operators.md#hnsw-method), by replacing an algorithm with a number that represents the size of the dynamic candidate list. This index is recommended when a small loss of accuracy is acceptable in order to preserve performance. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,40|> $embeds ORDER BY distance; ``` Inside the Rust SDK we can put all four of these inside a single `.query()` call and then add a line to see if there are errors inside any of them. ```rust let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE;", ) .await?; for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } ```
### Generate Mistral AI embeddings At this point, you will need a [key](https://console.mistral.ai/api-keys) to interact with Mistral AI's platform. They offer a free tier for experimentation, after which you will be able to create a key to interact with it via the code below. The code in this page will still work without a proper code, but the request to the Mistral AI API will end up returning the following error message. ```text Error: ApiError: 401 Unauthorized: {"detail":"Unauthorized"} ```
The best way to set the key is as an environment variable, which we will set to be a static called `KEY`. The client will look for one called `MISTRAL_API_KEY`, though you can change this when setting up the Mistral AI Rust client if you like. ```rust // Looks for MISTRAL_API_KEY let client = Client::new(Some(KEY.to_string()), None, None, None)?; // Looks for OTHER_ENV_VAR let client = Client::new(Some(KEY.to_string()), Some("OTHER_ENV_VAR".to_string()), None, None)?; ``` Using a `LazyLock` will let us call it via `std::env::var()` function the first time it is accessed. You can of course simply put it into a `const` for simplicity when first testing, but always remember to never hard-code API keys in your code in production. ```rust static KEY: LazyLock = LazyLock::new(|| { std::env::var("MISTRAL_API_KEY").unwrap() }); ```
And then run the code like this: ```bash MISTRAL_API_KEY=whateverthekeyis cargo run ```
Or like this if you are using PowerShell on Windows. ```powershell $env:MISTRAL_API_KEY = "whateverthekeyis" cargo run ```
We can also create a `const MODEL` to hold the Mistral AI model used, which in this case is an `EmbedModel::MistralEmbed`. ```rust const MODEL: EmbedModel = EmbedModel::MistralEmbed; ``` Inside `main()`, we will then [create a client](https://docs.rs/mistralai-client/0.14.0/mistralai_client/v1/client/struct.Client.html#method.new) from the `mistralai-client` crate. ```rust let client = Client::new(Some(KEY.to_string()), None, None, None)?; ```
We'll use that to generate a Mistral AI embedding using the [`mistral-embed`](https://docs.mistral.ai/getting-started/models/models_overview/#premier-models) model. The `mistralai-client` has both sync and async functions that take a `Vec`, and since SurrealDB uses the tokio runtime, we'll call the async `.embeddings_async()` method. ```rust let input = vec!["Joram is the main character in the Darksword Trilogy.".to_string()]; let result = client.embeddings_async(MODEL, input, None).await?; println!("{:?}", result); ```
The output in your console should show a massive number of floats, 1024 of them to be precise. That's the embedding for this input! ### Store embeddings in database Now that we have the embedding returned from the Mistral AI client, we can store it in the database. The [response](https://docs.rs/mistralai-client/0.14.0/mistralai_client/v1/embedding/struct.EmbeddingResponse.html) returned from the mistralai-client crate looks like this, with a `Vec` of `EmbeddingResponseDataItem` structs that hold a `Vec`. ```rust pub struct EmbeddingResponse { pub id: String, pub object: String, pub model: EmbedModel, pub data: Vec, pub usage: ResponseUsage, } pub struct EmbeddingResponseDataItem { pub index: u32, pub embedding: Vec, pub object: String, } ```
We know that our simple request only returned a single embedding, so `.remove(0)` will do the job. In a more complex codebase you would probably opt for a match on `.get(0)` to handle any possible errors. ```rust let embeds = result.data.remove(0).embedding; ```
There are a [number of ways](/docs/reference/rust/concepts/flexible-typing.md) to work with or avoid structs when using the Rust SDK, but we'll just go with two basic structs: one to represent the input into a `.create()` statement, and one to display the results. ```rust #[derive(SurrealValue)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, SurrealValue)] struct Document { id: RecordId, embedding: Vec, text: String, } ```
Once that is done, we can print out the created documents as a `Document` struct. We'll fiddle with the code a bit to have the `input` start as a `&str` which will be turned into a `String` in order to get the embedding, as well as to create a `Document` struct. ```rust let input = "Octopuses solve puzzles and escape enclosures, showing advanced intelligence."; let mut result = client .embeddings_async(MODEL, vec![input.to_string()], None) .await?; let embeds = result.data.remove(0).embedding; let in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; println!("{in_db:?}"); ```
We should now add some more `document` records. To do this, we'll move the logic to create them inside a function of its own. Since the `embeddings_async()` method takes a single `Vec`, we'll first clone it to keep the original `Vec` around, then zip it together with the embeddings returned so that they can be put into the database along with the original input. ```rust async fn create_embeds( input: Vec, db: &Surreal, client: &Client, ) -> Result<(), Error> { let cloned = input.clone(); let embeds = client.embeddings_async(MODEL, input, None).await?; let zipped = cloned .into_iter() .zip(embeds.data.into_iter().map(|item| item.embedding)); for (text, embeds) in zipped { let _in_db = db .create::>("document") .content(DocumentInput { text, embedding: embeds, }) .await?; } Ok(()) } ```
Then we'll create four facts for each of four topics: sea creatures, Korean and Japanese cities, historical figures, and planets of the Solar System (including the dwarf planet Ceres). ```rust let embeds = [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks exhibit learning behavior, but their intelligence is instinct-driven.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] .into_iter() .map(|s| s.to_string()) .collect::>(); create_embeds(embeds, &db, &client).await?; ```
### Semantic search Finally let's perform semantic search over the embeddings in our database. We'll go with this query that uses the KNN operator to return the closest four matches to an embedding. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance; ```
You can customise this [with other algorithms](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. We will then put this into a separate function called `ask_question()` which looks similar `create_embed()`, except that it first prints out its input, and then uses its embedding retrieved from Mistral to query the database against existing documents instead of creating a new document. ```rust async fn ask_question(input: &str, db: &Surreal, client: &Client) -> Result<(), Error> { println!("{input}"); let embeds = client .embeddings_async(MODEL, vec![input.to_string()], None) .await? .data .remove(0) .embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } ```
Finally, we will call this function a few times inside `main()` to confirm that the results are what we expect them to be, printing out the results of each so that we can eyeball them and make sure that they are what we expect them to be. ```rust ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; ```
The output shows that the facts that fit most to our questions end up displayed first, with differing distance depending on how close the other facts were. Octopuses end up smarter than sharks (which is true), but the "learning behaviour" part of our input does end up making sharks score pretty close. On the other extreme, Wilhelm II is clearly the only input that comes anywhere close to "Germany's last Kaiser", with Napoleon Bonaparte way behind. Poor Aristotle doesn't make it into any results, with "Sejong is South Korea's planned administrative capital" slightly closer semantically in terms of "Who was Germany's last Kaiser". ```text Which Korean city is just across the sea from Japan? [{ distance: 0.19170371029549582f, text: 'Busan is a major South Korean port located in the far southeast.' }, { distance: 0.2399314515762122f, text: 'Tokyo is Japan’s capital, known for innovation and dense population.' }, { distance: 0.2443623703771407f, text: 'Sejong is South Korea’s planned administrative capital.' }, { distance: 0.24488082839731895f, text: 'Seoul is South Korea’s capital and a global tech hub.' }] Who was Germany's last Kaiser? [{ distance: 0.11228576780228805f, text: 'Wilhelm II was Germany’s last Kaiser before World War I.' }, { distance: 0.2957177300085634f, text: 'Napoleon Bonaparte was a French emperor and brilliant military strategist.' }, { distance: 0.34394473621670896f, text: 'Cyrus the Great founded the Persian Empire with tolerant rule.' }, { distance: 0.34911517400935843f, text: 'Sejong is South Korea’s planned administrative capital.' }] Which sea animal is most intelligent? [{ distance: 0.2342596053829904f, text: 'Octopuses solve puzzles and escape enclosures, showing advanced intelligence.' }, { distance: 0.24131327939924785f, text: 'Sharks exhibit learning behavior, but their intelligence is instinct-driven.' }, { distance: 0.2426242772516931f, text: 'Clams have simple nervous systems with no known intelligent behavior.' }, { distance: 0.24474598154128135f, text: 'Sea cucumbers lack a brain and show minimal cognitive response.' }] Which planet's atmosphere has a part with the same temperature as Earth? [{ distance: 0.20653440713083582f, text: 'Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.' }, { distance: 0.23354208810464594f, text: 'Mars has a thin, cold atmosphere with seasonal dust storms.' }, { distance: 0.24560810032473468f, text: 'Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior' }, { distance: 0.2761595357544341f, text: 'Ceres has a tenuous exosphere with sporadic water vapor traces.' }] ```
As the database grows, you could also change the `<|4,COSINE|>` part of the query to something like `<|4,40|>` to see the results using the HNSW index instead of the brute force method. Finally, here is all of the code for you to run and modify as you wish. ```rust use std::sync::LazyLock; use anyhow::Error; use mistralai_client::v1::{client::Client, constants::EmbedModel}; use surrealdb::{ Surreal, engine::any::{Any, connect}, types::{RecordId, SurrealValue, ToSql, Value}, }; static KEY: LazyLock = LazyLock::new(|| std::env::var("MISTRAL_API_KEY").unwrap()); // Experiment plan const MODEL: EmbedModel = EmbedModel::MistralEmbed; #[derive(SurrealValue)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, SurrealValue)] struct Document { id: RecordId, embedding: Vec, text: String, } async fn create_embeds( input: Vec, db: &Surreal, client: &Client, ) -> Result<(), Error> { let cloned = input.clone(); let embeds = client.embeddings_async(MODEL, input, None).await?; let zipped = cloned .into_iter() .zip(embeds.data.into_iter().map(|item| item.embedding)); for (text, embeds) in zipped { let _in_db = db .create::>("document") .content(DocumentInput { text, embedding: embeds, }) .await?; } Ok(()) } async fn ask_question(input: &str, db: &Surreal, client: &Client) -> Result<(), Error> { println!("{input}"); let embeds = client .embeddings_async(MODEL, vec![input.to_string()], None) .await? .data .remove(0) .embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|4,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await.unwrap(); db.use_ns("ns").use_db("db").await.unwrap(); let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1024 DIST COSINE;", ) .await .unwrap(); for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } let client = Client::new(Some(KEY.to_string()), None, None, None)?; let embeds = [ "Octopuses solve puzzles and escape enclosures, showing advanced intelligence.", "Sharks exhibit learning behavior, but their intelligence is instinct-driven.", "Sea cucumbers lack a brain and show minimal cognitive response.", "Clams have simple nervous systems with no known intelligent behavior.", // "Seoul is South Korea’s capital and a global tech hub.", "Sejong is South Korea’s planned administrative capital.", "Busan is a major South Korean port located in the far southeast.", "Tokyo is Japan’s capital, known for innovation and dense population.", // "Wilhelm II was Germany’s last Kaiser before World War I.", "Cyrus the Great founded the Persian Empire with tolerant rule.", "Napoleon Bonaparte was a French emperor and brilliant military strategist.", "Aristotle was a Greek philosopher who shaped Western intellectual thought.", // "Venus’s atmosphere ranges from scorching surface to Earth-like upper clouds.", "Mars has a thin, cold atmosphere with seasonal dust storms.", "Ceres has a tenuous exosphere with sporadic water vapor traces.", "Saturn’s atmosphere spans cold outer layers to a deep metallic hydrogen interior", ] .into_iter() .map(|s| s.to_string()) .collect::>(); create_embeds(embeds, &db, &client).await?; ask_question("Which Korean city is just across the sea from Japan?", &db, &client).await?; ask_question("Who was Germany's last Kaiser?", &db, &client).await?; ask_question("Which sea animal is most intelligent?", &db, &client).await?; ask_question("Which planet's atmosphere has a part with the same temperature as Earth?", &db, &client).await?; Ok(()) } ```
**OpenAI** ## Using OpenAI Semantic search in Rust with SurrealDB and OpenAI This guide demonstrates how to store OpenAI embeddings as [SurrealDB vectors](/docs/learn/data-models/vector-search/overview.md) via the Rust SDK for the purposes of semantic search. It uses Rust's [async-openai](https://crates.io/crates/async-openai/0.28.3) crate to generate embeddings. ### Setup Setting up an embedded SurrealDB database only takes a few lines of code. After creating a new Cargo project with `cargo new project_name` and going into the project folder, we will then add the following dependencies inside `Cargo.toml`: ```toml anyhow = "1.0.98" async-openai = "0.34.0" surrealdb = { version = "3.0.4", features = ["kv-mem"] } tokio = "1.45.0" ```
They can also be added on the command line using this command: ```bash cargo add anyhow async-openai tokio surrealdb --features surrealdb/kv-mem ```
Inside `main()`, we can call the `connect` function with `"memory"` to instantiate an embedded database in memory. With the possibility of error types from various sources, using `anyhow` is the easiest way to get started. ```rust use anyhow::Error; use surrealdb::engine::any::connect; #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; Ok(()) } ```
If you have a Cloud or local instance to connect to, you can pass that path into the connect function instead. ```rust // Cloud address let db = connect("wss://cloud-docs-068rp16e0hsnl62vgooa7omjks.aws-euw1.staging.surrealdb.cloud").await?; // Local address let db = connect("ws://localhost:8000").await?; ```
After connecting, we will select a namespace and database name, such as `ns` and `db`. ```rust db.use_ns("ns").use_db("db").await?; ```
### Create a vector table Next we'll create a table to store documents and embeddings, along with an index for the embeddings. The statements look like this: ```surql DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536; ```
Inside the SDK we can put all four of these inside a single `.query()` call and then add a line to see if there are errors inside any of them. ```rust let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536;", ) .await?; for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } ```
The important piece to understand is the relationship between the `embedding` field, a simple array of floats, and the `hnsw_embed` index. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since OpenAI's `text-embedding-3-small` model in this example uses [1536 as its default length](https://platform.openai.com/docs/guides/embeddings), we set the vector size to 1536. The [HNSW index](/docs/learn/data-models/vector-search/vector-indexes.md) is not strictly necessary to use the KNN operator (`<||>`) to find an embedding's closest neighbours, and for our small sample code we will use the simple [brute force method](/docs/reference/query-language/language-primitives/operators.md#brute-force-method) which chooses [an algorithm](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. The following is the code that we will use, which uses the cosine of an embedding to find the four closest neighbours. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance; ``` As the dataset grows, if some loss of accuracy is acceptable then the syntax can be changed to use [the HNSW index](/docs/reference/query-language/language-primitives/operators.md#hnsw-method), by replacing an algorithm with a number that represents the size of the dynamic candidate list. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,40|> $embeds ORDER BY distance; ``` ### Generate OpenAI embeddings At this point, you will need an [OpenAI API key](https://platform.openai.com/api-keys) to interact with the OpenAI API. You can still check the code to see if it works if you don't have a key, and you will get as far as this error message. ```text Error: invalid_request_error: Incorrect API key provided: blah. You can find your API key at https://platform.openai.com/account/api-keys. (code: invalid_api_key) ```
The best way to set the key is as an environment variable, `OPENAI_API_KEY` in this case. Using a `LazyLock` will let us call it via `std::env::var()` function the first time it is accessed. You can of course simply put it into a `const` for simplicity when first testing, but always remember to never hard-code API keys in your code in production. ```rust static KEY: LazyLock = LazyLock::new(|| { std::env::var("OPENAI_API_KEY").unwrap() }); ```
And then run the code like this: ```bash OPENAI_API_KEY=whateverthekeyis cargo run ```
Or like this if you are using PowerShell on Windows. ```powershell $env:OPENAI_API_KEY = "whateverthekeyis" cargo run ```
Inside `main()`, we will then [create a client](https://docs.rs/async-openai/0.28.3/async_openai/struct.Client.html) from the async-openai crate holding this config inside `main()`. ```rust let config = OpenAIConfig::new().with_api_key(KEY); let client = Client::with_config(config); ```
We'll use that to generate an OpenAI embedding using [`text-embedding-3-small`](https://platform.openai.com/docs/guides/embeddings/embedding-models), as follows. ```rust let input = "What does the cat chase?"; let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let result = client.embeddings().create(request).await?; println!("{result:?}"); ```
The output in your console should show a massive number of floats, 1536 of them to be precise. That's the embedding for this input! ### Store embeddings in database Now that we have the embedding returned from the OpenAI client, we can store it in the database. The [response](https://docs.rs/async-openai/0.28.3/async_openai/types/struct.CreateEmbeddingResponse.html) returned from the async-openai crate looks like this, with a `Vec` of `Embedding` structs that hold a `Vec`. ```rust pub struct CreateEmbeddingResponse { pub object: String, pub model: String, pub data: Vec, pub usage: EmbeddingUsage, } pub struct Embedding { pub index: u32, pub object: String, pub embedding: Vec, } ```
We know that our simple request only returned a single embedding, so `.remove(0)` will do the job. In a more complex codebase you would probably opt for a match on `.get(0)` to handle any possible errors. ```rust let embeds = result.data.remove(0).embedding; ```
There are a [number of ways](/docs/reference/rust/concepts/flexible-typing.md) to work with or avoid structs when using the Rust SDK, but we'll just go with two basic structs: one to represent the input into a `.create()` statement, and another to show the results. ```rust #[derive(SurrealValue)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, SurrealValue)] struct Document { id: RecordId, embedding: Vec, text: String, } ```
Once that is done, we can print out the created documents as a `Document` struct. ```rust let in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec() }) .await?; println!("{in_db:?}"); ```
We should now add some more `document` records. To do this, we'll move the logic to create them inside a function of its own: ```rust async fn create_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let result = client.embeddings().create(request).await?; let embeds = &result.data.get(0).unwrap().embedding; let _in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; Ok(()) } ```
And then call it a few times inside `main()`. See if you can guess the answers yourself! ```rust for input in [ "What does the cat chase?", "What do Fraggles love to eat?", "Which planet rotates slowly on its axis?", "Which Greek general helped Cyrus the Younger?", "What is the largest inland sea?"] { create_embed(input, &db, &client).await? } ```
### Semantic search Finally let's perform semantic search over the embeddings in our database. With that done, it's time to test the database out. We'll go with this query that uses the KNN operator to return the closest two matches to an embedding. ```surql SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance; ```
You can customise this [with other algorithms](/docs/learn/data-models/vector-search/similarity-search.md#computation-on-vectors-vector-package-of-functions) such as Euclidean, Hamming, and so on. We will then put this into a separate function called `test_embed()` which looks similar `create_embed()`, except that it uses its embedding retrieved from OpenAI to query the database against existing documents instead of creating a new document. ```rust async fn test_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{as_val}\n"); Ok(()) } ```
Finally, we will call this function a few times inside `main()` to confirm that the results are what we expect them to be, printing out the results of each so that we can eyeball them and make sure that they are what we expect them to be. ```rust println!("Venus is closest to:"); test_embed("Venus", &db, &client).await?; println!("Xenophon is closest to:"); test_embed("Xenophon", &db, &client).await?; println!("Mice are closest to:"); test_embed("mouse", &db, &client).await?; println!("Radishes are closest to:"); test_embed("radish", &db, &client).await?; println!("The Caspian Sea is closest to:"); test_embed("Caspian Sea", &db, &client).await?; ```
The output shows that in each case the closest document is returned first: * "Venus" to "Which planet rotates slowly on its axis?" * "Xenophon" to "Which Greek general helped Cyrus the Younger?" * "mouse" to "What does the cat chase?" * "radish" to "What do Fraggles love to eat?", and * "Caspian Sea" to "What is the largest inland sea?" Success! ```text Venus is closest to: [{ distance: 0.6495068000978139f, text: 'Which planet rotates slowly on its axis?' }, { distance: 0.8388033444017572f, text: 'Which Greek general helped Cyrus the Younger?' }] Xenophon is closest to: [{ distance: 0.4421917772479055f, text: 'Which Greek general helped Cyrus the Younger?' }, { distance: 0.873354690471173f, text: 'What does the cat chase?' }] Mice are closest to: [{ distance: 0.6945913095506092f, text: 'What does the cat chase?' }, { distance: 0.8249335430462937f, text: 'Which planet rotates slowly on its axis?' }] Radishes are closest to: [{ distance: 0.7256996315669555f, text: 'What do Fraggles love to eat?' }, { distance: 0.8812784798259233f, text: 'What does the cat chase?' }] The Caspian Sea is closest to: [{ distance: 0.49966454922547254f, text: 'What is the largest inland sea?' }, { distance: 0.8096568276647603f, text: 'Which Greek general helped Cyrus the Younger?' }] ```
At this point, you could give the HNSW index a try by changing the `<|2,COSINE|>` in the query to something like `<|2,40|>`. The distance numbers will end up looking quite different, but the ordering of the closest neighbours will probably be the same in this small example.
Finally, here is all of the code for you to run and modify as you wish. Any questions or thoughts about this or semantic search using SurrealDB? [Join our Discord](https://discord.gg/surrealdb) - `#rust` and `#all-ai` are good places to start for this topic, and `#help` or `#general` work for anything else. ```rust use std::sync::LazyLock; use anyhow::Error; use async_openai::{Client, config::OpenAIConfig, types::CreateEmbeddingRequestArgs}; use surrealdb::{ Surreal, engine::any::{Any, connect}, types::{RecordId, SurrealValue, ToSql, Value}, }; static KEY: LazyLock = LazyLock::new(|| std::env::var("OPENAI_API_KEY").unwrap()); #[derive(SurrealValue)] struct DocumentInput { text: String, embedding: Vec, } #[derive(Debug, SurrealValue)] struct Document { id: RecordId, embedding: Vec, text: String, } async fn create_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let _in_db = db .create::>("document") .content(DocumentInput { text: input.into(), embedding: embeds.to_vec(), }) .await?; Ok(()) } async fn test_embed( input: &str, db: &Surreal, client: &Client, ) -> Result<(), Error> { let request = CreateEmbeddingRequestArgs::default() .model("text-embedding-3-small") .input(input) .dimensions(1536u32) .build()?; let mut result = client.embeddings().create(request).await?; let embeds = result.data.remove(0).embedding; let mut response = db.query("SELECT text, vector::distance::knn() AS distance FROM document WHERE embedding <|2,COSINE|> $embeds ORDER BY distance;").bind(("embeds", embeds)).await?; let as_val: Value = response.take(0)?; println!("{}\n", as_val.to_sql()); Ok(()) } #[tokio::main] async fn main() -> Result<(), Error> { let db = connect("memory").await?; db.use_ns("ns").use_db("db").await?; let mut res = db .query( "DEFINE TABLE document; DEFINE FIELD text ON document TYPE string; DEFINE FIELD embedding ON document TYPE array; DEFINE INDEX hnsw_embed ON document FIELDS embedding HNSW DIMENSION 1536 DIST COSINE;", ) .await?; for (index, error) in res.take_errors() { println!("Error in query {index}: {error}"); } let config = OpenAIConfig::new().with_api_key(&*KEY); let client = Client::with_config(config); for input in [ "What does the cat chase?", "What do Fraggles love to eat?", "Which planet rotates slowly on its axis?", "Which Greek general helped Cyrus the Younger?", "What is the largest inland sea?", ] { create_embed(input, &db, &client).await? } println!("Venus is closest to:"); test_embed("Venus", &db, &client).await?; println!("Xenophon is closest to:"); test_embed("Xenophon", &db, &client).await?; println!("Mice are closest to:"); test_embed("mouse", &db, &client).await?; println!("Radishes are closest to:"); test_embed("radish", &db, &client).await?; println!("The Caspian Sea is closest to:"); test_embed("Caspian Sea", &db, &client).await?; Ok(()) } ``` --- Source: https://surrealdb.com/docs/learn/data-models # Data Models Store and query document, graph, vector, time-series, geospatial. All the data models SurrealDB supports, in one database. When you think of a database, you probably think of a table. A table is a collection of rows and columns, where each row represents a record and each column represents a field. In a relational database, these tables are connected by foreign keys, which are columns that reference the primary key of another table. SurrealDB is a multi-model database, which means that it can store data in different formats. This flexibility allows you to choose the most appropriate data model for your use case, whether you are storing [graph](/docs/learn/data-models/graph/overview.md), [document](/docs/learn/data-models/document/overview.md), [time-series](/docs/learn/data-models/time-series/overview.md), [vector](/docs/learn/data-models/vector-search/overview.md), [full-text search](/docs/learn/data-models/full-text-search/overview.md), or [geospatial](/docs/learn/data-models/geospatial/overview.md) data. Throughout this section, you will explore the different data models that SurrealDB supports and how they can be achieved using SurrealQL. You will also learn about how to think in SurrealDB whether you are a SQL, NoSQL, or graph developer. ## Resources - [Document](/docs/learn/data-models/document/overview.md) - [Graph and record links](/docs/learn/data-models/graph/overview.md) - [Vector](/docs/learn/data-models/vector-search/overview.md) - [Text and full-text search](/docs/learn/data-models/full-text-search/overview.md) - [Time series](/docs/learn/data-models/time-series/overview.md) - [Geospatial](/docs/learn/data-models/geospatial/overview.md) --- Source: https://surrealdb.com/docs/learn/data-models/architecture # Architecture How SurrealDB separates compute from storage. Every data model maps onto one storage engine, and namespaces, databases and tables give the structure. SurrealDB separates the query engine (compute) from the storage layer. The engine speaks one query language and exposes one API, while the storage layer decides how data is persisted, replicated and scaled. Because the two are decoupled, the same database, queries and SDK calls work from an embedded edge application through to a distributed cloud cluster. Diagram of SurrealDB's layered architecture: clients connect through [client].surrealdb.com to multiple compute nodes, all backed by centralised storage on AWS S3. This page describes the two layers, how the [data models](/docs/learn/data-models.md) share a single store, and how a SurrealDB deployment is organised. ## Query layer The query layer handles client requests and coordinates work against storage: - Parses and executes [SurrealQL](/docs/reference/query-language.md) - Authenticates connections and sessions, for example through `SIGNIN` and access methods - Enforces table- and field-level `PERMISSIONS` as records are read and written - Plans index-backed queries, updates index entries on writes, and coordinates [transactions](/docs/learn/querying/concepts-and-guides/transactions.md) against storage Incoming SurrealQL passes through a parser, an executor that groups statements into transactions, an iterator that plans data access and fetches keys from storage, and a document processor that applies permissions and persists changes through the storage API. Every transaction runs under snapshot isolation with write conflict detection on commit, whichever storage backend sits underneath. A [`SELECT ... FOR UPDATE`](/docs/reference/query-language/statements/select.md#the-for-update-clause) read extends that detection to records the transaction reads without writing. ## Storage layer The storage layer handles persistence and durability. It determines deployment characteristics such as scalability, temporal versioning, replication and fault tolerance. SurrealDB integrates with several engines depending on how you run the database: | Concern | Typical engine | | --- | --- | | Single-node production | [RocksDB](https://rocksdb.org/) (recommended for single node) | | Single-node or embedded | [SurrealKV](https://github.com/surrealdb/surrealkv) (beta) | | In-memory server or embedded | [SurrealMX](https://github.com/surrealdb/surrealmx) | | Browser persistence | [IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) | | Distributed multi-node | Shared distributed storage on [SurrealDB Cloud Scale](https://surrealdb.com/pricing/scale) and self-hosted Enterprise | Each engine must support transactional read and write of individual keys and key ranges. That is the whole contract, which is why the query layer can offer identical semantics across every deployment model. ## One store for every data model SurrealDB is a document database at its core. Each record is a document held on a key-value engine, and it can carry arbitrary nested objects and arrays. The other models come from how those keys are laid out rather than from separate subsystems: - **[Record IDs](/docs/reference/query-language/language-primitives/data-types/record-ids.md)** sort in a defined order, so a range read over a table returns records in that order. This is what makes [time-series](/docs/learn/data-models/time-series/overview.md) access a range scan rather than a scan-and-filter. - **Graph edges** are records of their own, created with [`RELATE`](/docs/reference/query-language/statements/relate.md). Traversal reads the edge table by key, so a [graph query](/docs/learn/data-models/graph/overview.md) stays a series of key lookups. - **Indexes** for [vector](/docs/learn/data-models/vector-search/overview.md) and [full-text](/docs/learn/data-models/full-text-search/overview.md) search are index entries in the same store, maintained by the query layer as records change. One consequence matters for application design: a write that touches a document, its edges and its index entries is one transaction against one store, so those parts cannot drift apart. ## System structure SurrealDB is a multi-tenant platform. Resources nest in four levels, and each level has its own [`DEFINE`](/docs/reference/query-language/statements/define/overview.md) statement: | Level | Statement | Purpose | | --- | --- | --- | | [Namespace](/docs/reference/query-language/statements/define/namespace.md) | `DEFINE NAMESPACE` | Isolation for an organisation, department or team. No limit on the number of namespaces. | | [Database](/docs/reference/query-language/statements/define/database.md) | `DEFINE DATABASE` | The unit that holds data. Each database has its own tables, indexes, schema, settings and permissions. No limit per namespace. | | [Table](/docs/reference/query-language/statements/define/table.md) | `DEFINE TABLE` | A collection of records. Called a collection in some other systems. | | [Field](/docs/reference/query-language/statements/define/field.md) | `DEFINE FIELD` | A typed field on a table, with optional assertions and permissions. | A row or document in SurrealDB is called a **record**, and a column is called a **field**. Records are [created](/docs/reference/query-language/statements/create.md), [read](/docs/reference/query-language/statements/select.md), [updated](/docs/reference/query-language/statements/update.md) and [deleted](/docs/reference/query-language/statements/delete.md), with further statements for common patterns: [`UPSERT`](/docs/reference/query-language/statements/upsert.md) writes a record whether or not it already exists, and [`RELATE`](/docs/reference/query-language/statements/relate.md) links two records through an edge table. ### Namespaces and databases A namespace is the outer container. It holds databases and nothing else, which makes it the natural boundary in multi-tenant deployments where separate applications or teams share one instance or cluster. Permissions and access methods can be granted at namespace level rather than per database. A database sits inside a namespace and is where data lives: tables, records, indexes, events, functions and access methods. Most work happens here. Both are defined with a unique name and an optional comment. The [`USE`](/docs/reference/query-language/statements/use.md) statement switches the session from one namespace or database to another. ```surql DEFINE NAMESPACE dev_namespace COMMENT "Internal use only: do not use in prod"; USE NAMESPACE dev_namespace; -- Now inside 'dev_namespace', define a database within it DEFINE DATABASE dev_db_1 COMMENT "First of many dev databases"; ``` Multiple [access](/docs/reference/query-language/statements/define/access.md) methods can be defined on a namespace or a database. The [record](/docs/reference/query-language/statements/define/access/record.md) access method authenticates your end users against your own tables, down to field level. ### Inspecting a deployment The [`INFO`](/docs/reference/query-language/statements/info.md) statement reports what exists at each level: - `INFO FOR ROOT`: namespaces, root users, and system information such as allocated memory and physical cores. - `INFO FOR NAMESPACE`: the namespace's databases, users and access methods. - `INFO FOR DATABASE`: the database's tables, users, accesses, functions, analysers and more. Each resource is reported as the `DEFINE` statement that would recreate it, which makes the output a readable snapshot of the schema: ```surql title="Abridged output of INFO FOR DATABASE" { accesses: { }, analyzers: { blank_snowball: 'DEFINE ANALYZER blank_snowball TOKENIZERS BLANK FILTERS LOWERCASE, SNOWBALL(ENGLISH)' }, functions: { pound_to_usd: 'DEFINE FUNCTION fn::pound_to_usd($price: number) -> float { $price * 1.26f } PERMISSIONS FULL' }, tables: { order: 'DEFINE TABLE order TYPE RELATION IN person OUT product SCHEMAFULL PERMISSIONS NONE', user: 'DEFINE TABLE user TYPE ANY SCHEMALESS PERMISSIONS NONE' }, users: { Boris: "DEFINE USER Boris ON DATABASE PASSHASH '[REDACTED]' ROLES VIEWER DURATION FOR TOKEN 1h, FOR SESSION NONE" } } ``` Further `INFO` statements report the state of individual tables, users and indexes. ## Deployment models The separation of compute from storage gives four ways to run the same database: - **Embedded**, in memory through SurrealMX, on disk through RocksDB or SurrealKV, or in the browser through IndexedDB. - **Single-node self-hosted**, on RocksDB, or on SurrealKV while it is in beta. - **Multi-node self-hosted**, on [managed Kubernetes](/docs/manage/self-hosted/managed-kubernetes.md) such as EKS, GKE or AKS. - **Managed**, through [managed instances](/docs/manage/instances.md), from single-node **Start** instances to multi-node **Scale** clusters on distributed storage. Development can begin on an embedded or single-node deployment and move to a cluster later without changing application code or queries. For storage trade-offs and how to choose a model, see [Deployment models](/docs/manage/self-hosted/deployment-models.md). ## Where SurrealDB sits in your stack SurrealDB works as a conventional database behind a backend service, using the SDKs for Go, Python, Rust, C, Java, .NET, Node.js or PHP. It can also serve a frontend directly. Table-, record- and field-level permissions, combined with [record access](/docs/reference/query-language/statements/define/access/record.md) authentication, let a browser or mobile client connect to the database and still only see the data it is entitled to. The JavaScript SDK, WebAssembly and the framework integrations for React, Next.js, Vue, Svelte and others support this arrangement. Both approaches use the same query language and the same permission model, so a project can start with one and add the other later. --- Source: https://surrealdb.com/docs/learn/data-models/document/common-patterns # Common patterns Map document-database concepts to SurrealDB, compare SurrealQL with MongoDB-style operations, and find resources for CRUD and migration. When thinking in a document model database, you will often find that the concepts align closely with SurrealDB. The table below maps common terms. ## Concept mapping
Document model SurrealDB
database database
collection table
document record
field field
index index
Objectid record id
transactions transactions
reference and embedding record links, embedding and graph relations
## Benefits of using a document model database 1. **Flexibility**: You don’t need to define rigid schemas in advance. Changes to data structure are often just changes in the JSON object itself. 2. **Natural data representation**: Since you’re working with JSON-like objects, document databases align well with modern programming languages that manipulate data as objects or dictionaries. 3. **Simplicity of application code**: Because you can embed everything related to an entity in a single document, you often have fewer JOINs (or complex queries) and simpler code for retrieving complete objects. 4. **Easier horizontal scaling**: Many document databases are built for horizontal partitioning (sharding), making them easier to scale for large workloads. ## Using MongoDB syntax as a reference As a multi-model database, SurrealDB offers a lot of flexibility. SurrealQL often provides more than one way to achieve the same result, depending on developer preference. The mapping below focuses on syntax that most closely resembles the MongoDB query language (MQL). For SurrealQL equivalents to MongoDB data types and how to import MongoDB data into SurrealDB, see the Surreal Sync migration tool: * [Surreal Sync for MongoDB](https://github.com/surrealdb/surreal-sync/blob/main/docs/mongodb.md) * [MongoDB data types support in Surreal Sync](https://github.com/surrealdb/surreal-sync/blob/main/docs/mongodb-data-types.md) ## Syntax mapping ### Create As MongoDB is schemaless, only the SurrealQL schemaless approach is shown below. For a schemafull option, see the [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) page. For more SurrealQL examples, see the [`CREATE`](/docs/reference/query-language/statements/create.md) and [`INSERT`](/docs/reference/query-language/statements/insert.md) pages.
MQL SurrealQL
db.createCollection("person") CREATE person
db.person.insertMany([{ name: "John" }, { name: "Jane" }]) INSERT INTO person [{name: "John"}, {name: "Jane"}]
db.person.createIndex({ name: 1 }) DEFINE INDEX idx_name ON TABLE person FIELDS name
### Read For more SurrealQL examples, see the [`SELECT`](/docs/reference/query-language/statements/select.md), [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md) and [`RETURN`](/docs/reference/query-language/statements/return.md) pages.
MQL SurrealQL
db.person.find() SELECT * FROM person
db.person.find({}, { _id: 0, name: 1 }) SELECT name FROM person
db.person.find({ name: “Jane” }, { _id: 0, name: 1 }) SELECT name FROM person WHERE name = "Jane"
db.person.find({ name: “Jane” }, { _id: 0, name: 1 }).explain() SELECT name FROM person WHERE name = "Jane" EXPLAIN
db.person.aggregate([{ $count: “personCount” }]) SELECT count() AS person_count FROM person GROUP ALL
db.person.aggregate([{ $group: { _id: “$name” } }]) SELECT array::distinct(name) FROM person GROUP ALL
db.person.find().limit(10) SELECT * FROM person LIMIT 10
db.review.aggregate([{ “$lookup”: { “from”: “person”, “localField”: “person”, “foreignField”: “_id”, “as”: “person_detail” } }]) SELECT *, person.name as reviewer FROM review
### Update For more SurrealQL examples, see the [`UPDATE`](/docs/reference/query-language/statements/update.md) page.
MQL SurrealQL
db.person.updateMany({ name: “Jane” }, { $set: { last_name: “Doe” } }) UPDATE person SET last_name = "Doe" WHERE name = "Jane"
db.person.updateMany({ name: “Jane” }, { $unset: { last_name: 1 } }) UPDATE person UNSET last_name WHERE name = "Jane"
### Delete For more SurrealQL examples, see the [`DELETE`](/docs/reference/query-language/statements/delete.md) and [`REMOVE`](/docs/reference/query-language/statements/remove.md) pages.
MQL SurrealQL
db.person.deleteMany({ name: “Jane” }) DELETE person WHERE name = "Jane"
db.person.deleteMany({}) DELETE person
db.person.drop() REMOVE TABLE person
## Resources - [`CREATE` statement](/docs/reference/query-language/statements/create.md) - [`SELECT` statement](/docs/reference/query-language/statements/select.md) - [`UPDATE` statement](/docs/reference/query-language/statements/update.md) - [`DELETE` statement](/docs/reference/query-language/statements/delete.md) - [`RELATE` statement](/docs/reference/query-language/statements/relate.md) - [SurrealQL documentation](/docs/reference/query-language.md) - [SurrealDB University](/learn) --- Source: https://surrealdb.com/docs/learn/data-models/document/nested-objects-and-arrays # Nested objects and arrays Store nested JSON-like structures in SurrealDB records, use SurrealQL examples with addresses and record links, and relate documents via fields such as article authors. Documents in SurrealDB behave like structured objects you might already use in application code: fields can hold primitives, nested objects, and arrays of values. ## Example: a user with nested addresses The example below creates a `users` record with an `addresses` array of objects. ```surql CREATE users CONTENT { name: 'Alice Smith', email: 'alice@example.com', age: 29, addresses: [ { type: 'home', address_line: '123 Maple St', city: 'Springfield', country: 'USA' }, { type: 'work', address_line: '456 Oak Ave', city: 'Metropolis', country: 'USA' } ] }; ``` By clicking the **Run query** button, you will see a result similar to: ```surql [ { addresses: [ { address_line: '123 Maple St', city: 'Springfield', country: 'USA', type: 'home' }, { address_line: '456 Oak Ave', city: 'Metropolis', country: 'USA', type: 'work' } ], age: 29, email: 'alice@example.com', id: 'users:a2ndbh1hsquvkvthws09', name: 'Alice Smith' } ] ``` You may notice that the `id` field has a `users:` prefix. This is because SurrealDB uses an [id](/docs/reference/query-language/language-primitives/data-types/record-ids.md) to uniquely identify each record, and the combination of the table name and the record id is used as the [record link](/docs/reference/query-language/language-primitives/record-links.md). ## Embedding and linking Document model databases are designed to store data in a flexible, nested structure. **Data organisation** often means self-describing documents in JSON or a similar format. Relationships can be represented **inside** the document (embedding) or via **references** using record links to other documents. For example, if you wanted to associate a `person` with an `article` they wrote, you could assign the person's ID to the `author` field of the article document. This binds the `person` and `article` together, allowing you to query the `article` by the `person`'s ID. ```surql CREATE article SET created_at = time::now(), author = person:john, title = 'Lorem ipsum dolor', text = 'Donec eleifend, nunc vitae commodo accumsan, mauris est fringilla.'; CREATE person:john SET name.first = 'John', name.last = 'Adams', name.full = string::join(' ', name.first, name.last), age = 29, admin = true, signup_at = time::now() ; ``` ## Retrieving documents To read documents back, use a normal `SELECT`. For example, to return every field from the `users` table: ```surql SELECT * FROM users; ``` SurrealDB automatically generates a unique [`id`](/docs/reference/query-language/language-primitives/data-types/record-ids.md) for each document unless you supply your own identifiers. For more on schema options and CRUD patterns, see [Schema modes](/docs/learn/data-models/document/schema-modes.md) and [Common patterns](/docs/learn/data-models/document/common-patterns.md). --- Source: https://surrealdb.com/docs/learn/data-models/document/overview # Document model Learn how to think in a document model, how SurrealDB maps tables and records to documents, and where to find guides on nested data, schema modes, and common patterns including SurrealQL and MongoDB-style mappings. One of the most popular database models is that of a document database. It provides a flexible way to store data, allowing for nested structures and relationships to be stored within a single document. In a document database, data is stored in the form of documents (which usually resemble JSON objects) rather than in rows and columns. This model offers a level of simplicity and flexibility that can be especially appealing when your data does not naturally fit into a strict tabular format or when the structure of your data frequently changes. Over the last decade, we have seen a surge of NoSQL databases such as [MongoDB](https://www.mongodb.com/), [CouchDB](https://couchdb.apache.org/), and [DynamoDB](https://aws.amazon.com/dynamodb/), all of which are in the broad category of document stores (although with varying specific features). But how do you “think” in a document model database? Thinking in a document model means orienting your data design around the entities as you naturally represent them in your applications, rather than forcing your data into normalised or heavily structured relational schemas. ## Core concepts of document-oriented modelling In a document-oriented system you typically work with: - **Self-describing records**: Data is grouped into records that carry their own structure (fields, nested objects, arrays), often similar to JSON in application code. - **Embedding vs linking**: Related data can live inside the same document (embedding) or be referenced with [record links](/docs/reference/query-language/language-primitives/record-links.md) or graph-style relations when you need shared entities or strict edges. - **Flexible schema**: Tables can mix different shapes when you choose a schemaless workflow, or you can tighten definitions with [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) and field types when you need validation and tooling support. ## Where to go next The guides in this section break these ideas down in more detail: - [Nested objects and arrays](/docs/learn/data-models/document/nested-objects-and-arrays.md): worked examples with nested structures, record IDs, and linking documents. - [Schema modes](/docs/learn/data-models/document/schema-modes.md): creating and retrieving documents, and how schemaless usage compares to defining a schema. - [Common patterns](/docs/learn/data-models/document/common-patterns.md): concept mapping to other document stores, benefits, MongoDB-style SurrealQL examples, and further resources. For statement-level reference and CRUD details, see the [`CREATE`](/docs/reference/query-language/statements/create.md), [`SELECT`](/docs/reference/query-language/statements/select.md), [`UPDATE`](/docs/reference/query-language/statements/update.md), and [`DELETE`](/docs/reference/query-language/statements/delete.md) statements, and the [SurrealQL documentation](/docs/reference/query-language.md). --- Source: https://surrealdb.com/docs/learn/data-models/document/schema-modes # Schema modes Create flexible documents without upfront column definitions, or tighten schemas with DEFINE TABLE; add and retrieve JSON-like records in SurrealQL. SurrealDB supports both **schemaless** workflows (insert records of varying shape into the same table) and **schemafull** definitions when you want validation and clearer contracts. This page focuses on the schemaless style that feels closest to typical document databases. ## Schema flexibility Document model databases are often chosen because: 1. **Data organisation**: Data is contained in self-describing documents, typically in JSON or a similar format. Relationships can be done within the document itself (embedding) or via [record links](/docs/reference/query-language/language-primitives/record-links.md) to other documents. 2. **Schema flexibility**: Schemas are often flexible, allowing for documents of varying shapes in the same collection (or table-like structure). When you need stricter rules, you can use [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) and [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md). ## Creating and managing documents In SurrealDB, you can create a database and then store collections of documents (often referred to as “tables”) without strict schema definitions. This is conceptually similar to creating a table in a relational database, but you do not need to define all columns upfront. Instead, you can insert JSON-like objects directly. ### Adding a document ```surql CREATE users CONTENT { name: "Alice Smith", email: "alice@example.com", age: 29, addresses: [ { type: "home", address_line: "123 Maple St", city: "Springfield", country: "USA" }, { type: "work", address_line: "456 Oak Ave", city: "Metropolis", country: "USA" } ] }; ``` Here, a user document includes nested objects (`addresses`) in the same record. You can add nested objects or properties without modifying a central schema first. ### Retrieving documents ```surql SELECT * FROM users; ``` The query returns all documents in the `users` table, similar to a traditional SQL `SELECT`: ```surql [ { addresses: [ { address_line: '123 Maple St', city: 'Springfield', country: 'USA', type: 'home' }, { address_line: '456 Oak Ave', city: 'Metropolis', country: 'USA', type: 'work' } ], age: 29, email: 'alice@example.com', id: 'users:xyz123', name: 'Alice Smith' } ] ``` SurrealDB automatically generates a unique [`id`](/docs/reference/query-language/language-primitives/data-types/record-ids.md) for the document. You can also specify your own custom IDs if you prefer more human-readable or domain-specific identifiers. ## Next steps For concept mapping to other document stores and MongoDB-style SurrealQL examples, see [Common patterns](/docs/learn/data-models/document/common-patterns.md). --- Source: https://surrealdb.com/docs/learn/data-models/full-text-search/analyzers-and-tokenizers # Analyzers and tokenizers Learn how analyzers turn raw text into searchable tokens: tokenizers, filters, and testing with search::analyze before you build an index. Full-text search does not compare your query string to the document byte for byte. Instead, SurrealDB tokenizer text into terms, optionally filters those terms (case folding, stemming, and more), and indexes what comes out. If you are new to FTS in SurrealDB, read the [overview](/docs/learn/data-models/full-text-search/overview.md) first. This guide walks through analyzers from the ground up; exact grammar, every clause, and diagrams live under [`DEFINE ANALYZER`](/docs/reference/query-language/statements/define/analyzer.md). ## What an analyzer does Roughly, processing flows like this: 1. Optional `FUNCTION`: transforms the raw input string once (for example normalising punctuation or stripping markup) via a [user-defined function](/docs/reference/query-language/statements/define/function.md) that accepts and returns a `string`. 2. Tokenizers - split the string into tokens (words, symbols, or other chunks) using one or more built-in tokenizers. 3. Filters - transform each token (lowercase, strip accents, stem, n-grams, and so on). The same analyzer is used when indexing and matching queries, so spending time here pays off for relevance and performance. ## See the tokens before you index Use [`search::analyze()`](/docs/reference/query-language/functions/database-functions/search.md#searchanalyze) to print the token array an analyzer would produce, which is ideal for experimentisation. Start with the simplest split, whitespace-only tokenization: ```surql DEFINE ANALYZER words TOKENIZERS blank; RETURN search::analyze("words", "hello world"); ``` ```surql title="Output" [ 'hello', 'world' ] ``` Once you are happy with the tokens, you attach the analyzer name to a full-text index and query with `@@` (covered on [Search indexes](/docs/learn/data-models/full-text-search/search-indexes.md) and [Scoring and ranking](/docs/learn/data-models/full-text-search/scoring-and-ranking.md)). ## Step 1 - Choose how to split text (tokenizers) Tokenizers answer: *where are the boundaries between tokens?* Some examples of tokenizers are `blank`, `camel`, and `class`. ```surql DEFINE ANALYZER example_blank TOKENIZERS blank; search::analyze("example_blank", "hello world"); DEFINE ANALYZER example_camel TOKENIZERS camel; search::analyze("example_camel", "helloWorld"); DEFINE ANALYZER example_class TOKENIZERS class; search::analyze("example_class", "123abc!XYZ"); ``` ## Step 2 - Normalise and enrich tokens (filters) Filters answer: *what should each token look like before indexing?* Some examples of filters are `ascii`, `snowball`, and `ngram`. ```surql DEFINE ANALYZER example_ascii TOKENIZERS class FILTERS ascii; search::analyze("example_ascii", "résumé café"); DEFINE ANALYZER english_snowball TOKENIZERS class FILTERS snowball(english); DEFINE ANALYZER german_snowball TOKENIZERS class FILTERS snowball(german); RETURN [ search::analyze("english_snowball", "Looking at some running cats"), search::analyze("german_snowball", "Sollen wir was trinken gehen?") ]; DEFINE ANALYZER example_ngram TOKENIZERS class FILTERS ngram(1, 3); search::analyze("example_ngram", "apple banana"); ``` ## Custom dictionaries with `mapper(path)` The `mapper(path)` filter rewrites tokens using a tab-separated file: canonical form first, variant second, one pair per line. That supports lemmatisation beyond what stemming alone catches, or normalising arbitrary phrasing (for example mapping multilingual error strings to a single code). The server reads the dictionary from the **host filesystem** when you define the analyzer. Configure [`SURREAL_FILE_ALLOWLIST`](/docs/reference/cli/surrealdb-cli/environment-variables.md#file-config) so the path lies under an allowed directory, without which no filesystem paths are permitted. See [DEFINE ANALYZER - `mapper(path)`](/docs/reference/query-language/statements/define/analyzer.md#mapperpath) for startup examples. Point `path` at a dictionary file under your allowlist. Here is a very short example dictionary: ```text drive driven drive drives swim swam ``` An analyzer making use of this dictionary can be defined as follows: ```surql DEFINE ANALYZER lemme_english TOKENIZERS blank, class FILTERS lowercase, mapper('/path/to/lemmatization-en.txt'); RETURN [ search::analyze("lemme_english", "He drove and swam"), ]; ``` ## Next steps - [Search indexes](/docs/learn/data-models/full-text-search/search-indexes.md) - attach `FULLTEXT ANALYZER` to a field. - [Scoring and ranking](/docs/learn/data-models/full-text-search/scoring-and-ranking.md) - `@@`, BM25, `search::score`, and highlights. - Reference: [`DEFINE ANALYZER`](/docs/reference/query-language/statements/define/analyzer.md), [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md), [Search functions](/docs/reference/query-language/functions/database-functions/search.md). ### Updating or creating analyzers safely To add an analyzer only if it is missing, or to replace an existing definition, use `IF NOT EXISTS` or `OVERWRITE` on `DEFINE ANALYZER`. Examples and caveats are in the [statement reference](/docs/reference/query-language/statements/define/analyzer.md#using-if-not-exists-clause). --- Source: https://surrealdb.com/docs/learn/data-models/full-text-search/other-ways-to-work-with-text # Other ways to work with text Compare and sort text with COLLATE and NUMERIC, use contains and fuzzy string metrics, regex matching, and how these complement full-text search and hybrid retrieval. Besides [full-text search](/docs/learn/data-models/full-text-search/overview.md), SurrealDB offers many tools for comparing, ordering, and matching strings: sorting with `COLLATE` / `NUMERIC`, substring checks, distance and similarity scores, regex, and ad-hoc string helpers. This page summarises those options so you can pick the right tool next to your FTS indexes and analyzers. ## Comparing and sorting text ### In SELECT queries The following example shows a few records created from an array of strings in an order that is sorted to the human eye: lowest to highest numbers, then A to Z. ```surql FOR $word IN ['1', '2', '11', 'Ábaco', 'kitty', 'Zoo'] { CREATE data SET val = $word; }; ``` Inside a `SELECT` query, an `ORDER BY` clause can be used to order the output by one or more field names. For the above data, an ordered `SELECT` query looks like this. ```surql SELECT VALUE val FROM data ORDER BY val; ``` However, in the case of strings, sorting is done by Unicode ordering which often leads to output that seems out of order to the human eye. The output of the above query shows the following: ```surql title="Output" [ '1', '11', '2', 'Zoo', 'kitty', 'Ábaco' ] ``` This is because: * '11' is ordered before '2', because the first character in the string '2' is greater than the first character in the string '1'. * 'Zoo' is ordered before 'kitty', because the first character in the string 'Zoo' is 'Z', number 0059 in the [list of Unicode characters](https://en.wikipedia.org/wiki/List_of_Unicode_characters#Basic_Latin). A lowercase 'k' is 0076 on the list and thus "greater", while the 'Á', registered as the "Latin Capital letter A with acute", is 0129 on the list. To sort strings in a more natural manner to the human eye, the keywords [`NUMERIC` and `COLLATE` (or both) can be used](/docs/reference/query-language/statements/select.md#sort-records-using-the-order-by-clause). `NUMERIC` will instruct strings that parse into numbers to be treated as such. ```surql SELECT VALUE val FROM data ORDER BY val NUMERIC; -- Output: ['1', '2', '11', 'Zoo', 'kitty', 'Ábaco'] ``` `COLLATE` instructs unicode strings to sort by alphabetic order, rather than Unicode order. ```surql SELECT VALUE val FROM data ORDER BY val COLLATE; -- Output: ['1', '11', '2', 'Ábaco', 'kitty', 'Zoo'] ``` And for the data in this example, `COLLATE NUMERIC` is likely what will be desired. ```surql SELECT VALUE val FROM data ORDER BY val COLLATE NUMERIC; -- Output: ['1', '2', '11', 'Ábaco', 'kitty', 'Zoo'] ``` ### Sorting functions The functions [`array::sort_natural()`, `array::sort_lexical()`, and `array::sort_natural_lexical()`](/docs/reference/query-language/functions/database-functions/array.md) can be used on ad-hoc data to return the same output as the `COLLATE` and `NUMERIC` clauses in a [`SELECT` statement](/docs/reference/query-language/statements/select.md). ## Contains functions and operators The most basic way to see if one string is contained inside another is to use the `IN` operator, or the [`string::contains()` function](/docs/reference/query-language/functions/database-functions/string.md#stringcontains). ```surql "Umple" IN "Rumplestiltskin"; //- false string::contains("Rumplestiltskin", "Umple"); //- false -- Same function using method syntax "Rumplestiltskin".contains("Umple"); //- false "umple" IN "Rumplestiltskin"; //- true string::contains("Rumplestiltskin", "umple"); //- true "Rumplestiltskin".contains("umple"); //- true ``` SurrealDB has a number of [operators](/docs/reference/query-language/language-primitives/operators.md) to determine if all or some of the values of one array are contained in another, such as `CONTAINSALL` and `CONTAINSANY`, or `ALLINSIDE` and `ANYINSIDE`. The queries with `CONTAINS` and `INSIDE` perform the same operation, just in the opposite order. ```surql -- If 1,2,3 contains each item in 1,2... [1,2,3] CONTAINSALL [1,2]; -- then each item in 1,2 is inside 1,2,3 [1,2] ALLINSIDE [1,2,3]; ``` Because strings are essentially arrays of characters, these operators work with strings as well. Both of these queries will return `true`: ```surql "Rumplestiltskin" CONTAINSALL ["umple", "kin"]; "kin" ALLINSIDE "Rumplestiltskin"; ["kin", "someotherstring"] ANYINSIDE "Rumplestiltskin"; ``` ## Equality and fuzzy equality SurrealDB offers quite a few algorithms inside the [string functions module](/docs/reference/query-language/functions/database-functions/string.md) for distance or similarity comparison. They are: * `string::distance::damerau_levenshtein()` * `string::distance::normalized_damerau_levenshtein()` * `string::distance::hamming()` * `string::distance::levenshtein()` * `string::distance::normalized_levenshtein()` * `string::distance::osa()` * `string::similarity::jaro()` * `string::similarity::jaro_winkler()` Which of these functions to choose depends on your personal use case. For example, fuzzy similarity and distance scores are not a measure of absolute equality and ordered similarity scores should only be used in comparisons against the same string. Take the following queries for example which return the score for the string "United" and "Unite": ```surql -- return 131 and 111 string::similarity::fuzzy("United Kingdom", "United"); string::similarity::fuzzy("United Kingdom", "Unite"); -- also return 131 and 111 string::similarity::fuzzy("United", "United"); string::similarity::fuzzy("United", "Unite"); ``` While the word "Unite" is clearly closer to the word "United" than it is to "United Kingdom", the algorithm used for this function only considers how much of the second string is found in the first string. However, the `string::similarity::jaro()` function returns an output that approaches 1 if two strings are equal, making it a more apt solution when the first and second string may be entirely different. Using the same input strings as above shows that "Unite" is clearly the most similar of the strings that are not outright equal to "United". ```surql string::similarity::jaro("United Kingdom", "United"); //- 0.8095238095238096f string::similarity::jaro("United Kingdom", "Unite"); //- 0.7857142857142857f string::similarity::jaro("United", "United"); //- 1 string::similarity::jaro("United", "Unite"); //- 0.9444444444444445f ``` Another example of the large difference between algorithms is the Hamming distance algorithm, which only compares strings of equal length. ```surql string::distance::hamming("United Kingdom", "United"); //- Error: strings have different length string::distance::hamming("United", "United"); //- 0 string::distance::hamming("United", "Unitéd"); //- 1 string::distance::hamming("United", "uNITED"); //- 6 ``` ## Regex matching The `string::matches()` function can be used to perform regex matching on a string. ```surql string::matches("Cat", "[HC]at"); //- true string::matches("Hat", "[HC]at"); //- true ``` ## Other string functions SurrealDB has a large number of [string functions](/docs/reference/query-language/functions/database-functions/string.md) that can be used manually to refine string searching, such as `string::lowercase()`, `string::starts_with()`, and `string::ends_with()`. ```surql SELECT $this AS word, $this.lowercase() = "sleek" AS is_sleek FROM ["sleek", "SLEEK", "Sleek", "sleeek"]; ``` ```surql title="Output" [ { is_sleek: true, word: 'sleek' }, { is_sleek: true, word: 'SLEEK' }, { is_sleek: true, word: 'Sleek' }, { is_sleek: false, word: 'sleeek' } ] ``` ## Combining full-text and vector search SurrealDB has functions that combine full-text and vector search results. For more on that pattern, see [Hybrid search](/docs/learn/data-models/vector-search/hybrid-search.md) in the vector search section. ## Resources - [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md) - [`DEFINE ANALYZER`](/docs/reference/query-language/statements/define/analyzer.md) - [Search functions](/docs/reference/query-language/functions/database-functions/search.md) - [SurrealDB search functions](/docs/reference/query-language/functions/database-functions/search.md) - [SurrealDB operators](/docs/reference/query-language/language-primitives/operators.md) - Blog post: [Create a Search Engine with SurrealDB full-text Search](/blog/create-a-search-engine-with-surrealdb-full-text-search) --- Source: https://surrealdb.com/docs/learn/data-models/full-text-search/overview # Full-text search model Learn how full-text search differs from literal matching, why SurrealDB fits FTS workloads, and where to find guides on analyzers, indexes, scoring, and non-FTS text operations. A full-text search database is designed to index and retrieve text-based data (like articles, messages, or comments) based on tokenized and modified parts of the text itself, rather than exact, literal matches. This allows you to: * Find documents containing certain keywords. * Search for phrases or words with variants (e.g., “run,” “runs,” “running”). * Rank results by relevance, not just by literal string matches. As a multi-model database, SurrealDB has integrated full-text search capabilities so that you can store your data and query it with advanced text search features. Note: SurrealDB has many other built-in ways of working with text besides full-text search. For more details, see [Other ways to work with text](/docs/learn/data-models/full-text-search/other-ways-to-work-with-text.md). ## How full-text search differs from keyword matching In traditional databases, you might do something like: ```surql SELECT * FROM articles WHERE 'fox' IN title; ``` This approach: - Doesn’t rank results by relevance; it just returns every article containing “fox.” - Ignores language variations, e.g., “Foxes,” “FoX,” or synonyms like “vixen.” - May scan an entire table, making it slower for large datasets. Full-text search, by contrast, uses an inverted index or other specialised structures for fast lookups and can handle a variety of linguistic transformations. It can highlight results and rank them by how relevant or frequent the terms are. ## Advantages of SurrealDB for FTS - **Unified model**: You can keep your data, relationships, and search logic in a single engine. - **Flexible schema**: SurrealDB can be schemaless, so adding new fields or text columns doesn’t require schema migrations. - **Powerful query language**: SurrealQL blends SQL-like syntax with searching syntax (the [`@@` matching operator](/docs/reference/query-language/language-primitives/operators.md#matches-a-idmatchesa) for FTS queries, advanced indexing features, and so on). - **Real-time updates**: SurrealDB can handle real-time changes, so newly inserted or updated text becomes searchable quickly. ## Implementing full-text search There are three steps involved in full-text search: - Defining an analyzer - Defining an index that uses the analyzer - Querying using syntax that specifically uses full-text search The guides in this section walk through each part. ## Where to go next - [Analyzers and tokenizers](/docs/learn/data-models/full-text-search/analyzers-and-tokenizers.md): `DEFINE ANALYZER`, tokenizers, filters, stemming, and `search::analyze`. - [Search indexes](/docs/learn/data-models/full-text-search/search-indexes.md): `FULLTEXT ANALYZER` on a single field per index. - [Scoring and ranking](/docs/learn/data-models/full-text-search/scoring-and-ranking.md): the `@@` operator, BM25, highlights, and `search::score` / `search::highlight`. - [Other ways to work with text](/docs/learn/data-models/full-text-search/other-ways-to-work-with-text.md): sorting and collating text, contains, fuzzy matching, regex, and related string tools. For reference, see [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md), [`DEFINE ANALYZER`](/docs/reference/query-language/statements/define/analyzer.md), and [Search functions](/docs/reference/query-language/functions/database-functions/search.md). --- Source: https://surrealdb.com/docs/learn/data-models/full-text-search/scoring-and-ranking # Scoring and ranking Query with the MATCHES operator, combine BM25 and HIGHLIGHTS on indexes, and use search::score and search::highlight with numbered match clauses. After you have [defined analyzers](/docs/learn/data-models/full-text-search/analyzers-and-tokenizers.md) and attached [search indexes](/docs/learn/data-models/full-text-search/search-indexes.md), you can run full-text queries. ## Querying Once an index that uses a full-text analyzer is in place, use the `@@` operator (the `MATCHES` operator) to query it. - **Ranking / Scoring**: Once matches are found, an FTS engine ranks them to show the most relevant results first. Algorithms such as BM25 or TF-IDF look at how often terms appear in a document, or whether those terms appear in the title vs. the body, etc. - **Highlighting** : A good search experience shows where in the text the matches occur, often by wrapping matched terms in HTML tags or otherwise emphasising them. ```surql DEFINE ANALYZER my_analyzer TOKENIZERS class FILTERS lowercase, ascii; -- Two statements as full-text indexes must be defined on only one field DEFINE INDEX body_index ON TABLE article FIELDS body FULLTEXT ANALYZER my_analyzer; DEFINE INDEX title_index ON TABLE article FIELDS title FULLTEXT ANALYZER my_analyzer; CREATE article SET title = "Machine Learning!", body = "Machine learning, or ML, is all the rage these days. Developers are..."; CREATE article SET title = "History of machines", body = "The earliest 'machine' used by our ancestors was a simple sharpened stone tool. It was..."; SELECT body, title FROM article WHERE body @@ "machine" OR title @@ "machine"; ``` ```surql title="Output" [ { body: 'Machine learning, or ML, is all the rage these days. Developers are...', title: 'Machine Learning!' }, { body: "The earliest 'machine' used by our ancestors was a simple sharpened stone tool. It was...", title: 'History of machines' } ] ``` To use highlighting and best match scoring on searches, the `BM25` and `HIGHLIGHTS` clauses can be added to the `DEFINE INDEX` statement. These enable you use the [`search::highlight`](/docs/reference/query-language/functions/database-functions/search.md#searchhighlight) and [`search::score`](/docs/reference/query-language/functions/database-functions/search.md#searchscore) functions. Inside a query, the `@@` operator takes a number that is matched with the same number passed into one of these functions. In the example below, the `WHERE text @0@ "night"` part of the query will match with `search::highlight("->", "<-", 0)` and `search::score(0) AS text_score`, while `title @1@ "hound"` will match with `search::score(1) AS title_score`. ```surql DEFINE ANALYZER my_analyzer TOKENIZERS class, blank FILTERS lowercase, ascii; DEFINE INDEX text_index ON TABLE article FIELDS text FULLTEXT ANALYZER my_analyzer BM25 HIGHLIGHTS; DEFINE INDEX title_index ON TABLE article FIELDS title FULLTEXT ANALYZER my_analyzer BM25 HIGHLIGHTS; INSERT INTO article (title, text) VALUES ("A Study in Scarlet", "IN the year 1878 I took my degree of Doctor of Medicine of the University of London, and proceeded to Netley to go through the course prescribed for surgeons in the army.") ("A Study in Scarlet", "Having completed my studies there, I was duly attached to the Fifth Northumberland Fusiliers as Assistant Surgeon.") ("The Sign of the Four", "SHERLOCK HOLMES took his bottle from the corner of the mantel-piece and his hypodermic syringe from its neat morocco case.") ("The Hound of the Baskervilles", "MR. SHERLOCK HOLMES, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all night, was seated at the breakfast table.") ("The Hound of the Baskervilles", "I stood upon the hearth-rug and picked up the stick which our visitor had left behind him the night before."); SELECT text, title, search::highlight("->", "<-", 0) AS title, search::score(0) AS text_score, search::score(1) AS title_score FROM article WHERE text @0@ "night" OR title @1@ "hound"; ``` ```surql title="Output" [ { text: 'MR. SHERLOCK HOLMES, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all night, was seated at the breakfast table.' text_score: 0.30209195613861084f, title: 'MR. SHERLOCK HOLMES, who was usually very late in the mornings, save upon those not infrequent occasions when he was up all ->night<-, was seated at the breakfast table.' title_score: 0.32491400837898254f }, { text: 'I stood upon the hearth-rug and picked up the stick which our visitor had left behind him the night before.', text_score: 0.35619309544563293f, title: 'I stood upon the hearth-rug and picked up the stick which our visitor had left behind him the ->night<- before.', title_score: 0.32491400837898254f } ] ``` ## Why a score can be 0 BM25 weights every term by its inverse document frequency, a measure of how rare that term is across the indexed documents. SurrealDB computes it as `ln((N - n + 0.5) / (n + 0.5))`, where `N` is the number of indexed documents and `n` is how many of them contain the term, and then clamps the result at zero so that very common terms never carry a negative weight. That clamp is reached as soon as `n` is at least half of `N`. A term held by half or more of the indexed documents weighs exactly `0`, and because that weight multiplies the rest of the formula, `search::score()` returns `0` for it. Small datasets reach the threshold easily. Three documents, two of which contain the term being searched for, are already enough: ```surql DEFINE ANALYZER simple TOKENIZERS class FILTERS lowercase; DEFINE INDEX doc_text ON document FIELDS text FULLTEXT ANALYZER simple BM25; CREATE document:1 SET text = "a database for graphs"; CREATE document:2 SET text = "a database for documents"; CREATE document:3 SET text = "an engine for vectors"; SELECT id, search::score(0) AS score FROM document WHERE text @0@ 'database'; ``` ```surql title="Output" [ { id: document:1, score: 0f }, { id: document:2, score: 0f } ] ``` Four more documents that never mention the term leave `n` at 2 while `N` rises to 7, putting the term under the threshold and giving it a positive weight. The index, the query and the two matching records are all unchanged: ```surql CREATE document:4 SET text = "an engine for time series"; CREATE document:5 SET text = "an engine for documents"; CREATE document:6 SET text = "a store for vectors"; CREATE document:7 SET text = "a store for graphs"; SELECT id, search::score(0) AS score FROM document WHERE text @0@ 'database'; ``` ```surql title="Output" [ { id: document:1, score: 0.7997389435768127f }, { id: document:2, score: 0.7997389435768127f } ] ``` Matching is unaffected either way, so the same rows come back and only their ranking weight is zero. A score of `0` across a handful of records is the clamp behaving as designed, which makes a small dataset a poor place to measure relevance. It is preferable to benchmark on a larger corpus (the size of the real one if possible), and read `ORDER BY score DESC` over a toy dataset as unordered. The `AND` and `OR` clauses can be used inside the `@@` as well. This allows a single string to be compared against instead of needing to specify individual parts of the string. ```surql CREATE document:1 SET text = "It is rare that I find myself penning a personal note in my chronicles."; DEFINE ANALYZER simple TOKENIZERS blank,class FILTERS lowercase; DEFINE INDEX some_index ON document FIELDS text FULLTEXT ANALYZER simple; -- @AND@ and @OR@: can use the entire string SELECT * FROM document WHERE text @AND@ "personal rare"; SELECT * FROM document WHERE text @OR@ "personal nice weather today"; -- Separate AND and OR outside of matches operator: -- Must specify parts of string to check for match SELECT * FROM document WHERE text @@ "personal" AND text @@ "rare"; SELECT * FROM document WHERE text @@ "personal note"; SELECT * FROM document WHERE text @@ "personal" OR text @@ "nice weather today"; ``` In addition to full-text search capabilities, SurrealDB has a number of other ways of working with text such as string similarity, regex matching, and functions that order text in different ways. See [Other ways to work with text](/docs/learn/data-models/full-text-search/other-ways-to-work-with-text.md). --- Source: https://surrealdb.com/docs/learn/data-models/full-text-search/search-indexes # Search indexes Apply FULLTEXT ANALYZER indexes to single fields per index, and understand common parse errors when listing multiple columns. A search index makes a field searchable through an analyzer that has already been defined. This page covers defining one, why each index takes a single field, and the parse error that follows from listing several. ## Defining an index that uses an analyzer Once a search analyzer is defined, it can be applied to the fields of a table to make them searchable by [defining an index](/docs/reference/query-language/statements/define/indexes.md#full-text-search-fulltext-index) that uses the `FULLTEXT ANALYZER` clause. ```surql DEFINE ANALYZER my_analyzer TOKENIZERS class FILTERS lowercase, ascii; DEFINE INDEX body_index ON TABLE article FIELDS body FULLTEXT ANALYZER my_analyzer; DEFINE INDEX title_index ON TABLE article FIELDS title FULLTEXT ANALYZER my_analyzer; ``` An index can only be defined on a single field (column). ```surql DEFINE ANALYZER my_analyzer TOKENIZERS class FILTERS lowercase, ascii; DEFINE INDEX body_index ON TABLE article FIELDS body, title FULLTEXT ANALYZER my_analyzer; ``` ```surql title="Output" 'Parse error: Expected one column, found 2 //- [5:55] | 5 | ...LDS body, title FULLTEXT ANALYZER my_analyzer; | ^^^^^ ' ``` For querying with `@@`, BM25, and highlights, continue to [Scoring and ranking](/docs/learn/data-models/full-text-search/scoring-and-ranking.md). --- Source: https://surrealdb.com/docs/learn/data-models/geospatial/distance-and-proximity # Distance and proximity Measure distance between points, reason about buffers and proximity, and use geohash helpers for hierarchical location keys alongside SurrealDB geo functions. Many applications need more than boolean inside/outside tests: they need how far apart two geometries are, whether entities fall within a buffer around a route, or a compact geohash key for bucketing nearby points. ## Distance The shortest path between two points on the Earth is often approximated in databases using great-circle or similar metrics, depending on your CRS and functions available. SurrealDB’s [`geo::distance()`](/docs/reference/query-language/functions/database-functions/geo.md#geodistance) calculates the distance between two geolocation points, and can be useful when ranking “nearest” results or filtering by a maximum radius. When you only need ordering, you may still use distance in an `ORDER BY` clause. When you need a hard cutoff, combine it with a `WHERE` threshold. ## Geohashes Geohashes approximate location with a short string that prefixes share when points are nearby, and are useful for rough bucketing and cache keys. SurrealDB provides [`geo::hash::encode()` and `geo::hash::decode()`](/docs/reference/query-language/functions/database-functions/geo.md) for moving between points and geohash strings. ## Next steps For end-to-end examples that tie these ideas to product features, see [Location-based patterns](/docs/learn/data-models/geospatial/location-based-patterns.md). --- Source: https://surrealdb.com/docs/learn/data-models/geospatial/geometry-types # Geometry types Use GeoJSON-style geometry values in SurrealDB, including Point, LineString, Polygon, and collections, with longitude-before-latitude ordering. SurrealDB’s `geometry` type follows the [GeoJSON](https://en.wikipedia.org/wiki/GeoJSON) model in which both (and only) a `type` and `coordinates` field must be specified. A tuple shorthand (e.g. `(50.0, 75.7)`) is also accepted when creating a `point`. ## Coordinate order Points are defined with **longitude before latitude**, as required by the GeoJSON spec. Many map services display latitude first, so be sure to confirm the order when ingesting external data. ```surql -- Full GeoJSON object form CREATE city:london SET centre = { type: "Point", coordinates: [-0.118092, 51.509865], }; -- Tuple shorthand for a point (lon, lat) CREATE city:london SET centre = (-0.118092, 51.509865); ``` See the [Geometries](/docs/reference/query-language/language-primitives/data-types/geometries.md) reference for `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`, and `Collection`, including examples for each type. ## Why geometry types matter - **Points** capture a single location (venues, vehicles, users). - **Line strings** represent paths (routes, boundaries along a corridor). - **Polygons** represent regions (service areas, countries, floor plans projected to the globe). Choosing the right type makes spatial predicates (`INTERSECTS`, `CONTAINS`, and so on) accurate and keeps indexes meaningful when you add spatial indexing in your schema. --- Source: https://surrealdb.com/docs/learn/data-models/geospatial/location-based-patterns # Location-based patterns Apply geospatial modelling in SurrealDB to store-of-interest search, delivery zones, asset tracking, and other location-first product features. Once you can store [geometry types](/docs/learn/data-models/geospatial/geometry-types.md) and run [spatial queries](/docs/learn/data-models/geospatial/spatial-queries.md) with [distance](/docs/learn/data-models/geospatial/distance-and-proximity.md) in mind, most product features follow a small set of patterns. ## Store finder (“near me”) Store each site's coordinates as a `Point`, optionally with a category and opening hours. At query time, supply the user's location, compute distance or bounding-box filters, and sort by proximity. For dense datasets, consider geohash prefixes or spatial indexes (where supported) to avoid scanning the whole table. For example, consider records used to store millions of events by location. These can use array-based record IDs that start with a geohash, making it easy to query every record with exactly this hash and no need to use any further filtering. ```surql DEFINE FUNCTION fn::create_geo_record($point: point) -> object { LET $hash = geo::hash::encode($point, 4); CREATE ONLY event:[$hash, $point, rand::ulid()]; }; fn::create_geo_record((50.0, 50.1)); fn::create_geo_record((50.1, 50.1)); fn::create_geo_record((50.5, 50.1)); SELECT * FROM event:['v0gt', NONE, NONE]..['v0gt', .., ..]; ``` ```surql title="Output: returns two of three above events" [ { id: event:[ 'v0gt', (50f, 50.1f), '01KNTYS85HQZ7T4D066MAVA3G9' ] }, { id: event:[ 'v0gt', (50.1f, 50.1f), '01KNTYS85H8JHM3T9YKGMT2BV8' ] } ] ``` ## Service areas and delivery zones Represent coverage as polygons (or multipolygons for islands). New customers or orders are classified with point-in-polygon tests. For a single zone record, store the boundary as a [polygon geometry](/docs/reference/query-language/language-primitives/data-types/geometries.md) and test whether a customer’s location falls inside it using [`CONTAINS`](/docs/reference/query-language/language-primitives/operators.md#contains) (polygon on the left, point on the right): ```surql DEFINE TABLE service_area SCHEMAFULL; DEFINE FIELD name ON service_area TYPE string; DEFINE FIELD boundary ON service_area TYPE geometry; CREATE service_area:greater_london SET name = "Greater London", boundary = { type: "Polygon", coordinates: [[ [-0.38314819, 51.37692386], [0.1785278, 51.37692386], [0.1785278, 51.61460570], [-0.38314819, 51.61460570], [-0.38314819, 51.37692386] ]] }; LET $customer = (-0.118092, 51.509865); SELECT name FROM service_area WHERE boundary CONTAINS $customer; ``` ```surql title="Output" [ { name: 'Greater London' } ] ``` A point outside the ring (for example further north) returns no records. For overlapping territories, you may return several matches and disambiguate with priority, postal rules, or extra fields on `service_area`. ## Tracking and corridors For vehicles or assets reporting on a schedule, store recent positions as points with timestamps and use line strings or buffers when you care about adherence to a route corridor rather than a single snapshot. ## Resources - [Geospatial data types](/docs/reference/query-language/language-primitives/data-types/geometries.md) - [Geo functions](/docs/reference/query-language/functions/database-functions/geo.md) - [SurrealDB University](/learn/book/chapter-10) --- Source: https://surrealdb.com/docs/learn/data-models/geospatial/overview # Geospatial model Learn how geospatial data differs from plain numeric fields, core ideas such as CRS and geometry types, and where to find guides on types, queries, distance, and location-based patterns. A geospatial database is specifically designed (or extended) to store and query data related to the Earth’s surface. Rather than just focusing on numeric or textual data, these systems handle coordinates, polygons, linestrings, and other spatial objects that define locations and shapes. Queries typically revolve around spatial relationships, like finding points within a boundary, measuring distances, or detecting intersections between shapes. Historically, geospatial capabilities were mostly the domain of specialised systems such as PostGIS (an extension of PostgreSQL) or dedicated GIS software. SurrealDB offers geospatial features to unify location-related data with other data models (relational, document, graph, vector, and so on) under one roof with no extensions required. In this section, you’ll learn how to think like a geospatial database user: represent location with geometry or geography types, index and query with spatial predicates, and combine SurrealDB’s [geo functions](/docs/reference/query-language/functions/database-functions/geo.md) with the rest of your data. ## Core concepts of geospatial data - Coordinate reference systems (CRS): Coordinates can be stored in different projections. A common choice is WGS84 for latitude and longitude (EPSG:4326), used by GPS. Other systems exist for local or projected coordinates. - Geometry types: Including points, linestrings, polygons, and multi-variants - see [Geometry types](/docs/learn/data-models/geospatial/geometry-types.md). - Spatial relationships: Operations such as contains, intersects, within, touches, or disjoint - see [Spatial queries](/docs/learn/data-models/geospatial/spatial-queries.md). - Distance and proximity: Measuring distance between geometries, buffers, and geohash-style encodings - see [Distance and proximity](/docs/learn/data-models/geospatial/distance-and-proximity.md). For reference types and function signatures, see [Geometries](/docs/reference/query-language/language-primitives/data-types/geometries.md) and [Geo functions](/docs/reference/query-language/functions/database-functions/geo.md). --- Source: https://surrealdb.com/docs/learn/data-models/geospatial/spatial-queries # Spatial queries Combine geometry fields with SurrealQL SELECTs and geo functions to answer contains, intersects, and distance-style questions over location data. Geospatial workloads usually start with a question in plain language, such as "which stores are inside this polygon?", "which trips cross this corridor?", or "which assets are within 10 km of this point?". They are then translated into **spatial relationships** between stored geometries. ## Relationships you model most often - **Contains / within**: Is a point inside a polygon, or is one region inside another? - **Intersects**: Do two geometries share any area or touch along an edge? - **Touches / disjoint**: Boundary contact only, or no overlap at all? SurrealDB exposes these ideas through geometry values and the [`geo::`](/docs/reference/query-language/functions/database-functions/geo.md) function family. ```surql title="Example of geo:: function" let $london = (-0.04592553673505285, 51.555282574465764); let $harare = (30.463880214538577, -17.865161568822085); RETURN geo::distance($london, $harare); ``` Exact operators available to your version are documented alongside geometry types and index definitions. You can also use [`EXPLAIN`](/docs/reference/query-language/clauses/explain.md) to confirm your query plan when tuning performance. ## Tip: combine with non-spatial filters As location queries are rarely done in isolation, you can combine spatial predicates with ordinary fields to form queries for tasks like "coffee shops in this polygon that are open now", not every record in the table. --- Source: https://surrealdb.com/docs/learn/data-models/graph/creating-relations # Creating relations Create graph edges with RELATE, store data on edge tables, define TYPE RELATION for safety and tooling, and handle symmetric relations with unique indexes. In SurrealDB, nodes are typically records in ordinary tables (users, posts, companies, and so on). Graph edges are also **real tables**, created and queried with [`RELATE`](/docs/reference/query-language/statements/relate.md). ## Creating nodes and edge records SurrealDB uses [`RELATE`](/docs/reference/query-language/statements/relate.md) to form the usual graph triple: subject → predicate → object. Using `RELATE`, you can model primary relationships from something like an e-commerce flow: wish list, cart, order, and review. These become your edge tables. ```surql RELATE person:billy->wishlist->product:01HGAR7A0R9BETTCMATM6SSXPT; RELATE person:vanessa->cart->product:01GXRS3FZG8Y8SDBNHMC14N25X; RELATE person:loki->order->product:tesseract; RELATE person:u1cczojntb5kvos2ugue->review->product:8g8ftj1mblza2vikm680; ``` Here, existing record IDs from the `person` and `product` tables are used. The edge tables sit between them and are created by these `RELATE` statements. After you run `RELATE`, edge records gain two standard fields: `in` and `out`. The `in` field holds the ID of the record on the left side of the statement, and `out` the ID on the right. The triple can also be read as: - `in -> id -> out` where the first node is `in`, the edge is the record’s `id`, and the second node is `out`. ## Adding data to edge tables with `SET` and `CONTENT` What sets SurrealDB apart from graph-only databases is that **edges are tables**, so you can store fields on them like any other row. ```surql RELATE person:j151lkm3k1dytd53y0i5->wishlist->product:r5pgmgjmr57kjvm2j1g1 SET time.created_at = time::now(); CREATE product:crystal_cave SET name = "The Crystal Cave", price = 5; CREATE person:brian SET address = "555 Brian Street"; RELATE person:brian->order-> product:crystal_cave SET quantity = 2; ``` You can both create an `order` relationship and query through it to pull data from `product` and `person`: ```surql SELECT quantity, out.name AS product_name, out.price * quantity AS price, in.address AS shipping_address FROM order; ``` ```surql title="Output" [ { price: 10, product_name: 'The Crystal Cave', quantity: 2, shipping_address: '555 Brian Street' } ] ``` That shows how to read fields on the edge table directly. For traversing with arrow syntax from nodes, see [Graph traversal](/docs/learn/data-models/graph/graph-traversal.md). ## Creating a graph relation with metadata on the edge This pattern is like the examples above, except the `user` row does not keep a `comments` field. A `wrote` edge connects user and comment, and **the edge row** stores context (location, device, mood): ```surql LET $new_user = CREATE ONLY user SET name = "User McUserson"; LET $new_comment = CREATE ONLY comment SET text = "I learned something new!", created_at = time::now(); RELATE $new_user->wrote->$new_comment SET location = "Arizona", os = "Windows 11", mood = "happy"; ``` Once the edge exists, you can traverse it with the arrow operator (forward, backward, recursively, and more. See [Graph traversal](/docs/learn/data-models/graph/graph-traversal.md) for patterns such as: ```surql SELECT ->wrote->comment FROM user; SELECT <-wrote<-user FROM comment; SELECT <-wrote<-user->wrote->comment FROM comment; ``` ## Define a table as a relation for type safety and SurrealDB Studio Defining a table as `TYPE RELATION` restricts it to valid graph edges between records. Adding `TYPE RELATION` to [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) is enough to enforce that behaviour. ```surql DEFINE TABLE likes TYPE RELATION; ``` Constraining `in` and `out` record types ensures only intended record types can be linked: ```surql DEFINE TABLE likes TYPE RELATION IN person OUT blog_post | book; ``` Strict relation definitions also feed [SurrealDB Studio](/docs/explore/studio.md)’s Designer view. Create some data and relate it: ```surql CREATE person:one, book:one, blog_post:one; RELATE person:one->likes->book:one; RELATE person:one->likes->blog_post:one; ``` Before the table is defined as a relation, SurrealDB Studio only sees schemaless tables: ![SurrealDB Studio designer view showing four schemaless tables without a specified connection between them.](~/assets/img/surrealdb/models/schema1.png) Defining the table as a `TYPE RELATION` clarifies that `likes` is a graph table: ```surql DEFINE TABLE likes TYPE RELATION; CREATE person:one, book:one, blog_post:one; RELATE person:one->likes->book:one; RELATE person:one->likes->blog_post:one; ``` ![SurrealDB Studio designer view showing three schemaless tables together with a likes table that has been defined as a relation.](~/assets/img/surrealdb/models/schema2.png) With `IN` and `OUT` specified, Designer can show the full relation: ```surql DEFINE TABLE likes TYPE RELATION IN person OUT blog_post | book; CREATE person:one, book:one, blog_post:one; RELATE person:one->likes->book:one; RELATE person:one->likes->blog_post:one; ``` ![SurrealDB Studio designer view showing four schemafull tables: three regular tables and one relation table linking them.](~/assets/img/surrealdb/models/schema3.png) ## Unique index for relations “between equals” When an edge represents friendship, partnership, sister cities, and similar **symmetric** roles, it may not matter which side is `in` or `out`), but you still want at most one edge for the pair. ```surql CREATE person:one, person:two; -- Relate them like this? RELATE person:one->friends_with->person:two; -- Or like this? RELATE person:two->friends_with->person:one; ``` Define a field from the sorted `in` and `out` values and put a **unique** index on it: ```surql DEFINE FIELD key ON TABLE friends_with VALUE array::sort([in, out]); DEFINE INDEX only_one_friendship ON TABLE friends_with FIELDS key UNIQUE; ``` Then a second `RELATE` from the other direction hits the index: ```surql CREATE person:one, person:two; RELATE person:one->friends_with->person:two; RELATE person:two->friends_with->person:one; ``` ```surql title="Output of RELATE statements" -------- Query -------- [ { id: friends_with:dblidwpc44qqz5bvioiu, in: person:one, key: '[person:one, person:two]', out: person:two } ] -------- Query -------- "Database index `only_one_friendship` already contains '[person:one, person:two]', with record `friends_with:dblidwpc44qqz5bvioiu`" ``` ## Duplicate record IDs Enforced in `INSERT RELATION` statements since SurrealDB 3.1.5, another way to ensure at most one edge between two records is to choose a specific ID. Because an `INSERT RELATION` statement only requires the `in` and `out` paths to be present, a statement like the one below will create a random ID for the edge such as `likes:3lmr630sb72awbzhh4cl`. ```surql -- INSERT RELATION INTO likes { in: person:one, out:person:two }; ``` If a specific ID is declared, then an attempt to insert a second edge will return an error. ```surql INSERT RELATION INTO likes { in: person:one, out:person:two, id: 1 }; INSERT RELATION INTO likes { in: person:one, out:person:two, id: 1 }; ``` This can be used as an ad-hoc unique index, especially if the ID itself contains the IDs of the records being linked. The [`array::sort()`](/docs/reference/query-language/functions/database-functions/array.md#arraysort) function can also be used here for symmetric relations. ```surql LET $in = person:three; LET $out = person:four; INSERT RELATION INTO married_to { in: $in, out: $out, id: [$in, $out].sort() }; INSERT RELATION INTO married_to { in: $out, out: $in, id: [$in, $out].sort() }; ``` The final query returns the following error, thanks to the duplicate ID, despite the attempt to double the relation by putting `in` at the `out` field and vice versa. ```surql title="Output" 'Database record `likes:[person:four, person:three]` already exists' ``` Querying symmetric edges with `<->` is covered in [Graph traversal](/docs/learn/data-models/graph/graph-traversal.md). For friendship-oriented examples, see [Social network patterns](/docs/learn/data-models/graph/social-network-patterns.md). ## `RELATE` before both endpoints exist Graph edge rows can exist **before** the two endpoint records are created: ```surql -- Works fine RELATE person:one->likes->person:two; person:one->likes->person; //- [] -- Finally create the 'person' records CREATE person:one, person:two; person:one->likes->person; //- [person:two] ``` To forbid that, add [`ENFORCED`](/docs/reference/query-language/statements/define/table.md) on the relation table: ```surql DEFINE TABLE likes TYPE RELATION IN person OUT person ENFORCED; ``` ```surql title="Output" "The record 'person:one' does not exist" ``` Some workflows rely on creating the edge first (for example, a street with predictable house IDs before houses exist. A [`DEFINE FIELD ... VALUE`](/docs/reference/query-language/statements/define/field.md) can describe the path from `house` to `street` and resolve once the house exists: ```surql DEFINE FIELD street ON house VALUE $this<-contains<-street; CREATE street:frankfurt_road; RELATE street:frankfurt_road->contains->[ house:["Frankfurt Road", 200], house:["Frankfurt Road", 205], house:["Frankfurt Road", 210], ]; -- Twelve months later once the house is built and size is known... CREATE house:["Frankfurt Road", 200] SET sq_m = 110.5; ``` ```surql title="Output" [ { id: house:[ 'Frankfurt Road', 200 ], sq_m: 110.5f, street: [ street:frankfurt_road ] } ] ``` --- Source: https://surrealdb.com/docs/learn/data-models/graph/graph-traversal # Graph traversal Use SurrealQL arrow syntax, bidirectional edges, traversals from record IDs, automatic flattening, graph paths in schema fields, and SurrealDB Studio’s Explorer to debug paths step by step. Graph queries in SurrealDB use SurrealQL’s `->` arrow syntax to walk relationships between records. ## Basic forward and reverse traversals ```surql SELECT ->wrote->post.* AS userPosts FROM users:alice; ``` - `FROM users:alice` starts at that node. - `->wrote->posts.*` follows the `wrote` edge to posts and returns full post records as `userPosts`. Traverse in the reverse direction (for example, from a post to its authors): ```surql SELECT <-wrote<-author AS authors FROM post:helloworld; ``` `<-wrote<-author` means “follow any `wrote` edge **into** this post from an `author`”. ## Chaining paths From comments, find authors and then all of their comments: ```surql SELECT <-wrote<-user->wrote->comment FROM comment; ``` ## Querying symmetric (“between equals”) relations For edges like `friends_with`, it may be unclear whether a `person` is on the `in` or `out` side. The `<->` operator traverses **both** directions on that edge table: ```surql SELECT *, <->friends_with<->person AS friends FROM person; ``` Each row lists the other people in the relation, regardless of which side they were on: ```surql [ { friends: [ person:one, person:two ], id: person:one }, { friends: [ person:one, person:two ], id: person:two } ] ``` To drop the current record’s own id from the list, use [`array::complement()`](/docs/reference/query-language/functions/database-functions/array.md#arraycomplement): ```surql SELECT *, array::complement(<->friends_with<->person, [id]) AS friends FROM person; ``` ```surql title="Output" [ { friends: [ person:two ], id: person:one }, { friends: [ person:one ], id: person:two } ] ``` For more on this pattern, see the [`RELATE` statement](/docs/reference/query-language/statements/relate.md#bidirectional-relation-querying) and [Chapter 7 of Aeon's Surreal Renaissance](/learn/book/chapter-07#bidirectional-querying-when-a-relationship-is-equal). ## Traverse directly from a record id Traversal can start at one or more record IDs **without** wrapping everything in `SELECT`: ```surql CREATE ONLY user:mcuserson SET name = "User McUserson"; CREATE ONLY comment:one SET text = "I learned something new!", created_at = time::now(); CREATE ONLY cat:pumpkin SET name = "Pumpkin"; RELATE user:mcuserson->wrote->comment:one SET location = "Arizona", os = "Windows 11", mood = "happy"; RELATE user:mcuserson->likes->cat:pumpkin; ``` ```surql -- Equivalent to: -- SELECT VALUE <-wrote<-user FROM ONLY comment:one; comment:one<-wrote<-user; -- Equivalent to: -- SELECT VALUE ->likes->cat FROM ONLY user:mcuserson; user:mcuserson->likes->cat; ``` ```surql title="Output" -------- Query -------- [user:mcuserson] -------- Query -------- [cat:pumpkin] ``` To include fields when starting from an id, use destructuring: ```surql -- Equivalent to: -- SELECT name, ->likes->cat AS cats FROM ONLY user:mcuserson; user:mcuserson.{ name, cats: ->likes->cat }; ``` ## Automatic flattening in graph queries When two lookups follow each other, or a filter follows a lookup, results can **flatten** in a way that mirrors the graph shape. This example builds a small org chart and walks up and down `works_for`: ```surql CREATE -- One president person:president, -- Two managers person:manager1, person:manager2, -- Four employees person:employee1, person:employee2, person:employee3, person:employee4; -- Employees work two to a manager, managers work two to a president RELATE [person:manager1, person:manager2]->works_for->person:president; RELATE [person:employee1, person:employee2]->works_for->person:manager1; RELATE [person:employee3, person:employee4]->works_for->person:manager2; [person:employee1, person:employee2, person:employee3, person:employee4] ->works_for->person ->works_for->person <-works_for<-person <-works_for<-person; ``` The result is four arrays, each reflecting the up-and-down path through the president. You can approximate the same with nested `SELECT` / `map`, but nesting grows quickly: ```surql [person:employee1, person:employee2, person:employee3, person:employee4] -- Each $p is a single record: an array .map(|$p| SELECT VALUE out FROM works_for WHERE in = $p.id) -- Each $p is an array, so now you have to map each item inside that .map(|$p| $p.map(|$p| SELECT VALUE out FROM works_for WHERE in = $p.id)) -- Now an array>> .map(|$p| $p.map(|$p| $p.map(|$p| SELECT VALUE in FROM works_for WHERE out = $p.id))) -- Now an array>>> .map(|$p| $p.map(|$p| $p.map(|$p| $p.map(|$p| SELECT VALUE in FROM works_for WHERE out = $p.id)))); ``` Graph syntax keeps structure aligned with the traversal. Calling `flatten()` at each step produces a **flat** list of every record seen along the way instead: ```surql [person:employee1, person:employee2, person:employee3, person:employee4] .map(|$p| SELECT VALUE out FROM works_for WHERE in = $p.id).flatten() .map(|$p| SELECT VALUE out FROM works_for WHERE in = $p.id).flatten() .map(|$p| SELECT VALUE in FROM works_for WHERE out = $p.id).flatten() .map(|$p| SELECT VALUE in FROM works_for WHERE out = $p.id).flatten(); ``` Treat graph flattening as preserving the **shape** of the walk, not necessarily a single post-hoc array. ## Graph paths in schema fields Paths are not limited to `SELECT`; they can appear in [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md): ```surql DEFINE FIELD employers ON TABLE person VALUE SELECT VALUE <-employs<-company FROM ONLY $this; CREATE person:1, person:2, company:1; RELATE company:1->employs->person:1; person:1.*; ``` A plain `VALUE` field is computed when the record is created or updated; if the `RELATE` runs later, the field can be stale until an `UPDATE`: ```surql UPDATE person:1; ``` ```surql title="Output" [ { employers: [ company:1 ], id: person:1 } ] ``` A [computed field](/docs/reference/query-language/statements/define/field.md#restrictions-on-computed-fields) is recomputed on read: ```surql DEFINE FIELD employers ON TABLE person COMPUTED <-employs<-company; CREATE person:1, person:2, company:1; RELATE company:1->employs->person:1; person:1.*; ``` ```surql title="Output" { employers: [ company:1 ], id: person:1 } ``` ## Using SurrealDB Studio Explorer to debug paths SurrealDB Studio’s [Explorer view](/docs/explore/studio.md) lets you step through records and relations one hop at a time, useful for building intuition for queries like `SELECT ->wrote->comment FROM user`. Example setup with two outgoing edges: ```surql CREATE user:mcuserson SET name = "User McUserson"; CREATE comment:one SET text = "I learned something new!", created_at = time::now(); CREATE cat:pumpkin SET name = "Pumpkin"; RELATE user:mcuserson->wrote->comment:one SET location = "Arizona", os = "Windows 11", mood = "happy"; RELATE user:mcuserson->likes->cat:pumpkin; ``` Walkthrough: * Open `user`, then `user:mcuserson`. * Open the **Relations** tab (outgoing `->`). * Follow `wrote` into its edge row, then `comment` to the comment row, matching `->wrote->comment`. Working backward in the Explorer is a good way to assemble a path while learning the syntax. ## Further reading * [Recursive traversals](/docs/learn/data-models/graph/recursive-traversals.md) for `@.{n}` and nested recursive shapes. * [Idioms: recursive paths](/docs/reference/query-language/language-primitives/idioms.md#recursive-paths) --- Source: https://surrealdb.com/docs/learn/data-models/graph/knowledge-graph-patterns # Knowledge graph patterns Model entities and relationships as first-class data for traversal, search, and AI workflows using SurrealDB's graph features. A knowledge graph models your domain as entities (people, products, documents, concepts) connected by typed relationships: - `document -> cites -> regulation` - `person -> works_at -> organisation` - `symptom -> associated_with -> condition` This structure gives you a map of meaning that you can do the following with: * traverse via single- or multi-hop queries * enrich with metadata that applies to the relation between one record and another * combine with search and AI workflows In SurrealDB, knowledge graphs are built using: - records as nodes (`person`, `document`, `concept`) - graph relations as edges, created with [`RELATE`](/docs/reference/query-language/statements/relate.md) Unlike simple links, edges are first-class records, so you can attach rich metadata to relationships. ## Core pattern: entities + typed relationships A typical pattern looks like this: ```surql DEFINE TABLE person SCHEMAFULL; DEFINE TABLE organisation SCHEMAFULL; DEFINE TABLE works_at SCHEMAFULL TYPE RELATION FROM person TO organisation; -- Require relationship-specific data on each edge DEFINE FIELD role ON works_at TYPE string; DEFINE FIELD started_at ON works_at TYPE datetime; ``` Relationships can then be created using a `RELATE` statement: ```surql RELATE person:alice->works_at->organisation:acme SET role = "Engineer", started_at = time::now(); ``` This lets you query both the connection and the context of the connection. ## Design tips ### Use stable, explicit identities Use clear table types (`person`, `document`, `clause`) and stable record IDs so relationships remain meaningful over time. ### Map out queries ahead of time and give thought to edge names Mapping out queries ahead of time can help make a decision between how coarse- or fine-grained records and edges should be. For example, you might want to use a single `wrote` edge to connect a `person` or similar record to their writing: ```surql SELECT *, ->wrote->blog AS blogs, ->wrote->comment AS comments FROM person; SELECT *, ->wrote->(blog, comment) AS all_written FROM person; ``` The more generic the type, the greater the chance that you may need to use a filter. ```surql SELECT *, ->(wrote WHERE type = "blog")->item AS blogs, ->(wrote WHERE type = "comment")->item AS comments FROM person; ``` A more fine-grained approach with separate blogged and commented edges can be more effective for knowledge-graph and AI use cases, because the relationship semantics are explicit in the graph itself rather than inferred from filters or record types. ```surql SELECT *, ->blogged->blog AS blogs, ->commented->comment AS comments FROM person; ``` As multiple edges can always be combined into a single query, you can make your logic as fine-grained as you can conceptually manage without needing to worry about this having an effect on the queries you put together. The only extra addition to such queries is that of the names of the edge tables involved. ```surql SELECT *, ->(blogged, commented)->(blog, comment) AS all_written FROM person; ``` ## Industry-oriented examples You can see these patterns applied in real-world schemas in our [sample industry schemas](/docs/learn/schema-management/schema-design/sample-industry-schemas.md) page. --- Source: https://surrealdb.com/docs/learn/data-models/graph/overview # Graph model Learn how to think in a graph model and how SurrealDB represents nodes, edges, and properties, with links to guides on creating relations, traversal, record links, and more. A graph database is specifically designed to store data as nodes and edges (relationships between nodes). With this model, connections are front and center, making it easier (and often faster) to query highly connected datasets, like [social networks](/blog/using-surrealdb-to-expose-organized-influence-campaigns), [planning and supply chain relationships](/docs/learn/schema-management/schema-design/sample-industry-schemas.md#project-planning), recommendation engines, or [fraud detection graphs](/blog/fraud-detection-with-surrealdb). When using SurrealDB as a graph database, you typically care about both the entities in a system and how they relate to each other. It might be a user that “follows” another user, a product that “belongs” to a category, or a web page that “links to” another page. These relationships are first-class citizens, rather than just foreign keys or nested objects. That enables powerful, intuitive traversal-based queries that reflect real-world systems. But how do you “think” in a graph database? Instead of focusing on how to break data into tables (relational) or embed data in documents (document model), you concentrate on expressing data as nodes and defining the edges that describe relationships. This mindset puts the connections at the core of your design: each data point is a node with properties, and edges hold properties too, representing the context or metadata about those relationships. ## Core concepts of graph-oriented modelling In any graph database, you deal with three fundamental elements: - **Nodes (Vertices)**: Represent main entities or “things”. In a social network for example, nodes might be people. In a knowledge graph, nodes might be concepts. In a product recommendation engine, nodes might be items or customers. - **Edges**: Represent relationships between nodes. These could be `follows`, `buys`, `likes`, `friend_of`, `in_category`, etc. They often include properties like timestamps or weights. - **Properties**: Both nodes and edges can contain key-value pairs (properties). For a `user` node, properties might be `name` or `age`. For a `likes` edge, a property might be `strength` to indicate how strong the affinity between users is. When you think in a graph, the modelling process shifts toward identifying the main entities in your application (the nodes) and how they relate to one another (the edges). Rather than flattening these relationships into foreign keys or embedding them in nested structures, you give them explicit representation and, often, explicit properties. ## Modelling data as a graph How this works in practice in most graph databases is through "semantic triples", which describe a graph in a three-part structure: - `subject -> predicate -> object` Or: - `node -> edge -> node` Another way to think about this is in terms of nouns connected by verbs, such that it forms a sentence. - `noun -> verb -> noun` Or: - `person -> order -> product` ## Where to go next The guides in this section break down how SurrealDB implements these ideas in practice: - [Creating relations](/docs/learn/data-models/graph/creating-relations.md): [`RELATE`](/docs/reference/query-language/statements/relate.md), edge tables, `in` / `out`, schema with `TYPE RELATION`, and edge cases such as relations before records exist. - [Graph traversal](/docs/learn/data-models/graph/graph-traversal.md): arrow syntax, traversing from record IDs, flattening behaviour, graph paths in schema, and using SurrealDB Studio’s Explorer. - [Recursive traversals](/docs/learn/data-models/graph/recursive-traversals.md): `@.{n}` paths and nested shapes along a graph. - [Record links vs graph relations](/docs/learn/data-models/graph/record-links-vs-graph-relations.md): when to use each, metadata on edges, weighting, and delete behaviour. - [Social network patterns](/docs/learn/data-models/graph/social-network-patterns.md): friendship-style edges, bidirectional queries, and interaction weighting examples. - [Knowledge graph patterns](/docs/learn/data-models/graph/knowledge-graph-patterns.md): orienting nodes and edges for knowledge-style and industry schemas. For statement-level reference and deeper query details, see: * The [`RELATE` statement](/docs/reference/query-language/statements/relate.md#querying-graphs) * [Idioms](/docs/reference/query-language/language-primitives/idioms.md) (including recursive paths) * The [SurrealDB Fundamentals course](/learn/fundamentals) * [Aeon's Surreal Renaissance](/learn/book), chapters 5 to 8 in particular --- Source: https://surrealdb.com/docs/learn/data-models/graph/record-links-vs-graph-relations # Record links vs graph relations Choose between record links and graph edges for performance, schema, bidirectional references, metadata on relationships, weighting, and delete behaviour. The first question when modelling connections is whether **graph edges** are the right tool: SurrealDB can also connect records with **record links**. ## Record links A record link is a pointer from one record to another: any field that holds a record id. Record ids are efficient direct pointers and avoid table scans. Example: one `user` with two `comment` rows linked from the user: ```surql LET $new_user = CREATE ONLY user SET name = "User McUserson"; -- Create a new comment, use the output to update the user UPDATE $new_user SET comments += (CREATE ONLY comment SET text = "I learned something new!", created_at = time::now()) .id; UPDATE $new_user SET comments += (CREATE ONLY comment SET text = "I don't get it, can you explain?", created_at = time::now()) .id; ``` Querying is like reading any other field: ```surql SELECT name, comments.{ created_at, text } FROM user; ``` ```surql title="Output" [ { comments: [ { created_at: d'2024-12-12T02:39:07.644Z', text: 'I learned something new!' }, { created_at: d'2024-12-12T02:39:07.645Z', text: "I don't get it, can you explain?" } ], name: 'User McUserson' } ] ``` ### Reverse direction without edges Historically, record links were unidirectional; the other direction often needed a subquery; while graph edges made reverse walks easy: ```surql SELECT *, -- Check the `user` table's `comments` field -- for the id of the current comment (SELECT id, name FROM user WHERE $parent.id IN comments) AS author FROM comment; -- Equivalent graph query is much easier -- to read and write SELECT *, <-wrote<-author FROM comment; ``` ```surql [ { author: [ { id: user:f3t90z8uvns76sr3nxrd, name: 'User McUserson' } ], created_at: d'2024-12-12T02:39:07.645Z', id: comment:gj1vtsd9d19z9afrc14j, text: "I don't get it, can you explain?" }, { author: [ { id: user:f3t90z8uvns76sr3nxrd, name: 'User McUserson' } ], created_at: d'2024-12-12T02:39:07.644Z', id: comment:zhnbfopxspekknsi6vx6, text: 'I learned something new!' } ] ``` ### Bidirectional record links (3.0+) Since version `3.0.0`, a record link can be bidirectional by defining a field with the [`REFERENCE`](/docs/reference/query-language/language-primitives/record-references.md) clause and using a computed back-reference: ```surql DEFINE FIELD comments ON user TYPE option>> REFERENCE; DEFINE FIELD author ON comment COMPUTED <~user; LET $new_user = CREATE ONLY user SET name = "User McUserson"; -- Create a new comment, use the output to update the user UPDATE $new_user SET comments += (CREATE ONLY comment:one SET text = "I learned something new!", created_at = time::now()) .id; UPDATE $new_user SET comments += (CREATE ONLY comment:two SET text = "I don't get it, can you explain?", created_at = time::now()) .id; -- 'author' field is populated with the 'user' who wrote the comment SELECT * FROM ONLY comment:one; -- Regular queries on incoming references work too SELECT text, <~user.id[0] AS commenter FROM comment; ``` ```surql title="Output" -------- Query -------- { author: [ user:igi77zrlewsqemgjz4zi ], created_at: d'2025-09-02T02:56:03.408Z', id: comment:one, text: 'I learned something new!' } -------- Query -------- [ { commenter: user:igi77zrlewsqemgjz4zi, text: 'I learned something new!' }, { commenter: user:igi77zrlewsqemgjz4zi, text: "I don't get it, can you explain?" } ] ``` ## When to prefer record links **Record links are preferred if:** * Performance is the top priority. * You do not need very complex multi-hop graph queries. * You want the schema to declare what happens when a linked row is deleted (cascade, refuse, ignore, etc.) via [`ON DELETE`](/docs/reference/query-language/language-primitives/record-references.md) and related options. ## When to prefer graph relations **Graph relations are preferred if:** * You want to create links quickly **without** pre-defining every field, or you link across many record types in one go (for example `RELATE person:one->wrote->[blog:one, book:one, comment:one]` versus several `DEFINE FIELD` steps. * You need expressive arrow syntax for multi-hop patterns such as `->wrote->comment<-wrote<-person->wrote->comment` starting from `person`. * You want SurrealDB Studio’s Designer to visualise edges clearly. Graph edges are **almost required** when the relationship itself carries metadata that belongs neither to the source nor target record alone (for example the moment a user posted a comment: ```surql { ip_addr_location: "Arizona", os: "Windows 11", current_mood: "Happy" } ``` That data is about the **link event**, not the whole user or the comment body, so it belongs on an edge table. The same applies to durable facts about the relationship: ```surql { friends_since: d'2024-12-31T06:43:21.981Z', friendship_strength: 0.4 } ``` ## Weighting and counting on graph tables Graph tables are a natural place to store **weights** on the edge row, or to **derive** interaction strength by **counting** how many edges exist between the same endpoints. If you need that kind of scoring or event metadata on the relationship itself, graph relations are usually the right abstraction. Worked examples (random NPC interactions, `knows` vs `greeted`, aggregations) live in [Social network patterns](/docs/learn/data-models/graph/social-network-patterns.md). ## When links are deleted Record links can encode rich behaviour on delete (for example, removing an id from an array while appending to a history field), via [`REFERENCE` `ON DELETE`](/docs/reference/query-language/language-primitives/record-references.md). Graph edges are simpler: if **either** endpoint record is deleted, the edge row is removed. ```surql -- likes record created without problems RELATE person:one->likes->person:two; CREATE person:one, person:two; DELETE person:one; -- 'likes' record is now gone SELECT * FROM likes; ``` Example of record-link cleanup: ```surql DEFINE FIELD comments ON person TYPE option>> REFERENCE ON DELETE THEN { UPDATE $this SET deleted_comments += $reference, comments -= $reference; }; ``` For more on `ON DELETE` and references, see [Record references](/docs/reference/query-language/language-primitives/record-references.md). --- Source: https://surrealdb.com/docs/learn/data-models/graph/recursive-traversals # Recursive traversals Use SurrealQL recursive path syntax and idioms to walk a graph to a chosen depth and return nested structures along a relationship. [Recursive queries](/docs/reference/query-language/language-primitives/idioms.md#recursive-paths) let you follow a path to a specific depth without spelling out every hop. ## Family tree example `person` records linked by `child_of`: ```surql CREATE |person:1..16|; -- parents of person:1 RELATE person:1->child_of->[person:2, person:3]; -- grandparents of person:1 RELATE person:2->child_of->[person:4, person:5]; RELATE person:3->child_of->[person:6, person:7]; -- great-grandparents of person:1 RELATE person:4->child_of->[person:8, person:9]; RELATE person:5->child_of->[person:10, person:11]; RELATE person:6->child_of->[person:12, person:13]; RELATE person:7->child_of->[person:14, person:15]; ``` You can repeat the path manually: ```surql SELECT ->child_of->person AS parents, ->child_of->person->child_of->person AS grandparents, ->child_of->person->child_of->person->child_of->person AS great_grandparents FROM ONLY person:1; ``` Or use recursive path repetition: ```surql SELECT @.{1}->child_of->person AS parents, @.{2}->child_of->person AS grandparents, @.{3}->child_of->person AS great_grandparents FROM ONLY person:1; ``` Recursive syntax is not only shorthand: it can return a **single nested object** that repeats down a path: ```surql -- Range to start at a depth of one, try to go down to depth of three SELECT @.{3}.{ id, -- At each depth, use this path to reach the next one parents: ->child_of->person.@ } FROM person:1; ``` Starting from a record id: ```surql person:1.{3}.{ id, parents: ->child_of->person.@ }; ``` Recursive paths are defined in idioms and are not limited to graph traversal. A record link recurses through the same syntax, with no edges involved: ```surql CREATE cat:one SET likes = cat:two; CREATE cat:two SET likes = cat:three; CREATE cat:three SET likes = cat:one; cat:one.{1..3}.likes; ``` ```surql title="Output" cat:one ``` What a recursive path needs at each step is a record, or an array of records. That is what a link and an edge have in common. A step landing on any other value ends the query rather than continuing. See [Behaviour of recursive queries](/docs/reference/query-language/language-primitives/idioms.md#behaviour-of-recursive-queries). ## Further reading * [Idioms: recursive paths](/docs/reference/query-language/language-primitives/idioms.md#recursive-paths) * [Chapter 8 of Aeon's Surreal Renaissance](/learn/book/chapter-08#longer-relational-queries) --- Source: https://surrealdb.com/docs/learn/data-models/graph/social-network-patterns # Social network patterns Model follows, friendships, and interaction histories with graph edges, using weighted relations and counted edges to rank “strongest” ties in a toy social graph. Social products are a classic graph use case: **people** (or accounts) are nodes, and **follows**, **likes**, **friendships**, and **messages** are edges, often with timestamps or weights on the edge row. SurrealDB fits this well because edges are tables: you can store how strong a tie is, or derive strength from repeated interactions, without cramming that into either user profile. ## Weight on the edge record These examples use a small `npc` population and random pairwise interactions. Each `knows` edge accumulates a `greeted` counter whenever two NPCs interact again. ```surql -- Create 4 'npc' records CREATE |npc:1..5|; FOR $npc IN SELECT * FROM npc { -- Give each npc 20 random interactions FOR $_ IN 0..20 { -- Looks for a random NPC, use array::complement to filter out self LET $counterpart = rand::enum(array::complement((SELECT * FROM npc), [$npc])); -- See if they have a relation yet LET $existing = SELECT * FROM knows WHERE in = $npc.id AND out = $counterpart.id; -- If relation exists, increase 'greeted' by one IF !!$existing { UPDATE $existing SET greeted += 1; -- Otherwise create the relation and set 'greeted' to 1 } ELSE { RELATE $npc->knows->$counterpart SET greeted = 1; } }; }; SELECT id, ->knows.{ like_strength: greeted, with: out } AS relations FROM npc; ``` ```surql title="Which NPC each NPC likes the most" [ { id: npc:1, relations: [ { like_strength: 8, with: npc:3 }, { like_strength: 8, with: npc:4 }, { like_strength: 4, with: npc:2 } ] }, { id: npc:2, relations: [ { like_strength: 10, with: npc:1 }, { like_strength: 4, with: npc:3 }, { like_strength: 6, with: npc:4 } ] }, { id: npc:3, relations: [ { like_strength: 6, with: npc:2 }, { like_strength: 3, with: npc:4 }, { like_strength: 11, with: npc:1 } ] }, { id: npc:4, relations: [ { like_strength: 7, with: npc:1 }, { like_strength: 6, with: npc:3 }, { like_strength: 7, with: npc:2 } ] } ] ``` ## Counting parallel edges instead If each interaction is its own edge row, **aggregate** to find the strongest ties: ```surql -- Create 4 'npc' records CREATE |npc:1..5|; FOR $npc IN SELECT * FROM npc { -- Give each npc 20 random interactions FOR $_ IN 0..20 { -- Looks for a random NPC, use array::complement to filter out self LET $counterpart = rand::enum(array::complement((SELECT * FROM npc), [$npc])); RELATE $npc->greeted->$counterpart; }; }; SELECT count() AS like_strength, in AS npc, out AS counterpart FROM greeted GROUP BY npc, counterpart; ``` ```surql title="Which NPC each NPC likes the most" [ { counterpart: npc:2, like_strength: 6, npc: npc:1 }, { counterpart: npc:3, like_strength: 9, npc: npc:1 }, { counterpart: npc:4, like_strength: 5, npc: npc:1 }, { counterpart: npc:1, like_strength: 9, npc: npc:2 }, { counterpart: npc:3, like_strength: 6, npc: npc:2 }, { counterpart: npc:4, like_strength: 5, npc: npc:2 }, { counterpart: npc:1, like_strength: 10, npc: npc:3 }, { counterpart: npc:2, like_strength: 7, npc: npc:3 }, { counterpart: npc:4, like_strength: 3, npc: npc:3 }, { counterpart: npc:1, like_strength: 6, npc: npc:4 }, { counterpart: npc:2, like_strength: 4, npc: npc:4 }, { counterpart: npc:3, like_strength: 10, npc: npc:4 } ] ``` ## Friendship and symmetric ties For mutual “friend” relations where direction should not matter, combine [unique indexes on sorted endpoints](/docs/learn/data-models/graph/creating-relations.md#unique-index-for-relations-between-equals) with [`<->` traversal](/docs/learn/data-models/graph/graph-traversal.md#querying-symmetric-between-equals-relations) so you query both orientations consistently. ## Related guides * [Record links vs graph relations](/docs/learn/data-models/graph/record-links-vs-graph-relations.md), when edges win for interaction metadata and scoring. * [Graph traversal](/docs/learn/data-models/graph/graph-traversal.md), arrow syntax for multi-hop social feeds. --- Source: https://surrealdb.com/docs/learn/data-models/time-series/aggregation-queries # Aggregation queries Build downsampling with pre-computed table views, live queries, drop tables, and DEFINE EVENT for anomaly detection; compare SurrealDB to specialised TSDBs. Time-series data is usually written at full resolution and read at a coarser one, which means aggregating as it arrives rather than on every query. This page covers the four SurrealDB features that do that, and how the result compares with a specialised time-series database. ## Modelling metrics For doing metrics in SurrealDB you can choose one or combine: - **Pre-computed table views** - **Live Queries** - **Drop tables** - **Custom events** ### Pre-computed table views Our [pre-computed table views](/docs/reference/query-language/statements/define/table.md#pre-computed-table-views) are most similar to event-based, incrementally updating, materialised views. Practically, this means our downsampled metrics will always be up to date as it incrementally updates in near real-time when we add more records to the sensor_readings table. ```surql -- Define a table view which aggregates data from the sensor_readings table DEFINE TABLE daily_measurements_by_location AS SELECT id[0] AS location, time::day(id[2]) AS day, math::mean(temperature_celsius) AS avg_temperature_celsius, math::mean(humidity) AS avg_humidity_percent FROM sensor_readings GROUP BY id[0]; SELECT * FROM daily_measurements_by_location; ``` ```surql title="Output" [ { avg_humidity_percent: 55dec, avg_temperature_celsius: 28.4dec, day: 13, id: daily_measurements_by_location:[ location:Longleat_House ], location: location:Longleat_House } ] ``` For real-time visualisation of our metrics we can then use [Live Queries](/docs/reference/query-language/statements/live-select.md) to stream real-time updates to our client, such as a BI dashboard or embedded analytics code. ```surql LIVE SELECT * FROM daily_measurements_by_location; ``` ### Drop tables [Drop tables](/docs/reference/query-language/statements/define/table.md#example-usage) are pretty unique tables that drop all writes once they have been written. A view can be defined on a table whether it is a `DROP` table or not, but `DROP` is recommended if you have no reason to directly query the table that provides the data for the aggregated view. ```surql -- Drop all writes to the sensor_readings table. -- We don't need every these readings and can view them -- via daily_measurements_by_location instead DEFINE TABLE sensor_readings DROP; ``` These tables can be very useful in a time series context if you want to capture very high-frequency data but only care about storing the aggregated downsampled metrics. They are typically used in combination with either the table views or custom events, such that the metrics are calculated then the underlying data is automatically dropped. When combining drop tables, table views and live queries, you have a very easy-to-set up, event-based and real-time solution from capturing events, creating metrics, dropping stale data and live selects for visualisation. ### Custom events If you have something even more bespoke in mind, you can even [create your own event triggers](/docs/reference/query-language/statements/define/event.md) based on when a record is created, updated or deleted. You can include any valid SurrealQL inside the event. For example, we can create a simple real-time anomaly detection and notification solution using just SurrealQL events and functions in 5 steps. 1. Define an event to trigger when a record is added to the sensor_readings table. 2. Get the desired time range you want to track. 3. Calculate both the upper and lower threshold for an outlier, using the standard Q1 - 1.5 * IQR formula for the low outliers and Q3 + 1.5 * IQR formula for the high outliers. 4. Check if the current temperature is a low or high outlier. 5. Send an [http::post](/docs/reference/query-language/functions/database-functions/http.md#httppost) request with the outlier details. ```surql -- Trigger the event on when a record is created DEFINE EVENT sensor_anomaly_notification ON sensor_readings WHEN $event = 'CREATE' THEN { -- Get the desired time range you want to track -- here we're grabing just the past hour LET $location = $after[0]; LET $sensor = $after[1]; LET $temp_past_hour = ( SELECT VALUE temperature_celsius FROM sensor_readings:[ $location, $sensor, time::now() - 1h, ]..=[ $location, $sensor, time::now() ]); -- Calculate both the upper and lower threshold for an outlier -- using the standard Q1 - 1.5 * IQR formula for the low outliers LET $low_outliers = ( RETURN math::percentile($temp_past_hour, 25) - 1.5 * math::interquartile($temp_past_hour) ); -- Q3 + 1.5 * IQR formula for the high outliers LET $high_outliers = ( RETURN math::percentile($temp_past_hour, 75) + 1.5 * math::interquartile($temp_past_hour) ); -- If a low outlier is found send a http post request -- with the outlier details IF $after.temperature_celsius < $low_outliers { http::post('https://jsonplaceholder.typicode.com/posts', { id: rand::ulid(), outlier: $after, message: 'Outlier Detected: low temperature' }); }; -- If a high outlier is found send a http post request -- with the outlier details IF $after.temperature_celsius > $high_outliers { http::post('https://jsonplaceholder.typicode.com/posts', { id: rand::ulid(), outlier: $after, message: 'Outlier Detected: high temperature' }); }; }; ``` ### Async events If you need an event to execute in the background out of the main transaction in which they are called, you can use an [async event](/docs/reference/query-language/statements/define/event.md#async-events). This can be useful for events that take a certain amount of time to process. The tradeoff is that your read data will not be consistent until the event or events have finished processing. This is what is known as *eventual consistency*. ## SurrealDB vs specialised time series databases There are many specialised time series databases out there, so where does SurrealDB fit in? The advantages SurrealDB has over specialised time series databases are: - That you can combine our time series functionality with the rest of our multi-model database features. For example, doing full-text search and vector search on your log data. - No need to learn another query language just for time series. SurrealDB has a unified query language for all its features. - Connect and enrich your metrics easily, instead of having them being siloed in a separate system. You can have all your data in one place with zero ETL for your various use cases. Whether you’re doing transactional, analytical, ML and AI applications, SurrealDB covers a lot of the use cases a modern application needs. The advantages specialised time series databases have over SurrealDB currently are: - More advanced time series features such as custom data retention policies and better data compression. Whether you pick SurrealDB for your time series use cases depends mostly on whether you are looking to lower your total system complexity or if you are looking for another specialised solution. For raw event storage and complex record ID patterns, see [IoT and telemetry patterns](/docs/learn/data-models/time-series/iot-and-telemetry-patterns.md). --- Source: https://surrealdb.com/docs/learn/data-models/time-series/iot-and-telemetry-patterns # IoT and telemetry patterns Model sensor readings with nested fields or complex record IDs, use range queries on array-based IDs, and attach metadata through record links. Sensor data arrives constantly and is almost always read back by time range, which makes the record ID the main modelling decision. This page works through an IoT example covering nested fields, array-based record IDs, range queries over them, and metadata attached through record links. ## Modelling time series data in SurrealDB Let’s explore a practical example using IoT sensor data. ### Modelling events The normal way of modelling data in SurrealDB would be as fields in a record, such as `coordinates` or `location`. ```surql CREATE sensor:ARF8394AAA SET coordinates = (-2.2743, 51.1857); CREATE location:Longleat_House SET built = d'1580-01-01'; CREATE sensor_readings CONTENT { timestamp: time::now(), location: location:Longleat_House, sensor: sensor:ARF8394AAA, temperature_celsius: 28.4, humidity_percent: 55 }; ``` ### Complex record IDs A more optimised way of working with the `sensor_readings` table in a time series context would however be using an array-based record ID, otherwise known as a complex record ID. ```surql -- Array-based record IDs CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now(), 28.4 ]; ``` Storing the same information as a record or as part of the ID might look similar at first glance, but under the hood, it’s optimised for efficient range selection, through the magic of [range queries](/docs/reference/query-language/language-primitives/data-types/record-ids.md#record-ranges) on record IDs. As all values in SurrealDB can be [compared and sorted](/docs/reference/query-language/language-primitives/data-types/values.md#comparing-and-ordering-values), putting the most crucial information into the record ID itself allows you to query only a range of record IDs instead of a whole table. This effectively means fewer worries about selecting the right indexes or partitions since the ID field already does that naturally in your data as you scale with the performance of a key-value lookup regardless of size! ```surql -- Select all the temperature readings from the start until now -- from a specific sensor SELECT id[0] AS location, id[2] AS at, id[3] AS temperature_celsius FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() - 1d ]..=[ location:Longleat_House, sensor:ARF8394AAA, time::now(), ]; ``` ```surql title="Output" [ { at: d'2025-04-17T04:04:54.842Z', location: location:Longleat_House, temperature_celsius: 28.4f } ] ``` This is however not the only way of doing it, you can have the metadata in the ID and sensor data in the record itself, like in the example below. Now the record ID only contains the fields useful in a range query, while the remaining fields are kept in the record itself. ```surql CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() ] CONTENT { temperature_celsius: 28.4, humidity: 55 }; -- Select all the temperature readings from the start until now -- from a specific sensor SELECT id[0] AS location, id[2] AS at, temperature_celsius FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() - 1d ]..=[ location:Longleat_House, sensor:ARF8394AAA, time::now(), ]; ``` ```surql title="Output" [ { at: d'2025-04-17T05:25:00.030Z', location: location:Longleat_House, temperature_celsius: 28.4f } ] ``` ### Ids inside ids The last thing to note here is that we’ve actually been using record IDs inside our complex record IDs! This is known as a [record link](/docs/reference/query-language/language-primitives/record-links.md), which allows us to reduce the fields in our main time series table to only the necessary ones by offloading most of the metadata to connected tables. In our case we have: - `location:Longleat_House`, which refers to the `Longleat_House` ID in the `location` table. There we put all the metadata about the location itself such as [geo coordinates](/docs/reference/query-language/language-primitives/data-types/geometries.md). - `sensor:ARF8394AAA`, which refers to the `ARF8394AAA` ID on the `sensor` table. There we could put all the metadata about the sensor such as location, firmware, when it was bought and when it needs maintenance. It’s very easy and performant to get the connected data, since you don’t have to do any table scans for that either since it links directly to a specific record on a specific table! In the example below, with time for the sensor readings set exactly to the hour (`2024-08-13T05:00:00`), the record ID is easy to predict. ```surql CREATE sensor:ARF8394AAA SET coordinates = (-2.2743, 51.1857); CREATE location:Longleat_House SET built = d'1580-01-01'; -- location + sensor + easy to predict timestamp CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ] CONTENT { temperature_celsius: 28.4, humidity: 55 }; -- Select all fields in the ID and the coordinates field from the sensor table SELECT id, id[1].coordinates AS sensor_coordinates FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ]; ``` ```surql title="Output" [ { id: sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ], sensor_coordinates: (-2.2743, 51.1857) } ] ``` For aggregations, materialised views, and retention-style patterns built on top of these events, see [Aggregation queries](/docs/learn/data-models/time-series/aggregation-queries.md). --- Source: https://surrealdb.com/docs/learn/data-models/time-series/overview # Time-series model Learn how time-series workloads centre on timestamps, tags, and aggregations in SurrealDB, and where to find guides on bucketing, IoT-style modelling, and metrics. A time series database is optimised for storing, querying, and managing time-stamped data. Rather than treating the timestamp as just another attribute, TSDBs place time at the forefront of data modelling, indexing, retention, and query optimisation. The queries in time series workloads often include: * Finding or aggregating values over time windows (e.g., last 5 minutes, last 30 days). * Detecting patterns or anomalies (peaks, troughs, rolling averages). * Combining real-time and historical data for a holistic view of a system’s behaviour. This section helps you understand how to think like a time series database user, and how SurrealDB can accommodate these needs. ## Core concepts of time series modelling - Timestamp: Every record includes at least one timestamp field, typically indicating when the data was created, measured, or observed. - Measurement / metrics: Time series data often revolves around measurements, like temperature, CPU usage, or stock prices. Each measurement can have multiple fields or tags. - Tags (or labels): Extra dimensions that categorise data. For instance, a sensor reading might have a `location` or a `device_id` tag. These tags help you filter or group data. - Aggregations: Because time series data can be massive (think thousands of data points per second), it is common to store or generate aggregated values (e.g., hourly averages or daily sums) for faster analytics. - Retention policies: Time series datasets can grow exponentially. Many TSDBs support retention policies that drop or downsample older data while keeping recent data at high granularity. - Time buckets: It is typical to group data into buckets (e.g., 1-minute, 5-minute, or 1-day intervals) for analytics. ## Benefits of using time series databases - Efficient ingestion: TSDBs are optimised for high write throughput, as time series data often arrives in large bursts or continuous streams. - Fast time-based queries: Data structures are specialised for retrieving data over specific time windows or intervals. - Built-in aggregation and downsampling: Many TSDBs offer native functions to summarise data by minute, hour, or other intervals. - Scalability: Time partitioning or sharding strategies make it easier to handle large volumes of time series data. - Retention control: You can automatically expire older data or keep only aggregated versions to save storage. ## Where to go next - [Time buckets and windowing](/docs/learn/data-models/time-series/time-buckets-and-windowing.md): bucketing, retention, and window-style thinking. - [Aggregation queries](/docs/learn/data-models/time-series/aggregation-queries.md): pre-computed table views, live queries, drop tables, events, and how SurrealDB compares to specialised TSDBs. - [IoT and telemetry patterns](/docs/learn/data-models/time-series/iot-and-telemetry-patterns.md): sensor readings, complex record IDs, and linking metadata through record IDs. ## Going further - [Using SurrealDB as a time series database](/docs/learn/data-models/time-series/using-as-time-series-database.md) - the full pattern end to end --- Source: https://surrealdb.com/docs/learn/data-models/time-series/time-buckets-and-windowing # Time buckets and windowing Group time-stamped records into buckets with time::floor and GROUP BY, plan retention and downsampling, and align queries with windowed analytics workloads. Time series analytics almost always relies on windows: “per minute”, “last 24 hours”, “calendar month”. In SurrealDB you express these ideas with ordinary SurrealQL - filtering on timestamps or on components of [complex record IDs](/docs/learn/data-models/time-series/iot-and-telemetry-patterns.md), and grouping with functions such as `time::day()` where appropriate. ## Time buckets A bucket is a fixed or sliding interval that groups raw events into summaries: counts, sums, averages, or percentiles. Buckets are how dashboards show smooth lines instead of millions of raw points. When you design buckets, choose the grain that matches how people consume the data (operations often want 1-minute or 5-minute data; executives may only need daily rollups). **Example - hourly energy readings:** [`time::floor()`](/docs/reference/query-language/functions/database-functions/time.md#timefloor) snaps each `ts` down to the start of a duration (here `1h`), and [`GROUP BY`](/docs/reference/query-language/clauses/group.md) collapses every row in that window into one aggregate. ```surql CREATE measurement SET ts = d"2025-04-10T09:15:00Z", kwh = 1.2; CREATE measurement SET ts = d"2025-04-10T09:42:00Z", kwh = 0.8; CREATE measurement SET ts = d"2025-04-10T10:05:00Z", kwh = 2.0; SELECT count(), time::floor(ts, 1h) AS hour_bucket FROM measurement GROUP BY hour_bucket; ``` ```surql title="Output" [ { count: 2, hour_bucket: d'2025-04-10T09:00:00Z' }, { count: 1, hour_bucket: d'2025-04-10T10:00:00Z' } ] ``` For calendar-aligned buckets (whole days or months), [`time::group()`](/docs/reference/query-language/functions/database-functions/time.md#timegroup) is often simpler than picking a duration. ## Aligning windows to business time Decide whether windows are wall-clock (calendar buckets) or relative (“last 15 minutes” rolling). Range queries on record IDs or timestamp fields should use the same convention across writers and readers so aggregates line up. For concrete sensor and metrics examples, continue with [IoT and telemetry patterns](/docs/learn/data-models/time-series/iot-and-telemetry-patterns.md) and [Aggregation queries](/docs/learn/data-models/time-series/aggregation-queries.md). --- Source: https://surrealdb.com/docs/learn/data-models/time-series/using-as-time-series-database # Using SurrealDB as a time series database In this guide, you will learn how to “think” in a time-series database and show how SurrealDB helps you implement these concepts seamlessly. A time series database is optimised for storing, querying, and managing time-stamped data. Rather than treating the timestamp as just another attribute, TSDBs place time at the forefront of data modelling, indexing, retention, and query optimisation. The queries in time series workloads often include: * Finding or aggregating values over time windows (e.g., last 5 minutes, last 30 days). * Detecting patterns or anomalies (peaks, troughs, rolling averages). * Combining real-time and historical data for a holistic view of a system’s behaviour. This guide will help you understand how to think like a time series database user, and how SurrealDB can accommodate these needs. ## Core concepts of time series modelling - **Timestamp**: Every record includes at least one timestamp field, typically indicating when the data was created, measured, or observed. - **Measurement / Metrics**: Time series data often revolves around measurements, like temperature, CPU usage, stock prices. Each measurement can have multiple fields or tags. - **Tags (or Labels)**: Extra dimensions that categorize data. For instance, a sensor reading might have a `location` or a `device_id` tag. These tags help you filter or group data. - **Aggregations**: Because time series data can be massive (think thousands of data points per second), it’s common to store or generate aggregated values (e.g., hourly averages or daily sums) for faster analytics. - **Retention Policies**: Time series datasets can grow exponentially. Many TSDBs support retention policies that drop or downsample older data while keeping recent data at high granularity. - **Time Buckets**: It’s typical to group data into buckets (e.g., 1-minute, 5-minute, or 1-day intervals) for analytics. ## Benefits of using time series databases - **Efficient Ingestion**: TSDBs are optimised for high write throughput, as time series data often arrives in large bursts or continuous streams. - **Fast Time-Based Queries**: Data structures are specialized for retrieving data over specific time windows or intervals. - **Built-In Aggregation and Downsampling**: Many TSDBs offer native functions to summarize data by minute, hour, or other intervals. - **Scalability**: Time partitioning or sharding strategies make it easier to handle large volumes of time series data. - **Retention Control**: You can automatically expire older data or keep only aggregated versions to save storage. ## Modelling time series data in SurrealDB Now that we’ve established a common understanding of time series data, let’s explore a practical example using IoT sensor data. ### Modelling events The normal way of modelling data in SurrealDB would be as fields in a record, such as `coordinates` or `location`. ```surql CREATE sensor:ARF8394AAA SET coordinates = (-2.2743, 51.1857); CREATE location:Longleat_House SET built = d'1580-01-01'; CREATE sensor_readings CONTENT { timestamp: time::now(), location: location:Longleat_House, sensor: sensor:ARF8394AAA, temperture_celsius: 28.4, humidity_percent: 55 }; ``` ### Complex record IDs A more optimised way of working with the `sensor_readings` table in a time series context would however be using an array-based record ID, otherwise known as a complex record ID. ```surql -- Array-based record IDs CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now(), 28.4 ]; ``` Storing the same information as a record or as part of the ID might look similar at first glance, but under the hood, it’s optimised for efficient range selection, through the magic of [range queries](/docs/reference/query-language/language-primitives/data-types/record-ids.md#record-ranges) on record IDs. As all values in SurrealDB can be [compared and sorted](/docs/reference/query-language/language-primitives/data-types/values.md#comparing-and-ordering-values), putting the most crucial information into the record ID itself allows you to query only a range of record IDs instead of a whole table. This effectively means fewer worries about selecting the right indexes or partitions since the ID field already does that naturally in your data as you scale with the performance of a key-value lookup regardless of size! ```surql -- Select all the temperature readings from the start until now -- from a specific sensor SELECT id[0] AS location, id[2] AS at, id[3] AS temperature_celsius FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() - 1d ]..=[ location:Longleat_House, sensor:ARF8394AAA, time::now(), ]; ``` ```surql title="Output" [ { at: d'2025-04-17T04:04:54.842Z', location: location:Longleat_House, temperature_celsius: 28.4f } ] ``` This is however not the only way of doing it, you can have the metadata in the ID and sensor data in the record itself, like in the example below. Now the record ID only contains the fields useful in a range query, while the remaining fields are kept in the record itself. ```surql CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() ] CONTENT { temperature_celsius: 28.4, humidity: 55 }; -- Select all the temperature readings from the start until now -- from a specific sensor SELECT id[0] AS location, id[2] AS at, temperature_celsius FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, time::now() - 1d ]..=[ location:Longleat_House, sensor:ARF8394AAA, time::now(), ]; ``` ```surql title="Output" [ { at: d'2025-04-17T05:25:00.030Z', location: location:Longleat_House, temperature_celsius: 28.4f } ] ``` ### IDs inside IDs The last thing to note here is that we’ve actually been using record IDs inside our complex record IDs! This is known as a [record link](/docs/reference/query-language/language-primitives/record-links.md), which allows us to reduce the fields in our main time series table to only the necessary ones by offloading most of the metadata to connected tables. In our case we have: - `location:Longleat_House`, which refers to the `Longleat_House` ID in the `location` table. There we put all the metadata about the location itself such as [geo coordinates](/docs/reference/query-language/language-primitives/data-types/geometries.md). - `sensor:ARF8394AAA`, which refers to the `ARF8394AAA` ID on the `sensor` table. There we could put all the metadata about the sensor such as location, firmware, when it was bought and when it needs maintenance. It’s very easy and performant to get the connected data, since you don’t have to do any table scans for that either since it links directly to a specific record on a specific table! In the example below, with time for the sensor readings set exactly to the hour (`2024-08-13T05:00:00`), the record ID is easy to predict. ```surql CREATE sensor:ARF8394AAA SET coordinates = (-2.2743, 51.1857); CREATE location:Longleat_House SET built = d'1580-01-01'; -- location + sensor + easy to predict timestamp CREATE sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ] CONTENT { temperature_celsius: 28.4, humidity: 55 }; -- Select all fields in the ID and the coordinates field from the sensor table SELECT id, id[1].coordinates AS sensor_coordinates FROM sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ]; ``` ```surql title="Output" [ { id: sensor_readings:[ location:Longleat_House, sensor:ARF8394AAA, d'2024-08-13T05:00:00Z' ], sensor_coordinates: (-2.2743, 51.1857) } ] ``` Now that we’ve explored a bit how to store and query event data, let’s turn this data into metrics. ### Modelling metrics For doing metrics in SurrealDB you can choose one or combine: - **Pre-computed table views** - **Live Queries** - **Drop tables** - **Custom events** #### Pre-computed table views Our [pre-computed table views](/docs/reference/query-language/statements/define/table.md#pre-computed-table-views) are most similar to event-based, incrementally updating, materialised views. Practically, this means our downsampled metrics will always be up to date as it incrementally updates in near real-time when we add more records to the sensor_readings table. ```surql -- Define a table view which aggregates data from the sensor_readings table DEFINE TABLE daily_measurements_by_location AS SELECT id[0] AS location, time::day(id[2]) AS day, math::mean(temperature_celsius) AS avg_temperture_celsius, math::mean(humidity) AS avg_humidity_percent FROM sensor_readings GROUP BY id[0]; SELECT * FROM daily_measurements_by_location; ``` ```surql title="Output" [ { avg_humidity_percent: 55dec, avg_temperture_celsius: 28.4dec, day: 13, id: daily_measurements_by_location:[ location:Longleat_House ], location: location:Longleat_House } ] ``` For real-time visualisation of our metrics we can then use [Live Queries](/docs/reference/query-language/statements/live-select.md) to stream real-time updates to our client, such as a BI dashboard or embedded analytics code. ```surql LIVE SELECT * FROM daily_measurements_by_location; ``` #### Drop tables [Drop tables](/docs/reference/query-language/statements/define/table.md#example-usage) are pretty unique tables that drop all writes once they have been written. A view can be defined on a table whether it is a `DROP` table or not, but `DROP` is recommended if you have no reason to directly query the table that provides the data for the aggregated view. ```surql -- Drop all writes to the sensor_readings table. -- We don't need every these readings and can view them -- via daily_measurements_by_location instead DEFINE TABLE sensor_readings DROP; ``` These tables can be very useful in a time series context if you want to capture very high-frequency data but only care about storing the aggregated downsampled metrics. They are typically used in combination with either the table views or custom events, such that the metrics are calculated then the underlying data is automatically dropped. When combining drop tables, table views and live queries, you have a very easy-to-set up, event-based and real-time solution from capturing events, creating metrics, dropping stale data and live selects for visualisation. #### Custom events If you have something even more bespoke in mind, you can even [create your own event triggers](/docs/reference/query-language/statements/define/event.md) based on when a record is created, updated or deleted. You can include any valid SurrealQL inside the event. For example, we can create a simple real-time anomaly detection and notification solution using just SurrealQL events and functions in 5 steps. 1. Define a event to trigger when a record is added to the sensor_readings table. 2. Get the desired time range you want to track. 3. Calculate both the upper and lower threshold for an outlier, using the standard Q1 - 1.5 * IQR formula for the low outliers and Q3 + 1.5 * IQR formula for the high outliers. 4. Check if the current temperature is a low or high outlier. 5. Send an [http::post](/docs/reference/query-language/functions/database-functions/http.md#httppost) request with the outlier details. ```surql -- Trigger the event on when a record is created DEFINE EVENT sensor_anomaly_notification ON sensor_readings WHEN $event = 'CREATE' THEN { -- Get the desired time range you want to track -- here we're grabing just the past hour LET $location = $after[0]; LET $sensor = $after[1]; LET $temp_past_hour = ( SELECT VALUE temperature_celsius FROM sensor_readings:[ $location, $sensor, time::now() - 1h, ]..=[ $location, $sensor, time::now() ]); -- Calculate both the upper and lower threshold for an outlier -- using the standard Q1 - 1.5 * IQR formula for the low outliers LET $low_outliers = ( RETURN math::percentile($temp_past_hour, 25) - 1.5 * math::interquartile($temp_past_hour) ); -- Q3 + 1.5 * IQR formula for the high outliers LET $high_outliers = ( RETURN math::percentile($temp_past_hour, 75) + 1.5 * math::interquartile($temp_past_hour) ); -- If a low outlier is found send a http post request -- with the outlier details IF $after.temperature_celsius < $low_outliers { http::post('https://jsonplaceholder.typicode.com/posts', { id: rand::ulid(), outlier: $after, message: 'Outlier Detected: low temperature' }); }; -- If a high outlier is found send a http post request -- with the outlier details IF $after.temperature_celsius > $high_outliers { http::post('https://jsonplaceholder.typicode.com/posts', { id: rand::ulid(), outlier: $after, message: 'Outlier Detected: high temperature' }); }; }; ``` ### Async events If you need an event to execute in the background out of the main transaction in which they are called, you can use an [async event](/docs/reference/query-language/statements/define/event.md#async-events). This can be useful for events that take a certain amount of time to process. The tradeoff is that your read data will not be consistent until the event or events have finished processing. This is what is known as *eventual consistency*. ## SurrealDB vs specialised time series databases There are many specialised time series databases out there, so where does SurrealDB fit in? The advantages SurrealDB has over specialised time series databases are: - That you can combine our time series functionality with the rest of our multi-model database features. For example, doing full-text search and vector search on your log data. - No need to learn another query language just for time series. SurrealDB has a unified query language for all its features. - Connect and enrich your metrics easily, instead of having them being siloed in a separate system. You can have all your data in one place with zero ETL for your various use cases. Whether you’re doing transactional, analytical, ML and AI applications, SurrealDB covers a lot of the use cases a modern application needs. The advantages specialised time series databases have over SurrealDB currently are: - More advanced time series features such as custom data retention policies and better data compression. Whether you pick SurrealDB for your time series use cases depends mostly on whether you are looking to lower your total system complexity or if you are looking for another specialised solution. --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/embedding-pipelines # Embedding pipelines Store vector fields on records, size embeddings for throughput and accuracy, and trim dimensions when models allow. Embeddings are produced **outside** SurrealDB by your chosen model (API or local). The database’s job is to store the resulting arrays and index them for retrieval. To store vectors in SurrealDB, you typically define a field within your data schema dedicated to holding the vector data. These vectors represent data points in embedding space and can be used for various applications, from recommendation systems to image recognition. Below is an example of how to create records with vector embeddings: ```surql CREATE Document:1 CONTENT { items: [ { content: "apple", embedding: [0.00995, -0.02680, -0.01881, -0.08697] } ] }; ``` There are no strict rules or limitations regarding the length of the embeddings, and they can be as large as needed. Just keep in mind that larger embeddings lead to more data to process and that can affect performance and query times based on your physical hardware. In fact, embeddings retrieved from a model can be cut down to any length you prefer if the accuracy is still acceptable for your use case. For end-to-end similarity examples, see [Similarity search](/docs/learn/data-models/vector-search/similarity-search.md). For HNSW, DISKANN, and brute-force trade-offs, see [Vector indexes](/docs/learn/data-models/vector-search/vector-indexes.md). --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/hybrid-search # Hybrid search Compare lexical full-text search with vector similarity on the same dataset and fuse rankings with search::rrf and related helpers. Full-text search and vector search find different things, and the two together usually rank better than either alone. This page compares them on one dataset, then fuses their rankings with `search::rrf` and related helpers. ## Vector search vs full-text search SurrealDB supports [full-text search](/docs/learn/data-models/full-text-search/overview.md) and Vector Search. Full-text search (FTS) involves indexing documents using an [FTS index](/docs/reference/query-language/statements/define/indexes.md#full-text-search-fulltext-index) that makes use of an [analyzer](/docs/reference/query-language/statements/define/analyzer.md) that breaks down text using [tokenizers](/docs/reference/query-language/statements/define/analyzer.md#tokenizers) and [filters](/docs/reference/query-language/statements/define/analyzer.md#filters). Google search for the word 'lead' The image above is a Google search for the word “lead”, a word with more than one definition (and pronunciation!). Lead can mean 'taking initiative', as well as the chemical element with the symbol 'Pb'. Let's consider this in the context of a database of liquid samples which note down harmful chemicals that are found in them. In the example below, we have a table called `liquids` with a `sample` field and a `content` field. Next, we can define a [full-text index](/docs/reference/query-language/statements/define/indexes.md#full-text-search-fulltext-index) on the `content` field by first defining an analyzer called `liquid_analyzer`. We can then [define an index](/docs/reference/query-language/statements/define/indexes.md) on the content field in the liquid table and set our [custom analyzer](/docs/reference/query-language/statements/define/analyzer.md) (`liquid_analyzer`)to search through the index. Then, using the select statement to retrieve all the samples containing the chemical lead will also bring up samples that mention the word `lead`. ```surql -- Insert a sample & content field into a liquids table INSERT INTO liquids [ {sample:'Sea water', content: 'The sea water contains some amount of lead'}, {sample:'Tap water', content: 'The team lead by Dr. Rose found out that the tap water in was potable'}, {sample:'Sewage water', content: 'High amounts of a were found in Sewage water'} ]; -- Define an analyzer for the liquid table and an index on the content field with the analyzer DEFINE ANALYZER liquid_analyzer TOKENIZERS blank,class,camel,punct FILTERS snowball(english); DEFINE INDEX liquid_content ON liquids FIELDS content FULLTEXT ANALYZER liquid_analyzer BM25 HIGHLIGHTS; -- Retrieve all the samples containing the chemical lead will also bring up samples that simply mention the word lead SELECT sample, content FROM liquids WHERE content @0@ 'lead'; ``` If you read through the content of the tap water sample, you’ll notice that it does not contain any lead in it but it has the mention of the word `lead` under “The team lead by Dr. Rose…” which means that the team was guided by Dr. Rose. The search pulled up both the records although the tap water sample had no lead in it. This example shows us that while full-text search does a great job at matching query terms with indexed documents, on its own it may not be the best solution for use cases where the query terms have deeper context and scope for ambiguity. For vector-side retrieval on the same story, see [Similarity search](/docs/learn/data-models/vector-search/similarity-search.md). ## Hybrid search functions As mentioned above, full-text search and vector search can both be used in SurrealDB. In addition, some functions exist inside the [`search::`](/docs/reference/query-language/functions/database-functions/search.md) namespace that take both full-text and vector arguments in order to produce a single unified output. Here is an example of one of them called [`search::rrf()`](/docs/reference/query-language/functions/database-functions/search.md#searchrrf) which does this using an algorithm called reciprocal rank fusion. ```surql -- Sample data -- CREATE test:1 SET text = "Graph databases are great.", embedding = [0.10, 0.20, 0.30]; CREATE test:2 SET text = "Relational databases store tables.", embedding = [0.05, 0.10, 0.00]; CREATE test:3 SET text = "This document mentions graphs.", embedding = [0.20, 0.10, 0.25]; -- Analyzer used by the full‑text index DEFINE ANALYZER simple TOKENIZERS class, punct FILTERS lowercase, ascii; -- Full‑text index DEFINE INDEX idx_text ON TABLE test FIELDS text FULLTEXT ANALYZER simple BM25; ``` **HNSW (in-memory)** ```surql DEFINE INDEX idx_embedding ON TABLE test FIELDS embedding HNSW DIMENSION 3 DIST COSINE; ``` **DISKANN (on-disk)** _(since v3.1.0)_ ```surql DEFINE INDEX idx_embedding ON TABLE test FIELDS embedding DISKANN DIMENSION 3 DIST COSINE TYPE F32; ``` For very large embedding sets that do not fit comfortably in RAM, prefer DISKANN. It is **not available on WASM** builds. ```surql -- Query vector (whatever your embedding model produced for "graph databases") LET $qvec = [0.12, 0.18, 0.27]; -- Vector search: top 2 nearest neighbours LET $vs = SELECT id FROM test WHERE embedding <|2,100|> $qvec; -- Full‑text search: top 2 lexical matches LET $ft = SELECT id, search::score(1) as score FROM test WHERE text @1@ 'graph' ORDER BY score DESC LIMIT 2; -- Fuse with Reciprocal Rank Fusion (k defaults to 60 if omitted) search::rrf([$vs, $ft], 2, 60); ``` > [!NOTE] > On a small dataset the lexical half of a hybrid query can contribute membership without contributing order. BM25 clamps the weight of any term appearing in half or more of the indexed documents to zero, so `search::score` returns `0` for every match and the `ORDER BY score DESC` above has nothing to sort on. Reciprocal rank fusion then folds in an arbitrary ordering of the matched rows. See [why a score can be 0](/docs/learn/data-models/full-text-search/scoring-and-ranking.md#why-a-score-can-be-0). --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/overview # Vector search model Learn how vector embeddings represent semantic similarity, how SurrealDB stores and queries vectors alongside other models, and where to find guides on pipelines, similarity, indexes, hybrid search, and RAG-style retrieval. A vector database is specialised for storing high-dimensional vectors and for efficiently performing queries on them. Rather than searching on exact values or text-based queries, vector databases let you search based on semantic similarity. For instance, in a text embedding scenario, you can find documents that are semantically similar to a given query, even if they do not share the same keywords. This allows for usage in areas such as: - Recommendation systems: Suggest items (movies, products, etc.) similar to what a user has liked based on learned embeddings. - Image or audio search: Identify images or audio clips semantically similar to a given sample. - Clustering and classification: Perform unsupervised clustering of data points or quickly identify which category a vector is close to. With SurrealDB’s query language you can define vector fields, store numeric arrays as embeddings, create indexes, and perform similarity queries. SurrealDB’s approach unifies these features to allow you to move from multiple data stores (a dedicated vector database plus separate document database and finally a graph database to stitch them together) to a single source of truth. But how do you “think” in a vector database? Unlike relational or document models, where the focus is on well-defined schemas and relationships, vector databases revolve around embeddings: numerical representations of objects (like text, images, audio snippets, etc.) in a continuous vector space. This allows you to design data structures and queries to exploit these embeddings for similarity search or AI-driven retrieval. ## Core concepts of vector-oriented modelling Vector search is a search mechanism that goes beyond traditional keyword matching and text-based search methods to capture deeper characteristics and similarities between data. Vector search isn't new to the world of data science. [Gerard Salton](https://en.wikipedia.org/w/index.php?title=Gerard_Salton&oldid=1334543561), known as the Father of Information Retrieval, introduced the Vector Space Model, cosine similarity, and TF-IDF for information retrieval around 1960. It converts data such as text, images, or sounds into numerical vectors, called vector embeddings. You can think of vector embeddings as cells. In the same way that cells form the basic structural and biological unit of all known living organisms, vector embeddings serve as the basic units of data representation in vector search. In practice, embeddings are typically dense vectors of real numbers that capture the semantic or contextual meaning of data. For instance, in Natural Language Processing (NLP), a word or sentence can be transformed into a vector of length 128, 256, 768, or even more dimensions. The idea is that similar objects (in meaning) end up having similar vector representations, making it possible to compute how close they are in the vector space. Embeddings themselves are not generated by the database. Instead, they depend on which model is used to generate them. Various companies such as OpenAI and Mistral have both free and paid models, while many other [free models](/docs/build/integrations/embeddings-providers/fastembed.md) exist to generate embeddings. Inside a database they will be stored in this sort of manner. ```surql [ { embedding: [ 0.0007022718782536685, 0.004178352188318968, 0.009888353757560253, -- and so on for 128, 256, 768, or even more numbers ], text: "To be, or not to be: that is the question." }, { embedding: [ -0.027426932007074356, 0.0008020889363251626, -0.02949262224137783 ], text: "All the world’s a stage, and all the men and women merely players." }, { embedding: [ -0.05859993398189545, -0.011999601498246193, -0.06185592710971832 ], text: "The course of true love never did run smooth." } ] ``` ## Where to go next The guides in this section break down how to store embeddings and query them in practice: - [Embedding pipelines](/docs/learn/data-models/vector-search/embedding-pipelines.md): defining vector fields, record shapes, and dimension trade-offs. - [Similarity search](/docs/learn/data-models/vector-search/similarity-search.md): worked examples, the `vector::` function family, and KNN-style filtering. - [Vector indexes](/docs/learn/data-models/vector-search/vector-indexes.md): brute force vs HNSW vs DISKANN, parameters, and the query cheat sheet. - [Hybrid search](/docs/learn/data-models/vector-search/hybrid-search.md): combining lexical full-text search with vector retrieval, including `search::rrf()`. - [RAG architecture patterns](/docs/learn/data-models/vector-search/rag-architecture-patterns.md): retrieval-oriented conclusions and further reading. For statement-level reference, see [Vector functions](/docs/reference/query-language/functions/database-functions/vector.md) and [`DEFINE INDEX` (vector search)](/docs/reference/query-language/statements/define/indexes.md#vector-search-indexes). --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/rag-architecture-patterns # RAG architecture patterns Map retrieval-augmented generation to SurrealDB using chunk storage, embeddings, hybrid search, filters, and operational concerns for RAG. Retrieval-augmented generation (RAG) combines retrieval from your own data with generation from a language model. Instead of relying only on the model’s training data, you ground answers in chunks of text (and sometimes structured facts) that you control. SurrealDB is then used for storage and retrieval, allowing you to keep chunk text, metadata, access rules, and vector embeddings in one database and query them with SurrealQL. ## What RAG changes A plain LLM call answers from parametric memory only. RAG adds a knowledge plane to this, in which documents are split into chunks, each chunk gets an embedding from a model you choose, and at query time you retrieve the most relevant chunks before (or while) the LLM writes an answer. ## Typical pipeline stages 1. Ingest: Load sources (files, web pages, tickets, database exports). 2. Chunk: Split text into segments with stable boundaries (headings, paragraphs, token limits) and optional overlap so ideas are not split awkwardly. 3. Embed: Call an embedding API or local model; store the resulting vector with each chunk. See [Embedding pipelines](/docs/learn/data-models/vector-search/embedding-pipelines.md) and the [embeddings integrations](/docs/build/integrations/embeddings-providers/fastembed.md) for options outside the database. 4. Index: Define [vector indexes](/docs/learn/data-models/vector-search/vector-indexes.md) (for example HNSW) on embedding fields, and optionally [full-text indexes](/docs/learn/data-models/full-text-search/overview.md) on chunk text for lexical search. 5. Retrieve: For a user question, embed the query (or use the same model family as the corpus), run similarity search, optionally fuse with keyword results (see [Hybrid search](/docs/learn/data-models/vector-search/hybrid-search.md)). 6. Generate: Pass the top chunks as context to the LLM, with instructions to cite or stay within that context. ## Retrieval patterns - Dense retrieval: KNN over embeddings with [`vector::distance::knn()`](/docs/reference/query-language/functions/database-functions/vector.md#vectordistanceknn) and the patterns in [Similarity search](/docs/learn/data-models/vector-search/similarity-search.md). Good for paraphrases and conceptual match. - Hybrid retrieval: Combine dense scores with [full-text search](/docs/learn/data-models/full-text-search/overview.md) when exact product names, error codes, or legislation matter; use [`search::rrf()`](/docs/learn/data-models/vector-search/hybrid-search.md) or related helpers as described under [Hybrid search](/docs/learn/data-models/vector-search/hybrid-search.md). Re-ranking (a second model that scores the top *k* candidates) is often implemented in the application layer; SurrealDB supplies the candidate set efficiently. ## Chunking, metadata, and citations There is no single best chunk size: smaller chunks improve precision but lose surrounding context; larger chunks add context but dilute embeddings. Many teams store title, heading path, or summary fields to improve retrieval without inflating the embedded body. For citations, persist enough metadata to map a chunk back to a human-readable source (page, anchor, ticket id). The LLM should only “see” what you retrieved; clear provenance reduces hallucinated references. ## Operations and quality - Embedding model changes usually require re-embedding the corpus or maintaining a version dimension; plan migrations before switching dimensions or distance metrics. - Staleness: when sources update, replace or invalidate affected chunks so answers do not quote obsolete text. - Evaluation: track retrieval hit rate, user thumbs-up/down, or offline benchmarks on labelled questions. ## Resources - [Vector search cheat sheet](/docs/learn/data-models/vector-search/vector-indexes.md#vector-search-cheat-sheet) - [Vector search overview](/docs/learn/data-models/vector-search/overview.md) - [Hybrid search](/docs/learn/data-models/vector-search/hybrid-search.md) - [Full-text search overview](/docs/learn/data-models/full-text-search/overview.md) - [Embedding pipelines](/docs/learn/data-models/vector-search/embedding-pipelines.md) - [Vector functions](/docs/reference/query-language/functions/database-functions/vector.md) - [Vector search indexes](/docs/reference/query-language/statements/define/indexes.md#vector-search-indexes) - [FastEmbed and embeddings integrations](/docs/build/integrations/embeddings-providers/fastembed.md) - [YouTube: Vector search intro](https://www.youtube.com/watch?v=MqddPmgKSCs) --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/similarity-search # Similarity search Run SurrealQL similarity queries with the liquids example, use vector:: distance and similarity helpers, and filter KNN results with predicates. Vector search finds records by meaning rather than by the words they contain. This page runs similarity queries over an example dataset, covering the `vector::` distance and similarity helpers and how to filter KNN results with predicates. ## Vector search in SurrealDB What is Vector Search The vector search feature of SurrealDB will help you do more and dig deeper into your data. This can be used in place of, or together with full-text search. For example, still using the same `liquids` table, you can store the chemical composition of the liquid samples in a vector format. ```surql -- Insert a sample & content field into a liquids table INSERT INTO liquidsVector [ { sample:'Sea water', content: 'The sea water contains some amount of lead', embedding: [0.1, 0.2, 0.3, 0.4] }, { sample:'Tap water', content: 'The team lead by Dr. Rose found out that the tap water in was potable', embedding:[1.0, 0.1, 0.4, 0.3] }, { sample:'Sewage water', content: 'High amounts of a were found in Sewage water', embedding : [0.4, 0.3, 0.2, 0.1] } ]; ``` Notice that we have added an `embedding` field to the table. This field will store the vector embeddings of the content field so we can perform vector searches on it. ```surql -- Define a vector index on the liquidsVector table for embedding field DEFINE INDEX mt_pts ON liquidsVector FIELDS embedding HNSW DIMENSION 4 DIST COSINE TYPE F32; -- Insert a sample & content field into a liquids table INSERT INTO liquidsVector [ { sample:'Sea water', content: 'The sea water contains some amount of lead', embedding: [0.1, 0.2, 0.3, 0.4] }, { sample:'Tap water', content: 'The team lead by Dr. Rose found out that the tap water in was potable', embedding:[1.0, 0.1, 0.4, 0.3] }, { sample:'Sewage water', content: 'High amounts of a were found in Sewage water', embedding : [0.4, 0.3, 0.2, 0.1] } ]; -- Add embeddings for what lead as a harmful substance should be. LET $lead_harmful = [0.15, 0.25, 0.35, 0.45]; -- Select the sample and content from the liquids table with cosine similarity SELECT sample, content, vector::similarity::cosine(embedding, $lead_harmful) AS dist FROM liquidsVector WHERE embedding <|2,COSINE|> $lead_harmful; ``` In the example above you can see that the results are more accurate. The search pulled up only the results in which the word "lead" was used to mean the material, while the final `liquidsVector` record had the lowest score. This is the advantage of using vector search over full-text search. Another use case for vector search is in the field of facial recognition. For example, if you wanted to search for an actor or actress who looked like you from an extensive dataset of movie artists, you would first use an LLM model to convert the artist's images and details into vector embeddings and then use SurrealQL to find the artist with the most resemblance to your face vector embeddings. The more characteristics you decide to include in your vector embeddings, the higher the dimensionality of your vector will be, potentially improving the accuracy of the matches but also increasing the complexity of the vector search. ## Computation on vectors: "vector::" package of functions SurrealDB provides [vector functions](/docs/reference/query-language/functions/database-functions/vector.md) for most of the major numerical computations done on vectors. They include functions for element-wise addition, division and even normalisation. They also include similarity and distance functions, which help in understanding how similar or dissimilar two vectors are. Usually, the vector with the smallest distance or the largest cosine similarity value (closest to 1) is deemed the most similar to the item you are trying to search for. Vector functions available in SurrealDB The choice of distance or similarity function depends on the nature of your data and the specific requirements of your application. In the liquids examples, we assumed that the embeddings represented the harmfulness of lead (as a substance). We used the [`vector::similarity::cosine`](/docs/reference/query-language/functions/database-functions/vector.md#vectorsimilaritycosine) function because cosine similarity is typically preferred when absolute distances are less important, but proportions and direction matter more. ## Filtering through vector search The [`vector::distance::knn()`](/docs/reference/query-language/functions/database-functions/vector.md#vectordistanceknn) function from SurrealDB returns the distance computed between vectors by the KNN operator. This operator can be used to avoid recomputation of the distance in every `select` query. Consider a scenario where you’re searching for actors who look like you but they should have won an Oscar. You set a flag, which is true for actors who’ve won the golden trophy. Let’s create a dataset of actors and define an approximate vector index on the embeddings field. This walkthrough uses **HNSW**; from SurrealDB 3.1 you can instead use **DISKANN** when your vectors no longer fit comfortably in memory. See the page on [vector indexes](/docs/learn/data-models/vector-search/vector-indexes.md) for trade-offs and supported `TYPE` / `DIST` combinations. ```surql -- Create a dataset of actors with embeddings and flags CREATE actor:1 SET name = 'Actor 1', embedding = [0.1, 0.2, 0.3, 0.4], flag = true; CREATE actor:2 SET name = 'Actor 2', embedding = [0.2, 0.1, 0.4, 0.3], flag = false; CREATE actor:3 SET name = 'Actor 3', embedding = [0.4, 0.3, 0.2, 0.1], flag = true; CREATE actor:4 SET name = 'Actor 4', embedding = [0.3, 0.4, 0.1, 0.2], flag = true; -- Define an embedding to represent a face LET $person_embedding = [0.15, 0.25, 0.35, 0.45]; -- Define an HNSW index on the actor table DEFINE INDEX hnsw_pts ON actor FIELDS embedding HNSW DIMENSION 4; -- Select actors who look like you and have won an Oscar SELECT id, flag, vector::distance::knn() AS distance FROM actor WHERE flag = true AND embedding <|2,40|> $person_embedding ORDER BY distance; ``` ```surql [ [ { distance: 0.09999999999999998f, flag: true, id: actor:1 }, { distance: 0.412310562561766f, flag: true, id: actor:4 } ] ]; ``` `actor:1` and `actor:4` have the closest resemblance with your query vector among those who have also won an Oscar. ### How the filter is applied Because `embedding` is indexed, the `flag = true` condition is *not* applied after the nearest neighbours have been gathered. Instead SurrealDB pushes it down *into* the approximate (HNSW or DISKANN) search, so candidates that fail the condition are discarded as the graph is traversed and never occupy one of the `K` result slots. This keeps the result both correct, and efficient: * Correct, because you still receive up to `K` records that match the condition. * Efficient, since the search does not spend its slots on records that would only be filtered out afterwards. ### Confirming the pushed-down filter with `EXPLAIN` _(since v3.1.5)_ The [`EXPLAIN`](/docs/reference/query-language/statements/explain.md) clause can be used in the same query to see the query plan. The pushed-down condition appears as a `predicate` attribute on the `KnnScan` operator: ```surql EXPLAIN SELECT id, flag, vector::distance::knn() AS distance FROM actor WHERE flag = true AND embedding <|2,40|> $person_embedding ORDER BY distance; ``` ```surql title="Output" 'SelectProject [ctx: Db] [projections: id, flag, distance] SortByKey [ctx: Db] [sort_keys: distance ASC] Compute [ctx: Db] [fields: distance = vector::distance::knn(...)] Filter [ctx: Db] [predicate: flag = true] KnnScan [ctx: Db] [index: hnsw_pts, k: 2, ef: 40, dimension: 4, predicate: flag = true]' ``` The `predicate: flag = true` on the `KnnScan` line is the condition being evaluated inside the index search. If the attribute is missing, the condition is not being pushed down (for example, when the vector field is not indexed). A DISKANN index produces an identical `KnnScan` line. For HNSW and DISKANN configuration and the KNN cheat sheet, see [Vector indexes](/docs/learn/data-models/vector-search/vector-indexes.md). --- Source: https://surrealdb.com/docs/learn/data-models/vector-search/vector-indexes # Vector indexes Choose brute force, HNSW, or DISKANN vector indexes, tune DIMENSION and distance metrics, and use the query cheat sheet with vector::distance::knn(). When it comes to search, you can always use brute force. In SurrealDB, you can use the [brute force approach](/docs/reference/query-language/statements/define/indexes.md#brute-force-method) to search through your vector embeddings and data. Brute force search compares a query vector against all vectors in the dataset to find the closest match. As this is a brute-force approach, you do not create an index for this approach. The brute force approach for finding the nearest neighbour is generally preferred in the following use cases: - Small datasets / limited query vectors: For applications with small datasets, the overhead of building and maintaining an index might outweigh its benefits. In such cases, the brute force approach is optimal. - Guaranteed accuracy: Since the brute force method compares the query vector against every vector in the dataset, it guarantees finding the exact nearest vectors based on the chosen distance metric (like Euclidean, Manhattan, etc.). - Benchmarking models: The brute force approach can be used as a reference to help benchmark the performance of other approximate alternatives like HNSW or DISKANN. While brute force can give you exact results, it's computationally expensive for large datasets. In most cases, you do not need a 100% exact match, and you can give it up for faster, high-dimensional searches to find the approximate nearest neighbour to a query vector. This is where vector indexes come in. ## HNSW and DISKANN SurrealDB offers two **approximate** graph indexes for k-nearest-neighbour search: | Index | Best for | Storage | | --- | --- | --- | | [HNSW](https://en.wikipedia.org/wiki/Hierarchical_navigable_small_world) ([`DEFINE INDEX … HNSW`](/docs/reference/query-language/statements/define/indexes.md#hnsw-hierarchical-navigable-small-world)) | Low-latency ANN when the graph fits comfortably in memory with headroom for the bounded vector cache | In-memory hot graph + persistence | | [DISKANN](https://arxiv.org/abs/1907.01668) ([`DEFINE INDEX … DISKANN`](/docs/reference/query-language/statements/define/indexes.md#diskann-disk-based-approximate-nearest-neighbours)) *(SurrealDB 3.1+)* | Very large corpora where RAM cannot hold the full graph - full-precision vectors and graph stay in the key-value store and are paged through a bounded cache | Key-value-backed graph + bounded in-memory cache (not on **WASM** targets) | Both are [proximity graph](/docs/reference/query-language/statements/define/indexes.md#hnsw-hierarchical-navigable-small-world)-style indexes. Queries use the same [`<|K, …|>` KNN operator shapes](/docs/reference/query-language/language-primitives/operators.md#knn); the optimiser picks the index when distances and types line up. ## Vector search cheat sheet - **HNSW** - efficient in-memory approximation for high dimensions or large in-RAM datasets. - **DISKANN** - disk-oriented approximation for embeddings that exceed practical memory for a pure HNSW graph. - **Brute force** - when you do not define an index, when you want exact nearest neighbours, or when you pass an explicit distance function to the query that does not route to your index. ### HNSW index | Parameter | Default | Options | Description | | ------------- | --------- | ----------------------- | ------------- | | DIMENSION | | | Size of the vector | DIST | EUCLIDEAN | EUCLIDEAN, COSINE, MANHATTAN | Distance function | TYPE | F32 | F64, F32, I64, I32, I16 | Vector type | EFC | 150 | | EF construction | M | 12 | | Max connections per element | M0 | 24 | | Max connections in the lowest layer | LM | 0.40242960438184466f | | Multiplier for level generation. This value is automatically calculated with a value considered as optimal. Examples: ```surql -- User statement: DEFINE INDEX hnsw_idx ON pts FIELDS point HNSW DIMENSION 4; -- Defaults to: DEFINE INDEX hnsw_idx ON pts FIELDS point HNSW DIMENSION 4 DIST EUCLIDEAN TYPE F32 EFC 150 M 12 M0 24 LM 0.40242960438184466f; -- Users are strongly suggested not to set an LM value, as -- it is computed based on other parameters. Only users -- completely versed in the field should manually set it ``` For more details, see the [`DEFINE INDEX` statement](/docs/reference/query-language/statements/define/indexes.md#hnsw-hierarchical-navigable-small-world) documentation. ### DISKANN index _(since v3.1.0)_ | Parameter | Default | Options | Description | | ---------- | --------- | ------- | ----------- | | DIMENSION | | | Vector dimension | | DIST | EUCLIDEAN | EUCLIDEAN, COSINE, INNER_PRODUCT, COSINE_NORMALIZED | Distance (narrower set than HNSW) | | TYPE | F32 | F32, F16, I8, U8 | Element encoding (`COSINE_NORMALIZED` requires `F32` or `F16`) | | DEGREE | 64 | > 0 | Target maximum graph degree | | L_BUILD | 100 | > 0 | Construction search-list size | | ALPHA | 1.2 | | DiskANN pruning parameter | | HASHED_VECTOR | off | | Optional hash-stabilised vector keys | ```surql DEFINE INDEX diskann_idx ON pts FIELDS point DISKANN DIMENSION 4 DIST COSINE TYPE F32; ``` DISKANN keeps its graph and full-precision vectors in the key-value store and answers queries through a bounded in-memory cache, so resident memory is bounded by the cache rather than by the dataset. (DISKANN does not apply product quantisation; per-vector size is governed only by the chosen `TYPE`.) The cache defaults to 256 MiB and is shared across all DISKANN indexes; cap it with [`SURREAL_DISKANN_CACHE_SIZE`](/docs/reference/cli/surrealdb-cli/environment-variables.md#cache-config). See [`DEFINE INDEX` → DISKANN](/docs/reference/query-language/statements/define/indexes.md#diskann-disk-based-approximate-nearest-neighbours) for the full memory model and platform notes (including **no WASM support**). ### Querying ```surql DEFINE INDEX hnsw_idx ON pts FIELDS point HNSW DIMENSION 4; LET $vector = [2,3,4]; SELECT id, vector::distance::knn() as dist -- distance from $vector -- knn reuses the value computed during -- the query, in this case the euclidean -- distance FROM pts WHERE point <|2|> -- return 2, in this case using the distance function defined in the -- index: euclidean $vector; ``` With a DISKANN index defined on `point`, the same `<|K, EF|>` approximate form applies; the second number bounds the dynamic candidate list for search (see the [KNN operator](/docs/reference/query-language/language-primitives/operators.md#knn)). | Functions | | | ------------------------------------------------ | --- | | `vector::distance::knn()` | reuses the value computed during the query | `vector::distance::chebyshev(point, $vec)` | | `vector::distance::euclidean(point, $vec)` | | `vector::distance::hamming(point, $vec)` | | `vector::distance::manhattan(point, $vec)` | | `vector::distance::minkowski(point, $vec, 3)` | third param is [𝑝](#notes) | `vector::distance::mahalanobis(point, $vec, $cov)` | third param is the covariance matrix, not an index distance | `vector::similarity::cosine(point, $vec)` | | `vector::similarity::jaccard(point, $vec)` | | `vector::similarity::pearson(point, $vec)` | | `vector::similarity::spearman(point, $vec)` | **WHERE statement**
Query HNSW index DISKANN index
<|2|> uses distance function defined in index same when the index distance matches
<|2, EUCLIDEAN|> brute force method brute force method
<|2, COSINE|> brute force method brute force method
<|2, MANHATTAN|> brute force method brute force method
<|2, MINKOWSKI, 3|> brute force method (third param is 𝑝) brute force method
<|2, CHEBYSHEV|> brute force method brute force method
<|2, HAMMING|> brute force method brute force method
<|2, 10|> second param is effort* same approximate form - second value bounds the candidate list
\* **effort** - for HNSW and DISKANN, the second number in `<|K, N|>` tells the engine how far to search along the graph. Both algorithms are approximate and may miss some vectors. ### Notes - Verify index utilisation in queries using the [`EXPLAIN FULL` clause](/docs/reference/query-language/statements/select.md#the-explain-clause). E.g: `SELECT id FROM pts WHERE point <|10|> [2,3,4,5] EXPLAIN FULL;` - 𝑝 values: (more about 𝑝 in [Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance)) - 20 = 1 → manhattan/diamond ◇ - 21 = 2 → euclidean/circle ○ - 22 = 4 → squircle ▢ - 2∞ = ∞ → square □ --- Source: https://surrealdb.com/docs/learn/extensions # Extensions Extend SurrealDB with custom modules and WASM plugins. Done through the Surrealism extension system. _(since v3.0.0)_ >[!NOTE] > Surrealism plugins remain an experimental functionality under active development. This guide assumes the latest version of SurrealDB is being used (3.1.0), which contains [many additions](https://github.com/surrealdb/surrealdb/pull/7082) to Surrealism's core functionality. We look forward to feedback on the plugin system, either via raising an issue or PR on the [SurrealDB repo](https://github.com/surrealdb/surrealdb), or on our [Discord server](https://discord.gg/surrealdb) - the `#surrealism` channel is a good place to start. SurrealDB's extension system, known as Surrealism, lets you add custom functionality to the database engine without modifying its core. Extensions are compiled to WebAssembly (WASM) and loaded at runtime, which means they run in a sandboxed environment with predictable performance characteristics. This section covers: - [Plugins](/docs/learn/extensions/plugins/overview.md) - an overview of the plugin architecture, how plugins are discovered and loaded, and the types of extensions you can create. - [Guides](/docs/learn/extensions/guides/creating-custom-modules.md) - step-by-step walkthroughs for creating custom modules, understanding the module architecture, and working with WASM plugins. - [Attribute reference](/docs/learn/extensions/guides/surrealism-attribute-reference.md) - options for the `#[surrealism]` attribute used to mark functions as Surrealism functions, such as `writeable`, `comment`, `init`, and namespaced exports. ## Guides - [Using WASM plugins](/docs/learn/extensions/guides/using-wasm-plugins.md) - load and call a compiled plugin --- Source: https://surrealdb.com/docs/learn/extensions/guides/creating-custom-modules # Creating custom modules Step-by-step guide to scaffolding a Surrealism module, exposing functions with attributes, compiling and loading into SurrealDB. This page walks you through the options available when building a [Surrealism](/docs/learn/extensions/plugins/overview.md) module from scratch. ## Prerequisites You need a recent [Rust toolchain](https://www.rust-lang.org/tools/install), the `wasm32-wasip2` target, and the SurrealDB CLI (`surreal`) on your `PATH`. ```sh rustup target add wasm32-wasip2 ``` Install SurrealDB from the [installation guide](/docs/running/installation.md) if you have not already. ## Scaffold a module project The fastest way to start is `surreal module init`: ```sh surreal module init ``` This creates the project scaffold (`Cargo.toml`, `surrealism.toml`, `.cargo/Config.toml` with flags needed for the WASI build, and `src/lib.rs`) ready for Surrealism builds. ```toml title=".cargo/Config.toml flags" [build] rustflags = ["--cfg", "tokio_unstable"] ``` To automatically set the name and organisation for a project, you can use the following flags: ```sh surreal module init --headless --org surrealdb --name my_surreal_module ./my_surreal_module ``` ## Create a Rust project manually Create a new library crate for your module: ```sh cargo new --lib my_surreal_module cd my_surreal_module ``` Configure the library as a dynamic WASM library and add the Surrealism SDK plus any other crates you need to `Cargo.toml`, following versions recommended for your SurrealDB release: ```toml [lib] crate-type = ["cdylib"] ``` For a full annotated example, follow the [quick tutorial](/docs/learn/extensions/plugins/quick-tutorial.md). ## Configure `surrealism.toml` Add a `surrealism.toml` file at the crate root next to `Cargo.toml`. A minimal example: ```toml [package] organisation = "surrealdb" name = "demo" version = "1.0.0" [capabilities] allow_scripting = true allow_arbitrary_queries = true allow_functions = ["fn::test"] allow_net = ["127.0.0.1:8080"] ``` Optional attached filesystem: ```toml [attach] fs = "fs" ``` When `[attach] fs = "fs"` is present, the `fs/` folder in your project is packed into the module archive as a read-only filesystem available to the module at runtime. ## Annotate exported functions Expose functions to SurrealQL by annotating them with `#[surrealism]`. Only functions you mark this way are included in the module and callable from the database. ```rust use surrealism::surrealism; #[surrealism] fn can_drive(age: i64) -> bool { age >= 18 } #[surrealism(writeable, comment = "Creates a value in module KV")] fn kv_set_value(key: String, value: String) -> bool { // implementation omitted true } ``` For the full attribute reference (`writeable`, `comment`, `init`, and namespaced modules), see [Surrealism attribute reference](/docs/learn/extensions/guides/surrealism-attribute-reference.md). ## Compile with `surreal module` From your project directory, compile to a `.surli` artefact with the Surreal CLI: ```sh surreal module build --out demo.surli . ``` For faster local iteration, use debug builds. This works in the same way as `cargo build --debug` in skipping optimisation passes to speed up compiling time in exchange for lower performance. ```sh surreal module build --debug --out demo.surli . ``` See the [module command reference](/docs/reference/cli/surrealdb-cli/commands/module.md) for all flags. ## Load into SurrealDB Upload the `.surli` file with [`DEFINE BUCKET`](/docs/reference/query-language/statements/define/bucket.md) so SurrealDB can store it, then register exported functions with [`DEFINE MODULE`](/docs/reference/query-language/statements/define/module.md). The exact names and paths must match your bucket and module identifiers. ## Test your functions Connect with the CLI, [SurrealDB Studio](/docs/explore/studio.md), or an SDK and invoke your functions from SurrealQL. Confirm return values and error handling match what you expect. If something fails: - verify that experimental capabilities include `surrealism` (and `files` if using buckets/files); - verify the module was rebuilt after source changes; - verify your SurrealDB build supports the Surrealism version your module targets. For a high-level picture of how these steps fit together, see the [Surrealism overview](/docs/learn/extensions/plugins/overview.md). --- Source: https://surrealdb.com/docs/learn/extensions/guides/module-architecture # Module architecture How Surrealism modules are packaged, sandboxed, and invoked by SurrealDB, including exports metadata, runtime limits, and load strategy. [Surrealism](/docs/learn/extensions/plugins/overview.md) modules are Rust crates packaged as `.surli` archives. Each archive contains your compiled component plus metadata about exported `#[surrealism]` functions. SurrealDB reads this metadata to map SurrealQL calls to module exports without recompiling the database. ## Component target and sandbox Surrealism builds target WASI Preview 2 (`wasm32-wasip2`) and run through the WASM component model runtime. Your code does not execute as a native library on the host. The runtime boundary provides sandboxing where memory, execution time, filesystem access, and host capabilities can be constrained by server policy. ## Interaction with the engine The module does not link directly into SurrealDB’s core. Instead, the engine loads it in the Surrealism runtime, resolves exports, and marshals values between SurrealQL and your Rust functions through the Surrealism ABI. You work with SDK-exposed types and host imports, not arbitrary process memory. At build time, Surrealism also extracts an exports manifest (names, argument types, return type, `writeable` flag, optional comment) and stores it in the archive. That allows metadata lookups without instantiating the module for every query-planning or introspection operation. ## Runtime execution model Surrealism uses pooled module controllers so invocations can reuse warm execution contexts instead of rebuilding runtime state for each call. - **Pool ceiling:** per-module controller pool size is capped by `SURREAL_SURREALISM_MAX_POOL_SIZE`. - **Timeout mode:** strict timeout behaviour can be configured by module/runtime settings. - **Resource ceilings:** memory, execution time, and module KV limits can be enforced with Surrealism environment variables. For a complete list of server-level variables, see [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md#surrealism-config). ## Module loading strategy By default, SurrealDB eagerly compiles defined Surrealism modules at startup to reduce first-call latency. If you prefer lazy loading, start the server with: ```bash surreal start --lazy-surrealism ``` Or set: **Bash** ```bash SURREAL_LAZY_SURREALISM=true surreal start ``` **PowerShell** ```powershell $env:SURREAL_LAZY_SURREALISM = "true" surreal start ``` ## Lifecycle The end-to-end lifecycle is: **build** your crate with [`surreal module`](/docs/reference/cli/surrealdb-cli/commands/module.md), **upload** the `.surli` archive (typically via [`DEFINE BUCKET`](/docs/reference/query-language/statements/define/bucket.md)), **define** the module ([`DEFINE MODULE`](/docs/reference/query-language/statements/define/module.md)), then **invoke** functions from queries. Operational teams often automate upload and definition in CI so test and production databases stay aligned. The [overview](/docs/learn/extensions/plugins/overview.md) summarises this flow. ## Security Treat every module as privileged code that can affect data your database can access. Sandboxing limits host access, but a module can still implement logic that leaks or corrupts data if you expose it to the wrong scopes. Load only WASM you trust, pin versions, and follow least privilege for namespaces, databases, and credentials that call into Surrealism functions. --- Source: https://surrealdb.com/docs/learn/extensions/guides/surrealism-attribute-reference # Surrealism attribute reference Reference for the #[surrealism] attribute options, including writeable functions, comments, init hooks, and namespaced exports. The `#[surrealism]` attribute controls which Rust items are exported from your module and how SurrealDB should treat them. ## Basic export Use `#[surrealism]` on a function to expose it to SurrealQL: ```rust use surrealism::surrealism; #[surrealism] fn can_drive(age: i64) -> bool { age >= 18 } ``` After packaging and `DEFINE MODULE`, this function is callable through your module path. ## `writeable` Mark functions that perform writes with `writeable`: ```rust #[surrealism(writeable)] fn kv_set_value(key: String, value: String) -> bool { // implementation snipped true } ``` This metadata is used by the query planner so read-only and write-capable functions can be scheduled with appropriate transaction modes. ## `comment = "..."` Attach human-readable metadata to an export: ```rust #[surrealism(comment = "Parses a decimal string into an integer")] fn parse_number(input: String) -> Result { input.parse::().map_err(|e| e.to_string()) } ``` The comment appears in module metadata outputs such as: - `surreal module info` - `INFO FOR DB STRUCTURE` module export metadata ## `init` Register one initialisation hook that runs once after module instantiation: ```rust #[surrealism(init)] fn warm_cache() { // implementation omitted } ``` Use this for one-time setup such as preloading attached files or priming in-module caches. ## Namespaced modules Apply `#[surrealism]` to modules to group exports under namespaces: ```rust #[surrealism] mod math { use surrealism::surrealism; #[surrealism] pub fn add(a: i64, b: i64) -> i64 { a + b } #[surrealism] pub mod util { use surrealism::surrealism; #[surrealism] pub fn negate(value: i64) -> i64 { -value } } } ``` Exports are prefixed by namespace, for example: - `math::add` - `math::util::negate` ## Related docs - [Creating custom modules](/docs/learn/extensions/guides/creating-custom-modules.md) - [Module architecture](/docs/learn/extensions/guides/module-architecture.md) - [Module command reference](/docs/reference/cli/surrealdb-cli/commands/module.md) --- Source: https://surrealdb.com/docs/learn/extensions/guides/using-wasm-plugins # Using WASM plugins How to load a pre-built Surrealism module archive into SurrealDB, register it, inspect exports, and call functions from SurrealQL. You do not have to author Rust yourself to benefit from Surrealism. Many teams consume a **pre-built `.surli` archive** produced by another team or pipeline. Authoring a module means writing Rust, configuring `surrealism.toml`, and running [`surreal module`](/docs/reference/cli/surrealdb-cli/commands/module.md) yourself. This page focuses on the consumer workflow. ## Storing the module archive SurrealDB needs access to the `.surli` binary. Use [`DEFINE BUCKET`](/docs/reference/query-language/statements/define/bucket.md) to create or configure bucket storage, then upload the `.surli` file according to your environment’s path and permissions model. ## Registering the module Once the binary is stored, [`DEFINE MODULE`](/docs/reference/query-language/statements/define/module.md) associates that artefact with a module name and exposes the compiled functions to SurrealQL. The module definition must line up with the exported function names and signatures from the build. ## Inspecting exports before use If you receive a module from another team, inspect it locally before upload: ```bash surreal module info demo.surli surreal module sig --fnc can_drive demo.surli ``` These commands let you verify function signatures and metadata (such as writeable/comment annotations) before registration. ## Calling functions from SurrealQL After the module is defined, call its exported functions by qualifying the function with the registered module path. Consult your module’s documentation for exact names and parameters. If you upgrade the module archive, repeat upload and module definition steps (or your deployment automation) so the running instance picks up the new build. ## Further reading - [Surrealism overview](/docs/learn/extensions/plugins/overview.md) - how compilation, buckets, and modules fit together. - [`DEFINE BUCKET` reference](/docs/reference/query-language/statements/define/bucket.md) - file storage for WASM. - [`DEFINE MODULE` reference](/docs/reference/query-language/statements/define/module.md) - registering Surrealism functions. --- Source: https://surrealdb.com/docs/learn/extensions/plugins/further-examples # Further examples Further examples of Surrealism attributes, namespaced exports, and CLI commands More examples of Surrealism in practice can be seen inside the [surrealism repo](https://github.com/surrealdb/surrealdb/tree/main/surrealism). ## Surrealism Rust code demo [This demo](https://github.com/surrealdb/surrealdb/tree/main/surrealism/demo) contains examples of the `#[surrealism]` attribute in real code. ### Renaming an export You can rename a function on the SurrealDB side: ```rust #[surrealism(name = "other")] fn can_drive_bla(age: i64) -> bool { age >= 18 } ``` ### Setting a default export You can make a function the default, allowing calls via the module path alone: ```rust #[surrealism(default)] fn def(age: i64) -> bool { age >= 18 } ``` ### Attaching export comments ```rust #[surrealism(comment = "Parses a string into a number")] fn parse_number(input: String) -> Result { input.parse::().map_err(|e| e.to_string()) } ``` ## CLI examples [This file](https://github.com/surrealdb/surrealdb/blob/main/surrealism/test/test.sh) shows ways to work with module archives directly from the CLI. ```bash # Scaffold a new module project surreal module init # Build quickly while iterating surreal module build --debug --out demo.surli . # See function signature surreal module sig --fnc can_drive demo.surli # Run function with argument surreal module run --fnc can_drive --arg 17 demo.surli ``` --- Source: https://surrealdb.com/docs/learn/extensions/plugins/overview # Surrealism plugins Surrealism is a plugin feature that allows users to compile Rust functions into modules that can be called from a SurrealDB instance. Surrealism extensions are written in Rust, compiled into `.surli` modules, and loaded into a running SurrealDB instance. This gives you access to the Rust ecosystem while keeping extensions sandboxed within the database. Surrealism was built in order to allow users to extend SurrealDB in ways that benefit them, without needing to make any changes to the code of SurrealDB itself. Some potential use cases are: * Adding functions to create fake / mock data for testing. * Accessing functionality in Rust crates that are too specific to merge into SurrealDB itself, such as [language-specific functionality](https://crates.io/crates/hangeul), [custom case conversions](https://docs.rs/convert_case/latest/convert_case/enum.Case.html), or [quantitative finance](https://docs.rs/crate/RustQuant/latest). * Anything else you have built in your own code that you would like to access inside SurrealDB. ## How Surrealism works Surrealism works by using the following steps: * Use the [`surreal module init`](/docs/reference/cli/surrealdb-cli/commands/module.md) command to begin a new project. This command works in a similar manner to `cargo new` in scaffolding a new project with its own `toml` file (`surrealism.toml`). * Annotate functions to export with `#[surrealism]`. * Use the `surreal module build` command to build the module. * In SurrealDB, [allow access to module files](/docs/reference/query-language/statements/define/bucket.md) and [define a module](/docs/reference/query-language/statements/define/module.md) to access the functions. A flowchart showing the steps involved to turn regular Rust code into a compiled WASM binary that can be accessed from a running SurrealDB instance. ## What is available in Surrealism Current Surrealism releases include: * function-level metadata such as `writeable` and `comment` via `#[surrealism(...)]`; * one-time module initialisation with `#[surrealism(init)]`; * namespaced exports with `#[surrealism] mod ...`; * optional read-only attached filesystem data packaged into the module archive; * server-side resource controls for execution time, memory, and module KV limits. The next pages walk through the workflow and APIs: - [Quick tutorial](/docs/learn/extensions/plugins/quick-tutorial.md) - [Creating custom modules](/docs/learn/extensions/guides/creating-custom-modules.md) - [Surrealism attribute reference](/docs/learn/extensions/guides/surrealism-attribute-reference.md) ## More examples - [Further examples](/docs/learn/extensions/plugins/further-examples.md) - additional plugins to read through --- Source: https://surrealdb.com/docs/learn/extensions/plugins/quick-tutorial # Quick tutorial A quick tutorial showing the steps involved in turning regular Rust code into SurrealDB-accessible WASM functions. This tutorial turns ordinary Rust code into a SurrealDB-accessible WASM module, from `surreal module init` through to calling the function from SurrealQL. ## Getting started: regular Rust code Starting a Surrealism project is best done using the `surreal module init` command, which is similar to `cargo run` when starting a project in Rust. The `Cargo.toml` file will include the following line to instruct cargo to generate a `.wasm` file instead of the standard `.rlib` file when compiling Rust. ```toml [lib] crate-type = [ "cdylib" ] ``` The command will also ask for the project and organisation name, which will lead to the following `surrealism.toml` file. ```toml target = "rust" abi = 2 [package] organisation = "my_organisation" name = "surrealism_fun" version = "1.0.0" [capabilities] allow_scripting = false allow_arbitrary_queries = false allow_functions = [] allow_net = [] strict_timeout = true [attach] ``` You'll also notice a few sample functions inside `lib.rs`. To internalise how Surrealism works, let's delete them and begin with three of our own functions that we would like to call using SurrealQL. One is a simple function that returns a bool. ```rust fn can_drive(age: i64) -> bool { age >= 18 } ``` The second returns a `Result`. ```rust fn parse_number(input: String) -> Result { input.parse::() } ``` And the third returns a `User` struct via a function called `random_user()` which uses the [fake](https://docs.rs/fake/4.4.0/fake/index.html) crate to create a user with a random English first name, French middle name, and German last name. This struct is annotated with the `SurrealValue` trait, allowing its return value to be understood on the SurrealDB side. ```rust #[derive(Debug, SurrealValue)] pub struct User { first_name: String, middle_name: String, last_name: String, age: i32 } pub fn random_user() -> User { User { first_name: FirstName(EN).fake(), middle_name: FirstName(FR_FR).fake(), last_name: LastName(DE_DE).fake(), age: random_range(10..=50) } } ``` Here is all of the code. ```rust use fake::faker::name::raw::*; use fake::{Fake, locales::*}; use rand::random_range; use surrealdb_types::SurrealValue; use std::num::ParseIntError; #[derive(Debug, SurrealValue)] pub struct User { first_name: String, middle_name: String, last_name: String, age: i32 } fn can_drive(age: i64) -> bool { age >= 18 } fn parse_number(input: String) -> Result { input.parse::() } pub fn random_user() -> User { User { first_name: FirstName(EN).fake(), middle_name: FirstName(FR_FR).fake(), last_name: LastName(DE_DE).fake(), age: random_range(10..=50) } } ``` ## Annotating functions The functions that we want to call from inside SurrealQL can now be exposed by adding the `#[surrealism]` annotation. ```rust use fake::faker::name::raw::*; use fake::{Fake, locales::*}; use rand::random_range; use surrealdb_types::SurrealValue; use surrealism::surrealism; use std::num::ParseIntError; #[derive(Debug, SurrealValue)] pub struct User { first_name: String, middle_name: String, last_name: String, age: i32 } #[surrealism] fn can_drive(age: i64) -> bool { age >= 18 } #[surrealism] fn parse_number(input: String) -> Result { input.parse::() } #[surrealism] pub fn random_user() -> User { User { first_name: FirstName(EN).fake(), middle_name: FirstName(FR_FR).fake(), last_name: LastName(DE_DE).fake(), age: random_range(10..=50) } } ``` With this done, the Rust code can now be compiled via the [`module build`](/docs/reference/cli/surrealdb-cli/commands/module.md) command in the CLI. This is followed with the `--out` flag and the output file name/path (e.g. `demo.surli`). If you are in a separate folder, you can follow this with the path to the Rust code with functions to expose. ```bash surreal module build --out demo.surli /Users/my_name/my_rust_code ``` For faster local iteration: ```bash surreal module build --debug --out demo.surli /Users/my_name/my_rust_code ``` The file will now be compiled at either your current location or the one specified like in the command above: `/Users/my_name/my_rust_code`. That's the module archive that SurrealDB can point to! ## Calling the Surrealism file from SurrealDB The Surrealism file is ready to be used. All that is left now is to start a database with two environment variables. They are: * `SURREAL_CAPS_ALLOW_EXPERIMENTAL=files,surrealism`, which allows the [files](/docs/reference/query-language/statements/define/bucket.md) and surrealism features to be used in the first place. Files can be used in SurrealDB once a `DEFINE BUCKET` statement has been used to set up the location for the files to be stored. * `SURREAL_BUCKET_FOLDER_ALLOWLIST="/Users/my_name/my_rust_code/"`, which tells the database that it's okay to access this folder when using the files feature. Now it's time to [start the database](/docs/reference/cli/surrealdb-cli/commands/start.md) with the `surreal start` command and these two env vars. **Bash** ```bash SURREAL_CAPS_ALLOW_EXPERIMENTAL=files,surrealism SURREAL_BUCKET_FOLDER_ALLOWLIST="/Users/my_name/my_rust_code/" surreal start --user root --pass secret ``` **PowerShell** ```powershell $env:SURREAL_CAPS_ALLOW_EXPERIMENTAL = "files,surrealism" $env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "C:\Users\my_name\my_rust_code" surreal start --user root --pass secret ``` You can then connect through [SurrealDB Studio](/docs/explore/studio.md) or the CLI with the [surreal sql](/docs/reference/cli/surrealdb-cli/commands/sql.md) command: **Bash** ```bash SURREAL_CAPS_ALLOW_EXPERIMENTAL=files,surrealism SURREAL_BUCKET_FOLDER_ALLOWLIST="/Users/my_name/my_rust_code/" surreal sql --user root --pass secret ``` **PowerShell** ```powershell $env:SURREAL_CAPS_ALLOW_EXPERIMENTAL = "files,surrealism" $env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "C:\Users\my_name\my_rust_code" surreal sql --user root --pass secret ``` We're almost there! Only two statements left and we can start accessing these functions. The first is a [DEFINE BUCKET](/docs/reference/query-language/statements/define/bucket.md) statement to create a bucket called `test`, linked to the folder that we have allowed the database to access. The "file:/" prefix in this statement tells the `DEFINE BUCKET` statement that this path is to a file, not storage in memory. ```surql DEFINE BUCKET test BACKEND "file:/users/my_name/my_rust_code"; ``` Now that we have a bucket called `test`, the `demo.surli` file will be available to us at the f"test:/demo.surli" path. The `f` there is an instruction to treat the string as a file path instead of just a regular string. Then we can [define a module](/docs/reference/query-language/statements/define/module.md) which will hold all the tests. We'll call it `mod::test` and connect it to the file. The keyword that does the connecting changed in 3.3, so pick the tab that matches your version. **Before SurrealDB 3.3** `AS` connects the module name to the file. ```surql DEFINE MODULE mod::test AS f"test:/demo.surli"; ``` **SurrealDB 3.3 and later** `FROM` replaces `AS`, and `UNSIGNED` is required because only modules published to Silo carry a signature. ```surql DEFINE MODULE mod::test FROM f"test:/demo.surli" UNSIGNED; ``` And now the magic begins! Let's give the `parse_number()` function a try, now available at the `mod::test::parse_number` path. ```surql mod::test::parse_number("10"); //- 10 mod::test::parse_number("Hi I'm number"); //- 'Thrown error: WASM function returned error: invalid digit found in string' ``` Next, we can use `random_user()` to create some random users. ```surql CREATE user CONTENT mod::test::random_user(); //- [{ age: 18, first_name: 'Thomas', id: user:hr8ohmn36zrpv3zthhnf, last_name: 'Meier', middle_name: 'Ninon' }] CREATE user CONTENT mod::test::random_user(); //- [{ age: 13, first_name: 'Verda', id: user:zg0ucdjizdp71fzq9syc, last_name: 'Schuster', middle_name: 'Clarisse' }] CREATE user CONTENT mod::test::random_user(); //- [{ age: 45, first_name: 'Zelda', id: user:rulp2bf82twrh94ifhsh, last_name: 'Berger', middle_name: 'Noah' }] ``` That leaves us with one function left to try out, the `can_drive()` function. ```surql SELECT first_name + ' ' + middle_name + ' ' + last_name AS name, mod::test::can_drive(age) AS can_drive FROM user; ``` Two of them can drive, but not Verda Clarisse Schuster who is far too young. ```surql [ { can_drive: true, name: 'Thomas Ninon Meier' }, { can_drive: true, name: 'Zelda Noah Berger' }, { can_drive: false, name: 'Verda Clarisse Schuster' } ] ``` --- Source: https://surrealdb.com/docs/learn/querying # Querying SurrealQL, the SDKs and GraphQL: SQL-like syntax for SurrealDB. Graphs, links, and practical querying tips. In this section we will learn about the multitude of ways to write and send queries to your SurrealDB database. The main focus is on the SurrealQL query language, which strongly resembles traditional SQL but differs in a number of ways to accommodate patterns such as graph traversal, record links, and other features unique to SurrealDB. Queries can also be written without using direct SurrealQL through methods such as SDKs in your preferred programming language, [GraphQL](/docs/learn/querying/graphql/overview.md), or [GQL](/docs/learn/querying/gql/overview.md) (ISO graph pattern queries on the `/gql` endpoint). The [Postgres wire protocol](/docs/reference/rest-api/postgres-protocol.md) lets standard Postgres clients (`psql`, JDBC, etc.) connect and run SurrealQL or GQL with tabular results. Built-in [`eval::*`](/docs/reference/query-language/functions/database-functions/eval.md) functions can also run SurrealQL or GQL query strings from inside a transaction when explicitly enabled. This section also includes tips and tricks to get the most out of your queries. For even more information after reading this section, feel free to look into the [reference](/docs/reference/query-language.md) section of the documentation which has separate pages for each statement, data type, clause and more for the entire SurrealQL query language. ## Concepts and guides - [Parameterised queries](/docs/learn/querying/concepts-and-guides/parameterised-queries.md) - pass values as parameters instead of building query strings - [Working with types](/docs/learn/querying/concepts-and-guides/working-with-types.md) - how values are typed, cast and asserted - [Custom functions](/docs/learn/querying/concepts-and-guides/custom-functions.md) - name and reuse a query with `DEFINE FUNCTION` - [Subqueries and advanced patterns](/docs/learn/querying/concepts-and-guides/subqueries-and-advanced-patterns.md) - nest queries and compose results - [Transactions](/docs/learn/querying/concepts-and-guides/transactions.md) - group statements so they succeed or fail together - [Idempotent operations](/docs/learn/querying/concepts-and-guides/idempotent-operations.md) - write statements that are safe to retry - [Error handling](/docs/learn/querying/concepts-and-guides/error-handling.md) - what a failed statement returns, and how to react to it - [Sequences](/docs/learn/querying/concepts-and-guides/sequences.md) - generate monotonic numbers without a race - [Sessions and scoping](/docs/learn/querying/concepts-and-guides/sessions-and-scoping.md) - what a session carries, and how long it lasts - [Query optimisation](/docs/learn/querying/concepts-and-guides/query-optimisation.md) - read a query plan and index for it - [Bulk operations and data import](/docs/learn/querying/concepts-and-guides/bulk-operations-and-data-import.md) - load many records efficiently - [Connecting from serverless & edge](/docs/learn/querying/concepts-and-guides/connecting-from-serverless-and-edge.md) - connection patterns where processes are short-lived - [Representations and codecs](/docs/learn/querying/concepts-and-guides/representations-and-codecs.md) - how values are encoded over each protocol - [Testing](/docs/learn/querying/concepts-and-guides/testing.md) - assert query results as part of a test suite ## Performance - [Performance best practices](/docs/learn/querying/performance/performance-best-practices.md) - what to measure, and what usually costs the most ## Custom APIs - [Middleware](/docs/learn/querying/custom-apis/middleware.md) - run code before and after a custom API handler --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/bulk-operations-and-data-import # Bulk operations and data import Bulk INSERT patterns in SurrealQL, plus Surreal Sync, HTTP import, /sql for side effects, and SurrealDB Studio CSV. A bulk operation can either be one involving many records in one SurrealQL statement, or loading a file or stream from outside the database. ## In-query bulk patterns An [`INSERT`](/docs/reference/query-language/statements/insert.md) statement is most commonly used for bulk operations. An `INSERT` can even use the output from another query as its input values: ```surql INSERT INTO archive (SELECT * FROM readings WHERE city = 'London'); ``` An `INSERT` can use an array of objects or classic SQL tuple syntax. ```surql INSERT INTO person [ { id: "jaime", name: "Jaime" }, { id: "tobie", name: "Tobie" }, ]; INSERT INTO company (name, founded) VALUES ('Acme Inc.', '1967-05-03'), ('SurrealDB', '2021-09-10'); ``` The `ON DUPLICATE KEY UPDATE` clause can be used to add extra logic when a unique key clashes instead of failing the operation. ```surql INSERT INTO city (id, population, at_year) VALUES ("Calgary", 1665000, 2024) ON DUPLICATE KEY UPDATE population = $input.population, at_year = $input.at_year; ``` ## Tools and HTTP endpoints Importing data from external sources can be done through a number of methods. - **[Surreal Sync](https://github.com/surrealdb/surreal-sync/)** helps migrate from other databases and streams into SurrealDB. See the [migrations overview](/docs/build/migrating/from-other-databases/overview.md) for how it fits with the rest of the import story. - **`POST /import`** is the HTTP endpoint for importing SurrealQL at volume. Imports must include an `OPTION IMPORT` line which instructs the server to skip events, live queries, query output and so on. For details, see [`POST /import`](/docs/reference/rest-api/http-protocol.md#import). - **`POST /sql`** runs SurrealQL like a normal query session. Use this endpoint when you **want side effects** during a load such as events and live query behaviour. The [HTTP API](/docs/reference/rest-api/http-protocol.md#sql) documents the `/sql` endpoint; the [CLI `import`](/docs/reference/cli/surrealdb-cli/commands/import.md) page describes the same trade-off for file-based imports. - **SurrealDB Studio** can import SurrealQL files and **CSV** from the Explorer view (choose fields, map to a table, and create records). See [exploring database records / import](/docs/explore/studio.md). For large one-off file loads from the shell, the **[`surreal import`](/docs/reference/cli/surrealdb-cli/commands/import.md)** command is the usual companion to `POST /import`. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/connecting-from-serverless-and-edge # Connecting from serverless & edge Serverless and edge environments change how applications connect to databases. Instead of long-lived processes with persistent connections, serverless and edge code runs in short-lived, stateless executions, often across multiple regions. Some examples of environments in which this pattern is used are AWS Lambda, Vercel Functions / Edge Functions, Cloudflare Workers, and Deno Deploy. SurrealDB works well in these environments, but it's important to use the right connection patterns. ## Serverless vs. traditional The following chart summarises the differences between the two approaches. | Traditional server | Serverless / edge | |------------------|------------------| | Long-lived process | Short-lived execution | | Persistent connections | No guaranteed reuse | | Connection pooling | Not available | | Single region | Often multi-region | | Warm memory | Frequent cold starts | As many serverless and edge environments don't support TCP sockets, connecting over HTTP is generally used. As no persistent connection is required, this allows you to create a new client, connect, run queries and then exit. ```javascript import { Surreal } from 'surrealdb'; const db = new Surreal(); await db.connect('https://your-instance'); await db.signin({ user: 'your-user', pass: 'your-pass', }); await db.use({ namespace: 'app', database: 'prod', }); const result = await db.query('SELECT * FROM user'); ``` ## Authentication patterns Authentication by password uses [secure salted and hashed passwords](/learn/book/chapter-15#a-bit-of-cryptography) that rely on algorithms that are compute-heavy and purposely take a few hundred milliseconds per attempt. ```javascript // Basic approach: authenticate per request await db.signin({ user: '...', pass: '...' }); ``` If you can store pre-generated tokens can securely in environment variables or platform secrets, then using a token directly may be preferred. ```javascript await db.signin({ token: 'your-token' }); ``` ## Batching queries The flexibility of SurrealQL allows you to batch and optimise queries, performing multiple operations inside a single request instead of multiple requests. Take this example to create and link two records that might be done over three operations, returning the first two results through an SDK to finally perform a `RELATE` operation at the end. ```surql -- Request 1 CREATE person:one; -- Request 2 CREATE person:two; -- Request 3 RELATE person:one->likes->person:two; ``` This can not only be sent as a single request, but even performed over a single query that both creates and relates the records in question. ```surql RELATE (CREATE ONLY person:one RETURN VALUE id) ->likes-> (CREATE ONLY person:two RETURN VALUE id); ``` ## When not to use serverless patterns Serverless and edge connections are not always the best choice. Consider a long-lived backend service if you need: * Live queries or subscriptions * Real-time updates over WebSockets * High-frequency queries from the same process * Persistent connection state In these cases, a traditional backend with a persistent SurrealDB connection will be more efficient. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/custom-functions # Custom functions Learn how to define and call database functions with fn::, optional arguments, and permissions. [`DEFINE FUNCTION`](/docs/reference/query-language/statements/define/function.md) registers reusable SurrealQL under a name beginning with the `fn::` prefix. You need appropriate access (root / namespace / database owner or editor), and you must [`USE`](/docs/reference/query-language/statements/use.md) the right namespace and database first. For more clauses and edge cases, see the reference documentation. ## Define and call Custom functions are always named with the `fn::` prefix. ```surql -- Simple function: build a string from an argument DEFINE FUNCTION fn::greet($name: string) { "Hello, " + $name + "!" }; RETURN fn::greet("Tobie"); ``` ```surql -- More than one parameter; body can use LET, subqueries, RETURN, and so on DEFINE FUNCTION fn::relation_exists($in: record, $tb: string, $out: record) { LET $results = SELECT VALUE id FROM type::table($tb) WHERE in = $in AND out = $out; RETURN array::len($results) > 0; }; ``` ## Optional trailing arguments If the last parameters use `option`, callers may omit them. ```surql DEFINE FUNCTION fn::last_option($required: number, $optional: option) { RETURN { required_present: type::is_number($required), optional_present: type::is_number($optional), }; }; RETURN fn::last_option(1, 2); -- both set RETURN fn::last_option(1); -- optional omitted ``` ## Annotated return types Adding `-> type` is recommended for extra type safety, as well as readability. Though the first example can clearly only return a `number` due to its two arguments being of the same type, it is easier for a casual reader to know what sort of value is being returned. ```surql DEFINE FUNCTION fn::combine($one: number, $two: number) -> number { $one + $two }; -- Accept any input; coercion to the return type happens when returning DEFINE FUNCTION fn::combine_any($one: any, $two: any) -> number { $one + $two }; ``` You can also return [literal union types](/docs/reference/query-language/language-primitives/data-types/literals.md) for structured data or a fallback string instead of [throwing](/docs/reference/query-language/statements/throw.md): ```surql DEFINE FUNCTION fn::age_and_name($user_num: int) -> { age: int, name: string } | string { LET $user = type::record("user", $user_num); IF $user.exists() { $user.{ name, age } } ELSE { "Couldn't find user number " + $user_num + "!" } }; ``` ## Recursion Though not a commonly used pattern in SurrealQL, a function may call itself. Below, each step relates the first record in a list to all the rest, then recurses on the tail until fewer than two records remain. ```surql DEFINE FUNCTION fn::relate_all($records: array) { IF $records.len() < 2 { -- stop recursion } ELSE { LET $first = $records[0]; LET $remainder = $records[1..]; FOR $counterpart IN $remainder { RELATE $first->to->$counterpart; }; fn::relate_all($remainder); } }; CREATE |person:1..8|; fn::relate_all(SELECT VALUE id FROM person); SELECT id, ->to->? FROM person; ``` ## Permissions `PERMISSIONS` controls whether [record users](/docs/learn/security/authentication/users.md#record-users) may invoke the function: `FULL` (typical default), `NONE`, or `WHERE` with a boolean expression (often involving `$auth`). ```surql -- PERMISSIONS NONE: record users cannot call this (admins / root still can, depending on setup) DEFINE FUNCTION fn::fetchAllPaymentDetails() -> array { SELECT stored_cards.expiry_year FROM payment_details LIMIT 5 } PERMISSIONS NONE; ``` ```surql -- PERMISSIONS WHERE: only when the expression is true (here, admin flag on $auth) DEFINE FUNCTION fn::fetchAllProducts() -> array { SELECT * FROM product LIMIT 10 } PERMISSIONS WHERE $auth.admin = true; ``` ## Creating or replacing a definition Clauses like `IF NOT EXISTS` and `OVERWRITE' can be used when defining a function. ```surql -- IF NOT EXISTS: only define when no function with this name exists yet (no-op if it does) DEFINE FUNCTION IF NOT EXISTS fn::example() {}; -- OVERWRITE: replace an existing definition with this one DEFINE FUNCTION OVERWRITE fn::example() {}; ``` ## Functions as API middleware Handlers for [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) middleware are ordinary `fn::` functions with a specific signature. See [custom middleware](/docs/reference/query-language/statements/define/api.md#custom-middleware) and the learn page on [middleware](/docs/learn/querying/custom-apis/middleware.md) for more details. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/error-handling # Error handling Handle errors in SurrealQL with THROW, validation guards and ASSERT on fields to stop queries and return clear failures. Error handling in SurrealQL allows you to stop execution and return a meaningful error when something goes wrong. This is most commonly done using the `THROW` statement. ## Throwing an error You can use `THROW` to immediately stop a query and return an error to the client: ```surql THROW "Something went wrong"; ``` When executed, the query is aborted and the error is returned. `THROW` is often used to validate input or guard business logic: ```surql IF !$email { THROW "Email is required"; }; IF $amount <= 0 { THROW "Transfer amount must be greater than zero"; }; ``` `THROW` is especially useful inside manual transactions. ```surql BEGIN TRANSACTION; UPDATE account:one SET balance -= 150; IF account:one.balance < 0 { THROW "Insufficient funds"; }; COMMIT TRANSACTION; ``` ## Returning custom error messages The value passed to THROW is returned to the client. ```surql THROW "Invalid username or password"; ``` You can also include dynamic data: ```surql THROW "User not found: " + $username; ``` Or even structured data. ```surql THROW { code: 400, message: "Invalid request" }; ``` ## Avoiding errors Type safety and strict definitions can be used to avoid throwing errors based on custom logic. For example, the `ASSERT` clause in a [DEFINE FIELD](/docs/reference/query-language/statements/define/field.md) statement can be used to ensure that a statement will fail if the string for the `email` field provided is not a valid email. ```surql DEFINE FIELD email ON TABLE user TYPE string ASSERT string::is_email($value); ``` Working with a schema in this way allows throwing errors to be taken care of by the definitions themselves as opposed to writing custom logic. ## Assertions while debugging When you are iterating on a query in the CLI or SurrealDB Studio, it is not always obvious which step in a long [method chain](/docs/reference/query-language/functions/database-functions/#method-syntax) produced an unexpected value. The [`value::expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) function (_(since v3.1.0)_) checks a condition on the current value and returns that same value when the closure is `true`, or fails the statement with a clear error (and an optional custom message). ```surql CREATE person:one SET name = "Tommy", city = "London"; CREATE person:two SET name = "Billy", city = "London"; LET $records = SELECT * FROM person WHERE city = "London"; $records .expect(|$n| $n.len() > 0, "Expected at least one person in London") .map(|$person| { name: $person.name + " from London" }); ``` ```surql title="Output" [ { name: 'Tommy from London' }, { name: 'Billy from London' } ]; ``` This can be used for temporary invariants while debugging. For permanent rules, prefer `DEFINE FIELD … ASSERT` on the schema. `.expect()` clones the value it receives, so remove it from hot paths once you are finished debugging. ## SDK error handling SurrealDB uses a [single public API error type](https://github.com/surrealdb/surrealdb/blob/main/surrealdb/types/src/error.rs#L37) that is shared by SDKs. As the repo states, the error type is: > Designed to be returned from public APIs (including over the wire). It is wire-friendly and non-lossy: serialisation preserves `kind`, `message`, and optional `details`. Use this type whenever an error crosses an API boundary (e.g. server response, SDK method return). > > The `details` field is flattened into the serialised object, so the wire format contains `kind` (string) and optionally `details` (object) at the same level as `code` and `message`. The optional `cause` field allows error chaining so that SDKs can receive and display full error chains. This error will then be handled in a different manner depending on the programming language the SDK is written for. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/idempotent-operations # Idempotent operations Make SurrealQL idempotent with UPSERT, safe deletes and unique indexes so retries do not corrupt data. Idempotency is a key concept when building reliable applications, especially APIs, event-driven systems, and distributed architectures. In simple terms, an operation is **idempotent** if running it multiple times produces the same result as running it once. Fun fact: *idempotent* comes from the Latin word *idem* which means "same" (while "potent" means to have power). Most readers have seen this word in its abbreviated form at the bottom of academic journals: * United States v. Martinez-Fuerte, 428 U.S. 543, 545 (1976). * *Id.* at 547. ## Why idempotency matters In real-world systems, requests can be retried, messages can be delivered more than once, and network failures can interrupt operations. However, if your database queries are idempotent then you can safely retry them without corrupting data. ## Idempotent patterns in SurrealQL ### 1. Using `UPSERT` `UPSERT` is the most common way to ensure idempotency. ```surql UPSERT user:123 SET name = "Alice", age = 30; ``` The above query will create the record if it does not exist, and update it if it does exist. Running it multiple times will result in the same final state, at least for the fields `name` and `age`. ### 2. Deleting records A `DELETE` statement will remove the record(s) on first execution, while executing the same statement will have no effect. A `DELETE` statement returns an empty array by default. ```surql DELETE user:123; ``` ### 3. Creating relations (`RELATE`) The following statement is not idempotent on its own, as multiple relations can be created between records. ```surql RELATE user:123->likes->post:456; ``` However, it can be made idempotent by first defining a [unique index](/docs/reference/query-language/statements/define/indexes.md#composite-index) on the `in` and `out` fields. Doing so will result in the `RELATE` statement above always resulting in a single graph edge between the two records. ```surql DEFINE INDEX only_one ON likes FIELDS in, out UNIQUE; ``` Alternatively, from version 3.1.5, you can supply an explicit edge record ID in the `RELATE` path (for example `->likes:[person:one, post:one]->`) or in an [`INSERT RELATION`](/docs/reference/query-language/statements/insert.md#insert-relation-tables) statement. A duplicate explicit ID on `INSERT RELATION` returns an error unless you add `ON DUPLICATE KEY UPDATE`. For more, see [Explicit edge record IDs](/docs/reference/query-language/statements/relate.md#handling-duplicate-edge-record-ids). ## Non-idempotent examples The following examples will produce different results on every execution, and thus are not idempotent. ```sql UPDATE user:123 SET login_count += 1; UPDATE user:123 SET tags += "new"; ``` ## Using idempotency to simplify queries Instead of writing application logic like this: ```text if user exists: update else: create ``` You can simplify it with a single idempotent query. ```surql UPSERT user:123 SET name = "Alice", status = "active"; ``` --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/parameterised-queries # Parameterised queries Queries can make use of parameters which hold stored values, even those from the output of a previous query. Parameters are stored values which can then be used in subsequent queries. To define a parameter in SurrealQL, use the [`LET`](/docs/reference/query-language/statements/let.md) statement. Parameter names must begin with a `$` (dollar sign) character. ## Defining parameters within SurrealQL ```surql -- Define the parameter LET $suffix = "Morgan Hitchcock"; -- Use the parameter CREATE person SET name = "Tobie " + $suffix; -- (Another way to do the same) CREATE person SET name = string::join(" ", "Jaime", $suffix); ``` ```surql title="Output" [ { "id": "person:3vs17lb9eso9m7gd8mml", "name": "Tobie Morgan Hitchcock" } ] [ { "id": "person:xh4zbns5mgmywe6bo1pi", "name": "Jaime Morgan Hitchcock" } ] ``` A parameter can store any value, including the result of a query. ```surql -- Assuming the CREATE statements from the previous example LET $founders = SELECT * FROM person; $founders.{ name, company }; ``` ```surql title="Output" [ { company: 'SurrealDB', name: 'Jaime Morgan Hitchcock' }, { company: 'SurrealDB', name: 'Tobie Morgan Hitchcock' } ] ``` ## Scope of parameters Parameters persist across the current connection, and thus can be reused between different namespaces and databases. In the example below, a created `person` record assigned to a parameter is reused in a query in a completely different namespace and database. ```surql LET $billy = CREATE ONLY person:billy SET name = "Billy"; -- Fails as `person:billy` already exists CREATE person CONTENT $billy; USE NAMESPACE other_namespace; USE DATABASE other_database; -- Succeeds as `person:billy` does not yet -- exist in this namespace and database CREATE person CONTENT $billy; ``` Parameters can be defined using SurrealQL as shown above, or can be passed in using the client libraries as request variables. ## Redefining and shadowing parameters Parameters in SurrealQL are immutable. The same parameter can be redefined using a `LET` statement. ```surql LET $my_name = "Alucard"; LET $my_name = "Sypha"; RETURN $my_name; ``` ```surql title="Output" 'Sypha' ``` ## Defining parameters within client libraries SurrealDB's client libraries allow parameters to be passed in as JSON values, which are then converted to SurrealDB data types when the query is run. The following example show a variable being used within a SurrealQL query from the JavaScript library. ```javascript let people = await surreal.query("SELECT * FROM article WHERE status INSIDE $status", { status: ["live", "draft"], }); ``` ## Reserved variable names SurrealDB automatically predefines certain variables depending on the type of operation being performed. For example, `$this` and `$parent` are automatically predefined for subqueries so that the fields of one can be compared to another if necessary. In addition, the predefined variables `$access`, `$auth`, `$token`, and `$session` are protected variables used to give access to parts of the current database configuration and can never be overwritten. ```surql LET $access = true; LET $auth = 10; LET $token = "Mytoken"; LET $session = rand::int(0, 100); ``` ```surql title="Output" -------- Query 1 -------- "'access' is a protected variable and cannot be set" -------- Query 2 -------- "'auth' is a protected variable and cannot be set" -------- Query 3 -------- "'token' is a protected variable and cannot be set" -------- Query 4 -------- "'session' is a protected variable and cannot be set" ``` For a complete list of reserved parameter names, see [this section](/docs/reference/query-language/language-primitives/parameters.md#reserved-variable-names) in the API documentation. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/query-optimisation # Query optimisation Practical ideas for faster SurrealQL: EXPLAIN, record ranges, async events, denormalised flags, and indexes. This page contains a number of tips you can use to optimise your queries in SurrealDB. For more details on each pattern, see the linked pages in the API documentation. ## Inspecting the plan with `EXPLAIN` Prefix a read-only statement with `EXPLAIN` to see how the database plans to run it. Add `ANALYZE` if you want timing and row metrics as well. As the output of this statement is informational and may change between versions, be sure not to build tooling that depends on an exact shape. ```surql EXPLAIN SELECT * FROM person WHERE email = 'user@example.com'; ``` Full syntax and options can be found in the [`EXPLAIN` reference](/docs/reference/query-language/statements/explain.md). ## Record ranges When you can identify records by record ID order (for example numeric or time-ordered IDs), selecting with a range on the ID (`table:start..end`) avoids scanning the whole table. A `WHERE` filter over the same records can be much more expensive because it typically implies a wider scan. ```surql SELECT * FROM person:1..1000; ``` See [record IDs](/docs/reference/query-language/language-primitives/data-types/record-ids.md) and [record ranges in `SELECT`](/docs/reference/query-language/statements/select.md#record-ranges). ## Async events By default, events run in the same transaction as the write that triggers them, which keeps behaviour easy to reason about but can slow commits if event logic is heavy. Using the [`ASYNC` clause in a `DEFINE EVENT` statement](/docs/reference/query-language/statements/define/event.md#async-events) runs the handler **after** the triggering transaction. This leads to lower write latency, with the caveat that it is an opt out of the ACID guarantees by default in all transactions. As such, it should only be used when this tradeoff is acceptable. More context: [Reactive patterns](/docs/learn/schema-management/events-and-triggers/reactive-patterns.md). ## Pre-allocated fields (denormalised flags) If a query repeatedly does a lookup or subquery only to answer a yes/no question (“is this user registered?”), consider storing the answer in a field. For example, an `is_registered` field updated when the user completes registration is more efficiently written ahead of time as a boolean value as opposed to using an extra `SELECT` inside another query. This will still need a strategy to keep the flag up to date, but allows you to avoid paying the check cost on every read. ## Indexes and iterators - Define indexes that match real filter and sort patterns; see [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md). - The [`WITH`](/docs/reference/query-language/clauses/with.md) clause can force or restrict which index the planner uses when you need predictable behaviour (for example comparing plans with `EXPLAIN`). ## Fast table counts For `SELECT count() … GROUP ALL` over a whole table, a `COUNT` index maintains a running total instead of scanning every row each time. See the note under [`SELECT` - `COUNT` index](/docs/reference/query-language/statements/select.md#using-a-count-index-to-speed-up-count-in-group-all-queries). From 3.2.5, a bare `count()` projection can [imply `GROUP ALL`](/docs/reference/query-language/statements/select.md#bare-count-implies-group-all). --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/representations-and-codecs # Representations and codecs Choose the right built-in when moving data between strings, bytes, tokens, patches, and executable query text. _(since v3.2.0)_ Many SurrealDB built-ins do the same broad job: take a value in **representation A** and produce a value in **representation B**. As these tend to be found over various parts of the documentation, this page is a map to introduce them in a single location and help you pick the right one. ## Quick overview | I want to… | Use | Reference | | --- | --- | --- | | Serialise a value to JSON or CBOR (or Base64 bytes) | `encoding::json::*`, `encoding::cbor::*`, `encoding::base64::*` | [Encoding functions](/docs/reference/query-language/functions/database-functions/encoding.md) | | Parse JSON/CBOR bytes back into SurrealQL values | `encoding::json::decode`, `encoding::cbor::decode` | [Encoding functions](/docs/reference/query-language/functions/database-functions/encoding.md) | | Preview how an analyzer tokenizes text | `search::analyze` | [Search functions](/docs/reference/query-language/functions/database-functions/search.md#searchanalyze) | | Extract part of an email or URL string | `parse::email::*`, `parse::url::*` | [Parse functions](/docs/reference/query-language/functions/database-functions/parse.md) | | Diff or patch a value with JSON Patch | `value::diff`, `value::patch` | [Value functions](/docs/reference/query-language/functions/database-functions/value.md) | | Coerce or inspect types | `type::*`, casts (``, ``, …) | [Type functions](/docs/reference/query-language/functions/database-functions/type.md) | | **Run a query string at runtime** | `eval::surql`, `eval::gql` | [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md) | ## Serialisation (`encoding::*`) **Reversible codecs** for wire formats and storage: - [`encoding::json::encode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingjsonencode) / [`decode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingjsondecode) - JSON text - [`encoding::cbor::encode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingcborencode) / [`decode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingcbordecode) - [CBOR](/docs/reference/rest-api/cbor-protocol.md) bytes - [`encoding::base64::encode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingbase64encode) / [`decode`](/docs/reference/query-language/functions/database-functions/encoding.md#encodingbase64decode) - Base64 text for binary payloads Typical uses: [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) request bodies, file bucket payloads, and SDK interchange. Round-trip is the mental model. ```surql LET $payload = { event: 'signup', user: 'tobie' }; -- Value → JSON text → value again encoding::json::decode(encoding::json::encode($payload)); ``` ```surql title="Output" { event: 'signup', user: 'tobie' } ``` Related one-way or format-specific helpers elsewhere include `string::html::encode`, `geo::hash::encode`, and [`crypto::*`](/docs/reference/query-language/functions/database-functions/crypto.md) hashes. ## Text analysis (`search::analyze`) [`search::analyze`](/docs/reference/query-language/functions/database-functions/search.md#searchanalyze) runs a named [`DEFINE ANALYZER`](/docs/reference/query-language/statements/define/analyzer.md) pipeline on a string and returns an array of tokens. It is **lossy** (stemming, filtering) and mirrors what full-text indexing does - useful for debugging analyzers before you index. The "format" here is not JSON or CBOR; it is whatever pipeline you defined on the analyzer. ```surql DEFINE ANALYZER demo_blank TOKENIZERS blank; search::analyze("demo_blank", "SurrealDB graph queries"); ``` ```surql title="Output" ['SurrealDB', 'graph', 'queries'] ``` Compare the tokens above with what you get after adding `FILTERS lowercase, snowball(english)` - the same input string produces a different token list, which is why `search::analyze` is handy when tuning an analyzer before you create a [`FULLTEXT`](/docs/reference/query-language/statements/define/indexes.md) index. ## Structured parsing (`parse::*`) [`parse::url::*`](/docs/reference/query-language/functions/database-functions/parse.md) and [`parse::email::*`](/docs/reference/query-language/functions/database-functions/parse.md) extract one component from a structured string. There is no round-trip - you get a field value, not a reassembled URL. ```surql { domain: parse::url::domain("https://surrealdb.com/docs"), user: parse::email::user("tobie@surrealdb.com"), }; ``` ```surql title="Output" { domain: 'surrealdb.com', user: 'tobie' } ``` ## In-value transforms (`value::*`) [`value::diff`](/docs/reference/query-language/functions/database-functions/value.md#valuediff) and [`value::patch`](/docs/reference/query-language/functions/database-functions/value.md#valuepatch) move between a SurrealQL value and JSON Patch operations. They pair naturally with [changefeeds](/docs/learn/querying/real-time/changefeeds.md) and [`LIVE SELECT DIFF`](/docs/learn/querying/real-time/live-queries.md). ```surql LET $before = { title: 'Weekly update', status: 'draft' }; LET $after = { title: 'Weekly update', status: 'published' }; LET $patch = value::diff($before, $after); value::patch($before, $patch); ``` ```surql title="Output" { title: 'Weekly update', status: 'published' } ``` `value::diff` produced the patch; `value::patch` applied it. The same pair works when you receive patch operations from a client or a live diff stream. ## Dynamic evaluation (`eval::*`) _(since v3.2.0)_ [`eval::surql`](/docs/reference/query-language/functions/database-functions/eval.md#evalsurql) and [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) parse and **execute** query text in the caller's transaction. That is fundamentally different from encoding: - Input is **executable** SurrealQL or [ISO GQL](/docs/learn/querying/gql/overview.md), not a static wire format. - **Denied by default** - requires [`allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries) and the [arbitrary-query](/docs/learn/security/authorization/capabilities.md#arbitrary-queries) gate. - Nested writes affect the caller's transaction; transaction-control statements are rejected. Use `eval::*` when the query string is only known at runtime. Prefer [`DEFINE FUNCTION`](/docs/reference/query-language/statements/define/function.md), [`DEFINE API`](/docs/reference/query-language/statements/define/api.md), or normal client queries when the shape of the work is fixed at deploy time. ```surql -- Query text from a table, config row, or user input (requires allow-eval-query) LET $template = "$greeting + ', ' + $name"; eval::surql($template, { greeting: 'Hello', name: 'world' }); ``` ```surql title="Output" 'Hello, world' ``` For [ISO GQL](/docs/learn/querying/gql/overview.md) strings, use `eval::gql` instead - same bindings object, plus [`--allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries). See [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md) for setup and examples. ## See also - [Database functions catalogue](/docs/reference/query-language/functions/database-functions.md) - [Parameterised queries](/docs/learn/querying/concepts-and-guides/parameterised-queries.md) - `$parameters` at the SurrealQL layer (contrast with `eval` bindings) - [Capabilities](/docs/learn/security/authorization/capabilities.md) --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/sequences # Sequences SurrealDB sequences for monotonic numeric identifiers, and how they differ from default randomly generated record IDs. Sometimes you need a monotonic number: an order line, a ticket counter, or an audit sequence that always moves forward and never hands the same value to two different writers. Sequences are SurrealDB's answer to this: shared, durable generators that work on a single node or in a cluster without needing to construct the logic in application code. ## When a sequence helps Reach for a sequence when: * You care about strictly increasing integers (or a predictable step) rather than opaque identifiers. * Multiple clients or nodes may allocate values at the same time, and collisions would be painful. * You are happy for the value to be numeric and database-owned, not derived from your domain model alone. If you only need a unique id and ordering is secondary, other patterns (record IDs, ULIDs, hashes) may be simpler. Sequences shine when order and uniqueness are both part of the contract. ## Sequences vs. manual incrementing numbers Auto-incrementing numbers are quick to put together inside a SurrealQL query. ```surql DEFINE FUNCTION fn::get_next() -> number { UPSERT ONLY the:number SET val += 1 RETURN VALUE val }; fn::get_next(); -- 1 fn::get_next(); -- 2 fn::get_next(); -- 3 fn::get_next(); -- 4 ``` This produces a number that will always increase by 1, and is rolled back during a failed transaction. As such, if `fn::get_next()` returns the value 4, you can also be certain that values 1, 2, and 3 also exist. This works well on one server, but it gets awkward in distributed setups: everyone contends on the same record, and you pay for coordination on every allocation. SurrealDB sequences use a batch idea instead: each node reserves a range of values, hands them out locally, and only talks to shared storage when the range runs out. In practice that means less chatter under load and fewer surprises when you scale out. You declare a sequence with a name, then ask for the next value when you need it. Here is the shape of the workflow: ```surql DEFINE SEQUENCE order_line; -- Later, whenever you need the next number: sequence::nextval('order_line'); ``` Note that a sequence is never rolled back. Each number is guaranteed to be unique and a greater value than any of the ones before, but any sequences used in a failed transaction will simply not be used. This means that a sequence that returns the number 4 is only guaranteed to be the greatest number thus far, but not that the numbers 1, 2, or 3 have been used in any successful transactions. In other words, ordinary data changes can roll back with a transaction but sequence advances do not. The following sketch shows the behaviour in which the sequence moves forward even when a transaction aborts, but a counter field does not. ```surql DEFINE SEQUENCE seq; CREATE my:counter SET val = 0; sequence::nextval('seq'); -- first value my:counter.val; -- still 0 until we update it BEGIN TRANSACTION; sequence::nextval('seq'); -- second value (consumed) UPDATE my:counter SET val += 1; -- counter becomes 1 CANCEL TRANSACTION; -- counter rolls back to 0 sequence::nextval('seq'); -- third value: sequence did not roll back UPDATE my:counter SET val += 1; -- counter is now 1 ``` Design with that rule in mind: sequences are for identity and ordering, not for values that must stay in lock step unless you accept gaps. ## Optional configurations: batch size, start, and timeout You can tune how big each reserved batch is, where counting starts, and how long the database should wait when acquiring a new batch. Larger batches mean fewer coordination trips; smaller batches mean less “wasted” headroom if a node stops. A timeout that is too tight can make allocation fail under pressure. ## Where to read more * [`DEFINE SEQUENCE`](/docs/reference/query-language/statements/define/sequence.md) - full statement reference. * [Sequence functions](/docs/reference/query-language/functions/database-functions/sequence.md) - `sequence::nextval` and related usage. * [Transactions](/docs/learn/querying/concepts-and-guides/transactions.md) - how commits and rollbacks interact with ordinary writes. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/sessions-and-scoping # Sessions and scoping A session stores information about the current connection. A session holds the state a connection carries between queries: the active namespace and database, the authenticated user, and any metadata set on it. This page covers what that context contains and how it is read and changed. ## What is session context? A session stores information about the current connection, including: - The active namespace - The active database - Session metadata This context is automatically used by all queries. ## Switching context with `USE` You can change the current namespace and database using the `USE` statement: ```surql USE NS my_namespace DB my_database; ``` After running this, all queries will operate within that context. The name `main` is used as the default name for the current namespace and database [when starting SurrealDB](/docs/reference/cli/surrealdb-cli/commands/sql.md#default-namespace-and-database). ## Parameters As parameters are set on the connection level, setting a parameter name to a value is one way to persist a value across different namespaces and databases. In the following example, `person:one` is allowed to be created inside a different namespace and database as that record ID does not yet exist inside `other_ns/other_db`. The `$person` parameter is what enabled the `person:one` value to be reused. ```surql LET $person = CREATE ONLY person:one; USE NS other_ns DB other_db; CREATE $person; ``` Beyond this, the only way to persist values beyond the current session is to use [an SDK](/docs/languages/javascript.md) or [an extension](/docs/learn/extensions/plugins/overview.md). ## Accessing session information You can access session details using [`session::*`](/docs/reference/query-language/functions/database-functions/session.md) functions. ```surql -- Current namespace session::ns(); -- Current database session::db(); -- Session ID session::id(); ``` ## Using session context in queries You can use session information inside queries: ```surql IF session::db() != "production" { THROW "This query must run in the production database"; }; -- Debugging session state { namespace: session::ns(), database: session::db(), session: session::id() }; ``` --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/subqueries-and-advanced-patterns # Subqueries and advanced patterns Nested SELECTs, CREATE as a value, $parent and $this, latest record per group, and graph paths that behave like inner queries. A query that is nested inside another one is called a subquery. Subqueries are executed first, enabling their output to be used inside a larger query. ## Subquery examples ### `SELECT` inside `SELECT` A `SELECT` can be used to populate the value of a field, which can be given an alias via the `AS` keyword. ```surql SELECT *, SELECT * FROM events WHERE type = 'activity' LIMIT 5 AS history FROM user; ``` To refer to part of the outer query, the preset `$parent` parameter can be used. ```surql SELECT *, SELECT * FROM events WHERE host == $parent.id AS hosted_events FROM user; ``` ### Other statements inside a larger query Other statements such as `CREATE`, `UPDATE` and so on can be used as subqueries as well. This can be useful to combine multiple queries into one or keep statements that should either succeed or fail together inside a single transaction. ```surql CREATE ONLY person:billy SET father = CREATE ONLY person:pete RETURN VALUE id, mother = CREATE ONLY person:brenda RETURN VALUE id; ``` ```surql title="Output" { father: person:pete, id: person:billy, mother: person:brenda } ``` Patterns such as appending a new comment id to a record are covered in [record references](/docs/reference/query-language/language-primitives/record-references.md). See also [`RETURN`](/docs/reference/query-language/statements/return.md). ## `$parent` and `$this` In nested contexts, SurrealDB predefines: - **`$this`** - the current record in the **inner** scope. - **`$parent`** - the current record in the **enclosing** scope. They let an inner `SELECT` relate its `WHERE` clause to the record being processed outside. ```surql SELECT name, SELECT VALUE name FROM user WHERE member_of = $parent.member_of AS group_members FROM user WHERE name = "User1"; ``` ```surql SELECT *, SELECT VALUE id FROM person WHERE $this.name = $parent.name AS people_with_same_name FROM person; ``` Full detail: [Reserved variables - `$parent`, `$this`](/docs/reference/query-language/language-primitives/parameters.md#parent-this). ## Latest record per group A common pattern is to return the **most recently modified record** for each distinct value of a field - the equivalent of a `ROW_NUMBER() OVER (PARTITION BY … ORDER BY …)` window in SQL. In SurrealQL you can combine [`GROUP BY`](/docs/reference/query-language/clauses/group.md), [`.map()`](/docs/reference/query-language/functions/database-functions/array.md#arraymap), and a nested [`SELECT`](/docs/reference/query-language/statements/select.md): ```surql CREATE person:1 SET role = "user", modified_at = d'1970-01-01'; CREATE person:2 SET role = "admin", modified_at = d'1990-01-01'; CREATE person:3 SET role = "user", modified_at = d'1999-01-01'; CREATE person:4 SET role = "admin", modified_at = d'2999-01-01'; (SELECT id, role FROM person GROUP BY role).map(|$o| { SELECT * FROM ONLY $o.id ORDER BY modified_at DESC LIMIT 1 }); ``` ```surql title="Output" [ { id: person:4, modified_at: d'2999-01-01T00:00:00Z', role: 'admin' }, { id: person:3, modified_at: d'1999-01-01T00:00:00Z', role: 'user' } ] ``` How it works: 1. **`GROUP BY role`** collapses records into one row per role. Non-aggregated fields such as `id` become arrays of the grouped record ids. 2. **`.map(|$o| { … })`** runs the nested query once per grouped record. 3. **`SELECT * FROM ONLY $o.id ORDER BY modified_at DESC LIMIT 1`** fetches all of the records in that group, orders by `modified_at`, and returns the latest one. To partition by more than one field, include every non-aggregated field in both the projection and the `GROUP BY` clause: ```surql (SELECT id, role, status FROM person GROUP BY role, status).map(|$o| { SELECT * FROM ONLY $o.id ORDER BY modified_at DESC LIMIT 1 }); ``` See also the [`GROUP` clause](/docs/reference/query-language/clauses/group.md#latest-record-per-group) reference for a shorter summary. ## Graph paths and inner queries Graph traversal (`->edge->table`) can include a **parenthesised inner query** on the edge or node, similar to filtering or projecting in a subquery. This allows you to restrict or shape the edges before the traversal continues. ```surql SELECT ->(SELECT like_strength FROM likes WHERE like_strength > 10) AS likes FROM person; ``` Shorthand filters are often written without a nested `SELECT`, but the nested form is useful when you need full `SELECT` power (`ORDER BY`, `GROUP BY`, and so on): ```surql SELECT ->(likes WHERE like_strength > 10) AS likes FROM person; ``` More examples: [Selecting inside graph queries](/docs/reference/query-language/statements/select.md#selecting-inside-graph-queries) and [graph clauses](/docs/reference/query-language/statements/relate.md#graph-clauses) on the `RELATE` page. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/testing # Testing This page details a number of ways that SurrealDB can be tested. This page covers the ways a SurrealDB database can be tested, starting with the SDKs, which is where most testing happens. ## SDKs Database testing by users is most frequently done using an SDK, particularly for the languages in which SurrealDB can be used [in embedded mode](/docs/build/embedding.md). This allows testing to be conducted in just a few lines of code. ```rust use surrealdb::{ engine::any::connect, types::{ToSql, Value}, }; #[tokio::test] async fn test_db() { let db = connect("memory").await.unwrap(); db.use_ns("main").use_db("main").await.unwrap(); let created = db .query("CREATE ONLY person:one RETURN VALUE id") .await .unwrap() .take::(0) .unwrap() .to_sql(); assert_eq!(created, "person:one"); } ``` For a larger example of a test suite made by a SurrealDB user, see [this repo](https://github.com/kotolex/surrealist/tree/644a97f5d899f3ad7e011cb82d11ce7d1d482ebc/tests/unit_tests) which tests hundreds of assertions using the Python SDK. ## Testing via direct SurrealQL queries Testing can also be done through direct SurrealQL queries to the database. While the CLI and SurrealDB Studio are the most convenient tools to use, sending in queries via the [HTTP endpoints](/docs/reference/rest-api/http-protocol.md) may be more convenient when needing to test output depending on certain environments such as queries as a record user vs. as a database-level system user or a root user. While you are exploring behaviour in SurrealQL, [`.expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) (_(since v3.1.0)_) on intermediate results or at the end of a method chain makes it obvious which step failed before you wrap the same logic in an SDK test or a transaction with `THROW`. ### Using manual transactions for testing As failed transactions automatically roll back any changes made, a transaction with a final `THROW` statement can be used as a confirmation that no errors have taken place inside a group of queries. Take the following example that creates a unique index and then inserts some records to make sure that the database logic is functioning as expected. However, as names are not necessarily unique, the index soon gives an error and cancels the transaction before `THROW` can be reached. ```surql BEGIN TRANSACTION; DEFINE INDEX unique_name ON TABLE person FIELDS name UNIQUE; INSERT INTO person [ { name: 'Agatha Christie', born: d'1890-09-15' }, { name: 'Billy Billerson', born: d'1979-09-11' }, -- Pretend there are is 10,000 more objects here { name: 'Agatha Christie', born: d'1955-05-15' }, ]; THROW "Reached the end"; COMMIT TRANSACTION; ``` The output is not the expected 'An error occurred: Reached the end' message, showing that not all queries were successful. ```surql title="Output" "Database index `unique_name` already contains 'Agatha Christie', with record `person:qs4bpvl96sf9x40b3567`" ``` If the index is redefined to be less strict, the statements will work and the expected output will be reached, confirming that no errors occurred during the test. ```surql BEGIN TRANSACTION; DEFINE INDEX OVERWRITE unique_person ON TABLE person FIELDS name, born UNIQUE; INSERT INTO person [ { name: 'Agatha Christie', born: d'1890-09-15' }, { name: 'Billy Billerson', born: d'1979-09-11' }, { name: 'Agatha Christie', born: d'1955-05-15' }, ]; THROW "Reached the end"; COMMIT TRANSACTION; ``` ```surql title="Expected output" 'An error occurred: Reached the end' ``` ## Validating `.surql` files One or more `.surql` files can be validated by the CLI using the [`surreal validate`](/docs/reference/cli/surrealdb-cli/commands/validate.md) command. ## SurrealDB internal testing Each PR submitted to the SurrealDB repo runs a series of [language tests](https://github.com/surrealdb/surrealdb/tree/main/language-tests) that carries out thousands of assertions. As an internal tool, the language test suite is and will continue to be developed with only the needs of the SurrealDB developer team in mind. It is in no way meant to be a tool for community use. That said, the code for the suite is publicly available and SurrealDB users sometimes gravitate towards it due to the convenience of being able to run tests over multiple `.surql` files in a variety of situations without needing to recompile any code. If you are considering doing the same, be sure to note that these tests are unit tests with output that may differ slightly from that received as a response from a running database instance. --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/transactions # Transactions How SurrealDB transactions work: the default per-statement scope, manual BEGIN, COMMIT and CANCEL, and the snapshot isolation guarantees that apply under concurrency. SurrealDB is ACID-compliant, so the work inside a transaction either commits as a whole or rolls back with no partial effect. By default **each SurrealQL statement runs in its own transaction**, including side effects such as [defined events](/docs/reference/query-language/statements/define/event.md). A single `CREATE`, `UPDATE` or `SELECT` is therefore atomic on its own. This page covers manual transactions, which group statements that must succeed or fail together, and the **isolation** guarantees that apply when many clients read and write at the same time. ## Implicit transactions `BEGIN` is not the only thing that groups statements. Blocks and function bodies each run as one transaction, despite the lack of `BEGIN` or `COMMIT`, which is why they are described as implicit. Each behaves like any other transaction: it commits when it finishes, and rolls back if anything inside it fails. Implicit transactions commit on normal completion, including an early `RETURN`. An early return is control flow rather than a failure, so the work already done is kept: ```surql { CREATE shipment:shp_5501 SET order = order:ord_8412, carrier = 'DHL'; -- the caller only wants the id RETURN shipment:shp_5501; -- never runs UPDATE order:ord_8412 SET status = 'shipped'; }; ``` Implicit transactions also roll back back on errors, whether that is a [`THROW`](/docs/reference/query-language/statements/throw.md) or a statement that simply fails, such as a schema violation. Nothing inside survives: ```surql { CREATE order:ord_8412 SET item = product:keyboard, quantity = 2; -- rolled back THROW "Insufficient stock"; UPDATE product:keyboard SET stock -= 2; -- never runs }; ``` Without the braces the order would persist while the stock it reserved never moved, which is the inconsistency a transaction exists to prevent. In a bare sequence of statements, each one is its own transaction. Here the `THROW` reports an error, and both records still exist afterwards, because the statements either side of it committed independently: ```surql CREATE product:keyboard SET stock = 40; -- commits THROW "Stop the import"; -- errors, and stops nothing CREATE product:mouse SET stock = 25; -- commits ``` Both products exist afterwards. A bare `THROW` reports an error without halting the statements around it, so it cannot be used as a guard clause outside a block. So wrapping statements in braces is enough to make them atomic. A block does not, however, give a failure somewhere to be contained. There is one transaction per query rather than a stack of them, so a block inside a manual transaction is part of that transaction rather than a nested one. A `THROW` inside the block aborts the whole thing, including the statements that ran before it: ```surql BEGIN; UPDATE account:one SET balance -= 100; -- rolled back as well { CREATE ledger_entry:le_9001 SET amount = 100, account = account:one; THROW "Ledger rejected the entry"; }; UPDATE account:two SET balance += 100; -- never runs COMMIT; -- fails: the transaction was already aborted ``` ## Manual transactions Three statements control a manual transaction: ```surql title="Transaction statements" BEGIN [ TRANSACTION ]; COMMIT [ TRANSACTION ]; CANCEL [ TRANSACTION ]; ``` [`BEGIN`](/docs/reference/query-language/statements/begin.md) opens the transaction. [`COMMIT`](/docs/reference/query-language/statements/commit.md) makes every change inside it a permanent part of the database. [`CANCEL`](/docs/reference/query-language/statements/cancel.md) rolls those changes back instead. If any statement inside the transaction fails, the whole transaction is rolled back and no change survives. ```surql title="A transfer between two accounts" -- Create two accounts for bank customers CREATE account:one SET balance = 135605.16; CREATE account:two SET balance = 91031.31; -- Start a manual database transaction BEGIN TRANSACTION; -- Update the balances of each customer involved in the wire transfer UPDATE account:one SET balance += 300.00; UPDATE account:two SET balance -= 300.00; -- Apply both updates together. Had either statement failed, the database -- would remain in its initial state. COMMIT TRANSACTION; ``` Replacing `COMMIT TRANSACTION` with `CANCEL TRANSACTION` in the example above leaves both balances untouched. Client SDKs expose the same model through transaction handles. See the transactions guide for your language under [SDKs](/docs/languages/javascript.md). ## THROW to conditionally cancel a transaction A transaction rolls back on its own when a statement errors. [`THROW`](/docs/reference/query-language/statements/throw.md) breaks out of one deliberately, at any point. `THROW` can be followed by any value, which serves as the error message, and is usually a string. ```surql CREATE account:one SET dollars = 100; CREATE account:two SET dollars = 100; LET $transfer_amount = 150; BEGIN TRANSACTION; UPDATE account:one SET dollars -= $transfer_amount; UPDATE account:two SET dollars += $transfer_amount; IF account:one.dollars < 0 { THROW "Insufficient funds, would have $" + account:one.dollars + " after transfer" }; COMMIT TRANSACTION; SELECT * FROM account; ``` ```surql title="Output when $transfer_amount set to 150" 'An error occurred: Insufficient funds, would have $-50 after transfer' ``` ```surql title="Output when $transfer_amount set to 50" [ { dollars: 50, id: account:one }, { dollars: 150, id: account:two } ] ``` ## Snapshot isolation Every SurrealDB transaction runs under **snapshot isolation**. When a transaction starts, it sees a consistent point-in-time view of the database. That snapshot stays stable for the lifetime of the transaction, so reads inside the transaction do not observe concurrent writes from other transactions until commit. SurrealDB does not offer weaker isolation levels. You cannot downgrade to read committed or read uncommitted. On commit, the engine checks for **write conflicts**. If two concurrent transactions modified the same key, the later commit fails with a transaction conflict error and must be retried. There is no silent last-writer-wins merge at the storage layer. Records that a transaction only reads are outside that check by default. The [`FOR UPDATE`](#locked-reads-with-for-update) clause brings them into it. These semantics apply across deployment models and storage backends: embedded and server, single-node and distributed, RocksDB, SurrealKV, SurrealMX, and browser IndexedDB. The [query layer](/docs/learn/data-models/architecture.md#query-layer) enforces the same isolation contract regardless of which engine persists the keys underneath. ### What you get in practice Snapshot isolation with conflict detection on commit protects against the anomalies most application developers plan for: | Anomaly | Protected? | Notes | | --- | --- | --- | | Dirty reads | Yes | A transaction never reads uncommitted data from another transaction. | | Non-repeatable reads | Yes | Re-reading the same data inside a transaction returns the same values. | | Lost updates (same key) | Yes | Concurrent writes to the same key cannot both commit; one transaction must retry. | | Write skew | Opt in | Covered per record with [`FOR UPDATE`](#locked-reads-with-for-update); see [write skew](#write-skew) below. | In database terms, snapshot isolation sits between read committed and serialisable. It matches the default isolation level in MySQL (InnoDB `REPEATABLE READ`) and PostgreSQL's optional `REPEATABLE READ` level, with the important caveat that SurrealDB does not provide serialisable isolation. For the specific records a transaction names, [`FOR UPDATE`](#locked-reads-with-for-update) closes the remaining gap. ### Compared with PostgreSQL `REPEATABLE READ` PostgreSQL's `REPEATABLE READ` is snapshot isolation too, so the two protect against the same anomalies. The mechanisms differ in ways that decide where an application handles a conflict. | | PostgreSQL `REPEATABLE READ` | SurrealDB | | --- | --- | --- | | Snapshot taken | At the first statement in the transaction | At the start of the transaction | | Competing writer | Blocks on a row lock until the first transaction resolves | Never blocks; proceeds and commits | | Conflict surfaces | At the conflicting statement, as `could not serialize access due to concurrent update` | At `COMMIT`, as a transaction conflict error | | `SELECT ... FOR UPDATE` | Takes a row lock, and raises straight away if the row moved since the snapshot | Registers the record, and raises at `COMMIT` if it moved | | Write skew | Needs explicit locking, or the `SERIALIZABLE` level | Needs [`FOR UPDATE`](#locked-reads-with-for-update) | | Blanket serialisable level | `SERIALIZABLE` covers predicate-based anomalies as well | No equivalent; `FOR UPDATE` covers named records only | Two consequences follow for application code. Conflicts arrive later in SurrealDB: a `FOR UPDATE` read never fails on account of a concurrent write, so retry logic wraps the whole transaction rather than guarding individual statements. And because a predicate cannot be registered, an invariant that depends on a set of records rather than on named ones has no direct equivalent of `SERIALIZABLE` to fall back on. Such an invariant needs a record that stands in for the set, such as a counter or a parent record that every participant reads for update. ## Write conflicts and retries When a commit fails because another transaction wrote the same key first, SurrealDB returns a **transaction conflict** error. Your application, or your client's retry logic, should run the transaction again. This is normal under concurrent load, not a sign of data corruption. At scale, rising conflict rates show up in metrics such as `surrealdb_transaction_conflicts_total` - see [Observability](/docs/manage/observability/metrics.md) for monitoring guidance. Keep transactions short and touch the fewest keys necessary. Long-running transactions that overlap on hot keys see more conflicts. ## Write skew Snapshot isolation does not prevent **write skew**. Write skew occurs when two transactions each read overlapping state, make independent decisions, and both commit even though their combined effect breaks an invariant. An everyday example is one in which two doctors each check a schedule, see that only one shift is booked, and both book themselves. Each transaction read a consistent snapshot, but together they overbooked the day. Where write skew matters for your workload, read the records the rule depends on with [`SELECT ... FOR UPDATE`](#locked-reads-with-for-update). Each doctor's transaction then registers the `schedule` record it based its decision on, so whichever transaction commits second fails and retries against the booking the first one made. Writing every record the rule depends on has the same effect, because a write is conflict-checked already. That remains the natural choice where the transaction was going to update the record anyway, and conditional updates work on the same principle. ## Locked reads with `FOR UPDATE` _(since v3.3.0)_ The [`FOR UPDATE`](/docs/reference/query-language/statements/select.md#the-for-update-clause) clause on `SELECT` enrols a record that the transaction only reads into the same commit-time conflict check that already covers writes. The transaction commits only if no other transaction wrote that record after the snapshot was taken. Otherwise `COMMIT` fails with a transaction conflict error and the transaction can be retried. ```surql BEGIN; -- Register the schedule for commit-time conflict detection LET $schedule = SELECT * FROM ONLY schedule:monday FOR UPDATE; -- The decision below rests on a value that is read but never written, -- which is exactly the case FOR UPDATE covers IF $schedule.shifts_booked = 0 { UPDATE booking SET doctor = $auth.id, day = "monday"; }; COMMIT; ``` Three properties are worth knowing beyond the basic guarantee: - **Concurrent writers are not blocked.** The clause takes no lock. Another transaction can write a registered record and commit normally, and the cost lands on the reading transaction, whose own `COMMIT` then fails and must be retried. Readers coming from a database where `SELECT ... FOR UPDATE` holds a row lock should plan for retries rather than for waiting. - **An absent record is still covered.** Reading a record id that does not exist registers that id, so a concurrent transaction creating it also invalidates the commit. This is what makes `FOR UPDATE` safe for get-or-create paths. - **The enclosing statement becomes write-classified.** A `FOR UPDATE` read nested inside an expression promotes its statement to a write transaction, because the registration can only be validated at commit time. | Aspect | Behaviour | | --- | --- | | Targets | Record ids only, written literally or passed through a parameter. Tables, record ranges, subqueries and parameters holding a table are rejected. | | Concurrency model | Optimistic. No lock is taken and no transaction waits; a conflict surfaces at `COMMIT` on the transaction that read the record. | | Granularity | One record at a time. Tables, ranges and query predicates are not locked, so a concurrent transaction can still create a new record that matches a predicate. | | Transaction type | Requires a transaction that commits. A read-only context rejects the clause, since the registration would never be validated. | | Storage backends | Supported on every storage backend. Where an engine has no conflict-tracked read path the clause is refused outright, so the guarantee is never quietly downgraded. | | Clause conflicts | Cannot be combined with `VERSION`, `GROUP BY` or `SPLIT`, or with `LIMIT` across multiple targets. | > [!NOTE] > `FOR UPDATE` is used for a record the transaction reads but never writes. A record the transaction also writes is covered already by the write conflict check, so adding the clause there changes nothing. ## ACID at a glance | Property | In SurrealDB | | --- | --- | | **Atomicity** | A transaction's statements commit together or roll back together. | | **Consistency** | Schema, permissions, and statement semantics apply on every commit; you define business invariants in SurrealQL and application code. | | **Isolation** | Snapshot isolation on all storage backends; write conflicts abort on commit, and [`FOR UPDATE`](#locked-reads-with-for-update) extends that check to records the transaction only reads. | | **Durability** | Committed data persists according to your storage engine and sync settings - see [File-backed storage](/docs/running/file-backed.md) and [Deployment models](/docs/manage/self-hosted/deployment-models.md). | Some features deliberately step outside the triggering transaction's ACID boundary. [`ASYNC` events](/docs/reference/query-language/statements/define/event.md#async-events), for example, run after commit in a separate transaction. Use them only when that trade-off is acceptable. ## See also * [The `FOR UPDATE` clause](/docs/reference/query-language/statements/select.md#the-for-update-clause) * [Transaction statements in SurrealQL](/docs/reference/query-language/language-primitives/transactions.md) * [Using transactions to test code for errors](/docs/learn/querying/concepts-and-guides/testing.md#using-manual-transactions-for-testing) * [Architecture](/docs/learn/data-models/architecture.md) --- Source: https://surrealdb.com/docs/learn/querying/concepts-and-guides/working-with-types # Working with types Tips for working with data types in SurrealDB, including arrays, type safety, typed LET statements, and chaining array functions. This page contains a number of examples and tips for working with various data types that go beyond the general API documentation for each type. For general information on these data types, see the [data types](/docs/reference/query-language/language-primitives/data-types.md) page in the query language documentation. ## Arrays Working with arrays is one of the most important skills when working with SurrealDB, as [`SELECT`](/docs/reference/query-language/statements/select.md) statements return an array of values by default unless the `ONLY` keyword is used on an array that contains a single item. ```surql -- Even this returns an array SELECT * FROM 9; -- Use the `ONLY` clause to return a single item SELECT * FROM ONLY 9; -- `ONLY` errors when the array holds more than one item SELECT * FROM ONLY [1,9]; ``` ```surql title="Output" -------- Query 1 -------- [ 9 ] -------- Query 2 -------- 9 -------- Query 3 -------- 'Expected a single result output when using the ONLY keyword' ``` This also means that a `SELECT` statement used to fetch records from a datastore can be used unchanged for any other array of values. ```surql LET $ten_items = [8,5,3,2,6,4,76,9,8,5]; SELECT * FROM $ten_items START 5 LIMIT 5; ``` ```surql title="Output" [ 4, 76, 9, 8, 5 ] ``` Other syntax can be used to achieve the same result, such as pulling from a range of indexes. ```surql LET $ten_items = [8,5,3,2,6,4,76,9,8,5]; $ten_items[5..10]; ``` ## Type safety and type conversion ### Using typed `LET` statements Using typed `LET` statements is a good practice when prototyping code or when getting used to SurrealQL for the first time. Take the following example that attempts to count the number of `true` values in a field by filtering out values that are not `true`, without noticing that the field actually contains strings instead of booleans. The query output ends up being 0, rather than the expected 2. ```surql CREATE some:record SET vals = ["true", "false", "true"]; some:record.vals.filter(|$val| $val = true).len(); ``` ```surql title="Output" 0 ``` Breaking this into multiple typed `LET` statements shows the error right away. ```surql LET $vals: array = some:record.vals; LET $len: number = $vals.filter(|$val| $val = true).len(); $len; ``` ```surql title="Output" -------- Query 1 -------- "Tried to set `$vals`, but couldn't coerce value: Expected `bool` but found `'true'` when coercing an element of `array`" -------- Query 2 -------- 'There was a problem running the filter() function. no such method found for the none type' -------- Query 3 -------- NONE ``` With the location of the error in clear sight, a fix is that much easier to implement. ```surql LET $vals: array = some:record.vals.map(|$val| $val); LET $len: number = $vals.filter(|$val| $val = true).len(); $len; ``` ```surql title="Output" 2 ``` ### Mapping items in an array The [`array::map()`](/docs/reference/query-language/functions/database-functions/array.md#arraymap) function provides access to each item in an array, allowing an operation to be performed on it before being passed on. Other similar functions can also be used, such as [`array::filter()`](/docs/reference/query-language/functions/database-functions/array.md#arrayfilter) which passes on an array that only contains the items that return `true` to an expression of your choice. The following example shows how to chain these functions to validate and modify data in a single statement. The example below removes any items with a `NONE`, checks to see if the location data is a valid geometric point, and then returns the remaining items as objects with a different structure. ```surql [ NONE, { at: (98, 65.7), name: "Some city" }, { at: (-190.7, 0), name: NONE }, { name: "Other city", at: (0.0, 0.1) }, { name: "Nonexistent city", at: (200.0, 66.5) } ] .filter(|$v| $v != NONE AND $v.name != NONE) .filter(|$v| $v.at.is_valid()) .map(|$v, $i| { item: $i, name: $v.name, coordinates: $v.at }); ``` ```surql title="Output" [ { coordinates: (98, 65.7), item: 0, name: 'Some city' }, { coordinates: (0, 0.1), item: 1, name: 'Other city' } ] ``` While refining pipelines like this, [`.expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) (_(since v3.1.0)_) on the result of `.filter()` or `.map()` can confirm the shape you expect before the query continues. For example, `.expect(|$records| $records.len() = 2, "Expected exactly two valid cities")`. --- Source: https://surrealdb.com/docs/learn/querying/custom-apis/managing-apis # Managing APIs Defining custom HTTP-style endpoints on SurrealDB so clients hit a narrow surface instead of arbitrary queries. Custom APIs let you expose a small, deliberate set of routes on top of SurrealDB. Instead of handing every caller a generic query channel, you can define named endpoints that return a familiar HTTP-shaped result. That pattern pairs well with tightening access such as varying querying timeouts for free or entry-level users on an app. To accomplish this, you can combine [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) with [capabilities](/docs/learn/security/authorization/capabilities.md) or server [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md) so certain users only see the APIs you designed. ## Where endpoints live Each definition maps to a path under `/api/:namespace/:database/...`, followed by the path you gave in the statement. As such, an endpoint path like `get_users` in namespace `my_namespace` and database `my_database` becomes something like `/api/my_namespace/my_database/get_users` over HTTP. ## Reading the incoming request Inside the handler you can use the built-in [`$request`](/docs/reference/query-language/language-primitives/parameters.md#request) value. This parameter holds the `method`, `body`, `headers`, `query`, `params` (from your path pattern), and `context` you or middleware may have set. ## A minimal endpoint Here is a small endpoint that echoes part of the body and sets a couple of headers. The precise clauses and permissions are spelled out in the [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) reference page. ```surql title="Defining an API endpoint" DEFINE API "/test" FOR get, post MIDDLEWARE api::timeout(1s) THEN { { status: 200, body: { request: $request.body, response: "The server works" }, headers: { 'last-modified': time::now(), 'expires': (time::now() + 4d) } }; }; ``` You can test the endpoint from SurrealQL without needing to deploy by calling the [`api::invoke`](/docs/reference/query-language/functions/database-functions/api.md) function. This function takes either the path alone or the path plus a body object. ```surql api::invoke("/test"); api::invoke("/test", { body: { hi: "please", give: "me", the: "information" } }); ``` ## Path patterns: one segment or the rest of the URL The path string in `DEFINE API` can be static, or it can capture pieces of the URL. A segment like `"/users/:id"` binds one path component - anything in that slot shows up on `$request.params`. A trailing pattern with `*` instead of `:` matches everything from that point - handy for nested paths or file-like routes. ```surql DEFINE API OVERWRITE "/test/:anything_goes" FOR get THEN { RETURN { body: { some: "data" } } }; api::invoke("/test/this_matches"); api::invoke("/test/same_here"); api::invoke("/test/but/this/wont/match"); ``` Here the first two calls hit the handler; the third does not, because `:anything_goes` only covers a single segment - extra slashes mean “no matching route”, which surfaces as a 404-style result from `api::invoke`. To accept multiple trailing segments, switch the capture to the `*` form: ```surql DEFINE API OVERWRITE "/test/*anything_goes" FOR get THEN { RETURN { body: { some: "data" } } }; api::invoke("/test/this_matches"); api::invoke("/test/same_here"); api::invoke("/test/works/with/multiple/paths/now"); ``` All three calls succeed because the remainder of the path is treated as one captured piece. ## Middleware Built-in helpers such as `api::timeout` sit in the `MIDDLEWARE` list before your `THEN` block. For more details on how to use middleware, see the [next page](/docs/learn/querying/custom-apis/middleware.md). ## Where to read more * [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) - full statement reference, including `PERMISSIONS`. * [API functions](/docs/reference/query-language/functions/database-functions/api.md) - `api::invoke` and related helpers. * [Custom functions](/docs/learn/querying/concepts-and-guides/custom-functions.md) - how `fn::` functions fit into larger designs. --- Source: https://surrealdb.com/docs/learn/querying/custom-apis/middleware # Middleware Chaining custom functions before your API handler so you can share logic on DEFINE API routes. When you use [`DEFINE API`](/docs/reference/query-language/statements/define/api.md), the `THEN` block is the handler that decides the HTTP-shaped response. Middleware is everything you run before that handler, such as timeouts, logging, auth checks, or tweaks to the outgoing payload. SurrealDB runs built-in helpers from the [API functions](/docs/reference/query-language/functions/database-functions/api.md) package alongside your own `fn::` functions, in the order you list them. Custom middleware functions are ordinary [user-defined functions](/docs/reference/query-language/statements/define/function.md). Each one receives: * The current request object (conventionally `$req`). * A `next` closure that continues the chain and eventually runs `THEN` when no middleware remains. You can add extra parameters after those two;such values are supplied when you attach the function to `DEFINE API` (for example passing in `time::now()` to return the time at which a request was processed). The function must return an object - that is the response object passed to the next middleware or returned to the client. Names like `$req` and `$next` are only conventions; what matters is the order of arguments and that you actually call `$next($req)` when you want the pipeline to proceed. ## Starting from a handler with no middleware ```surql DEFINE API "/custom_response" FOR get THEN { { status: 200, body: { num: 1 } }; }; ``` Calling `api::invoke("/custom_response")` returns the body you would expect: ```surql { body: { num: 1 }, context: {}, headers: {}, status: 200 }; ``` ## Adding a function that adjusts the response This middleware calls `$next($req)` to obtain the handler's result, then bumps `body.num` by one before returning: ```surql DEFINE FUNCTION fn::increment_num($req: object, $next: function) -> object { LET $res = $next($req); $res + { body: { num: $res.body.num + 1 } } }; DEFINE API "/custom_response" FOR get MIDDLEWARE fn::increment_num() THEN { { status: 200, body: { num: 1 } }; }; ``` Now `api::invoke("/custom_response")` yields `num: 2`, showing that the outer middleware has reshaped what the caller sees. ```surql { body: { num: 2 }, context: {}, headers: {}, status: 200 }; ``` ## Stacking two pieces of middleware Order matters when adding middleware, so be sure to list outer effects first if they should wrap everything that follows. Here a timer middleware adds a timestamp into `context`, and increment still adjusts the body: ```surql DEFINE FUNCTION fn::start_timer($req: object, $next: function, $called_at: datetime) -> object { LET $res = $next($req); $res + { context: { called_at: $called_at }} }; DEFINE FUNCTION fn::increment_num($req: object, $next: function) -> object { LET $res = $next($req); $res + { body: { num: $res.body.num + 1 } } }; DEFINE API "/custom_response" FOR get MIDDLEWARE fn::start_timer(time::now()), fn::increment_num() THEN { { status: 200, body: { num: 1 } }; }; api::invoke("/custom_response"); ``` Possible output: ```surql { body: { num: 2 }, context: { called_at: d'2026-01-16T01:49:44.115351Z' }, headers: {}, status: 200 }; ``` ## See also * [Managing APIs](/docs/learn/querying/custom-apis/managing-apis.md) - defining routes, path captures, and `$request`. * [Custom functions](/docs/learn/querying/concepts-and-guides/custom-functions.md) - patterns for `fn::` definitions used in middleware. --- Source: https://surrealdb.com/docs/learn/querying/gql/mutations # Mutations ISO GQL data-modifying statements - INSERT, SET, REMOVE, and DELETE - on the /gql endpoint. _(since v3.2.0)_ ISO GQL on SurrealDB supports the four **data-modifying** statements from the standard - **`INSERT`**, **`SET`**, **`REMOVE`**, and **`DELETE`** - in addition to read-only `MATCH … RETURN` queries. Mutations run on the same [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql) endpoint and through [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) (which still needs [`--allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries)). > [!NOTE] > From **3.3.0**, GQL is enabled by default. On **3.2.x**, enable it with [`--allow-experimental gql`](/docs/reference/cli/surrealdb-cli/commands/start.md#experimental-capabilities). Mutation-bearing queries open a **write transaction** - the same permissions, field validation, events, indexes, references, and live-query behaviour as native SurrealQL `CREATE` / `UPDATE` / `DELETE` / `RELATE` apply. ## Linear programs A GQL query is a **linear program**: an ordered sequence of `MATCH` / `OPTIONAL` read clauses and data-modifying statements, in **any interleaving**, optionally ending in `RETURN`. - The **binding table** threads through every step in textual order. - A **`MATCH` or `OPTIONAL` after a mutation** re-scans **live** (post-write) state in the same transaction - a clause after `SET` or `DELETE` sees updated or removed records; a clause after `INSERT` sees created records and may bind variables the `INSERT` introduced. - **`RETURN` is optional** when the query mutates. Read-only queries must still end with `RETURN`. - A mutation-only query (no `RETURN`) returns an **empty** result. The examples below assume that [the following seed data](/docs/learn/querying/gql/via-http.md#load-sample-data) has been first uploaded. ### Via `eval::gql` The same mutation strings work in the REPL when both **`gql`** and **`eval`** are allowed - see [Try without HTTP](/docs/learn/querying/gql/via-http.md#try-without-http---evalgql): ```cypher eval::gql("MATCH (n:person WHERE n.name = 'A') SET n.age = 99 RETURN n.age AS age"); -- { age: 99 } ``` ## `SET` Update properties on a bound node or edge. ```gql MATCH (n:person WHERE n.name = 'A') SET n.age = 99 RETURN n.age AS age ``` - **`SET a.p = v`** - set one property. - **`SET a = { … }`** - replace all **user** properties (a `CONTENT`-style replace). Properties absent from the map are dropped. The record `id`, and an edge's `in` / `out`, are preserved. Setting reserved keys **`id`**, **`in`**, or **`out`** is rejected on both the per-property form (`SET a.id = …`) and the object form. **`SET a:Label`** is rejected - a SurrealDB record belongs to exactly one table; labels are immutable. ## `REMOVE` Unset a property without deleting the record. ```gql MATCH (n:person WHERE n.name = 'A') REMOVE n.age RETURN n.name AS name ``` **`REMOVE a:Label`** is rejected for the same one-table-per-record rule as `SET` label mutations. ## `DELETE` Delete a matched node or edge. ```gql MATCH (n:person WHERE n.name = 'A') DETACH DELETE n ``` - **`NODETACH DELETE`** (ISO default) - errors if the node still has connected edges. - **`DETACH DELETE`** - deletes the node and cascades connected edges. Bound edge variables for cascaded edges become `null` in a trailing `RETURN`. A deleted binding becomes **`null`** in post-mutation projections. ## `INSERT` Create nodes and edges. Each new node requires a **label** (table name). **Leading insert** (no preceding `MATCH`) - runs once: ```gql INSERT (p:person {name: 'Z', age: 1}) RETURN p.name AS name ``` **After `MATCH`** - runs **once per binding row**. Relate existing endpoints or create new nodes: ```gql MATCH (a:person WHERE a.name = 'A') MATCH (b:person WHERE b.name = 'C') INSERT (a)-[:likes]->(b) RETURN a.name AS src, b.name AS dst ``` Graph-pattern form: `INSERT (a:Label {…})-[:Edge {…}]->(b:Label {…})`. A node with no label and no properties references a variable already bound by a preceding `MATCH` (an existing endpoint). ## Read after write Mutations and reads can interleave in one query: ```gql MATCH (n:person WHERE n.name = 'C') SET n.age = 20 MATCH (m:person WHERE m.age = 20) RETURN m.name AS name ORDER BY name ``` The second `MATCH` sees the updated age on `C` and returns both `B` and `C` (both age 20). ## `RETURN` after mutations When present, `RETURN` projects the **post-mutation** binding table: - **`SET` / `INSERT`** rebind the mutated or created record (the **after** image). - Fan-out that binds the same record more than once applies writes **per row**, in order (last-write-wins for row-dependent values). - **`DELETE`** nulls deleted bindings; **`DETACH DELETE`** also nulls cascaded edge bindings. ## Rejected forms The lowering layer rejects several ISO forms that do not map cleanly to SurrealDB's one-table-per-record model, including: - Label mutations (`SET a:Label`, `REMOVE a:Label`) - Mutations on unbound variables, groups, or path patterns - Reserved property keys on `SET` (`id`, `in`, `out`) - Malformed `INSERT` graph patterns (e.g. undirected edges, anonymous nodes without labels where required) Parse and semantic errors return the same error envelope as read queries. ## SurrealQL vs GQL for writes | Task | Prefer | | --- | --- | | Schema changes, bulk load, `RELATE` with arbitrary SurrealQL | [SurrealQL](/docs/reference/query-language.md) on [`POST /sql`](/docs/reference/rest-api/http-protocol.md) | | Graph-pattern read + write in one ISO GQL program | GQL mutations on [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql) | | Nested GQL inside an open SurrealQL transaction | [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) | ## Next steps - [GQL overview](/docs/learn/querying/gql/overview.md) - capabilities, wire surfaces, and syntax differences from openCypher - [Sample queries](/docs/learn/querying/gql/sample-queries.md) - read-only pattern examples - [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql) - HTTP headers and response envelope --- Source: https://surrealdb.com/docs/learn/querying/gql/overview # GQL Query SurrealDB graph data with ISO GQL - a Cypher-like graph pattern language over HTTP and RPC. _(since v3.2.0)_ SurrealDB supports **[ISO/IEC 39075 GQL](https://www.iso.org/standard/76120.html)** for **graph pattern matching and data modification** over your existing tables and `RELATE` edges. The surface syntax is closer to **Cypher-style `MATCH … RETURN`** than to SurrealQL `SELECT`, but it runs on the same storage model: node labels map to tables, edge types map to relation tables, and properties map to record fields. > [!NOTE] > From **3.3.0**, ISO GQL is enabled by default on `POST /gql`, the `gql` RPC method, and MCP - no experimental capability is required. On **3.2.x**, enable it with `--allow-experimental gql` (or `SURREAL_CAPS_ALLOW_EXPERIMENTAL=gql`). The surface includes read queries (`MATCH … RETURN`) and data-modifying statements (`INSERT`, `SET`, `REMOVE`, `DELETE`) - see [GQL mutations](/docs/learn/querying/gql/mutations.md). ## GQL is not GraphQL SurrealDB exposes two different graph query languages on **separate** endpoints. Do not abbreviate [GraphQL](/docs/learn/querying/graphql/overview.md) to `gql` in product identifiers - the short name **`gql`** refers to ISO GQL only. | | **GQL** (this guide) | **[GraphQL](/docs/learn/querying/graphql/overview.md)** | | --- | --- | --- | | Standard | [ISO/IEC 39075 GQL](https://www.iso.org/standard/76120.html) | [GraphQL](https://graphql.org/) | | Syntax | `MATCH (a)-[:knows]->(b) RETURN …` | `query { people { name } }` | | HTTP | [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql) | [`POST /graphql`](/docs/reference/rest-api/http-protocol.md#graphql) | | WebSocket RPC | `method: "gql"` | `method: "graphql"` | | Setup | On by default from 3.3.0 (experimental `gql` on 3.2.x) | [`DEFINE CONFIG GRAPHQL`](/docs/reference/query-language/statements/define/config.md) | | Schema | Tables and `RELATE` edges you already have | Auto-generated GraphQL schema from your database | ## When to use GQL - You already think in **graph patterns** such as `(a)-[:knows]->(b)`, `SHORTEST`, and `ALL SHORTEST`, and have yet to learn how to query such paths in SurrealQL. - You are migrating a database from Neo4j to SurrealDB and want to test to ensure that existing Cypher queries map to the same output. - You want a **stable graph query surface** aligned with the ISO GQL standard. For general-purpose schema changes, bulk load, and full SurrealQL expressiveness, keep using [SurrealQL](/docs/reference/query-language.md) and the [`/sql`](/docs/reference/rest-api/http-protocol.md) endpoint. For ISO graph-pattern **writes** (`INSERT`, `SET`, `REMOVE`, `DELETE`), see [GQL mutations](/docs/learn/querying/gql/mutations.md). ## Wire surfaces | Surface | How to call | | --- | --- | | **HTTP** | [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql) - raw GQL query in the request body | | **WebSocket RPC** | `{ "method": "gql", "params": ["", { "var": value }] }` - use for typed `$variables` | | **Postgres wire** | [`SET dialect = 'gql'`](/docs/reference/rest-api/postgres-protocol.md) (or `options=-c dialect=gql` at connect) on a Postgres client connection | | **MCP** | `gql` tool (when the server exposes MCP) | | **SurrealQL** | [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) - nested GQL in the caller's transaction (capability-gated) | Session headers match `/sql`: `Surreal-NS`, `Surreal-DB`, and authentication. Responses use the same JSON envelope as `/sql` (`status`, `result`, `time`). ## Try from SurrealQL (`eval::gql`) To experiment in the REPL without `curl` or `POST /gql`, wrap a GQL string in [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql). The function runs the same engine as the HTTP endpoint and participates in the caller's transaction (including [mutations](/docs/learn/querying/gql/mutations.md)). `eval::gql` needs the **`eval`** capability that `--allow-all` does not enable: [`--allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries) (denied for every subject by default). From 3.3.0 you do **not** also need `--allow-experimental gql` (that flag is still required on 3.2.x). **Embedded REPL** (`surreal sql` against `memory` or a file path - no separate `surreal start`): **Bash** ```bash echo 'eval::gql("MATCH (n:person) RETURN n.name AS name ORDER BY name");' | surreal sql --user root --pass secret --allow-eval-query --pretty --hide-welcome ``` **PowerShell** ```powershell 'eval::gql("MATCH (n:person) RETURN n.name AS name ORDER BY name");' | surreal sql --user root --pass secret --allow-eval-query --pretty --hide-welcome ``` To explore interactively instead, open the same REPL without piping: ```bash surreal sql --user root --pass secret --allow-eval-query ``` **Remote server** (`surreal start` + `surreal sql -e ws://…`): pass `--allow-eval-query` on **`surreal start`** only. The client REPL does not enable `eval` at runtime. ```bash surreal start --user root --pass secret --allow-eval-query ``` Optional bindings use an object as the second argument: `eval::gql("… WHERE n.age > $min …", { min: 18 })`. Full setup, seed data, and more examples can be found on the [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) page. ## Data model mapping | GQL | SurrealDB | | --- | --- | | `(:person)` | Rows in table `person` | | `-[k:knows]->` | Rows in relation table `knows` (`in` / `out` record IDs) | | `n.name` | Field `name` on the bound record | | `$min` | Parameter - bind via RPC `params` object (typed JSON values) | GQL is **not** lowered to SurrealQL text - it compiles to an internal match plan and runs on the streaming engine. The [sample queries](/docs/learn/querying/gql/sample-queries.md) page shows **SurrealQL that returns the same shape** where a close equivalent exists; some patterns (optional match blocks, path selectors) are much more concise in GQL. ## Notable syntax differences from openCypher - `--` starts a **line comment**, not an undirected edge. - Inequality is **`<>`**, not `!=`. - Label conjunction uses **`&`** (`:person&employee`), not `:person:employee`. - Variable-length quantifiers are **postfix on the edge**: `-[e:knows]->{1,3}(b)`, not `*1..3` inside the brackets. - No `IN` list membership operator in this subset. Design notes, the supported subset, and mutation semantics are documented in the SurrealDB source tree under `doc/opengql/` (`REFERENCE.md`, `LOWERING.md`). The parser grammar is vendored from the upstream [opengql/grammar](https://github.com/opengql/grammar) project; that name refers to the grammar repository, not the ISO standard SurrealDB implements. ## Next steps - [GQL via HTTP](/docs/learn/querying/gql/via-http.md) - enable GQL, load data, and call `POST /gql` with cURL or `eval::gql` from the REPL - [Postgres wire protocol](/docs/reference/rest-api/postgres-protocol.md) - run GQL from `psql` or Postgres drivers with `SET dialect = 'gql'` - [GQL mutations](/docs/learn/querying/gql/mutations.md) - `INSERT`, `SET`, `REMOVE`, `DELETE` - [`POST /gql` HTTP reference](/docs/reference/rest-api/http-protocol.md#gql) - headers, response envelope, and parameters - [Sample GQL and SurrealQL queries](/docs/learn/querying/gql/sample-queries.md) - side-by-side examples on the seed graph - [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md) - run GQL from inside SurrealQL with `eval::gql` --- Source: https://surrealdb.com/docs/learn/querying/gql/sample-queries # Sample queries Compare common GQL graph patterns against similar SurrealQL queries on the same seed graph. _(since v3.2.0)_ The examples below use the same **person / knows / city** seed graph as the GQL language tests. SurrealQL snippets return the **same or equivalent row shape**, but are not a mechanical translation of the GQL compiler; they are idiomatic reads you can compare while learning. ## Prerequisites Start a local instance with GQL enabled and load the seed graph - see [GQL via HTTP](/docs/learn/querying/gql/via-http.md#load-sample-data). You can run the GQL examples in either of these ways: | Approach | How | | --- | --- | | **HTTP** | cURL to `POST /gql` (shown in each GQL tab) - GQL is on by default from 3.3.0; on 3.2.x use `--allow-experimental gql` | | **REPL** | [`eval::gql("…")`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) in `surreal sql` or SurrealDB Studio - also needs [`--allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries) on the process that runs the engine (embedded `surreal sql`, or `surreal start` for remote) | Example REPL equivalent for the first query: ```cypher eval::gql("MATCH (n:person) RETURN n.name AS name ORDER BY name"); -- [{ name: 'A' }, { name: 'B' }, { name: 'C' }] ``` ## List nodes by label **GQL** ```gql title="Query" MATCH (n:person) RETURN n.name AS name ORDER BY name ``` ```bash title="cURL" curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d 'MATCH (n:person) RETURN n.name AS name ORDER BY name' \ http://localhost:8000/gql ``` ```json title="Output" [ { "name": "A" }, { "name": "B" }, { "name": "C" } ] ``` **SurrealQL** ```surql title="Query" SELECT name FROM person ORDER BY name; ``` ```surql title="Output" [ { name: 'A' }, { name: 'B' }, { name: 'C' } ] ``` ## Traverse an edge with a property filter Only **person → person** `knows` edges with `since > 2020`. **GQL** ```gql title="Query" MATCH (a:person)-[k:knows]->(b:person) WHERE k.since > 2020 RETURN a.name, b.name ORDER BY a.name ``` ```bash title="cURL" curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d 'MATCH (a:person)-[k:knows]->(b:person) WHERE k.since > 2020 RETURN a.name, b.name ORDER BY a.name' \ http://localhost:8000/gql ``` ```json title="Output" [ { "a.name": "A", "b.name": "B" } ] ``` **SurrealQL** ```surql title="Query" SELECT in.name AS `a.name`, out.name AS `b.name` FROM knows WHERE since > 2020 AND record::tb(out) = 'person' ORDER BY in.name; ``` ```surql title="Output" [ { 'a.name': 'A', 'b.name': 'B' } ] ``` ## Optional match Every `person`, with an optional `knows` edge to a `city` when one exists. **GQL** ```gql title="Query" MATCH (a:person) OPTIONAL MATCH (a)-[k:knows]->(b:city) RETURN a.name AS name, b.name AS city ORDER BY name ``` ```bash title="cURL" curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d 'MATCH (a:person) OPTIONAL MATCH (a)-[k:knows]->(b:city) RETURN a.name AS name, b.name AS city ORDER BY name' \ http://localhost:8000/gql ``` ```json title="Output" [ { "name": "A", "city": "London" }, { "name": "B", "city": null }, { "name": "C", "city": null } ] ``` **SurrealQL** ```surql title="Query" SELECT name, array::first(->knows->city.name) AS city FROM person ORDER BY name; ``` ```surql title="Output" [ { name: 'A', city: 'London' }, { name: 'B', city: NONE }, { name: 'C', city: NONE } ] ``` ## Aggregation with GROUP BY Count outgoing **person → person** `knows` edges per person. **GQL** ```gql title="Query" MATCH (a:person)-[:knows]->(b:person) RETURN a.name AS name, count(*) AS c GROUP BY a.name ORDER BY name ``` ```bash title="cURL" curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d 'MATCH (a:person)-[:knows]->(b:person) RETURN a.name AS name, count(*) AS c GROUP BY a.name ORDER BY name' \ http://localhost:8000/gql ``` ```json title="Output" [ { "name": "A", "c": 1 }, { "name": "B", "c": 2 }, { "name": "C", "c": 1 } ] ``` **SurrealQL** ```surql title="Query" SELECT in.name AS name, count() AS c FROM knows WHERE record::tb(out) = 'person' GROUP BY in.name ORDER BY name; ``` ```surql title="Output" [ { name: 'A', c: 1 }, { name: 'B', c: 2 }, { name: 'C', c: 1 } ] ``` ## What to try next The language-test corpus under `language-tests/tests/opengql/` in the SurrealDB repository covers variable-length quantifiers (`->{1,3}`, `->*`), path search (`ALL SHORTEST`, `SHORTEST k`), multi-pattern comma joins, and [mutations](/docs/learn/querying/gql/mutations.md). Variable-length and path patterns have no single-line SurrealQL equivalent - they are the main reason to reach for GQL for reads; mutations are the ISO-native way to change graph data in the same program as `MATCH`. --- Source: https://surrealdb.com/docs/learn/querying/gql/via-http # Via HTTP Enable ISO GQL on a SurrealDB instance and run queries through POST /gql. _(since v3.2.0)_ The **`POST /gql`** endpoint accepts a **raw GQL query** in the request body (not JSON-wrapped). Authentication and namespace selection use the same headers as [`POST /sql`](/docs/reference/rest-api/http-protocol.md). For the full HTTP reference (headers, response envelope, limits), see [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql). ## Start SurrealDB with GQL From **3.3.0**, start a normal instance - ISO GQL is on by default: ```bash surreal start --log info --user root --pass secret memory ``` On **3.2.x**, pass the experimental capability (storage path after the flag, or use `memory`): ```bash surreal start --log info --user root --pass secret \ --allow-experimental gql memory ``` **Bash** ```bash export SURREAL_CAPS_ALLOW_EXPERIMENTAL=gql surreal start --log info --user root --pass secret memory ``` **PowerShell** ```powershell $env:SURREAL_CAPS_ALLOW_EXPERIMENTAL = "gql" surreal start --log info --user root --pass secret memory ``` > [!NOTE] > This is **not** [GraphQL](/docs/learn/querying/graphql/overview.md) - use [`POST /graphql`](/docs/reference/rest-api/http-protocol.md#graphql) for GraphQL queries. ## Load sample data Use [`POST /sql`](/docs/reference/rest-api/http-protocol.md) with namespace **`main`** and database **`main`** (set via headers below): ```bash curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d "CREATE person:1 SET name = 'A', age = 30, active = true, city = 'London'; CREATE person:2 SET name = 'B', age = 20, active = false, city = 'Paris'; CREATE person:3 SET name = 'C', city = 'London'; CREATE city:1 SET name = 'London'; INSERT RELATION INTO knows [ { id: knows:k12, in: person:1, out: person:2, since: 2021 }, { id: knows:k21, in: person:2, out: person:1, since: 2018 }, { id: knows:k23, in: person:2, out: person:3, since: 2020 }, { id: knows:k1c, in: person:1, out: city:1, since: 2019 }, { id: knows:k31, in: person:3, out: person:1 } ];" \ http://localhost:8000/sql ``` ## `POST /gql` Send the GQL query as the **raw body** with `Content-Type: text/plain` (or omit; UTF-8 text is expected). ```bash curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d 'MATCH (n:person) RETURN n.name AS name ORDER BY name' \ http://localhost:8000/gql ``` Example response (same envelope as `/sql`): ```json [ { "status": "OK", "result": [ { "name": "A" }, { "name": "B" }, { "name": "C" } ], "time": "1.5ms" } ] ``` ### Response formats Set `Accept` to: - `application/json` (default) - `application/cbor` for CBOR-encoded results Parse errors return **HTTP 400** with an error payload. ## Mutations The same endpoint accepts **data-modifying** GQL - `INSERT`, `SET`, `REMOVE`, and `DELETE` - interleaved with `MATCH` / `OPTIONAL` in one query. Mutation-bearing requests run in a **write transaction** and enforce the same permissions as SurrealQL writes. Example - update a property and return the new value: ```bash curl -sS -X POST -u "root:secret" \ -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -H "Content-Type: text/plain" \ -d "MATCH (n:person WHERE n.name = 'A') SET n.age = 99 RETURN n.age AS age" \ http://localhost:8000/gql ``` See [GQL mutations](/docs/learn/querying/gql/mutations.md) for `INSERT`, `REMOVE`, `DELETE`, read-after-write interleaving, and rejected forms. ## WebSocket RPC On an authenticated WebSocket session, send the query as the first parameter. Pass **typed** variables as an optional second object: ```json { "id": 1, "method": "gql", "params": [ "MATCH (n:person) RETURN n.name AS name ORDER BY name" ] } ``` With parameters: ```json { "id": 2, "method": "gql", "params": [ "MATCH (n:person) WHERE n.age > $min RETURN n.name AS name", { "min": 18 } ] } ``` ## Try without HTTP - `eval::gql` If you prefer the CLI or SurrealDB Studio over cURL, run GQL through [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md#evalgql) inside SurrealQL. You still need [`--allow-eval-query`](/docs/learn/security/authorization/capabilities.md#eval-queries), as `eval::*` is denied by default even under `--allow-all`. From 3.3.0 you do not need `--allow-experimental gql` as well (required on 3.2.x only). One process (embedded): pass `--allow-eval-query` on `surreal sql`, load the [seed data](#load-sample-data) with ordinary SurrealQL, then: ```cypher eval::gql("MATCH (n:person) RETURN n.name AS name ORDER BY name"); ``` ```bash surreal sql --user root --pass secret --allow-eval-query ``` **Two processes (remote):** enable eval on **`surreal start`**, then connect with `surreal sql` as usual - capability flags on the client do not turn on `eval` for a remote engine. ```bash surreal start --user root --pass secret --allow-eval-query ``` Mutations work the same way: `eval::gql("MATCH (n:person WHERE n.name = 'A') SET n.age = 99 RETURN n.age AS age")` runs in the open SurrealQL transaction. See [GQL mutations](/docs/learn/querying/gql/mutations.md) and [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md#evalgql). --- Source: https://surrealdb.com/docs/learn/querying/graphql/overview # GraphQL In this section, you will explore GraphQL, an industry-wide recognised protocol for interacting with your data, allowing you to query your data using any preferred method which offers precision and efficiency in data retrieval. SurrealDB supports [GraphQL](https://graphql.org/) through the [`/graphql`](/docs/reference/rest-api/http-protocol.md#graphql) endpoint, which can be accessed via [SurrealDB Studio](https://studio.surrealdb.com/), GraphiQL, Postman, or any other GraphQL client. > [!NOTE] > **GraphQL is not GQL.** [ISO GQL](/docs/learn/querying/gql/overview.md) (`MATCH … RETURN …`) is a separate language on [`POST /gql`](/docs/reference/rest-api/http-protocol.md#gql). Do not abbreviate GraphQL to `gql` - in SurrealDB, `gql` always means ISO GQL. ## Key features GraphQL offers a number of key features that make it a powerful tool for working with SurrealDB: - **Declarative Data Fetching**: GraphQL allows you to request exactly the data you need, no more and no less. This reduces over-fetching and under-fetching of data, leading to more efficient queries. - **Strongly Typed Schema**: GraphQL uses a strong type system to define the capabilities of an API. This schema serves as a contract between the client and the server, ensuring that queries are valid before execution. - **Hierarchical Structure**: GraphQL queries mirror the shape of the data they return, making it intuitive to work with nested data structures. - **Single Endpoint**: When using GraphQL over HTTP, it typically uses a single endpoint, simplifying API architecture and reducing network overhead. - **Ecosystem and Tools**: GraphQL has a rich ecosystem of tools for development, testing, and monitoring, including GraphiQL for query exploration and Apollo Client for state management. ## Getting started Enabling GraphQL queries for SurrealDB can be done through a single statement: [`DEFINE CONFIG GRAPHQL`](/docs/reference/query-language/statements/define/config.md). The simplest clause to follow this with is `AUTO`, which will automatically include all tables in the GraphQL schema. ```surql DEFINE CONFIG GRAPHQL AUTO; ``` A query for `(INFO FOR DB).configs` will show that the above statement ends up having the clauses `TABLES AUTO FUNCTIONS AUTO`. ```surql { GraphQL: 'GRAPHQL TABLES AUTO FUNCTIONS AUTO' } ``` For more fine tuning of the configuration, the `TABLES` and `FUNCTIONS` clauses can be followed by [other clauses](/docs/reference/query-language/statements/define/config.md#tables-configuration). ## Schema naming _(since v3.1.0)_ From SurrealDB 3.1.0, the auto-generated GraphQL schema follows a single **Apollo-style** naming convention (there is no `NAMING` switch): | Operation | Example for table `person` | | --- | --- | | Fetch one record | `person(id: ID!)` | | List records | `people(filter, where, order, limit, start, version)` - pluralised table name | | Aggregate | `people_aggregate(filter, groupBy, …)` | | Create / update / delete | `createPerson`, `updatePerson`, `deletePerson` (bulk: `createPeople`, …) | Field names on types mirror SurrealQL unless you set [`GRAPHQL_ALIAS`](/docs/reference/query-language/statements/define/field.md) or [`GRAPHQL_DEPRECATED`](/docs/reference/query-language/statements/define/field.md) on [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md) / [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) / [`DEFINE FUNCTION`](/docs/reference/query-language/statements/define/function.md). List queries also support Relay-style **`peopleConnection`** cursor pagination alongside offset `limit` / `start`. If you upgrade from 3.0.x, regenerate client stubs and saved queries - names such as `_get_person` and `createManyPerson` are no longer generated for typical tables. ## Next steps The next page introduces some common GraphQL patterns and their SurrealQL equivalents or near equivalents. This is followed by the following pages that detail the various tools by which GraphQL queries can be run on a SurrealDB database: - Using the [`/graphql`](/docs/reference/rest-api/http-protocol.md#graphql) endpoint [via HTTP (cURL)](/docs/learn/querying/graphql/via-http.md) - Using the [`/graphql`](/docs/reference/rest-api/http-protocol.md#graphql) endpoint [via Bruno](/docs/learn/querying/graphql/via-bruno.md) - Using [SurrealDB Studio](https://studio.surrealdb.com/), SurrealDB's interactive environment for experimenting with GraphQL queries and seeing results immediately in the UI. Other tools such as Postman and many others can be used against the `/graphql` endpoint. ## Examples - [Sample queries](/docs/learn/querying/graphql/sample-queries.md) - queries to run against a GraphQL-enabled database --- Source: https://surrealdb.com/docs/learn/querying/graphql/sample-queries # Sample queries Compare common GraphQL queries against similar SurrealQL SELECT patterns _(since v3.1.0)_ From SurrealDB 3.1.0, SurrealDB’s GraphQL layer uses **Apollo-style** names: a **pluralised list** field (for example `people` for table `person`), a **singular fetch** field `person(id: …)`, and `people_aggregate` for aggregates. Under the hood these map to SurrealQL-style reads (typically `SELECT`). The SurrealQL here is a rough equivalent for the same or similar data shape. See [GraphQL overview](/docs/learn/querying/graphql/overview.md#schema-naming-since-vv310-) for the full naming table. Before trying the examples, enable GraphQL and define data in the current namespace and database (see [GraphQL overview](/docs/learn/querying/graphql/overview.md) using [`DEFINE CONFIG GRAPHQL AUTO`](/docs/reference/query-language/statements/define/config.md)). The snippets below assume: - Namespace **`main`**, database **`main`** - Root authentication **`root`** / **`secret`** - A `person` table with `name` and `age`, and records `person:simon` and `person:marcus` as in [GraphQL via HTTP](/docs/learn/querying/graphql/via-http.md) ```surql title="Schema and sample data" DEFINE TABLE person SCHEMAFULL; DEFINE FIELD name ON TABLE person TYPE string; DEFINE FIELD age ON TABLE person TYPE number; CREATE person:simon SET name = "Simon", age = 23; CREATE person:marcus SET name = "Marcus", age = 28; DEFINE CONFIG GRAPHQL AUTO; ``` ## List records and choose fields GraphQL here returns a list of objects, similar to `SELECT` without `ONLY`. **GraphQL** ```graphql title="Query" query { people { name age } } ``` ```graphql title="Output" { "data": { "people": [ { "age": 28, "name": "Marcus" }, { "age": 23, "name": "Simon" } ] } } ``` **SurrealQL** ```surql title="Query" SELECT name, age FROM person; ``` ```surql title="Output" [ { age: 28, name: 'Marcus' }, { age: 23, name: 'Simon' } ] ``` **cURL (HTTP)** ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d '{ "query": "query { people { name age } }" }' http://localhost:8000/graphql ``` ## Fetch a single record by id Use **`person(id: …)`** with the **record key** (`simon`, not `person:simon` in the argument). The GraphQL `id` field on the result is the full record id. **GraphQL** ```graphql title="Query" query { person(id: "simon") { id name age } } ``` ```graphql title="Output" { "data": { "person": { "id": "person:simon", "name": "Simon", "age": 23 } } } ``` **SurrealQL** ```surql title="Query" SELECT * FROM ONLY person:simon; ``` ```surql title="Output" { age: 23, id: person:simon, name: 'Simon' } ``` **cURL (HTTP)** ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d '{ "query": "query { person(id: \"simon\") { id name age } }" }' http://localhost:8000/graphql ``` ## Limit how many records are returned GraphQL uses **`limit`** (and optional **`start`** for offset). SurrealQL uses `LIMIT` / `START` and `ONLY` to return a single record as opposed to an array containing a single record. **GraphQL** ```graphql title="Query" query { people(limit: 1) { name age } } ``` ```graphql title="Output" { "data": { "people": [ { "age": 28, "name": "Marcus" } ] } } ``` **SurrealQL** ```surql title="Query" SELECT name, age FROM ONLY person LIMIT 1; ``` ```surql title="Output" { age: 28, name: 'Marcus' } ``` **cURL (HTTP)** ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d '{ "query": "query { people(limit: 1) { name age } }" }' http://localhost:8000/graphql ``` ## Filter records GraphQL accepts **`filter`** or **`where`** with the generated input type for the table. For a scalar field, use comparison keys such as **`eq`**, **`ne`**, **`gt`**, and **`lt`** where the schema allows them. **GraphQL** ```graphql title="Query" query { people(where: { age: { eq: 23 } }) { name } } ``` ```graphql title="Output" { "data": { "people": [ { "name": "Simon" } ] } } ``` **SurrealQL** ```surql title="Query" SELECT name FROM person WHERE age = 23; ``` ```surql title="Output" [ { name: 'Simon' } ] ``` **cURL (HTTP)** ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d '{ "query": "query { people(where: { age: { eq: 23 } }) { name } }" }' http://localhost:8000/graphql ``` ## Order results Use an **`order`** argument with **`asc`** or **`desc`** and a field name. **GraphQL** ```graphql title="Query" query { people(order: { asc: age }) { name age } } ``` ```graphql title="Output" { "data": { "people": [ { "age": 23, "name": "Simon" }, { "age": 28, "name": "Marcus" } ] } } ``` **SurrealQL** ```surql title="Query" SELECT name, age FROM person ORDER BY name ASC; ``` ```surql title="Output" [ { age: 23, name: 'Simon' }, { age: 28, name: 'Marcus' } ] ``` **cURL (HTTP)** ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d '{ "query": "query { people(order: { asc: age }) { name age } }" }' http://localhost:8000/graphql ``` ## Next steps - Query the same endpoint from HTTP clients: [GraphQL via HTTP](/docs/learn/querying/graphql/via-http.md) - Optional: [Bruno](/docs/learn/querying/graphql/via-bruno.md) or [SurrealDB Studio](/docs/learn/querying/graphql/via-studio.md) For the full configuration surface (tables, functions, limits), see [`DEFINE CONFIG GRAPHQL`](/docs/reference/query-language/statements/define/config.md). --- Source: https://surrealdb.com/docs/learn/querying/graphql/via-bruno # Via Bruno In this section, you will explore querying SurrealDB using Bruno. Bruno is an API client that can send GraphQL queries to SurrealDB over HTTP. This page covers starting a server and querying it from Bruno. ## Getting started Before you can start making queries, you need to start SurrealDB. You can do this by starting a new instance of SurrealDB with the [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) command, Docker, or SurrealDB Studio. **CLI** ```bash surreal start --log debug --user root --password secret ``` **Docker** ```bash docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:nightly start -u root -p secret ``` ## Create a new Bruno collection Create a new collection by hand, or copy the following files into a new folder, for example `graphql-test`. If you are creating your collection copying the following files, make sure to create `bruno.json` with the following content: ```json { "version": "1", "name": "surrealdb graphql", "type": "collection", "ignore": [ "node_modules", ".git" ] } ``` Set authentication to basic, using your credentials from above (username: `root`, password: `secret`). Or save the following in a new file as `collection.bru`: ```text auth { mode: basic } auth:basic { username: root password: secret } ``` ## Create a script to populate the DB Save this file as `import.bru`. We are going to import this script using Bruno: ```text meta { name: import type: http seq: 2 } post { url: http://localhost:8000/import body: text auth: inherit } headers { Surreal-NS: test Surreal-DB: graphql Accept: application/json } body:text { DEFINE TABLE item SCHEMAFULL; DEFINE TABLE tag SCHEMAFULL; DEFINE TABLE container SCHEMAFULL; DEFINE TABLE space SCHEMAFULL; -- Enable GraphQL for the table DEFINE CONFIG GRAPHQL AUTO; -- Define some fields DEFINE FIELD name ON TABLE item TYPE string; DEFINE FIELD description ON TABLE item TYPE string; DEFINE FIELD space ON TABLE item TYPE record; DEFINE FIELD time ON TABLE item TYPE object; DEFINE FIELD time.createdAt ON TABLE item TYPE datetime; DEFINE FIELD tags ON TABLE item TYPE array; DEFINE FIELD tags.*.name ON TABLE item TYPE string; DEFINE FIELD tags.*.color ON TABLE item TYPE string; DEFINE FIELD name ON TABLE tag TYPE string; DEFINE FIELD name ON TABLE container TYPE string; DEFINE FIELD name ON TABLE space TYPE string; DEFINE TABLE is_in TYPE RELATION IN item | container OUT container | space; DEFINE TABLE tagged TYPE RELATION IN item OUT tag; -- Create tags CREATE tag:electronics CONTENT { name: 'Electronics' }; CREATE tag:tools CONTENT { name: 'Tools' }; CREATE tag:cleaning CONTENT { name: 'Cleaning' }; -- Create spaces CREATE space:home CONTENT { name: 'Home' }; -- Create containers LET $c_living = CREATE container CONTENT { name: 'Living room' } \ RETURN id; LET $c_desk = CREATE container CONTENT { name: 'Work desk' } RETURN id; LET $c_box = CREATE container CONTENT { name: 'Box 1' } RETURN id; LET $c_box_2 = CREATE container CONTENT { name: 'Small box 2' } \ RETURN id; RELATE $c_box_2->is_in->$c_box SET time = { updatedAt: time::now() \ }; RELATE $c_box->is_in->$c_desk SET time = { updatedAt: time::now() }; RELATE $c_desk->is_in->$c_living SET time = { updatedAt: time::now() }; RELATE $c_living->is_in->space:home SET time = { updatedAt: \ time::now() }; -- Create an item LET $item = CREATE item CONTENT { name: 'Lens wipes', description: 'box of lens wipes', space: space:home, tags: [{name: "comsumable", color: "#FF0000"}, {name: "cleaning", color: "#0000FF"}], time: { createdAt: time::now() } } RETURN id; RELATE $item->is_in->$c_box SET time = { updatedAt: time::now() }; RELATE $item->tagged->tag:cleaning; -- Create an item LET $item2 = CREATE item CONTENT { name: 'HDD', description: 'external hard drive samsung white', space: space:home, tags: [{name: "electronics", color: "#00FFFF"}], time: { createdAt: time::now() } } RETURN id; RELATE $item2->is_in->$c_box_2 SET time = { updatedAt: time::now() \ }; RELATE $item2->tagged->tag:electronics; } settings { encodeUrl: true timeout: 0 } ``` ## Create a script to query using GraphQL Save this one as `query.graphql` ```text meta { name: test type: graphql seq: 1 } post { url: http://localhost:8000/graphql body: graphql auth: inherit } headers { Surreal-NS: test Surreal-DB: graphql Accept: application/json } body:graphql { { item(filter: { name: { ne: "HDD" } }) { # -- example "not equals" filter name space { id } } } } settings { encodeUrl: true timeout: 0 } ``` ## Now open and run in Bruno Your collection folder should contain the following files: - `bruno.json` - `collection.bru` - `import.bru` - `query.bru` Open the collection in Bruno, run the "import" request, and then "query". You should then see a result like this: ```json { "data": { "item": [ { "name": "Lens wipes", "space": { "id": "space:home" } } ] } } ``` ## Troubleshooting - if you see this error: `InvalidRequest(NotConfigured)`, make sure you have included this line in the import `DEFINE CONFIG GRAPHQL AUTO`. --- Source: https://surrealdb.com/docs/learn/querying/graphql/via-http # Via HTTP In this section, you will explore querying SurrealDB using the GraphQL HTTP endpoint. The HTTP API is designed to be simple and intuitive, with any interface that provides a consistent way to interact with the database. SurrealDB provides a powerful HTTP API that allows you to interact with the database programmatically. This API can be used to perform a wide range of database operations, from querying data to modifying records and managing database structures. The HTTP API is designed to be simple and intuitive, with a RESTful interface that provides a consistent way to interact with the database. You can use the API to perform a wide range of database operations, from querying data to modifying records and managing database structures. ## Starting a new connection Before you can start making queries, you need to start SurrealDB with the GraphQL module enabled. You can do this by starting a new instance of SurrealDB with the [`surreal start` command](/docs/reference/cli/surrealdb-cli/commands/start.md). ```bash surreal start --log debug --user root --password secret ``` In order to allow querying the created table using GraphQL, you will need to explicitly enable GraphQL using the [`DEFINE CONFIG`](/docs/reference/query-language/statements/define/config.md) statement. This will allow you to query the table using GraphQL on a per-database basis. ## `POST /sql` To use the GraphQL API, you first need to enable it using the `DEFINE CONFIG` statement. This will allow you to query the table using GraphQL on a per-database basis. To do this, you can send a `POST` request to the `/sql` endpoint with a RAW body containing the `DEFINE CONFIG` statement. For example: ```surrealql title="Enabling GraphQL" DEFINE CONFIG GRAPHQL AUTO; ``` Here are three commands to define this configuration, along with some table and field definitions and sample data. They also assume a root user with the name `root` and the password `secret`. ```bash curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d 'DEFINE TABLE person SCHEMAFULL; DEFINE FIELD name ON TABLE person TYPE string; DEFINE FIELD age ON TABLE person TYPE number;' \ http://localhost:8000/sql curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d 'CREATE person:simon SET name = "Simon", age = 23; CREATE person:marcus SET name = "Marcus", age = 28;' \ http://localhost:8000/sql curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" \ -H "Accept: application/json" -d 'DEFINE CONFIG GRAPHQL AUTO' http://localhost:8000/sql ``` ## `POST /graphql` To use the GraphQL API, you can send a `POST` request to the `/graphql` endpoint with a JSON body containing the GraphQL query via Postman or any other HTTP client. For example, to query the `person` table for all records, you can send the following request: ```json curl -X POST -u "root:secret" -H "Surreal-NS: main" -H "Surreal-DB: main" -H "Accept: application/json" \ -d '{ "query": "query { person { name } }" }' http://localhost:8000/graphql ``` ```json title="Response" {"data":{"person":[{"name":"Marcus"},{"name":"Simon"}]}} ``` The GraphQL endpoint enables use of GraphQL queries to interact with your data. ### Headers
Header Description
Authorization Sets the root, namespace, database, or record authentication data
Accept Sets the desired content-type of the response
surreal-ns Sets the selected Namespace for queries
surreal-db Sets the selected Database for queries
### Example usage ```bash title="Request" curl -X POST \ -u "root:secret" \ -H "Surreal-NS: main" \ -H "Surreal-DB: main" \ -H "Accept: application/json" \ -d '{"query": "query { person(where: { age: { gt: 18 } }) { id name age } }"}' \ http://localhost:8000/graphql ``` ```json title="Response" [ { "time": "14.357166ms", "status": "OK", "result": [ { "age": "23", "id": "person:simon" "name": "Simon", }, { "age": "28", "id": "person:marcus" "name": "Marcus", }, ] } ] ``` --- Source: https://surrealdb.com/docs/learn/querying/graphql/via-studio # Via SurrealDB Studio In this section, you will explore querying SurrealDB using SurrealDB Studio. The GraphQL query view in [SurrealDB Studio](https://studio.surrealdb.com/query) provides syntax highlighting, query validation, and real-time execution, with results displayed as the JSON structure returned by GraphQL. ## Getting started Before you can start making queries, you need to start SurrealDB with the GraphQL module enabled. You can do this by starting a new instance of SurrealDB with the [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) command. ```bash surreal start --log debug --user root --password secret ``` After starting the SurrealDB instance, you can navigate to SurrealDB Studio to start a new connection. ### Start a new connection In the top left corner of the SurrealDB Studio, start a new connection. Ensure that the connection information is the same as the one you used to start the SurrealDB instance. In the example above we have set the user to `root` and the password to `secret`. > [!IMPORTANT] > Querying via GraphQL is not supported in the SurrealDB Studio sandbox. Learn more about starting a connection in the [SurrealDB Studio documentation](/docs/explore/studio.md). ### Setting a namespace and database Before you can start writing queries, you need to set the [namespace and database](/docs/learn/data-models/architecture.md#namespaces-and-databases) you want to use. For example, you can set the namespace to `test` and the database to `test`. This will set the namespace and database for the current connection. Additionally, you can start [a serving in SurrealDB Studio](/docs/explore/studio.md) which also enables GraphQL automatically, starting a server on `http://localhost:8000` by default for a root user with username and password `root`. SurrealDB Studio connection settings ### Preparing your database Next, use the [SurrealQL query editor](/docs/explore/studio.md) to create some data. For example, you can create a new `user` table with fields for `firstName`, `lastName`, and `email` and add a new user to the database. In order to allow querying the created table using GraphQL, you will need to explicitly enable GraphQL using the [`DEFINE CONFIG`](/docs/reference/query-language/statements/define/config.md) statement. This will allow you to query the table using GraphQL on a per-database basis. This must be followed by statements to explicitly define the resources to query. That is, you must use the [`DEFINE TABLE` statement](/docs/reference/query-language/statements/define/table.md) to define the table, and [`DEFINE FIELD` statement](/docs/reference/query-language/statements/define/field.md) to define the fields for the table. This is because GraphQL differs from SurrealDB itself in requiring resources to be defined before they can be used. ```surql title="Creating a user table" DEFINE TABLE user SCHEMAFULL; -- Enable GraphQL for the user table. DEFINE CONFIG GRAPHQL AUTO; -- Define some fields. Not strictly necessary for -- SurrealDB itself, but required for GraphQL DEFINE FIELD firstName ON TABLE user TYPE string; DEFINE FIELD lastName ON TABLE user TYPE string; DEFINE FIELD email ON TABLE user TYPE string ASSERT string::is_email($value); DEFINE INDEX userEmailIndex ON TABLE user FIELDS email UNIQUE; -- Create a new User CREATE user CONTENT { firstName: 'Jon', lastName: 'Doe', email: 'Jon.Doe@surrealdb.com', }; ``` ## Write your first GraphQL query After you have created some data, you can start writing GraphQL queries. You can use the [SurrealDB Studio GraphQL editor](/docs/explore/studio.md) to write your GraphQL queries. For example, to query the `person` table for all records, you can write the following GraphQL query: ```graphql { user { firstName lastName email } } ``` SurrealDB Studio GraphQL query And to get the person with the email "Jon.Doe@surrealdb.com", you can write the following GraphQL query: ```graphql { user(filter: {email: {eq: "Jon.Doe@surrealdb.com"}}) { firstName lastName } } ``` SurrealDB Studio will automatically validate the query and provide you with the results. ## Introspection SurrealDB Studio also supports introspection with GraphQL. This means that you can query the database and SurrealDB Studio will automatically infer the type of the data you are querying. For example, if you query the `user` table for all records, SurrealDB Studio will automatically infer the type of the data to be `user`. SurrealDB Studio GraphQL type inference ## Learn more To learn more about the GraphQL view in SurrealDB Studio, check out the [SurrealDB Studio documentation](/docs/explore/studio.md). --- Source: https://surrealdb.com/docs/learn/querying/performance/performance-best-practices # Performance best practices This guide outlines some key performance best practices for using SurrealDB. While SurrealDB offers powerful and flexible features to support you in meeting your desired performance standards, the use that you make of those features will ultimately determine whether or not you meet them. To achieve the best performance from SurrealDB, there are a number of configuration options and runtime design choices to consider. The following is a non-exhaustive list of best practices to help you address common performance challenges and avoid frequent pitfalls. ## SurrealDB architecture While SurrealDB is a [multi-model database](/blog/what-are-multi-model-databases), at its core, SurrealDB stores data in documents on transactional key-value stores. This means that SurrealDB is a general-purpose databases optimised for a combination of various workloads such as operational, AI and real-time workloads. While SurrealDB can perform well with real-time and advanced analytical workloads, its architecture is not a columnar one. As such, it is not optimised for large ad-hoc analytical queries in the same way as specialised columnar data warehouses. SurrealDB is built using a layered approach, with compute separated from the storage. This allows you, if necessary, to scale up the compute and storage layers independently from each other. Read more about [SurrealDB architecture](/docs/learn/data-models/architecture.md) and [deployment models](/docs/manage/self-hosted/deployment-models.md). ## Running SurrealDB ### Using SurrealDB Cloud The easiest way to deploy and scale up SurrealDB is by using SurrealDB Cloud, which allows you to focus on building great products while we take care of running and maintaining it in the most performant and scalable way. Read more about running SurrealDB [using SurrealDB Cloud ](/cloud). ### Running SurrealDB as a server When starting the SurrealDB server, it is important to run the server using the correct configuration options and settings. For production environments or for performance benchmarking, the `--log` command-line argument or the `SURREAL_LOG` environment variable should be set to `info` (the default option when not specified), `warn`, or `error`. Other log verbosity levels (such as `debug`, `trace`, or `full`) are only recommended for use in debugging, testing, or development scenarios. This is because verbosity of the log level impacts the performance by increasing the amount of information being logged for each single operation. The same applies for the SurrealDB Docker container, in which the `--log` argument should be omitted or specifically set to a log verbosity level that does not impact performance. ```sh docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start --log info rocksdb://path/to/mydatabase ``` Read more about running SurrealDB as a [single-node server](/docs/running/file-backed.md) or [multi-node cluster](/docs/running/multi-node.md). For performance comparisons between storage engines and deployment modes, see the [SurrealDB benchmarks](https://surrealdb.com/benchmarks). Additionally, ensure that the `rocksdb` storage engine is used to store data. ```sh surreal start --log info rocksdb://path/to/mydatabase ``` ### Running SurrealDB embedded in Rust It is common knowledge among Rust developers that the `--release` flag is used to ensure that a build is optimised for performance as opposed to compilation speed. However, the SurrealDB source code also contains a few additional flags when the `--release` flag is passed in as seen below. As this configuration will not be present by default inside the `Cargo.toml` for your own project when adding the `surrealdb` dependency, be sure to add it if performance is crucial. ```toml [profile.release] codegen-units = 1 lto = true opt-level = 3 panic = 'abort' strip = true ``` In addition, using the correct memory allocator can greatly improve the performance of the database core engine when running SurrealDB as an embedded database within Rust. Using an optimised asynchronous runtime configuration can also help speed up concurrent queries and increase database throughput. In your project's `Cargo.toml` file, ensure that the `allocator` feature is enabled on the `surrealdb` dependency: ```toml surrealdb = { version = "2", features = ["allocator", "storage-mem", \ "storage-surrealkv", "storage-rocksdb", "protocol-http", \ "protocol-ws", "rustls"] } ``` When running SurrealDB within your Rust code, ensure that the asynchronous runtime is configured correctly, making use of multiple threads, an increased stack size, and an optimised number of threads: ```toml tokio = { version = "1.49.0", features = ["sync", "rt-multi-thread"] } ``` ```rs fn main() { tokio::runtime::Builder::new_multi_thread() .enable_all() .thread_stack_size(10 * 1024 * 1024) // 10MiB .build() .unwrap() .block_on(async { // Your application code }) } ``` Read more about [running SurrealDB embedded in Rust.](/docs/build/embedding/by-language/rust.md) ### Running SurrealDB embedded in Tauri When running SurrealDB as an embedded database within Rust, default options of Tauri can make SurrealDB run slower, as it processes and outputs the database information logs. Configuring Tauri correctly, can result in much improved performance with the core database engine and any queries which are run on SurrealDB. When building a desktop application with Tauri, ensure that the Tauri plugin log is disabled by configuring the `tauri.conf.json` file: ```json { "plugins": { "logger": { "enabled": false } } } ``` Alternatively you can disable logs at compile time when building your Tauri app: **Bash** ```bash TAURI_LOG_LEVEL=off cargo tauri build ``` **PowerShell** ```powershell $env:TAURI_LOG_LEVEL = "off" cargo tauri build ``` ## Performing queries ### Selecting single records Certain queries in SurrealDB can be more efficiently written in certain ways which ensure that full table scans or indexes are not necessary when executing the query. In traditional SQL, the following query can be used to query for a particular row from a table: ```surql SELECT * FROM user WHERE id = 19374837491; ``` In SurrealDB 3.0 and above, the query planner is able to identify this as a record ID scan if the `id` field after the `WHERE` clause is a record ID. ```surql SELECT * FROM user WHERE id = user:19374837491; ``` However, using a `WHERE` clause will always perform a table scan in versions of SurrealDB before 3.0. If this is the case, just remove the `WHERE` clause and select directly from the record ID itself. ```surql SELECT * FROM user:19374837491; ``` ### Selecting multiple records In traditional SQL, the following queries can be used to query for getting particular rows from a table: ```surql -- Selecting individual IDs SELECT * FROM user WHERE id = 19374837491 OR id = 12647931632; ``` ```surql -- Selecting a range of IDs SELECT * FROM user WHERE id >= 12647931632 AND id <= 19374837491; ``` However, currently in SurrealDB this query will perform a scan to find the record, although this is not necessary and you don't need to index the id field when using SurrealDB. Instead the following query can be used to select the specific record without needing to perform any scan: ```surql -- Selecing indiviudal IDs SELECT * FROM user:19374837491, user:12647931632; ``` ```surql -- Selecting a range of IDs SELECT * FROM user:12647931632..=19374837491; ``` ### Simplifying logic in `WHERE` clauses If a `WHERE` clause cannot be avoided, performance can still be improved by optimising the portion after the `WHERE` clause. As a boolean check is the simplest possible operation, having a boolean field that can be used in a `WHERE` clause can significantly improve performance. ```surql DEFINE FIELD data_length ON person VALUE random_data.len(); DEFINE FIELD is_short ON person VALUE random_data.len() < 10; -- Fill up the database a bit with 10,000 records CREATE |person:10000| SET random_data = rand::string(1000) RETURN NONE; -- Add one outlier with short random_data CREATE person:one SET random_data = "HI!" RETURN NONE; -- Function call + compare operation: slowest SELECT * FROM person WHERE random_data.len() < 10; -- Compare operation: much faster SELECT * FROM person WHERE data_length < 10; -- Boolean check: even faster SELECT * FROM person WHERE is_short; -- Direct record access: almost instantaneous SELECT * FROM person:one; ``` ## Using indexes SurrealDB has native built-in support for a number of different index types, without leveraging external libraries or implementations. With native support for indexes in the core database engine, SurrealDB leverages indexes where possible within the SurrealQL query language, without pushing queries down to a separate indexing engine. In addition, data is indexed in the same way for embedded systems, single-node database servers, and multi-node highly-available clusters, ensuring that the same indexing functionality is available regardless of the SurrealDB deployment model. Indexing support in SurrealDB is in active development, with work focusing on increased support for additional operators, compound indexes, additional index types, and overall improved index performance. > [!NOTE] > Currently no indexes are used when performing `UPDATE` or `DELETE` queries on large table, even where indexes are defined. > We'll be adding support for indexes within `UPDATE`, `UPSERT`, and `DELETE` statements in SurrealDB release `v2.3.0`. In the meantime, you can improve the performance of `UPDATE` and `DELETE` statements by combining these with a `SELECT` statement. To improve the performance of an `UPDATE` statement, use a `SELECT` statement within a subquery, selecting only the `id` field. This will use any defined indexes: ```surql UPDATE (SELECT id FROM user WHERE age < 18) SET adult = false; ``` To improve the performance of an `DELETE` statement, use a `SELECT` statement within a subquery, selecting only the `id` field. This will use any defined indexes: ```surql DELETE (SELECT id FROM user WHERE age < 18); ``` ### Index strategies explained When using `SELECT`, SurrealDB uses a query planner whose role is to identify if it can use the index to speed the execution of the query. Without indexes, SurrealDB will operate a `SELECT` query on a table by using the table iterator. It mainly scans every record of a given table. If there is a condition (`WHERE ...`), an ordering (`ORDER BY ...`), or an aggregation (`GROUP BY ...`), it will load the value in memory and execute the operation. This process is commonly called a "table full scan". ```surql SELECT * FROM user WHERE age < 18 EXPLAIN; ``` ```surql title="Output" [ { detail: { table: 'user' }, operation: 'Iterate Table' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` Under certain conditions, if an index exists, and the condition or ordering involves exclusively fields that are indexed, the query planner will suggest an execution plan that involves one or multiple indexes to achieve these potential optimisations: - Only collect records that match the condition(s), as opposed to performing a table full scan. - As the index already stores the records in order, the scanning collects the records pre-ordered, sparing an additional ordering phase. ```surql DEFINE INDEX idx_user_age ON user FIELDS age; SELECT age FROM user WHERE age > 18 EXPLAIN; ``` ```surql title="Output" [ { detail: { plan: { from: { inclusive: false, value: 18 }, index: 'idx_user_age', to: { inclusive: false, value: NONE } }, table: 'user' }, operation: 'Iterate Index' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` If there are several clauses separated with `OR` operators, the query planner may do several index-based iterations: ```surql SELECT age FROM user WHERE age < 7 OR age > 77 EXPLAIN; ``` ```surql title="Output" [ { detail: { plan: { from: { inclusive: false, value: NONE }, index: 'idx_user_age', to: { inclusive: false, value: 7 } }, table: 'user' }, operation: 'Iterate Index' }, { detail: { plan: { from: { inclusive: false, value: 77 }, index: 'idx_user_age', to: { inclusive: false, value: NONE } }, table: 'user' }, operation: 'Iterate Index' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` ### Use `UPSERT` to take advantage of unique indexes `UPSERT` statements have a unique performance advantage when paired with a unique index. A unique index on its own is used to prevent more than one record from containing the same data, such as a name or email address. ```surql DEFINE INDEX email_index ON user FIELDS email UNIQUE; CREATE user SET email = "bob@bob.com"; CREATE user SET email = "bob@bob.com"; ``` ```surql title="Output" "Database index `email_index` already contains 'bob@bob.com', with record `user:g7s070gqvh3lj7fdp26w`" ``` An `UPSERT` statement works like a `CREATE` statement in this case as well, except that if the value for `email` is already present, it will modify the existing record instead of creating a new one. An `UPSERT` will only fail in this case if a user attempts to upsert to a certain record ID (like `user:bob` instead of just the `user` table) when another record holds this value. The key point here is that in either case, `UPSERT` is using the index to find the record instead of a table scan. ```surql DEFINE INDEX email_index ON user FIELDS email UNIQUE; CREATE user SET email = "bob@bob.com"; -- Checks index, finds existing user via email "bob@bob.com", modifies it UPSERT user SET email = "bob@bob.com", name = "Bob Bobson"; -- Checks index, fails as a new `user:bob` cannot be created with the same email UPSERT user:bob SET email = "bob@bob.com", name = "Bob Bobson"; ``` As such, when updating a single record on a table that contains a unique index, `UPSERT` is much more performant than `UPDATE`. ```surql DEFINE INDEX email_index ON user FIELDS email UNIQUE; -- Create 50,000 users to fill up the database CREATE |user:50000| RETURN NONE; -- Create Bob CREATE user SET email = "bob@bob.com"; -- Don't do this: full table scan to find and update a record UPDATE user SET name = "Bob Bobson" WHERE email = "bob@bob.com"; -- Do this instead: use the index instead to go directly to the record, no table scan UPSERT user SET name = "Bob Bobson", email = "bob@bob.com"; ``` ### Index lookup on remote fields SurrealDB document record IDs store both the table name and the record identifier. This design provides a straightforward and consistent way to reference records across the database. One particularly powerful feature is the ability to filter a table based on conditions that relate to a referenced table. Here is a concrete example, where the statement `SELECT * FROM access WHERE user.role = 'admin'` will retrieve records from the `access` table for which the referenced record in the `user` table has the `name` field set to 'admin'. Consider the following example: ```surql DEFINE FIELD user ON TABLE access TYPE record; CREATE user:1 SET name = 'foo', role = 'admin'; CREATE user:2 SET name = 'bar', role = 'admin'; CREATE access:A SET user = user:1; CREATE access:B SET user = user:2; SELECT * FROM access WHERE user.role = 'admin' ``` The query retrieves records from the `access` table whose associated record in the `user` table has the role `field` set to 'admin'. ```surql title="Output" [ { id: access:A, user: user:1 }, { id: access:B, user: user:2 } ] ``` To optimise this query, you can create indexes on both the `user.role` field and the `access.user` field. With these indexes, the query planner can leverage an index-based join strategy: ```surql DEFINE INDEX idx_user_role ON TABLE user FIELDS role; DEFINE INDEX idx_access_user ON TABLE access FIELDS user; SELECT * FROM access WHERE user.role = 'admin' EXPLAIN; ``` ```surql title="Output" [ { detail: { plan: { index: 'idx_access_user', joins: [ { index: 'idx_user_role', operator: '=', value: 'admin' } ], operator: 'join' }, table: 'access' }, operation: 'Iterate Index' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` --- Source: https://surrealdb.com/docs/learn/querying/real-time/changefeeds # Changefeeds Replaying table changes over time with changefeeds and SHOW CHANGES - useful for sync, pipelines, and external systems. Changefeeds are how you ask SurrealDB for a history of writes to a table (or database) so you can replay them elsewhere: search indexes, warehouses, cache warmers, or any system that needs to catch up after downtime. You define how long changes are retained when you set up the feed by adding a duration like `3d` or `100h`. Changes are then read with the [`SHOW`](/docs/reference/query-language/statements/show.md) statement. ## What you need first Before `SHOW CHANGES` returns anything useful, the table or database must opt in with a `CHANGEFEED` definition (duration, scope). Changefeeds are usually defined using a `DEFINE TABLE` statement, but can be defined on the entire database if desired via `DEFINE DATABASE`. ## Replaying from a point in time or a versionstamp A typical flow is: 1. Note when you last processed changes (a datetime) or where you stopped (a versionstamp). 2. Ask for everything since that marker, optionally with a limit so you page through large backlogs. The [`SHOW` statement reference](/docs/reference/query-language/statements/show.md) spells out the full syntax. In practice you are choosing: “give me the next batch of mutations after this cursor.” ```surql -- Define the changefeed and its duration DEFINE TABLE reading CHANGEFEED 3d; -- Create some records in the reading table CREATE reading SET story = "Once upon a time"; CREATE reading SET story = "there was a database"; -- Replay changes to the reading table since a date SHOW CHANGES FOR TABLE reading SINCE d"2025-09-07T01:23:52Z" LIMIT 10; -- Replay changes to the reading table since a versionstamp SHOW CHANGES FOR TABLE reading SINCE 1 LIMIT 10; ``` If the datetime is after the changefeed was created and lines up with your retention window, you get a list of entries, each with `changes` and a `versionstamp` (where you are in the log). How each mutation is encoded depends on your [`DEFINE TABLE ... CHANGEFEED`](/docs/reference/query-language/statements/define/table.md#example-usage) options; when differences are stored (`INCLUDE ORIGINAL`), an update to an existing row carries a **reverse diff** from the current state to the state immediately before that write. The [`SHOW`](/docs/reference/query-language/statements/show.md) reference covers this in full. The following shape is representative: ```surql title="Output" [ { changes: [ { define_table: { changefeed: { expiry: 3d, original: false }, drop: false, id: 0, kind: { kind: 'ANY' }, name: 'reading', permissions: { create: false, delete: false, select: false, update: false }, schemafull: false } } ], versionstamp: 116372747574247424 }, { changes: [ { update: { id: reading:1up24usipwi7wjs4nbgk, story: 'Once upon a time' } } ], versionstamp: 116372747574247425 }, { changes: [ { update: { id: reading:6q3845opi3nikxr7jvgg, story: 'there was a database' } } ], versionstamp: 116372747574312960 } ] ``` --- Source: https://surrealdb.com/docs/learn/querying/real-time/live-queries # Live queries Subscribing to inserts, updates, and deletes with LIVE SELECT so clients stay in sync over a persistent connection. Live queries are SurrealDB's way to push changes to a client. Running a `LIVE SELECT` opens a subscription that sends notifications whenever matching records appear, change, or disappear, without needing to directly poll the database to do so. That fits interactive UIs, dashboards, and small collaboration features. For replaying history or catching up a batch pipeline, use [changefeeds](/docs/learn/querying/real-time/changefeeds.md) and `SHOW CHANGES` instead. The full grammar and edge cases live under [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md) in the reference. Below is how to think about subscriptions and what to watch for in production. ## Starting a subscription By default, creates and updates deliver the full record; deletes return nothing unless you add a clause like `RETURN BEFORE`. You receive a UUID as soon as the live query is registered. This UUID can later be passed in to a [`KILL`](/docs/reference/query-language/statements/kill.md) statement when you want to stop. ```surql LIVE SELECT * FROM person; ``` ```surql title="Output" u'b1f1d115-ad0f-460d-8cbf-dbc7ce48851c' ``` Message layout on the wire is described in the [live query / WebSocket protocol](/docs/reference/rest-api/rpc-protocol.md#live) section. ## DIFF mode If you prefer patches instead of whole documents on update, use `DIFF`. Updates arrive as [JSON Patch](https://jsonpatch.com/)-style arrays, which can be smaller and easier to merge on the client when records are large. ```surql LIVE SELECT DIFF FROM person; ``` ```surql title="Output" 'b87cbb0d-ca15-4f0a-8f86-caa680672aa5' ``` ## Filtering with WHERE You can narrow the subscription the same way you would a normal `SELECT`: only rows that match your predicate participate. ```surql LIVE SELECT * FROM person WHERE age > 18; ``` ## Consistency and security Notifications reflect committed work, so a live event will not take place if a transaction is rolled back. Under heavy concurrency, the system makes a best effort to preserve sensible ordering, but you should not assume a total order across all writers that matches commit order in every edge case. ## Parameters in live queries From v3.0.0 onwards you can use session parameters in live queries, such as by binding names with `LET` before you open the subscription. ```surql LET $table = 'measurement'; LET $location = 'Tallinn'; LIVE SELECT * FROM type::table($table) WHERE location == $location; ``` ## Where to read more * [`LIVE SELECT` reference](/docs/reference/query-language/statements/live-select.md) * [Changefeeds](/docs/learn/querying/real-time/changefeeds.md) - replay and `SHOW CHANGES` * [Real-time best practices](/docs/learn/querying/real-time/real-time-best-practices.md) - wider event-driven patterns --- Source: https://surrealdb.com/docs/learn/querying/real-time/real-time-best-practices # Real-time best practices This guide outlines some best practices for using SurrealDB in an event-driven and real-time manner. This page details some of the patterns you'll want to use when using SurrealDB when time is of the essence, reacting autonomously to changes in the database is required, or when you want to query on aggregated data updated automatically over time. ## Temporal querying and record ranges Temporal querying is generally done to answer two questions: what/where, and when? For example: * What is the weather like in London at the timestamp `2026-01-28T01:02:56Z`? * What is the weather like in London between the timestamps `2026-01-28T00:00:00Z` (28 January 2026) and `2026-01-29T00:00:00Z` (the next day)? Databases that store this sort of data tend to be fairly massive. Data on weather, traffic, user activity, and anything else that holds one record per event at a certain point in time very quickly builds up. SurrealQL has a built-in method to keep querying time down in cases like these: a complex record ID made out of an array. ```surql CREATE weather:["London", time::now()] SET temperature = 9.0; CREATE weather:["London", time::now() + 1d] SET temperature = 8.0; ``` To see what makes an array special in a case like this, let's compare it with a more general approach that holds the time and location inside regular fields instead of the ID itself. We'll call this table `weather2` to differentiate one between the other. ```surql CREATE weather2 SET location = "London", at = time::now(), temperature = 9.0; CREATE weather2 SET location = "London", at = time::now() + 1d, temperature = 8.0; ``` To see all the data for London between yesterday and tomorrow, you would add a number of `WHERE` clauses. ```surql SELECT * FROM weather2 WHERE location = "London" AND at > time::now() - 1d AND at < time::now() + 1d; ``` The query works just fine, but let's now add the `EXPLAIN` clause to the end of it to see how the operation was performed. ```surql [ { detail: { direction: 'forward', table: 'weather2' }, operation: 'Iterate Table' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` The 'Iterate Table' part here means that the operation iterated through the entire table to find matching records. This is fine to a certain extent, but becomes less efficient as other data for times and locations begins to build up. To speed this up, an index can be added. ```surql DEFINE INDEX weather ON weather2 FIELDS location, at; ``` With the index added, a query followed by `EXPLAIN` will show that we now iterated over the index, which is much more efficient. ```surql [ { detail: { plan: { index: 'weather', prefix: [ 'London' ], ranges: [ { operator: '>', value: d'2026-01-27T02:02:18.367Z' }, { operator: '<', value: d'2026-01-29T02:02:18.367Z' } ] }, table: 'weather2' }, operation: 'Iterate Index' }, { detail: { type: 'Memory' }, operation: 'Collector' } ] ``` As you can see, there is nothing at all with using an index - and this may very well fit your own use case. But let's now compare it to the approach that uses a complex record ID. ```surql CREATE weather:["London", time::now()] SET temperature = 9.0; CREATE weather:["London", time::now() + 1d] SET temperature = 8.0; ``` Technically, you could use the same query as before with a `WHERE` clause on each of the parts of the array-based ID. But that would result in another full table iteration. ```surql SELECT * FROM weather WHERE id[0] = "London" AND id[1] > time::now() - 1d AND id[1] < time::now() + 1d; ``` Instead, we can query over a range of IDs: all the IDs that fall in between the starting point `["London", time::now() - 1d]` and the end point `["London", time::now() + 1d]`. ```surql SELECT * FROM weather:["London", time::now() - 1d]..=["London", time::now() + 1d]; ``` This is what is known as a record range query, or a table partition scan. It works by querying over just the fraction of record IDs that fall within this range, instead of every record in the table. And you don't need to define a separate index for it because a record ID is a direct pointer to the data itself. If you imagine a database that holds thousands of weather observations over time for thousands of cities, the space between `weather:["London", time::now() - 1d]` and `["London", time::now() + 1d]` is just a very tiny slice! Querying on a partition this small is like going into a large bookstore knowing that the books you want are all in section C7 as opposed to walking through the entire place looking at each book along the way. We can demonstrate the difference in performance by adding 100,000 random records using both formats over a period of time between now and one year ago. To make the output nice, we'll use a little bit of SurrealQL magic. First we'll put the two queries creating the records inside their own scope, and use the [value::chain()](/docs/reference/query-language/functions/database-functions/value.md#chain) function to grab the output, ignore it, and turn it into a string showing what operation has just been performed. Then we'll do the same for a regular `SELECT` query over the `weather2` data, compared to the record range query for the `weather` data. If you need the query to stop when a step does not look right, use [`.expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) on that value instead of `.chain()` to return the original value when your closure returns `true`, otherwise failing with an error. ```surql { -- 100,000 weather records FOR $_ IN 0..100000 { CREATE weather:["London", time::now() - rand::duration(0ns, 1y)] SET temperature = 9.0; }; -- 100,000 weather2 records FOR $_ IN 0..100000 { CREATE weather2 SET location = "London", at = time::now() - rand::duration(0ns, 1y), temperature = 9.0; }; }.chain(|$_| "Sample data added!"); "Regular select" + (SELECT * FROM weather2 WHERE location = "London" AND at > time::now() - 1d AND at < time::now() + 1d).chain(|$_| ""); "Record range select" + (SELECT * FROM weather:["London", time::now() - 1d]..=["London", time::now() + 1d]).chain(|$_| ""); ``` You should see a result showing that the record range is over a hundred times faster, which makes sense as it is only iterating over a surface area about 1/365th the size of the one that uses a full table scan. ## ULIDs and UUIDs The standard record ID for SurrealDB is twenty characters long and composed of underscore letters and numbers. ```surql qrfz62eovom9f6p9j0fk ``` If you want a record ID with temporal information you might want to try using a datetime, but datetimes can't be used as IDs in SurrealQL. ```surql CREATE weather SET id = time::now(); -- Won't work ``` Technically you can cheat the system a little bit by setting the ID to a stringified datetime, or an array with a single datetime. ```surql CREATE weather SET id = time::now(); CREATE weather:[time::now()]; ``` However, when a record ID is composed of only a datetime, there is a very slight chance that the ID won't be unique. Though datetimes have nanosecond precision, you might be using SurrealDB on a system that rounds its datetimes to the millisecond or microsecond and creating multiple records at the same time may result in an error from two records having the same ID. Fortunately, there is another method: you can set the ID of the record to be a ULID or a UUID. Both of these contain the datetime at which they were created, but are always unique and have no chance of collision. ```surql LET $now = time::now(); CREATE weather:uuid() SET location = "London"; CREATE weather:ulid() SET location = "London"; ``` A ULID and UUID can be created from a datetime, and both ULIDs and UUIDs can be turned back into one as well. ```surql LET $uuid = rand::uuid(d'1997-08-29'); time::from_uuid($uuid); //- d'1997-08-29T00:00:00Z' LET $ulid = rand::ulid(d'1997-08-29'); time::from_ulid($ulid); //- d'1997-08-29T00:00:00Z' ``` However, because datetimes have nanosecond precision but ULIDs and UUIDs have millisecond precision, a roundtrip from datetime to ULID/UUID and back will not be exactly the same as the original datetime. To remove the precision from the original datetime, you can use the [`time::floor()`](/docs/reference/query-language/functions/database-functions/time.md#timefloor) function. ```surql LET $now = time::now(); LET $ulid = rand::ulid($now); LET $now_again = time::from_ulid($ulid); $now == $now_again; -- false time::floor($now, 1ms) = $now_again; -- true ``` ## Live queries If you don't want to make a query every time you want to see the latest updates to a table, you can use a [live query](/docs/reference/query-language/statements/live-select.md) instead. Since results for live queries show up the moment a record is created or updated, you don't necessarily need a complex ID - though either way works. ```surql -- Just used for live queries? WHERE location = "London" is fine LIVE SELECT * FROM weather WHERE location = "London"; CREATE weather SET location = "London", temperature = 9.0; -- Querying 'weather' manually too? Probably opt for an array-based ID LIVE SELECT * FROM weather WHERE id[0] = "London"; CREATE weather:["London", time::now()] SET temperature = 9.0; ``` A `LIVE SELECT` will always return a UUID, like this one. ```surql u'20372f84-714c-412b-b259-c8703574d4f1' ``` If making a live query inside SurrealDB Studio, you'll also see a notification asking you if you would like to move to live mode to see the changes to a table as they come in. After clicking on this, you can open up another window to make queries (or use the CLI, an SDK, or anything else) and watch the events as they come in. Live query results can be listened for via SDKs as well. Here is one example showing how to listen for results on the `weather` table using the Rust SDK. ```rust use futures::StreamExt; use surrealdb::engine::any::connect; use surrealdb::opt::auth::Root; use surrealdb::types::{RecordId, SurrealValue, ToSql}; #[derive(Debug, SurrealValue)] struct Weather { id: RecordId, temperature: f64, location: String, } #[tokio::main] async fn main() -> surrealdb::Result<()> { let db = connect("ws://localhost:8000").await?; db.signin(Root { username: "root".to_string(), password: "secret".to_string(), }) .await?; db.use_ns("main").use_db("main").await?; let mut weather = db.select("weather").live().await?; while let Some(result) = weather.next().await { println!("Got something! {result:?}"); match result { Ok(data) => { let weather: Weather = data.data; println!("{}", weather.into_value().to_sql()); } Err(error) => eprintln!("{error}"), } } Ok(()) } ``` ## Defining events Defining events is a bit similar to live queries, except that this time we are able to have the database automatically respond on its own instead of requiring an external listener to take care of it. If everything that you need to respond to an event can be done at the database level, then defining an event is the best way to handle it. Here is one example of an event that not only creates an `alert` whenever a weather condition is set to critical, but can even use an [http function](/docs/reference/query-language/functions/database-functions/http.md) to let an external service know about it. ```surql DEFINE EVENT alert ON weather WHEN severity = "critical" THEN { LET $alert = CREATE ONLY alert SET at = time::now(), body = "Alert! " + $input.conditions + " in " + $input.location RETURN VALUE body; http::post('https://jsonplaceholder.typicode.com/posts/', { body: $alert }); }; CREATE weather:uuid() SET location = "London", severity = "critical", conditions = "Too many tourists"; SELECT * FROM alert; ``` Note that you will need to use the [--allow-net](/docs/learn/security/authorization/capabilities.md#network) flag when starting up a SurrealDB instance to allow functions like `http::post()` to work, as they are disabled by default. ### When and how not to use an event Generally it is best to keep event logic within a single event that does not itself lead to another event. For an extreme example, take the following event that creates a record that triggers another event, which itself triggers the first event. ```surql DEFINE EVENT goes_forever ON weather THEN { CREATE other; }; DEFINE EVENT also_goes_forever ON other THEN { CREATE weather; }; CREATE weather; ``` Doing so will not cause the database to freeze, but instead will quickly lead to a maximum computation depth after which the query will fail. This is thanks to the fact that every operation in SurrealDB is done inside its own transaction. Since each event triggered by the event before is part of the original transaction, the entire operation is cancelled and rolled back. Note that the error output will show the chain of events at each depth before the limit was reached. > 'Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Error while processing event goes_forever: Error while processing event also_goes_forever: Reached excessive computation depth due to functions, subqueries, or computed values' If you find yourself using an event to update an aggregate like in the example below, a table view - introduced in the next section - might be what you are looking for. ```surql CREATE sum SET date = "2026-01-29", amount = 0; DEFINE EVENT update_sum ON purchase WHEN $event = "CREATE" THEN { UPSERT sum SET date = $after.date, amount += $after.amount WHERE date = $after.date; }; CREATE purchase SET amount = 100, date = "2026-01-29"; CREATE purchase SET amount = 200, date = "2026-01-29"; CREATE purchase SET amount = 200, date = "2026-01-30"; SELECT * FROM sum; -- Output: [ { amount: 300, date: '2026-01-29', id: sum:m4qv5iqpr9lfmpe0sgqh }, { amount: 200, date: '2026-01-30', id: sum:vc7dxvrj5mdmm75bbqs5 } ] ``` ## Table views Sometimes you might not want either a real-time query or an instantaneous reaction to an event, but prefer instead to have an aggregate of all the data in a certain table whenever you need to know it. This can be done by defining a separate table as a `SELECT` expression preceded by the `AS` clause. The table that it draws from can be a regular table, like in the `purchase` example above. Alternatively, it can be followed by the `DROP` keyword if you only want to use it as a table view and don't need to query it directly. ```surql DEFINE TABLE purchase; -- DEFINE TABLE purchase DROP <- Use DROP if you never want to query 'purchase' DEFINE TABLE sum AS SELECT math::sum(amount) AS amount, date FROM purchase GROUP BY date; CREATE purchase SET amount = 100, date = "2026-01-29"; CREATE purchase SET amount = 200, date = "2026-01-29"; CREATE purchase SET amount = 200, date = "2026-01-30"; SELECT * FROM sum; ``` With the table view set up, you don't need to do anything but select from `sum` to see the aggregated results. ```surql [ { amount: 300, date: '2026-01-29', id: sum:[ '2026-01-29' ] }, { amount: 200, date: '2026-01-30', id: sum:[ '2026-01-30' ] } ] ``` The function `math::sum()` is one of a number of functions that can be used both on its own and as an aggregate function. For a full list of aggregate functions, see [this page](/docs/reference/query-language/functions/database-functions.md#aggregate-functions). ### Using predictable IDs for table views One nice thing about table views is how predictable the IDs of the grouped values can be. Take the following example which has a `traffic_snapshot` table that holds a timestamp along with the number of cars and trucks at a certain location. On top of it we have a table called `traffic` that groups the results according to hourly intervals by using the `time::format()` function to turn a timestamp into a string output like `'2026-01-28:06:00:00'`. ```surql DEFINE TABLE traffic_snapshot; DEFINE TABLE traffic AS SELECT location, time::format(at, "%Y-%m-%d:%H:00:00") AS at, math::sum(cars) AS cars, math::sum(trucks) AS trucks FROM traffic_snapshot GROUP BY location, at; CREATE traffic_snapshot SET location = "53rd St", at = time::now(), cars = 10, trucks = 3; CREATE traffic_snapshot SET location = "53rd St", at = time::now() + 30m, cars = 20, trucks = 5; CREATE traffic_snapshot SET location = "53rd St", at = time::now() + 1h, cars = 50, trucks = 10; CREATE traffic_snapshot SET location = "53rd St", at = time::now() + 1h30m, cars = 34, trucks = 6; ``` Because the table view groups by `location` and `at`, we know that that will be the format of the `traffic` record IDs. As such, if you want to know what the traffic was like for a certain day, you can just pull it directly from the record ID - an operation which is close to instantaneous. ```surql SELECT * FROM traffic:['53rd St', '2026-01-28:06:00:00']; SELECT * FROM traffic:['53rd St', '2026-01-28:07:00:00']; ``` ```surql title="Output" -------- Query -------- [ { at: '2026-01-28:06:00:00', cars: 70, id: traffic:[ '53rd St', '2026-01-28:06:00:00' ], location: '53rd St', trucks: 15 } ] -------- Query -------- [ { at: '2026-01-28:07:00:00', cars: 34, id: traffic:[ '53rd St', '2026-01-28:07:00:00' ], location: '53rd St', trucks: 6 } ] ``` ## Combining real-time and event-driven functionality In practice, you will probably want to combine many or even all of the approaches mentioned above. For example, you could have a schema that uses an array-based ID for quick record range queries, a defined event to respond to high volumes of traffic, and a table view to see aggregate traffic for certain units of time. ```surql -- Define an event DEFINE EVENT alert ON traffic_snapshot WHEN $after.cars + $after.trucks > 50 THEN { LET $alert = CREATE ONLY alert SET message = "Heavy traffic at " + $after.id[0] + " with " + ($after.cars + $after.trucks) + " vehicles reported at " + time::now(); http::post('https://jsonplaceholder.typicode.com/posts/', { body: { message: $alert.message } }); }; -- Define a table view DEFINE TABLE traffic AS SELECT id[0] AS location, time::format(id[1], "%Y-%m-%d:%H:00:00") AS at, math::sum(cars) AS cars, math::sum(trucks) AS trucks FROM traffic_snapshot GROUP BY location, at; -- Use an ID ideal for record range queries CREATE traffic_snapshot:["53rd St", time::now() - 1h30m] SET cars = 34, trucks = 6; CREATE traffic_snapshot:["53rd St", time::now() - 1h] SET cars = 50, trucks = 10; CREATE traffic_snapshot:["53rd St", time::now() - 30m] SET cars = 20, trucks = 5; CREATE traffic_snapshot:["53rd St", time::now()] SET cars = 10, trucks = 3; SELECT * FROM traffic_snapshot:["53d St", time::now() - 1h]..; SELECT * FROM alert; SELECT * FROM traffic; ``` --- Source: https://surrealdb.com/docs/learn/querying/surrealql/executing-queries # Executing queries In this section, you will explore methods to query data in SurrealDB using SurrealQL, GraphQL or any of the available SDKs. This allows you to retrieve, filter, and manipulate data efficiently and in the best way for your use case. SurrealDB supports multiple query interfaces: [SurrealQL](/docs/learn/querying/surrealql/what-is-surrealql.md) for full-featured database queries, [GraphQL](/docs/learn/querying/graphql/overview.md) for schema-driven access, and any of the [available SDKs](/docs/languages.md) for language-native integration. ## SurrealQL [SurrealQL](/docs/reference/query-language.md) is our powerful database query language that closely resembles traditional SQL but comes with unique differences and improvements. Designed to provide developers with an intuitive way to interact with SurrealDB, SurrealQL offers a familiar syntax and supports various statement types, allowing you to perform complex database operations efficiently. To get started with SurrealQL, explore the [SurrealQL documentation](/docs/reference/query-language.md) to learn about the [statements](/docs/reference/query-language/statements/overview.md) available. These statements enable you to perform a wide range of database operations, from querying data to modifying records and managing database structures. Additionally, SurrealQL provides a rich set of [database functions](/docs/reference/query-language/functions/database-functions.md) that enhance its capabilities. These functions allow you to perform advanced operations and leverage the full potential of SurrealDB's features. Whether you are working with data retrieval, manipulation, or complex computations, SurrealQL's functions offer the tools you need to build robust and scalable applications. ### Getting started The easiest way to get started with SurrealQL is to use the [SurrealDB Studio](https://studio.surrealdb.com/) UI. This interactive environment allows you to experiment with SurrealQL statements and see the results in a tabular format. You can begin with the built-in Sandbox, or connect to a remote or local instance. The Sandbox is non-persistent; to persist your data while still experimenting, click on **Deploy to Cloud** in SurrealDB Studio to create a free SurrealDB Cloud instance. You can use SurrealDB Studio to learn about the syntax and features of SurrealQL and to develop your queries and scripts. To familiarize yourself with SurrealQL, explore the various [statements](/docs/reference/query-language/statements/overview.md) and their syntax. The statements pages provides comprehensive examples and explanations for each statement type, helping you understand how to construct queries and interact with SurrealDB effectively. SurrealQL empowers you to leverage the full potential of SurrealDB and enables you to build robust and scalable applications. Let's dive into the world of SurrealQL and unlock the capabilities of SurrealDB together! ### Querying options When using SurrealQL, there are several options available to interact with your database instance depending on your environment and needs. - **SurrealDB Studio**: [SurrealDB Studio](https://studio.surrealdb.com/) is an interactive environment that allows you to experiment with SurrealQL statements and see the results in a tabular format. It is a great tool for learning about the syntax and features of SurrealQL and for developing your queries and scripts. - **CLI**: The SurrealDB Command Line Interface (CLI) provides a powerful way to [interact with the database directly from your terminal](/docs/reference/cli/surrealdb-cli/overview.md). You can execute SurrealQL queries, manage database structures, and perform administrative tasks using the CLI. - **WebSocket**: SurrealDB supports WebSocket connections, allowing you to execute SurrealQL queries in real-time. This option is ideal for applications that require low-latency communication and real-time updates. - **HTTP**: You can send HTTP requests to the [`/sql` endpoint](/docs/reference/rest-api/http-protocol.md#sql) to execute SurrealQL queries. This method is useful for integrating SurrealDB with web applications and services that communicate over HTTP. - **Postman**: Using [Postman](https://www.postman.com/) or any other HTTP client, you can send a `POST` request to the `/sql` endpoint with your SurrealQL query in the body. This method provides flexibility and can be useful for testing and automation purposes. - **RPC**: SurrealDB also supports RPC, allowing you to interact with the database programmatically over a network. #### Using RPC SurrealDB also supports RPC, allowing you to interact with the database programmatically over a network. 1. **Set up an RPC client**: Depending on your programming language, you can use various libraries to create an RPC client. Here is an example using JavaScript with the `node-fetch` library: ```javascript const fetch = require('node-fetch'); async function querySurrealDB() { // Create a new person record await fetch('http://localhost:8000/rpc', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ method: 'query', params: ['CREATE person SET name = "John Doe"; SELECT * FROM person;'], }), }); const data = await response.json(); } querySurrealDB(); ``` 2. **Execute the RPC call**: Run your script to send the RPC request and receive the response from SurrealDB. ## GraphQL SurrealDB also supports GraphQL, allowing you to interact with the database using the familiar syntax. Currently, you can use the [GraphQL integration via SurrealDB Studio](/docs/learn/querying/graphql/via-studio.md), our intuitive user interface specifically designed for SurrealDB, or [over HTTP](/docs/learn/querying/graphql/via-http.md) via a GraphQL Client such as [GraphiQL](https://github.com/graphql/graphiql) or [Postman](/docs/explore/tutorials/tutorials/http-via-postman.md) at the `localhost:8000/graphql` endpoint. With SurrealDB Studio, you can easily connect to any SurrealDB instance, execute queries in real time, explore your tables, and design your schemas - all in one place. The GraphQL view allows you to define and retrieve only the data you need, giving you more control and efficiency in how you interact with your database. ### Getting started To get started with GraphQL, you can use the GraphQL integration in SurrealDB Studio. Once you have connected to your SurrealDB instance, you can explore your tables, execute queries, and manage your data in a user-friendly interface. ### Querying options When using GraphQL, there are several options available to interact with your database instance depending on your environment and needs. - **SurrealDB Studio**: [SurrealDB Studio](https://studio.surrealdb.com/) is an interactive environment that allows you to experiment with SurrealQL statements and see the results in a tabular format. It is a great tool for learning about the syntax and features of SurrealQL and for developing your queries and scripts. - **HTTP**: You can [send HTTP requests to the `/graphql` endpoint](/docs/learn/querying/graphql/via-http.md) to execute GraphQL queries. This method is useful for integrating SurrealDB with web applications and services that communicate over HTTP. ## SDKs Another way to interact with SurrealDB is to use one of the available SDKs. The SDKs provide a convenient way to query data, insert records, and manage database operations programmatically. SurrealDB offers a variety of SDKs that allow you to interact with the database programmatically. These SDKs provide a convenient and efficient way to query data, insert records, and manage database operations using your preferred programming language. ### Available SDKs SurrealDB supports SDKs for several popular programming languages, including: - **JavaScript/TypeScript**: The [JavaScript/TypeScript SDK](/docs/reference/javascript/concepts/connecting-to-surrealdb.md) allows you to interact with SurrealDB from web applications, Node.js environments, and other JavaScript-based platforms. It provides a comprehensive set of methods to perform queries, manage records, and handle database transactions. - **Python**: The [Python SDK](/docs/reference/python.md) offers a seamless way to integrate SurrealDB with your Python applications. Whether you are building web applications, data analysis tools, or automation scripts, the Python SDK provides the necessary functions to interact with the database efficiently. - **.NET**: The [.NET SDK](/docs/reference/dotnet.md) allows you to interact with SurrealDB using the .NET framework. It provides a comprehensive set of methods to perform queries, manage records, and handle database transactions, making it suitable for building robust and scalable applications. - **Rust**: The [Rust SDK](/docs/reference/rust.md) leverages the safety and performance features of the Rust programming language. It allows you to interact with SurrealDB in a type-safe manner, ensuring that your database operations are both efficient and reliable. ### Getting started with SDKs To get started with any of the SDKs, you need to install the appropriate package for your programming language. Once installed, you can initialise the SDK and connect to your SurrealDB instance. Here is an example of how to get started with the JavaScript SDK: ```javascript import { Surreal } from 'surrealdb'; const db = new Surreal('http://localhost:8000'); await db.connect('root', 'root'); type Person = { id: string; name: string; }; // Assign the variable on the connection const result = await db.query<[Person[], Person[]]>( 'CREATE person SET name = "John"; SELECT * FROM type::table($tb);', { tb: 'person' } ); // Get all of the results from the second query const people = result[1].result; ``` ## Learn more To learn more about the available options for querying data in SurrealDB, explore the following resources: - [SurrealQL documentation](/docs/reference/query-language.md) - [SurrealDB Studio documentation](/docs/explore/studio.md) --- Source: https://surrealdb.com/docs/learn/querying/surrealql/executing-queries/via-cli # Via CLI In this section, you will explore SurrealQL queries using the SurrealDB CLI. The SurrealDB CLI provides a powerful command-line interface for writing, executing, and visualising SurrealQL queries in real-time. To get started, [install the SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) on your local machine. ## Getting started After installing the SurrealDB CLI, you can start writing SurrealQL queries by running the [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) command in your terminal. You can also add the `--help` flag to view the available options and commands. To start a SurrealDB server, run the surreal start command, using the options below. This example serves the database at the default location (http://localhost:8000), with a username and password. ```bash surreal start --user root --pass secret ``` The server is actively running, and can be left alone until you want to stop hosting the SurrealDB server. Terminal start ## Running queries Open a second terminal and connect with [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md). You can stay in the interactive REPL, or pipe a one-shot query. To open a REPL: ```bash title="Start a SurrealDB shell (local endpoint)" surreal sql --endpoint http://localhost:8000 --ns main --db main --user root --pass secret --pretty ``` ```bash title="Start a SurrealDB shell (in-memory)" surreal sql --endpoint memory --ns main --db main --user root --pass secret --pretty ``` To run a simple `SELECT` without staying in the REPL: **Bash** ```bash echo 'SELECT * FROM person;' | surreal sql --endpoint http://localhost:8000 --ns main --db main --user root --pass secret --pretty --hide-welcome ``` **PowerShell** ```powershell 'SELECT * FROM person;' | surreal sql --endpoint http://localhost:8000 --ns main --db main --user root --pass secret --pretty --hide-welcome ``` Terminal SQL ## Learn more Learn more about the available commands and options in the [SurrealDB CLI documentation](/docs/reference/cli/surrealdb-cli/overview.md). --- Source: https://surrealdb.com/docs/learn/querying/surrealql/executing-queries/via-http # Via HTTP In this section, you will explore querying SurrealDB using HTTP. The HTTP API is designed to be simple and intuitive, with a RESTful interface that provides a consistent way to interact with the database. SurrealDB provides a [RESTful HTTP API](/docs/reference/rest-api/http-protocol.md) for interacting with the database programmatically. This is useful when you need to integrate SurrealDB into existing HTTP-based workflows, scripts, or environments where an SDK is not available. ## Using curl `POST /sql` The `/sql` endpoint enables use of SurrealQL queries. > [!IMPORTANT] > This HTTP endpoint expects the HTTP body to be a set of SurrealQL statements. ### Headers
Header Description
Authorization Sets the root, namespace, database, or record authentication data
Accept Sets the desired content-type of the response
surreal-ns Sets the selected Namespace for queries.
surreal-db Sets the selected Database for queries.
### Example usage > [!IMPORTANT] > The `-u` in the example below is a shorthand used by curl to send an Authorization header. ```bash title="Request" curl -X POST -u "root:secret" -H "surreal-ns: main" -H "surreal-db: main" -H "Accept: application/json" -d "SELECT * FROM person WHERE age > 18" http://localhost:8000/sql ``` ```json title="Response" [ { "time": "14.357166ms", "status": "OK", "result": [ { "age": "23", "id": "person:6r7wif0uufrp22h0jr0o" "name": "Simon", }, { "age": "28", "id": "person:6r7wif0uufrp22h0jr0o" "name": "Marcus", }, ] } ] ``` ## Using Postman Postman is a popular tool for testing APIs. You can use it to send HTTP requests to your SurrealDB instance and perform various database operations. 1. **Set up Postman**: Download and install Postman from the [official website](https://www.postman.com/). 2. **Create a new request**: Open Postman and create a new HTTP request. 3. **Configure the request**: - Set the request method to `POST`. - Enter the URL of your SurrealDB instance, e.g., `http://localhost:8000/sql`. - In the [headers section](#headers), add a `Content-Type` header with the value `application/json`. - In the Body section, select `raw` and set the type to `Text`. Enter your SQL query, for example: ```surql INFO FOR DB; ``` 4. **Send the request**: Click the `Send` button to execute the query. You will see the response from SurrealDB in the lower section of the Postman interface. ## Learn more Learn more about other [HTTP Endpoints](/docs/reference/rest-api/http-protocol.md) available in SurrealDB. For a more detailed tutorial on using Postman with SurrealDB, refer to the [working with SurrealDB over HTTP via Postman tutorial](/docs/explore/tutorials/tutorials/http-via-postman.md). --- Source: https://surrealdb.com/docs/learn/querying/surrealql/executing-queries/via-sdks # Via SDKs Query SurrealDB programmatically using one of the official SDKs available for Rust, JavaScript, Python, Go, Java, .NET, and PHP. SurrealDB supports a number of methods for connecting to the database and performing data queries. Each SDK has its own set of methods for connecting to the database and performing data queries. In each SDK, you can connect to the database using a local or remote connection. Once you are connected, you can start performing data queries. Here is a list of all the Supported SDKs: - [Rust](/docs/reference/rust.md) (available) - [JavaScript](/docs/reference/javascript/concepts/connecting-to-surrealdb.md) (available) - [TypeScript](/docs/reference/javascript/concepts/connecting-to-surrealdb.md) (available) - [Python](/docs/reference/python.md) (available) - [Node.js](/docs/reference/javascript/engines/node.md) (available) - [.NET](/docs/reference/dotnet.md) (available) - [Golang](/docs/reference/golang.md) (available) - [Java](/docs/reference/java.md) (available) - [PHP](/docs/reference/php.md) (available) ## Writing SurrealQL queries in SDKs In addition to the variety of methods provided by the SDKs to perform data queries, the `query` method works as a catch-all way to run [SurrealQL statements](/docs/reference/query-language.md) against the database. **Javascript** ```ts title="Method Syntax" async db.query(query, vars) ``` ### Arguments
Arguments Description
query Specifies the SurrealQL statements.
vars Assigns variables which can be used in the query.
### Example usage ```ts type Person = { id: string; name: string; }; // Assign the variable on the connection const result = await db.query<[Person[], Person[]]>( 'CREATE person SET name = "John"; SELECT * FROM type::table($tb);', { tb: 'person' } ); // Get the first result from the first query const created = result[0].result[0]; // Get all of the results from the second query const people = result[1].result; ``` `.query_raw()` With `.query_raw()`, you will get back the raw RPC response. In contrast to the `.query()` method, this will not throw errors that occur in individual queries, but will rather give those back as a string, and this will include the time it took to execute the individual queries. **PHP** ```php title="Method Syntax" $db->query($query, $vars) ``` ### Arguments
Arguments Description
$query Specifies the SurrealQL statements.
$vars Assigns variables which can be used in the query.
### Example usage ```php // Assign the variable on the connection $result = db->query( 'CREATE person SET name = "John"; SELECT * FROM type::table($tb);', [ "tb" => "person" ] ); // Get the first result from the first query $created = $result[0]->result[0]; // Get all of the results from the second query $people = $result[1]->result; ``` **Python** ```python title="Method Syntax" db.query(sql, vars) ``` ### Arguments
Arguments Description
sql Specifies the SurrealQL statements.
vars Assigns variables which can be used in the query.
### Example usage ```python # Assign the variable on the connection result = await db.query('CREATE person; SELECT * FROM type::table($tb)', { 'tb': 'person', }) # Get the first result from the first query result[0]['result'][0] # Get all of the results from the second query result[1]['result'] ``` **.NET** ```csharp title="Method Syntax" await db.Query(sql) ``` ### Arguments
Arguments Description
sql Specifies the SurrealQL statements.
cancellationToken The cancellationToken enables graceful cancellation of asynchronous operations.

### Example usage ```csharp // Execute query with params const string table = "person"; var result = await db.Query($"CREATE person; SELECT * FROM type::table({table});"); // Get the first result from the first query var created = result.GetValue(0); // Get all of the results from the second query var people = result.GetValue>(1); ```
`.RawQuery()` : Runs a set of SurrealQL statements against the database, based on a raw SurrealQL query. ```csharp title="Method Syntax" await db.RawQuery(sql, params) ``` ### Arguments
Arguments Description
sql Specifies the SurrealQL statements.
params Assigns variables which can be used in the query.
cancellationToken The cancellationToken enables graceful cancellation of asynchronous operations.
### Example usage ```csharp // Assign the variable on the connection var @params = new Dictionary { { "table", "person" } }; var result = await db.RawQuery("CREATE person; SELECT * FROM type::table($table);", @params); // Get the first result from the first query var created = result.GetValue(0); // Get all of the results from the second query var people = result.GetValue>(1); ``` **Golang** ```go title="Method Syntax" db.Query(sql, vars) ``` ### Arguments
Arguments Description
sql Specifies the SurrealQL statements.
vars Assigns variables which can be used in the query.
### Example usage ```go // Assign the variable on the connection result, err := db.Query("CREATE person; SELECT * FROM type::table($tb);", map[string]string{ "tb": "person" }); ``` **Rust** ```rust title="Method Syntax" db.query(sql).bind(vars) ``` ### Arguments
Arguments Description
sql Specifies the SurrealQL statements.
vars Assigns variables which can be used in the query.
### Example usage ```rust // Run some queries let sql = " CREATE person; SELECT * FROM type::table($table); "; let mut result = db .query(sql) .bind(("table", "person")) .await?; // Get the first result from the first query let created: Option = result.take(0)?; // Get all of the results from the second query let people: Vec = result.take(1)?; ``` ## Learn more Learn more about the [SurrealQL query language](/docs/reference/query-language.md). --- Source: https://surrealdb.com/docs/learn/querying/surrealql/executing-queries/via-studio # Via SurrealDB Studio In this section, you will explore SurrealQL queries using SurrealDB Studio, the official query editor for SurrealDB. SurrealDB Studio is a powerful tool that allows you to write, execute, and visualise SurrealQL queries in real-time. The [SurrealDB Studio](https://studio.surrealdb.com/query) query editor provides syntax highlighting, query validation, and real-time execution. Query results are displayed in a tabular format, making it easy to analyse and visualise data. ## Getting started To get started with SurrealQL in SurrealDB Studio, go to the [SurrealDB Studio Query Editor](https://studio.surrealdb.com/query) and start writing your SurrealQL queries. You can use the query editor to write queries, execute them, and view the results in real-time. ### Setting up a connection Within the connection list you will find a special connection called Sandbox, which is always available and allows you to test and experiment without storing data persistently. The Sandbox is useful for learning SurrealQL, testing queries, and more. This connection is designed for simple testing and not for evaluating performance, as it is limited to a single thread within the browser's WebAssembly engine. To persist your data while still experimenting, click on **Deploy to Cloud** in SurrealDB Studio to create a free SurrealDB Cloud instance. In order to interact with a SurrealDB database by any other means you must first create a connection. Connections store the details required to connect to a database, such as the endpoint, namespace, database, and authentication details. When you select an active connection in SurrealDB Studio, you will connect to the database and be able to interact with it using the available interface views. After opening a connection, you can switch to another connection at any time by pressing the connection name in the top left of the interface. This will open the connection list allowing you to switch to another connection, or create a new one. You can also create a new custom connection which will allow you to connect to a remote or local SurrealDB instance. ### Setting a namespace and database for connections. If you are using a connection (local or remote), you need to set the [namespace and database](/docs/learn/data-models/architecture.md#namespaces-and-databases) for the connection before you can start writing queries. For example, you can set both the namespace to `main` and the database to `main`. This will set the namespace and database for the current connection. ### Writing a query Once you have a connection open, you can use the SurrealQL query editor to create some data. For example, you can create a new record in the person table. ```surql CREATE person SET name = "John", age = 30; ``` In the query editor, you can use syntax highlighting, code completion, and validation to help you write your queries more efficiently. To execute a query, press the run query button at the bottom of the query editor. ## Learn more To learn more about SurrealQL and how to write queries using SurrealDB Studio, check out the [SurrealDB Studio documentation](/docs/explore/studio.md). --- Source: https://surrealdb.com/docs/learn/querying/surrealql/sample-queries # Sample queries Runnable SurrealQL examples for CREATE, SELECT, UPDATE and DELETE, each editable in an embedded SurrealDB Studio. This page walks through the four statements that do most of the work in [SurrealQL](/docs/reference/query-language.md): `CREATE`, `SELECT`, `UPDATE` and `DELETE`. Every example runs against a live database in an embedded [SurrealDB Studio](/docs/explore/studio.md), so you can edit a query and run it again to see what changes. SurrealQL's basic shape is largely equivalent to SQL, but includes additional syntax to allow conveniences such as record links, subqueries and nested field access. The examples below introduce each of those in turn. ### Creating data with CREATE Before we can start querying data, we need to create some records. This can be done using the [CREATE statement](/docs/reference/query-language/statements/create.md), which is used to add new records to the database. The following example demonstrates how to create a record in the `category` table, initialised with a `name` field and a `created_at` field. Press the *"Run query"* button to execute the query and view the response.
```surql CREATE category SET name = 'Technology', created_at = time::now(); ``` After executing this statement, the `category` record is created in the database, and a randomly generated unique id known as a [Record ID](/docs/reference/query-language/language-primitives/data-types/record-ids.md) is assigned to it. This ID represents the primary key of our record, and can be used to reference the record in future queries. When creating records, you can also explicitly set the record ID. This can be useful when you are able to use predictable unique record IDs such as `company:surrealdb` or `planet:earth`. In the following example, we create a person record with the ID `john`, and set the `first`, `last`, `age`, `admin`, and `signup_at` fields.
```surql CREATE person:john SET first = 'John', last = 'Adams', age = 29, admin = true, signup_at = time::now(); ``` SurrealDB also supports subqueries, used in the following example to populate the `category` field of the `article` record with the ID of the `Technology` category.
```surql CREATE article SET created_at = time::now(), author = person:john, title = 'Lorem ipsum dolor', text = 'Donec eleifend, nunc vitae commodo accumsan, mauris est fringilla.', category = SELECT VALUE id FROM ONLY category WHERE name = 'Technology' LIMIT 1; ``` ### Querying data with SELECT After inserting records into your database, you can now use the [SELECT statement](/docs/reference/query-language/statements/select.md) to retrieve data. While this statement will be familiar to anyone who has used traditional SQL before, SurrealDB's SELECT statement adds features drawn from NoSQL databases. For example, in addition to selecting records from a single table, you can also select records from multiple tables, or select specific records by their Record ID.
```surql -- Select all records from a table SELECT * FROM article; -- Select records from multiple tables SELECT * FROM category, person; -- Selecting specific records SELECT * FROM person:john; ``` The [SELECT statement](/docs/reference/query-language/statements/select.md) can filter on fields, resolve the contents of a record link, and reach data through a Record ID directly, with no JOIN planning or indexes needed. The following query combines a number of such features: - **Filtering**: Use the `WHERE` clause to only include records where the author's age is less than 30. - **Fetching**: Use the `.*` idiom to replace record ids with their actual field values. - **Specific fields**: Only want to retrieve the title and author fields from the article table. - **Record links**: Structure the field data from the author in a preferred format, including an alias for the field `name.full`. ```surql SELECT title, category.*, author.{ age, name: name.full, } FROM article WHERE author.age < 30; ``` ### Modifying data with UPDATE Records can be updated using the [UPDATE](/docs/reference/query-language/statements/update.md) statement, which allows you to modify the contents of existing records. Much like the `SELECT` statement, you can pass both table names and individual record IDs to the `UPDATE` statement. This allows you to update specific records, or update multiple records at once. ```surql UPDATE person:john SET age += 1, admin = false; ``` The `UPDATE` statement offers a variety of features to further filter down records, and apply different update strategies. The following example demonstrates how we can merge new data into records matching a specific condition.
```surql UPDATE person MERGE { age: 30, admin: false } ``` In addition to the `UPDATE` statement, SurrealDB also offers an [UPSERT statement](/docs/reference/query-language/statements/upsert.md), which has the added functionality of creating a record if it does not already exist. This can be useful when you want to update a record if it exists, or create it if it does not. ### Deleting data with DELETE You can also delete records from your database using the [DELETE statement](/docs/reference/query-language/statements/delete.md). This statement allows you to remove records from your database, either by specifying the record ID, or by using specific conditions. The following example demonstrates the use of the `RETURN` clause, which instructs SurrealDB to return the records before they are deleted.
```surql DELETE article WHERE author.name.first = 'David' RETURN BEFORE; ``` ## Where next - [Writing SurrealQL](/docs/learn/querying/surrealql/writing-surrealql.md) for the syntax rules behind these examples. - [Statements and values](/docs/learn/querying/surrealql/statements-and-values.md) for choosing between `CREATE`, `INSERT`, `UPDATE`, `UPSERT` and `RELATE`. - [Executing queries](/docs/learn/querying/surrealql/executing-queries.md) for running the same statements from the CLI, an SDK or over HTTP. - [Transactions](/docs/learn/querying/concepts-and-guides/transactions.md) for grouping statements that must succeed or fail together. --- Source: https://surrealdb.com/docs/learn/querying/surrealql/statements-and-values # Statements and values SurrealQL statements grouped as resource definitions, control flow with transactions, and CRUD-style query operations. SurrealDB has a variety of statements that let you configure and query a database. In this section, we'll look at the different types of statements that are available. ## Types of statements SurrealDB has a large variety of statements. They can be divided into three types: * Statements that define and access database resources, * Statements used for control flow and handling manual transactions, * Statements used in the context of queries, usually in CRUD (create, read, update, delete) operations. ### Database resource statements These statements pertain to defining, removing, altering, and rebuilding database resources. Some examples are: * [`DEFINE`](/docs/reference/query-language/statements/define/overview.md) statements to define database resources, * [`ALTER`](/docs/reference/query-language/statements/alter/overview.md) statements to alter certain resources, * [`REBUILD`](/docs/reference/query-language/statements/rebuild.md) to rebuild an index. Some other statements pertain to using defined resources. They are: * [`USE`](/docs/reference/query-language/statements/use.md) to move from one namespace or database to another, * [`INFO`](/docs/reference/query-language/statements/info.md) statements to see the definitions for resources. ### Control flow statements These statements are used to describe how query execution should progress. Some control flow statements only pertain to manual transactions. While all statements in SurrealDB are conducted inside their own transaction, these statements can be used to manually set up a larger transaction composed of multiple statements. They are: * [`BEGIN`](/docs/reference/query-language/statements/begin.md) to begin a manual transaction, * [`COMMIT`](/docs/reference/query-language/statements/commit.md) to commit a transaction, * [`CANCEL`](/docs/reference/query-language/statements/cancel.md) to cancel a transaction. Other control flow statements are used in the same manner as in other programming languages. Some examples are: * [`FOR`](/docs/reference/query-language/statements/for.md) to begin a for loop, * [`CONTINUE`](/docs/reference/query-language/statements/continue.md) to continue to the next iteration of a loop, * [`BREAK`](/docs/reference/query-language/statements/break.md) to break out of a for loop, internal scope, function, etc., * [`THROW`](/docs/reference/query-language/statements/throw.md) to cancel execution and return an error. ### Query statements These statements are used to execute queries, most often but not always in the context of a CRUD operation. Some examples of query statements are: * [`CREATE`](/docs/reference/query-language/statements/create.md) to create one or more records of one or more types of tables, * [`INSERT`](/docs/reference/query-language/statements/insert.md) to create one or more regular records or graph edges, * [`RELATE`](/docs/reference/query-language/statements/relate.md) to create a single graph edge between two records, * [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md) to stream all the changes to a table, * [`DELETE`](/docs/reference/query-language/statements/delete.md) to delete one or more records. The following flowchart can be used to get a sense of when it makes sense to use `CREATE`, `INSERT`, `UPDATE`, `UPSERT`, and `RELATE`. A flowchart that explains in which cases to use the statements CREATE, INSERT, UPDATE, UPSERT, and RELATE. ## Statement parameters A number of parameters prefixed with `$` are automatically available within a statement that provide access to relevant context inside the statement. These are known as reserved variable names. For example: * [$before](/docs/reference/query-language/language-primitives/parameters.md#before-after) and [$after](/docs/reference/query-language/language-primitives/parameters.md#before-after) can be accessed in statements that mutate values to see the values before and after an update, * [$session](/docs/reference/query-language/language-primitives/parameters.md#session) provides context on the current session, * [$parent](/docs/reference/query-language/language-primitives/parameters.md#parent-this) provides access to the value in a primary query while inside a subquery. For a full list of these automatically generated parameters, see the [parameters](/docs/reference/query-language/language-primitives/parameters.md#reserved-variable-names) page. ## Values Each of the types mentioned in the data model is a subset of an all-encompassing type called a value. ## Comparing and ordering values As every data type a subset of value, any value can be compared with another one. ```surql 9 > 1; // Returns true null > none; // Also returns true ``` Being able to compare a value with any other value is what makes SurrealDB's record range syntax possible. ```surql CREATE time_data:[d'2024-07-23T00:00:00.000Z']; CREATE time_data:[d'2024-07-24T00:00:00.000Z']; CREATE time_data:[d'2024-07-25T00:00:00.000Z']; -- Records from the 24th to the 25th SELECT * FROM time_data:[d'2024-07-24']..[d'2024-07-25']; -- Records from the 24th SELECT * FROM time_data:[d'2024-07-24']..; -- All records SELECT * FROM time_data:[NONE]..; ``` The `..` open-range syntax also represents an infinite value inside a record range query, making it the greatest possible value and the inverse of `NONE`, the lowest possible value. A part of a record range query that begins with `NONE` and ends with `..` will thus filter out nothing. ```surql CREATE temperature:['London', d'2025-02-19T00:00:00.000Z'] SET val = 5.5; CREATE temperature:['London', d'2025-02-20T00:00:00.000Z'] SET val = 5.7; -- Return all records as long as index 0 = 'London' SELECT * FROM temperature:['London', NONE]..=['London', ..]; ``` ```surql title="Output" [ { id: temperature:[ 'London', d'2025-02-19T00:00:00Z' ], val: 5.5f }, { id: temperature:[ 'London', d'2025-02-20T00:00:00Z' ], val: 5.7f } ] ``` Inside a schema, the keyword `any` is used to denote any possible value. ```surql DEFINE FIELD anything ON TABLE person TYPE any; ``` ## Values and truthiness Any value is considered to be truthy if it is not NONE, NULL, or a default value for the data type. A data type at its default value is one that is empty, such as an empty string or array or object, or a number set to 0. The following example shows the result of the `array::all()` method, which checks to see if all of the items inside an array are truthy or not. ```surql array::all(["", 1, 2, 3]); // false because of "" array::all([{}, 1, 2, 3]); // false because of {} array::all(["SurrealDB", { is_nice_database: true }, 1, 2, 3]); // true ``` As [the ! operator](/docs/reference/query-language/language-primitives/operators.md) reverses the truthiness of a value, a doubling of this operator can also be used to check for truthiness. ```surql [ !!"Has a value", !!"", // true, false !!true, !!false, // true, false !!{ is_nice_database: true }, !!{} // true, false ]; ``` The following example shows how `!!` can be conveniently used along with the [`object::entries()`](/docs/reference/query-language/functions/database-functions/object.md#objectentries) and [`object::from_entries()`](/docs/reference/query-language/functions/database-functions/object.md#objectfrom_entries) function to set fields from one table in another as long as they are not empty, `NULL`, or `NONE`. The filtering itself is done using the [`array::filter()`](/docs/reference/query-language/functions/database-functions/array.md#arrayfilter) function which returns any values that match a pattern such as the `!!` operator. ```surql CREATE person:one CONTENT { this: NULL, that: "that", and: "and", the: NONE, other: "" }; CREATE person:two CONTENT { this: "this", that: NONE, and: NULL, the: "", other: "other" }; FOR $person IN SELECT * FROM person { LET $filtered = $person.*.entries().filter(|$entry| !!$entry[1]); CREATE new_table CONTENT object::from_entries($filtered); -- Or all in one line: -- CREATE new_table CONTENT object::from_entries($person.*.entries().filter(|$n| !!$n[1])) }; SELECT * FROM new_table; ``` The output shows two new records that only contain fields that were truthy in the original `person` records. ```surql title="Output" [ { and: 'and', id: new_table:one, that: 'that' }, { id: new_table:two, other: 'other', this: 'this' } ] ``` --- Source: https://surrealdb.com/docs/learn/querying/surrealql/what-is-surrealql # What is SurrealQL? SurrealQL is SurrealDB’s SQL-like language for queries, schemas, graph relationships and optimised execution. In this section, you will explore [SurrealQL](/docs/reference/query-language.md), a powerful database query language that closely resembles traditional SQL but comes with unique differences and improvements. SurrealQL is designed to provide developers with an intuitive way to interact with SurrealDB. It offers a familiar syntax and supports various statement types, allowing you to perform complex database operations efficiently. While SurrealQL shares similarities with traditional SQL, it introduces enhancements and optimisations that make it well-suited for working with SurrealDB's advanced features. Whether you are querying data, modifying records, or managing database structures, SurrealQL provides a comprehensive set of capabilities to meet your needs. ## Key features SurrealQL offers several key features that make it a powerful tool for working with SurrealDB: - **Familiar syntax**: SurrealQL adopts a syntax similar to traditional SQL, making it easy for developers familiar with SQL to transition to SurrealDB seamlessly. - **Advanced querying**: SurrealQL supports a wide range of querying capabilities, including filtering, sorting, aggregating, and joining data from multiple tables. - **Data manipulation**: With SurrealQL, you can easily insert, update, and delete records in your SurrealDB database, allowing you to manage your data effectively. - **Graph relationships**: SurrealQL supports graph relationships, allowing you to define and query relationships between records in your database. - **Schema management**: SurrealQL provides features for creating and modifying database schemas, allowing you to define the structure of your data and enforce data integrity. - **Performance optimisation**: SurrealQL incorporates optimisations specific to SurrealDB, ensuring efficient execution of queries and minimising resource usage. SurrealDB Studio query view --- Source: https://surrealdb.com/docs/learn/querying/surrealql/writing-surrealql # Writing SurrealQL Ways of executing SurrealQL queries such as SurrealDB Studio, CLI, HTTP, SDKs, and GraphQL. [SurrealQL](/docs/reference/query-language.md) is the query language for SurrealDB. How they reach the database depends on your tool or integration. The sections below describe each path in more detail. ## SurrealQL execution paths The following four approaches are the most commonly used ways to execute SurrealQL queries directly or indirectly via SDKs in one of many programming languages. | Approach | Typical use | | -------- | ----------- | | [SurrealDB Studio](/docs/learn/querying/surrealql/executing-queries/via-studio.md) | Interactive editing, Sandbox, and visual results in the browser. | | [CLI](/docs/learn/querying/surrealql/executing-queries/via-cli.md) | Local development, scripts, and [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md) against a running server. | | [HTTP](/docs/learn/querying/surrealql/executing-queries/via-http.md) | Services and integrations that call the [`/sql`](/docs/reference/rest-api/http-protocol.md#sql) endpoint. | | [SDKs](/docs/learn/querying/surrealql/executing-queries/via-sdks.md) | Application code using the official clients (WebSocket or HTTP under the hood, depending on SDK and configuration). | In SurrealDB Studio, the built-in Sandbox does not persist data. To keep experimental work, use **Deploy to Cloud** in the app to create a free SurrealDB Cloud instance. For a single entry point into these guides, see **[Executing queries](/docs/learn/querying/surrealql/executing-queries.md)**. Interactive clients (including the SDKs) usually speak to the database over **WebSocket** or **HTTP**. The [RPC protocol](/docs/reference/rest-api/rpc-protocol.md) describes how queries and responses are framed on the wire. Language syntax, statements, and functions are documented in the **[SurrealQL reference](/docs/reference/query-language.md)**. ## GraphQL [GraphQL](/docs/learn/querying/graphql/overview.md) is a separate query interface on top of SurrealDB: you describe fields and shapes in GraphQL, not in SurrealQL. You can use it from [SurrealDB Studio](/docs/learn/querying/graphql/via-studio.md), over [HTTP](/docs/learn/querying/graphql/via-http.md), or from tools such as Postman or Bruno. --- Source: https://surrealdb.com/docs/learn/schema-management # Schema management Tables, fields, indexes and events: how to shape data in SurrealDB. Tenancy and related patterns too. Schema management is where you describe what lives in the database: tables and fields, indexes, events, optional files and buckets, and the namespace / database layout for multi-tenant setups. What makes SurrealDB convenient is that schema management can be almost entirely dispensed with at the outset, because standard CRUD (Create, Read, Update, Delete) operations do not require a schema to be defined at the outset. This allows you to experiment at your leisure until it is time to firm up your database's expected record and data types, at which point schema definition comes to the fore. Schema is not all about defining data, however. For example, statements like [`DEFINE EVENT`](/docs/reference/query-language/statements/define/event.md) set up effects that take place after a statement is executed. The learn pages here explain how and why to use these pieces together. For exact statement grammar, defaults, and edge cases, use the SurrealQL reference ([`DEFINE`](/docs/reference/query-language/statements/define/overview.md) and related pages). If you are new to SurrealDB, it helps to read [tables](/docs/learn/schema-management/tables-and-fields/tables.md) and [fields](/docs/learn/schema-management/tables-and-fields/fields-and-validation.md) first, then [indexes](/docs/learn/schema-management/indexes/index-types-and-strategies.md) and [events](/docs/learn/schema-management/events-and-triggers/defining-events.md) when you need behaviour beyond simple storage. For managing schema files across environments, synchronising locally and running phased rollouts in production, see [SurrealKit schema migration](/docs/manage/schema-migration.md). ## Record IDs - [Record IDs and addressing](/docs/learn/schema-management/tables-and-fields/record-ids-and-addressing.md) - how a record id is built, and what it can address - [Record ID best practices](/docs/learn/schema-management/tables-and-fields/record-id-best-practices.md) - choosing ids that stay useful as data grows ## Design and architecture - [Schema design](/docs/learn/schema-management/schema-design.md) - choosing between schemafull and schemaless, and what each costs - [Namespace & database architecture](/docs/learn/schema-management/multi-tenancy/namespace-and-database-architecture.md) - how to lay out namespaces and databases for multiple tenants --- Source: https://surrealdb.com/docs/learn/schema-management/computed-data/closures # Closures Anonymous functions in SurrealQL, colloquially known as closures. Closures are small inline functions. They are generally used in two ways: * By binding them with `LET` as a parameter for reuse, * Passed directly into methods like `.map()`, `.filter()`, or `.chain()`. Parameters passed into closures are contained within `||`, after which the body of the function is written. They are not a replacement for [`DEFINE FUNCTION`](/docs/reference/query-language/statements/define/function.md) when you need a named, reusable routine with permissions. ## Basic examples ## Basic function definitions ```surql -- Define an anonymous function that doubles a number LET $double = |$n: number| $n * 2; RETURN $double(2); -- Returns 4 -- Define a function that concatenates two strings LET $concat = |$a: string, $b: string| $a + $b; RETURN $concat("Hello, ", "World!"); -- Returns "Hello, World!" ``` ## Error handling and type enforcement Like regular functions, you can also enforce type constraints within your functions to prevent type mismatches. ```surql -- Define a function that greets a person, returning a string LET $greet = |$name: string| -> string { "Hello, " + $name + "!" }; RETURN $greet("Alice"); -- Returns "Hello, Alice!" -- Define a function with a return type LET $to_upper = |$text: string| -> string { string::uppercase($text) }; RETURN $to_upper("hello"); -- Returns "HELLO" RETURN $to_upper(123); -- Error: type mismatch -- Define a function that accepts only numbers LET $square = |$num: number| $num * $num; RETURN $square(4); -- Returns 16 RETURN $square("4"); -- Error: type mismatch ``` ## Closures in functions Many of SurrealDB's functions require a closure to be passed in, making it easy to use complex logic on a value or the elements of an array. The `chain` function which performs an operation on a value before passing it on: ```surql "Two" .replace("Two", "2") .chain(|$num| $num * 1000); ``` ```surql title="Output" 2000 ``` We can see that the input to the `.chain()` method is indeed a closure by creating our own that is assigned to a parameter. This closure can be passed into `.chain()`, returning the same output as above. ```surql LET $my_func = |$num| $num * 1000; "Two" .replace("Two", "2") .chain($my_func); ``` The following example shows a chain of array functions used to remove useless data, followed by a check to see if all items in the array match a certain condition, and then a cast into another type. The [`array::filter`](/docs/reference/query-language/functions/database-functions/array.md#arrayfilter) call in the middle ensures that the [`string::len`](/docs/reference/query-language/functions/database-functions/string.md#stringlen) function that follows is being called on string values. ```surql [NONE, NONE, "good data", "Also good", "important", NULL] .filter(|$v| $v.is_string()) .all(|$s| $s.len() > 5) .chain(|$v| $v); ``` ```surql title="Output" 'true' ``` To fail the query when the pipeline does not meet your assumptions (for example while debugging), end the chain with [`.expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) instead of casting to a string: ```surql [NONE, NONE, "good data", "Also good", "important", NULL] .filter(|$v| $v.is_string()) .all(|$s| $s.len() > 5) .expect(|$ok| $ok, "Expected at least one string longer than 5 characters"); ``` ## Capturing parameters Another aspect of closures is that they are able to capture parameters in their environment. This is in fact where the name "closure" comes from, as they are anonymous functions that are able to "enclose" such parameters without needing to pass them through the `||` parallel bars that hold its input arguments. ```surql LET $okay_nums = [1,2,3]; [1,5,6,7,0].filter(|$n| $n IN $okay_nums); ``` ## Closures and writes Whether a closure can modify database resources depends on your version. **Before SurrealDB 3.3** Closures work inside a read-only context, and cannot be used to modify database resources. This holds even when the write sits inside a function the closure calls. ```surql -- 1. Create a test table and function DEFINE TABLE test_table SCHEMAFULL; DEFINE FIELD name ON test_table TYPE string; DEFINE FUNCTION fn::test_create($name: string) -> object { CREATE test_table CONTENT { name: $name }; { created: true, name: $name }; }; -- 2. Call the function directly - works fn::test_create("direct_call"); -- 3. Call the function inside .map() - fails LET $names = ["Alice", "Bob", "Charlie"]; $names.map(|$n| fn::test_create($n)); ``` ```surql title="Output" Error: "Couldn't write to a read only transaction" ``` In many cases, a closure can be substituted by another operation such as a `FOR` loop or a regular `SELECT` statement. ```surql DEFINE TABLE test_table SCHEMAFULL; DEFINE FIELD name ON test_table TYPE string; DEFINE FUNCTION fn::test_create($name: string) -> object { CREATE test_table CONTENT { name: $name }; { created: true, name: $name }; }; -- Function to create a record called for each name SELECT VALUE fn::test_create($this) FROM ["Alice", "Bob", "Charlie"]; ``` **SurrealDB 3.3 and later** A closure that writes makes the expression holding it a write, so it can modify database resources like any other statement. This applies wherever the closure is invoked, including closure-taking functions such as `map`, `filter` and `fold`, and it follows a write reached through a function the closure calls. ```surql -- 1. Create a test table and function DEFINE TABLE test_table SCHEMAFULL; DEFINE FIELD name ON test_table TYPE string; DEFINE FUNCTION fn::test_create($name: string) -> object { CREATE test_table CONTENT { name: $name }; { created: true, name: $name }; }; -- 2. Call the function directly fn::test_create("direct_call"); -- 3. Call the function inside .map() LET $names = ["Alice", "Bob", "Charlie"]; $names.map(|$n| fn::test_create($n)); ``` ```surql title="Output" [ { created: true, name: 'Alice' }, { created: true, name: 'Bob' }, { created: true, name: 'Charlie' } ] ``` A closure whose body only reads still resolves as a read, so `map`, `filter` and `fold` over a pure closure keep their read-only path. A `FOR` loop or a `SELECT` over a list remains a good choice where it reads more clearly: ```surql -- Function called once for each name SELECT VALUE fn::test_create($this) FROM ["Alice", "Bob", "Charlie"]; ``` ## Conclusion These anonymous functions provide a flexible way to define small, reusable pieces of logic that can be used throughout your queries. By leveraging them, you can write more modular and maintainable SurrealQL code. --- Source: https://surrealdb.com/docs/learn/schema-management/computed-data/computed-fields # Computed fields Fields that are derived on read with COMPUTED, traversing record links, and how they differ from VALUE and stored data. Most fields are stored: you write a value and it stays on the record until you change it. A computed field differs from a normal field in that it is recalculated whenever the record is read, using an expression you attach in the schema. That makes it ideal for derived data that should always reflect a certain state instead of needing a separate update job. ## Compared to `VALUE` and normal storage * `VALUE` on a field sets what gets stored (often normalising input); it runs on write, not on every read in the same way a computed field does. * `COMPUTED` stores an expression, as opposed to set data. * You cannot mark `id` or nested paths (for example `meta.score`) as computed - only top-level field names. For everything else about field definitions - types, `ASSERT`, `DEFAULT`, permissions - see [Fields and validation](/docs/learn/schema-management/tables-and-fields/fields-and-validation.md) and the [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md) reference. ## Example: always-fresh timestamp on read ```surql DEFINE TABLE user SCHEMAFULL; DEFINE FIELD name ON user TYPE string; DEFINE FIELD accessed_at ON user COMPUTED time::now(); CREATE user:one SET name = "Ada"; SELECT * FROM ONLY user:one; SLEEP 1s; SELECT * FROM ONLY user:one; ``` The second `SELECT` should show a new `accessed_at` each time, because the expression is evaluated when the record is projected. ## Record references A stored field can hold a [record link](/docs/reference/query-language/language-primitives/record-links.md) - a record ID that points at another record - often declared with `record` or `option>`, and optionally with [`REFERENCE`](/docs/reference/query-language/statements/define/field.md) so deletes on the target record are handled predictably. A computed field does not store a link itself (the `REFERENCE` clause applies to stored fields only), but the expression may traverse link fields already on the record and read fields from the related record. In the example below, `book.author` stores `person:ada`, while `author_name` is derived from that link. If you change the person’s `name`, the next read of the book shows the updated label without writing back to `book`. ```surql DEFINE TABLE person SCHEMAFULL; DEFINE FIELD name ON person TYPE string; DEFINE TABLE book SCHEMAFULL; DEFINE FIELD title ON book TYPE string; DEFINE FIELD author ON book TYPE record; DEFINE FIELD author_name ON book COMPUTED author.name; CREATE person:ada SET name = "Ada Lovelace"; CREATE book:one SET title = "Notes", author = person:ada; SELECT title, author_name FROM book:one; UPDATE person:ada SET name = "Augusta Ada King"; SELECT title, author_name FROM book:one; ``` ## When to use computed fields Use them when: * The result is cheap enough to evaluate on read and you want zero staleness. * The logic is purely relative to the record and its graph. Computed fields may not always be the best option. They should not be used when: * You need the value indexed or searched like a normal field (compute into a stored field via events or application writes instead). * The expression depends on external systems. ## See also * [Closures](/docs/learn/schema-management/computed-data/closures.md) - small anonymous functions in expressions. * [Record links](/docs/reference/query-language/language-primitives/record-links.md) - storing and traversing record IDs. * [Reactive patterns](/docs/learn/schema-management/events-and-triggers/reactive-patterns.md) - when updates should flow from writes instead of reads. --- Source: https://surrealdb.com/docs/learn/schema-management/events-and-triggers/defining-events # Defining events Running logic after creates, updates, and deletes with DEFINE EVENT. Events let the database react to record changes: write an audit log, normalise related rows, or enqueue follow-up work. They run after the record change but inside the same transaction (unless you opt into async), and they see [$before and $after](/docs/reference/query-language/language-primitives/parameters.md#before-after) snapshots of the record. > [!NOTE] > Events are a side effect of normal writes. They are not fired during a bulk [import](/docs/reference/cli/surrealdb-cli/commands/import.md). ## Key ideas * `$event`: `"CREATE"`, `"UPDATE"`, or `"DELETE"`. * `$before` / `$after`: state immediately before and after the change. * `$value`: the record as seen by the event (after create/update, before delete). * `WHEN`: optional filter so the body runs only when something meaningful changed. * `THEN`: the SurrealQL block to execute. You need the usual database privileges and [`USE`](/docs/reference/query-language/statements/use.md) scope. Full syntax and async options are in [`DEFINE EVENT`](/docs/reference/query-language/statements/define/event.md); [Reactive patterns](/docs/learn/schema-management/events-and-triggers/reactive-patterns.md) covers ASYNC, RETRY, and MAXDEPTH. ## Example usage - Email Change Detection: Create an event that logs whenever a user's email is updated. In this example: - The `WHEN` clause checks if the email has changed. - The `THEN` clause records this change in a `log` table. ```surql -- Create a new event whenever a user changes their email address -- One-statement event DEFINE EVENT OVERWRITE test ON TABLE user WHEN $before.email != $after.email THEN ( CREATE log SET user = $value.id, -- Turn events like "CREATE" into string "email created" action = 'email' + ' ' + $event.lowercase() + 'd', -- `email` field may be NONE, log as '' if so old_email = $before.email ?? '', new_email = $after.email ?? '', at = time::now() ); UPSERT user:test SET email = 'old_email@test.com'; UPSERT user:test SET email = 'new_email@test.com'; DELETE user:test; SELECT * FROM log ORDER BY at ASC; ``` ```surql title="Output" [ { action: 'email created', at: d'2025-11-25T02:59:41.003Z', id: log:e3thw1l0q7xiapznar1f, new_email: 'old_email@test.com', old_email: '', user: user:test }, { action: 'email updated', at: d'2025-11-25T02:59:41.003Z', id: log:uaarfyk191jgod06xobm, new_email: 'new_email@test.com', old_email: 'old_email@test.com', user: user:test }, { action: 'email deleted', at: d'2025-11-25T02:59:41.003Z', id: log:mlkag8h1xotglpz9wt2i, new_email: '', old_email: 'new_email@test.com', user: user:test } ] ``` ### More complex logic: The following event executes multiple actions to both log a purchase and establish relationships between the customer and product. ```surql DEFINE EVENT purchase_made ON TABLE purchase WHEN $before == NONE THEN { LET $customer = (SELECT * FROM customer WHERE id = $after.customer); LET $product = (SELECT * FROM product WHERE id = $after.product); RELATE $customer->bought->$product CONTENT { quantity: $after.quantity, total: $after.total, status: 'Pending', }; CREATE log SET customer_id = $after.customer, product_id = $after.product, action = 'purchase_created', timestamp = time::now(); }; ``` ## Specific events You can trigger events based on specific events. You can use the variable $event to detect what type of event is triggered on the table. ```surql -- UPDATE event -- Here we are creating a notification when a user is updated. DEFINE EVENT user_updated ON TABLE user WHEN $event = "UPDATE" THEN ( CREATE notification SET message = "User updated", user_id = $after.id, created_at = time::now() ); -- DELETE event is triggered when a record is deleted from the table. -- Here we are creating a notification when a user is deleted. DEFINE EVENT user_deleted ON TABLE user WHEN $event = "DELETE" THEN ( CREATE notification SET message = "User deleted", user_id = $before.id, created_at = time::now() ); -- You can combine multiple events based on your use cases. -- Here we are creating a log when a user is created, updated or deleted. DEFINE EVENT user_event ON TABLE user WHEN $event = "CREATE" OR $event = "UPDATE" OR $event = "DELETE" THEN ( CREATE log SET table = "user", event = $event, happened_at = time::now() ); ``` This longer example shows an event that updates all posts for a publication to "published" status once a publication containing them is created. ```surql -- Define an event DEFINE FIELD status ON post TYPE "submitted" | "published" DEFAULT "submitted"; DEFINE EVENT publish_post ON TABLE publication WHEN $event = "CREATE" THEN ( FOR $post IN $after.posts { UPDATE $post SET status = "published"; } ); CREATE post:one SET content = "I read the news today, oh boy..."; CREATE post:two SET content = "On the banks of Tuonela Bleach the skeletons of kings"; CREATE post:three SET content = "뭐 화끈한 일 뭐 신나는 일 없을까"; CREATE publication SET posts = [post:one, post:two, post:three]; SELECT * FROM post; ``` ```surql title="Output" [ { content: 'I read the news today, oh boy...', id: post:one, status: 'submitted' }, { content: '뭐 화끈한 일 뭐 신나는 일 없을까', id: post:three, status: 'submitted' }, { content: 'On the banks of Tuonela Bleach the skeletons of kings', id: post:two, status: 'submitted' } ] ``` ## Events and permissions Queries inside the event always execute without any permission checks, even when triggered by changes made by the currently authenticated user. Consider a CREATE query sent by a record user that has CREATE access to the `comment` table only: ```surql CREATE comment SET post = post:tomatosoup, content = "So delicious!", author = $auth.id; ``` Logic can be added to the event itself to modify the behaviour depending on a user's permissions or any other condition. ```surql DEFINE EVENT on_comment_created ON TABLE comment WHEN $event = "CREATE" THEN { -- Check if the post allows for adding comments. -- User record doesn't have access to the `post` table. IF $after.post.disable_comments { THROW "Can 't create a comment - Comments are disabled for this post"; }; -- Set the `approved` field on the new comment - automatically approve -- comments made by the author of the post. -- For security reasons, record users don't have any permissions for the `approved` field. UPDATE $after.id SET approved = $after.post.author == $after.author; }; ``` ## Accessing `$input` in events The behaviour of events can be further refined via the `$input` parameter, which represents the record in question for the event. ```surql -- Set CREATE in event to only trigger when record has `true` for `log_event` DEFINE EVENT something ON person WHEN $input.log_event = true THEN { CREATE log SET at = time::now(), of = $input; }; -- Set to `false`, does not trigger CREATE CREATE person:debug SET name = "Billy", log_event = false; -- Triggers CREATE CREATE person:real SET name = "Bobby", log_event = true; SELECT * FROM log; ``` Output: ```surql [ { at: d'2025-10-14T06:15:21.141Z', id: log:svbr2qhjywml20mufb0o, of: { log_event: true, name: 'Bobby' } } ] ``` ## Async events Events in SurrealDB are executed synchronously within the same transaction that triggers them. While this ensures consistency, it can lead to increased latency for write operations if the event logic is complex or resource intensive. To allow events to execute independently of the transaction that triggers them, the `ASYNC` clause can be used. ### How async events are processed Async events are processed in an interval dependant on the environment variable `SURREAL_ASYNC_EVENT_PROCESSING_INTERVAL` (or `--async-event-interval` when starting the server) which is set to 5 seconds as the default. Lowering this will reduce the latency between a document change and its events, while leading to more frequent polls by the background worker. Some more notes on the characteristics of async events: * Atomicity: The event is enqueued within the same transaction as the document change. If the transaction fails, the event is never queued. * Consistency: Asynchronous events run in a separate transaction from the original change. They see the database state at the time they are executed. * Ordering: Events are generally processed in the order they were created, though parallel processing may occur within a single batch. The easiest way to demonstrate that async events do not occur in the same transaction is by causing one to [throw](/docs/reference/query-language/statements/throw.md) an error. As an error inside any part of a transaction will cause the transaction to fail and roll back, the following event which fails about 50% of the time would cause the `CREATE` statement that follows to fail if it were not async. As an async event, however, the events that follow the statement are each run in their own transaction ```surql DEFINE TABLE did_not_throw; DEFINE EVENT may_throw ON person ASYNC THEN { IF rand::bool() { THROW "This message will never show"; } ELSE { CREATE did_not_throw; } }; CREATE |person:50|; count(SELECT * FROM did_not_throw); ``` ### The `MAXDEPTH` clause The `MAXDEPTH` clause is used to set the maximum number of times that an async event can be triggered. The number following this can range from 0 to 16. The default for `MAXDEPTH` is 3, as events defined on other events that lead to record creation can quickly spiral out of control at greater levels. Taking the following contrived example: ```surql DEFINE EVENT start ON start THEN { CREATE cat; }; DEFINE EVENT cat ON person ASYNC MAXDEPTH 4 THEN { CREATE |cat:9|; }; DEFINE EVENT person ON cat ASYNC MAXDEPTH 4 THEN { CREATE |person:9|; }; CREATE start; count(SELECT VALUE id FROM person, cat); ``` While the `MAXDEPTH` in this case is only one greater than the default, the sheer number of records created results in the final `count()` score being 20503, compared to 2278 if the default is used. This is somewhat similar to recursive queries which can also quickly add up. ```surql -- Create five people CREATE |person:1..=5|; -- Make each person friends with each of the four others UPDATE person SET friends = (SELECT VALUE id FROM person).complement([$this.id]); -- Count after five levels of depth is already 1024! count(person:1.{..5}.friends); ``` ### The `RETRY` clause The `RETRY` clause is suitable for events that may fail but can succeed on successive attempts. The example below shows two events that each have a 50% chance of failure and zero retries. ```surql DEFINE EVENT one ON account ASYNC RETRY 0 THEN { IF rand::bool() { THROW "Failed!" } ELSE { CREATE it:worked SET very = "well"; } }; DEFINE EVENT two ON account ASYNC RETRY 0 THEN { IF rand::bool() { THROW "Failed!" } ELSE { CREATE it:worked SET very = "well"; } }; CREATE account; ``` Following up these events with a `SELECT * FROM it` will most likely lead to the following input. ```surql [ { id: it:worked, very: 'well' } ] ``` However, with zero retries there is still a 25% chance that the query will only ever lead to the error `"The table 'it' does not exist"`. --- Source: https://surrealdb.com/docs/learn/schema-management/events-and-triggers/reactive-patterns # Reactive patterns Choosing between synchronous and asynchronous events, LIVE SELECT subscriptions, and evolving reactive behaviour safely. [Events](/docs/learn/schema-management/events-and-triggers/defining-events.md) are SurrealDB's built-in way to react to changes: you attach logic that runs after a create, update, or delete using [`DEFINE EVENT`](/docs/reference/query-language/statements/define/event.md). They do not run during bulk [import](/docs/reference/cli/surrealdb-cli/commands/import.md), so plan backfills separately or use the [sql](/docs/reference/cli/surrealdb-cli/commands/sql.md) endpoint if you rely on them for derived data. When deciding how to define an event, consider the following: * What should happen immediately, what can wait, and how do you keep the system predictable? ## Immediate vs deferred work Every event introduces work triggered by a write. The first and most important decision is whether that work belongs in the same transaction as the write or outside it. The sections below mirror that split; the [`DEFINE EVENT`](/docs/reference/query-language/statements/define/event.md) reference describes `ASYNC`, `MAXDEPTH`, and `RETRY` in full. ### Strong consistency (synchronous events) By default, events run in the same transaction as the triggering write (see [parameters](/docs/reference/query-language/language-primitives/parameters.md#before-after) such as `$before` and `$after` in the event body). That means: * If the event fails, the original write is rolled back * All changes are immediately consistent * The write includes the cost of the event You will want to use this pattern when: * The follow-up must always succeed * The database must never enter an inconsistent state * The logic is lightweight and predictable This might be the case when you need to enforce constraints across tables or when writing audit records that must exist. Remember that [queries inside events bypass permission checks](/docs/learn/schema-management/events-and-triggers/defining-events.md#events-and-permissions), so validate anything security-sensitive explicitly in the `THEN` block. The tradeoff with this pattern is that event logic should stay as small as practical to avoid added latency on every write. ### Deferred processing (asynchronous events) With [`ASYNC`](/docs/reference/query-language/statements/define/event.md#async-events), the event is queued and processed outside the original transaction. That implies: * The write succeeds independently of the event * The event runs in a separate transaction * Failures do not roll back the original change You will want to use this pattern when: * The write must be fast * The work is expensive or unreliable * Temporary inconsistency is acceptable For the background processing interval, [`MAXDEPTH`](/docs/reference/query-language/statements/define/event.md#the-maxdepth-clause), and [`RETRY`](/docs/reference/query-language/statements/define/event.md#the-retry-clause), see [Async events](/docs/reference/query-language/statements/define/event.md#async-events) in the `DEFINE EVENT` reference. ### `LIVE SELECT` While different from an event, a [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md) is reactive in that it streams notifications when records matching the selection change. While an event embeds what the database should do in a `THEN` block, a live query only tells subscribers that something changed, so your app (or another service) decides how to react - typically in [SDK code](/docs/reference/javascript/concepts/live-queries.md) using the connection's live-query APIs. For behaviour on the wire, subscriptions, and production caveats, read [Live queries](/docs/learn/querying/real-time/live-queries.md). To end a subscription, use [`KILL`](/docs/reference/query-language/statements/kill.md) with the query UUID the server returns when you register the live select. [Real-time best practices](/docs/learn/querying/real-time/real-time-best-practices.md#defining-events) also contrasts live queries with defining events when the database can do the work without a client listener. ### Building reactive systems incrementally The sheer convenience of reactive systems makes them tempting to reach for early. They are harder to debug when many are introduced together. For example, an error that occurs may be difficult to debug inside an [event](/docs/learn/schema-management/events-and-triggers/defining-events.md) that chains into further writes plus several [`LIVE SELECT`](/docs/reference/query-language/statements/live-select.md), especially if all of these reactive patterns were introduced all at once. For this reason, it is often best to introduce reactive behaviour gradually. A good rule of thumb is to do the following: * Start with explicit logic, and build your core behaviour in straightforward queries or application code. * Stabilize your data model and workflows. * Make sure the system behaves correctly before introducing automation. * Introduce reactivity where it adds clear value. * Move well-understood logic into [events](/docs/learn/schema-management/events-and-triggers/defining-events.md) or [live queries](/docs/learn/querying/real-time/live-queries.md) once the behaviour is predictable. * Avoid large or highly coupled chains of events. --- Source: https://surrealdb.com/docs/learn/schema-management/files/buckets # Buckets Defining file buckets: memory, disk, global backends, and permissions. A bucket is named storage that backs file values (`f"bucket:/path"`). You have a few options when defining a bucket: * Memory for non-peristent storage. * A folder on disk for persistence. * A global backend driven by environment variables. Once a bucket is defined, you can then read and write to files inside it through the [file functions](/docs/reference/query-language/functions/database-functions/file.md). Full statement detail: [`DEFINE BUCKET`](/docs/reference/query-language/statements/define/bucket.md). For how pointers look in queries, see [Working with files](/docs/learn/schema-management/files/working-with-files.md). ## Example usage A bucket backend can be set as "memory" for non-persistent in-memory storage, or as "file:/", followed by the path, for storage on disk. ### Memory backend The simplest way to experiment with a bucket for files is by using the memory backend: ```surql DEFINE BUCKET my_bucket BACKEND "memory"; ``` Once this is defined, `my_bucket` can be accessed by using a file pointer: a path prefixed by an `f`. ```surql -- Create a file by adding some content f"my_bucket:/my_book.txt".put("Once there were four children whose names were Peter, Susan, Edmund, and Lucy."); -- Copy it to a new file name f"my_bucket:/my_book.txt".copy("lion_witch_wardrobe.txt"); -- Read the file as bytes f"my_bucket:/lion_witch_wardrobe.txt".get(); -- Cast the bytes to a string f"my_bucket:/lion_witch_wardrobe.txt".get(); ``` ### File backend A file backend can be chosen for a bucket by typing `"file:"` and then the rest of the path, if necessary. ```surql DEFINE BUCKET my_bucket BACKEND "file:/some_directory"; DEFINE BUCKET my_bucket BACKEND "file:/some_directory"; ``` A check will then be made to see if the `SURREAL_BUCKET_FOLDER_ALLOWLIST` environment variable contains the path, without which the following error will be generated. ```surql 'File access denied: /some_directory' ``` The following command can be used to start running an instance in which a bucket with a file backend can be defined. ```bash # Unix SURREAL_BUCKET_FOLDER_ALLOWLIST="/" surreal start --user root --pass \ secret --allow-experimental files # Windows (PowerShell) $env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "/" surreal start --user root --pass secret --allow-experimental files ``` ### Global backend A global backend can also be selected, allowing all namespaces and databases access to the same file storage. If no backend is selected, the database will search for the environment variable `SURREAL_GLOBAL_BUCKET` and assign this as the global bucket. In this case, files will have a `namespace/database` prefix added (e.g. `my_global_bucket:/test_ns/test_db/somefile.txt`). A second `SURREAL_GLOBAL_BUCKET_ENFORCED` environment variable can also be used, which when set to `true` will enforce usage of the global bucket. If a global backend is set, then a `DEFINE BUCKET` statement can be as short as `DEFINE BUCKET` plus its local name, as the rest of the logic is done via environment variables. ```surql DEFINE BUCKET my_bucket; -- Writes to e.g. `my_global_bucket:/test_ns/test_db/my_bucket/my_book.txt` f"my_bucket:/my_book.txt".put("Once there were four children whose names were Peter, Susan, Edmund, and Lucy."); ``` ## Setting permissions on buckets By default, the permissions on a bucket will be set to FULL unless otherwise specified. ```surql DEFINE BUCKET my_bucket BACKEND "memory"; INFO FOR DB; ``` ```surql title="Output" { accesses: {}, analyzers: {}, apis: {}, buckets: { my_bucket: "DEFINE BUCKET my_bucket BACKEND 'memory' PERMISSIONS FULL" }, configs: {}, functions: {}, models: {}, modules: {}, params: {}, sequences: {}, tables: {}, users: {} } ``` You can set permissions on buckets to control who can perform operations on the files stored in them using the `PERMISSIONS` clause. In the clause three additional variables are available: - `$action`: The action to be executed (`put`, `get`, `head`, `delete`, `copy`, `rename`, `exists`, `list`) - `$file`: The [file pointer](/docs/reference/query-language/language-primitives/data-types/files.md) of the file to be accessed - `$target`: The target [file pointer](/docs/reference/query-language/language-primitives/data-types/files.md) in copy/rename operations ```surql -- Set permissions for the bucket DEFINE BUCKET admin_bucket BACKEND "memory" PERMISSIONS WHERE $auth.admin = true ``` --- Source: https://surrealdb.com/docs/learn/schema-management/files/working-with-files # Working with files File pointers, buckets, and putting or getting bytes from SurrealDB. File values look like ordinary paths, but are prefixed with an `f` so SurrealDB knows you mean binary storage, not a plain string. To use files, a bucket must first be defined with the [`DEFINE BUCKET`](/docs/learn/schema-management/files/buckets.md) statement. The easiest way to experiment with file storage is to start with a bucket that holds files in memory. ```surql DEFINE BUCKET my_bucket BACKEND "memory"; ``` Once a bucket has been defined, you can begin calling methods on file pointers such as `.put()` and `.get()` (see the [file functions](/docs/reference/query-language/functions/database-functions/file.md) reference). ```surql DEFINE BUCKET my_bucket BACKEND "memory"; f"my_bucket:/some_file.txt".put("Some text inside"); f"my_bucket:/some_file.txt".get(); f"my_bucket:/some_file.txt".get(); ``` ```surql title="Output" -------- Query -------- b"536F6D65207465787420696E73696465" -------- Query -------- 'Some text inside' ``` ## See also A combination of files and SurrealDB's [encoding functions](/docs/reference/query-language/functions/database-functions/encoding.md#encodingcbordecode) can be used to set up ad-hoc memory storage. This can be convenient when running an instance that saves data to disk but prefers to keep certain items in memory. The following example shows how this pattern might be used for temporary storage such as a user's shopping cart during a single session. **Bash** ```bash # Set the allowlist env var to allow the directory to be accessed SURREAL_BUCKET_FOLDER_ALLOWLIST="/users/your_user_name" surreal start \ --allow-experimental files ``` **PowerShell** ```powershell # Set the allowlist env var to allow the directory to be accessed $env:SURREAL_BUCKET_FOLDER_ALLOWLIST = "C:\Users\your_user_name" surreal start ` --allow-experimental files ``` ```surql -- Set up the in-memory backend DEFINE BUCKET my_bucket BACKEND "file:/users/your_user_name"; -- Convenience functions to save, decode back into -- SurrealQL type, and delete DEFINE FUNCTION fn::save_file($file_name: string, $input: any) { LET $file = type::file("shopping_carts", $file_name); $file.put(encoding::cbor::encode($input)); }; DEFINE FUNCTION fn::get_file($file_name: string) -> object { encoding::cbor::decode(type::file("shopping_carts", $file_name).get()) }; DEFINE FUNCTION fn::delete_file($file_name: string) { type::file("shopping_carts", $file_name).delete(); }; -- Save current shopping cart fn::save_file("temp_cart_user_24567", { items: ["shirt1"], last_updated: time::now() }); fn::get_file("temp_cart_user_24567"); //- { items: ['shirt1', 'deck_of_cards'], last_updated: d'2025-11-20T01:03:24.141080Z' } -- User adds item, save over file with newer information fn::save_file("temp_cart_user_24567", { items: ["shirt1", "deck_of_cards"], last_updated: time::now() }); fn::get_file("temp_cart_user_24567"); //- { items: ['shirt1', 'deck_of_cards'], last_updated: d'2025-11-20T01:06:02.752429Z' } -- Session is over, delete temp file fn::delete_file("temp_cart_user_24567"); ``` --- Source: https://surrealdb.com/docs/learn/schema-management/indexes/index-types-and-strategies # Index types and strategies Choosing indexes for lookups, uniqueness, full-text, vectors, and counts. Indexes speed up the shapes of queries you run often: equality filters, uniqueness checks, text search, and similarity over vectors. You need the same namespace/database scope and privileges as other `DEFINE` statements. Clause-by-clause detail can be found in the [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md) page. ## Index types SurrealDB offers a range of indexing capabilities designed to optimise data retrieval and search efficiency. ### Standard (non-unique) index An index with `FIELDS` or `COLUMNS` and no special clause (`UNIQUE`, `COUNT`, `FULLTEXT`, etc.) is a **standard B-tree index** - the default secondary index type. Let's create a non-unique index for an age field on a user table. ```surql -- optimise queries looking for users of a given age DEFINE INDEX userAgeIndex ON TABLE user FIELDS age; ``` ### Unique index A unique index ensures that each value in the index is unique. A unique index helps enforce uniqueness across records by preventing duplicate entries in fields such as user IDs, email addresses, and other unique identifiers. Let's create a unique index for the email address field on a user table. ```surql -- Makes sure that the email address in the user table is always unique DEFINE INDEX userEmailIndex ON TABLE user FIELDS email UNIQUE; ``` The created index can be viewed using the [`INFO` statement](/docs/reference/query-language/statements/info.md). ```surql INFO FOR TABLE user; ``` The `INFO` statement will help you understand what indexes are defined in your `TABLE`. ```surql { "events": {}, "fields": {}, "indexes": { "userEmailIndex": { sql: "DEFINE INDEX userEmailIndex ON user FIELDS email UNIQUE" } }, "lives": {}, "tables": {} } ``` As we defined a `UNIQUE` index on the `email` column, a duplicate entry for that column or field will throw an error. ```surql -- Create a user record and set an email ID. CREATE user:1 SET email = 'test@surrealdb.com'; -- Create another user record and set the same email ID. CREATE user:2 SET email = 'test@surrealdb.com'; ``` ```surql title="Output" "Database index `userEmailIndex` already contains 'test@surrealdb.com', with record `user:1`" ``` ### Composite index A composite index spans multiple fields of a table. Composite indexes are mainly used to create a unique index when the definition of what is unique pertains to more than one field. ```surql -- Create an index on the account and email fields of the user table DEFINE INDEX test ON user FIELDS account, email UNIQUE; ``` ### Count index _(since v3.0.0)_ A count index uses the `COUNT` special clause instead of the usual `FIELDS` / `COLUMNS` form (which defines a standard B-tree index). Use it with `count()` and `GROUP ALL`. From 3.2.5, a bare `count()` projection [implies `GROUP ALL`](/docs/reference/query-language/statements/select.md#bare-count-implies-group-all). - **`COUNT`** - full-table counts: `SELECT count() FROM
GROUP ALL`. - **`COUNT WHERE `** - filtered counts when the query `WHERE` exactly matches the index condition. An unconditional count index is optional for full-table counts; SurrealDB already uses a `CountScan` fast path that counts keys without deserialising rows. `COUNT WHERE` is most useful for repeated filtered counts without scanning the whole table. ```surql DEFINE INDEX idx ON indexed_reading COUNT; FOR $_ IN 0..100000 { CREATE reading SET temperature = rand::int(0, 10); }; FOR $_ IN 0..100000 { CREATE indexed_reading SET temperature = rand::int(0, 10); }; -- Wait a moment before running these two -- queries to ensure the index is built SELECT count() FROM reading GROUP ALL; SELECT count() FROM indexed_reading GROUP ALL; ``` Filtered example: ```surql DEFINE INDEX item_active_count ON item COUNT WHERE status = "active"; SELECT count() FROM item WHERE status = "active" GROUP ALL; ``` A **standard index** (`DEFINE INDEX … FIELDS …` on the same field) can also accelerate filtered counts when it fully covers the `WHERE` clause; you typically choose one approach, not both. For choosing between standard and count indexes (including storage tradeoffs), see [When to use which](/docs/reference/query-language/statements/define/indexes.md#when-to-use-which). For counting several statuses (`IN` lists vs per-status totals), see [Multiple conditions](/docs/reference/query-language/statements/define/indexes.md#multiple-conditions). ### Full-text search (`FULLTEXT`) index Enables efficient searching through textual data, supporting advanced text-matching features like proximity searches and keyword highlighting. The [Full-Text search](/docs/learn/data-models/full-text-search/overview.md) index helps implement comprehensive search functionalities in applications, such as searching through articles, product descriptions, and user-generated content. Let's create a full-text search index for a `name` field on a `user` table. ```surql -- Define the an analyzer with DEFINE ANALYZER example_ascii TOKENIZERS class FILTERS ascii; -- Since 3.0.0: only FULLTEXT used to benefit from concurrent full-text search DEFINE INDEX userNameIndex ON TABLE user FIELDS name FULLTEXT ANALYZER example_ascii BM25 HIGHLIGHTS; ``` - `SEARCH` or `FULLTEXT`: By using the `SEARCH` keyword, you enable full-text search on the specified column. - `ANALYZER ascii`: Uses a custom [analyzer](/docs/reference/query-language/statements/define/analyzer.md) called `example_ascii` which uses the class tokenizier and `ascii` filter to analysing the text input. - `BM25`: Ranking algorithm used for relevance scoring. - `HIGHLIGHTS`: Allows keyword highlighting in search results output when using the [`search::highlight`](/docs/reference/query-language/functions/database-functions/search.md#searchhighlight) function - `FIELDS`: a full-text search index can only be used on one field at a time. To use full-text search on more than one field, use a separate `DEFINE INDEX` statement for each one. ## Rebuilding indexes Indexes can be rebuilt using the [`REBUILD`](/docs/reference/query-language/statements/rebuild.md) statement. This can be useful when you want to update the index definition or when you want to rebuild the index to optimise performance. You may want to rebuild an index overtime to ensure that the index is up-to-date with the latest data in the table. ```surql REBUILD INDEX userEmailIndex ON user; ``` ## Using `CONCURRENTLY` clause Building indexes can be lengthy and may time out before they're completed. Without `CONCURRENTLY`, `DEFINE INDEX` blocks until the index is ready. They `CONCURRENTLY` clause can be used when the statement should return immediately while the build runs in the background, during which progress can be monitored with [INFO FOR INDEX](/docs/reference/query-language/statements/info.md#index-information). ```surql -- Create an INDEX concurrently DEFINE INDEX test ON user FIELDS email CONCURRENTLY; INFO FOR INDEX test ON user; INFO FOR INDEX test ON user; ``` ## The `DEFER` clause > [!WARNING] > `DEFER` is available in SurrealDB 2.5 through 2.x only. On 3.x the clause is rejected at parse time, and this section applies to 2.x deployments. On 3.x, use [`CONCURRENTLY`](/docs/reference/query-language/statements/define/indexes.md#using-concurrently-clause) to build an index in the background - note that it addresses initial builds, not the ongoing write-path queueing `DEFER` provided. Index updates in SurrealDB occur synchronously during document operations. This ensures immediate consistency, in which all reads return the most recent write. However, this can become a bottleneck during high-volume parallel ingestion, leading to write-write conflicts and increased latency, particularly with Full-Text or Vector indexes. The `DEFER` clause can be used in this case if eventual consistency is acceptable, namely a setting in which reads may return stale data for a short period, but will eventually converge to the most recent write. An index with this clause will be enqueued in a persistent background queue so that ingestion and indexing are decoupled. ```surql DEFINE ANALYZER simple TOKENIZERS blank,class FILTERS lowercase; DEFINE INDEX title_index ON blog FIELDS title SEARCH ANALYZER simple BM25(1.2,0.75) HIGHLIGHTS DEFER; ``` Note: As unique indexes offer a guarantee that no records that contravene the index will ever exist, the `UNIQUE` clause cannot be used together with `DEFER`. ## Performance implications When defining indexes, it's essential to consider the fields most frequently queried or used to optimise performance. Indexes may improve the performance of SurrealQL statements. This may not be noticeable with small tables but it can be significant for large tables; especially when the indexed fields are used in the `WHERE` clause of a [`SELECT`](/docs/reference/query-language/statements/insert.md) statement. Indexes can also impact the performance of write operations ([INSERT](/docs/reference/query-language/statements/insert.md), [UPDATE](/docs/reference/query-language/statements/update.md), [DELETE](/docs/reference/query-language/statements/delete.md)) since the index needs to be updated accordingly. Therefore, it's essential to balance the need for read performance with write performance. --- Source: https://surrealdb.com/docs/learn/schema-management/multi-tenancy/namespace-and-database-architecture # Namespace & database architecture Layering namespaces and databases for isolation, security, and strict mode. SurrealDB scopes data in two levels: a namespace holds one or more databases, and each database holds your tables and other resources. There is no fixed cap on how many namespaces or databases you create. Design is driven by who should be isolated from whom (tenants, environments, product lines) and who is allowed to administer each layer. In fact, there are some social media platforms that even run by creating an entire database per user. A more common SaaS pattern is one namespace per tenant so credentials and data never cross tenant boundaries; another pattern is one namespace per environment (`dev`, `staging`, `prod`) with multiple databases inside. Namespaces are created by root users. Databases live inside a namespace and are created by root or namespace owners/editors once you execute a [`USE`](/docs/reference/query-language/statements/use.md) statement to move to that namespace. Reference pages: [`DEFINE NAMESPACE`](/docs/reference/query-language/statements/define/namespace.md), [`DEFINE DATABASE`](/docs/reference/query-language/statements/define/database.md). For convenience, a new running instance will create a new namespace and database that each have the name `main`. This can be disabled by passing in a flag or setting an environment variable when using the [`surreal start`](/docs/reference/cli/surrealdb-cli/environment-variables.md) command. ## Namespaces ### Example: create a namespace Below shows how you can create a namespace using the `DEFINE NAMESPACE` statement. ```surql DEFINE NAMESPACE platform_ltd; ``` ## Databases A database is where your application schema actually lives. Options such as `STRICT` change whether undefined resources may be created implicitly to allow CRUD operations to perform. ### Example: create a database Below shows how you can create a database using the `DEFINE DATABASE` statement. ```surql -- Specify the namespace for the database USE NS platform_ltd; -- Define database DEFINE DATABASE app_vitalsense; ``` ## Defining a `STRICT` database A strict database is one that does not allow a resource to be used unless it has already been defined. The default behaviour in SurrealDB works otherwise, by allowing statements like [CREATE](/docs/reference/query-language/statements/create.md), [INSERT](/docs/reference/query-language/statements/insert.md), and [UPSERT](/docs/reference/query-language/statements/upsert.md) to work. ```surql CREATE some_new_table; INFO FOR DATABASE.tables; ``` The output of the [INFO](/docs/reference/query-language/statements/info.md) statement shows that a table called `some_new_table` has been created with a few default clauses. ```surql { some_new_table: 'DEFINE TABLE some_new_table TYPE ANY SCHEMALESS PERMISSIONS NONE' } ``` Such an operation within a strict database is simply not allowed. ```surql DEFINE DATABASE new_db STRICT; USE DATABASE new_db; CREATE some_new_table; ``` ```surql title="Output" "The table 'some_new_table' does not exist" ``` --- Source: https://surrealdb.com/docs/learn/schema-management/schema-design # Schema design What DEFINE does in SurrealDB. How to inspect what you have defined. Almost everything structural in SurrealDB starts with `DEFINE`: namespaces, databases, tables, fields, indexes, functions, access methods, parameters, and more. The [DEFINE overview](/docs/reference/query-language/statements/define/overview.md) in the API docs shows the syntax for each statment. Each such resource also has an `ALTER` and `REMOVE` statement if you need to change or remove a definition. ## Seeing what is defined A schema will quickly grow to the point that it is no longer possible to keep in your head. The [`INFO`](/docs/reference/query-language/statements/info.md) statement can be used to show what definitions exist. `INFO` statements can be used on a variety of resources, such as `INFO FOR DATABASE`, `INFO FOR TABLE table_name`, `INFO FOR INDEX index_name`, and so on. ```surql DEFINE FIELD name ON TABLE person TYPE string COMMENT "Todo: add assertion for maximum length"; INFO FOR TABLE person; ``` ```surql output="Response" { events: {}, fields: { name: "DEFINE FIELD name ON person TYPE string COMMENT 'Todo: add assertion for maximum length' PERMISSIONS FULL" }, indexes: {}, lives: {}, tables: {} } ``` Users and other global objects show up when you ask at database level: ```surql DEFINE USER db_user ON DATABASE PASSWORD "strongpassword" ROLES OWNER; DEFINE TABLE person SCHEMAFULL; INFO FOR DB; ``` ```surql output="Response" { accesses: {}, analyzers: {}, functions: {}, models: {}, params: {}, tables: { person: 'DEFINE TABLE person TYPE ANY SCHEMAFULL PERMISSIONS NONE' }, users: { db_user: "DEFINE USER db_user ON DATABASE PASSHASH '[REDACTED]' ROLES OWNER" } } ``` ## Where to go next * [Tables](/docs/learn/schema-management/tables-and-fields/tables.md) and [fields](/docs/learn/schema-management/tables-and-fields/fields-and-validation.md) * [Schema best practices](/docs/learn/schema-management/schema-design/schema-best-practices.md) * [Schema evolution](/docs/learn/schema-management/schema-design/schema-evolution.md) * [SurrealKit schema migration](/docs/manage/schema-migration.md) - manage `.surql` schema files and apply them with sync or rollouts --- Source: https://surrealdb.com/docs/learn/schema-management/schema-design/sample-industry-schemas # Sample industry schemas Copy-paste starter schemas for energy, finance, retail, medical, and other domains. These snippets are starting points that are realistic enough to learn from, short enough to paste into SurrealDB Studio or to lift into [SurrealKit](/docs/manage/schema-migration.md) `.surql` schema files and then reshape. They mix **tables**, **relations**, **computed** fields, **events**, and **indexes** the way a real app might. > [!NOTE] > Many fields use [`COMPUTED`](/docs/reference/query-language/statements/define/field.md#computed-fields) (SurrealDB 3.0.0 onward). On older versions, replace with a [`future`](/docs/reference/query-language/language-primitives/data-types/futures.md) and `VALUE { … }` as in the futures documentation. ## Adding to this page Have a sample schema of your own that you'd like to add? If it's about 50 lines in length then feel free to [make a PR](https://github.com/surrealdb/docs.surrealdb.com/edit/main/src/content/doc-surrealdb/reference-guide/sample-industry-schemas.mdx) and we'll credit the addition with a link to your profile on a code hosting platform (e.g. GitHub, GitLab, Codeberg). You can also [get in touch](/contact) with us if you'd like a sample schema that isn't in this page that fits the industry in which you work. ## Energy and manufacturing ### Project planning A comprehensive project management schema that demonstrates activity scheduling, milestone tracking, and dependency management using [graph relationships](/docs/learn/data-models/graph/overview.md). This schema shows how to model complex project workflows with interdependent tasks and progress tracking using [`COMPUTED` fields](/docs/reference/query-language/statements/define/field.md#computed-fields) for calculated values. ```surql DEFINE TABLE project; -- Activities in a project schedule DEFINE TABLE activity SCHEMAFULL; DEFINE FIELD name ON activity TYPE string; DEFINE FIELD description ON activity TYPE option; DEFINE FIELD start ON activity TYPE datetime; DEFINE FIELD end ON activity TYPE datetime; DEFINE FIELD duration ON activity COMPUTED end - start; DEFINE FIELD progress ON activity TYPE float ASSERT $value IN 0.0..=1.0; DEFINE FIELD assigned_to ON activity TYPE option>; DEFINE FIELD followed_by ON activity COMPUTED <-depends_on<-activity; -- Milestones DEFINE TABLE milestone SCHEMAFULL; DEFINE FIELD project ON milestone TYPE record; DEFINE FIELD activities ON milestone TYPE array>; DEFINE FIELD name ON milestone TYPE string; DEFINE FIELD last_updated ON milestone VALUE time::now(); DEFINE FIELD progress ON milestone COMPUTED math::mean(activities.progress); DEFINE FIELD is_complete ON milestone COMPUTED activities.all(|$a| $a.progress > 0.95); -- Graph-style dependency links DEFINE TABLE depends_on SCHEMAFULL TYPE RELATION IN activity OUT activity; DEFINE TABLE activity_of SCHEMAFULL TYPE RELATION IN activity OUT project; CREATE project:one SET name = "Construction project"; CREATE activity:one SET name = "Project kickoff", start = time::now(), end = time::now() + 2d, progress = 1.0; CREATE activity:two SET name = "Pour concrete", start = time::now() + 90d, end = time::now() + 100d, progress = 0.0; CREATE activity:three SET name = "Dry concrete", start = time::now() + 100d, end = time::now() + 107d, progress = 0.0; CREATE activity:four SET name = "Build on top of concrete", start = time::now() + 107d, end = time::now() + 150d, progress = 0.0; RELATE activity:two->depends_on->activity:one; RELATE activity:three->depends_on->activity:two; RELATE activity:four->depends_on->activity:three; RELATE [activity:one,activity:two,activity:three, activity:four]->activity_of->project:one; CREATE milestone:one SET project = project:one, activities = [activity:one], name = "Project start"; CREATE milestone:two SET project = project:one, activities = [activity:two, activity:three, activity:four], name = "Initial construction"; -- See all graph connections between activity and project records SELECT *, ->? AS joins_to, <-? AS joined_from FROM activity, project; -- View the current milestones SELECT * FROM milestone; ``` ### SCADA (oil and gas) Industrial monitoring and control system schema for oil and gas operations. Demonstrates real-time sensor data collection, automated alert generation using [events](/docs/reference/query-language/statements/define/event.md), and [time-series data management](/docs/learn/data-models/time-series/overview.md) with composite keys. Shows how to handle flexible external data integration and [live query monitoring](/docs/reference/query-language/statements/live-select.md). ```surql DEFINE TABLE sensor SCHEMAFULL; DEFINE FIELD type ON sensor TYPE array ASSERT $value ALLINSIDE ["pressure", "temperature", "flow", "level"]; DEFINE FIELD location ON sensor TYPE point; DEFINE TABLE reading SCHEMAFULL; DEFINE FIELD id ON reading TYPE [record, datetime]; DEFINE FIELD pressure ON reading TYPE float; -- Optional telemetry values DEFINE FIELD temperature ON reading TYPE option; DEFINE FIELD humidity ON reading TYPE option; -- Flexible object for weather or external data DEFINE FIELD weather ON reading TYPE option FLEXIBLE; DEFINE TABLE alert SCHEMAFULL; DEFINE FIELD equipment ON alert TYPE record; DEFINE FIELD severity ON alert TYPE string ASSERT $value IN ["critical", "high", "medium", "low", "info"]; DEFINE FIELD message ON alert TYPE string; DEFINE FIELD triggered_at ON alert TYPE datetime; -- Create a sensor CREATE sensor:one SET type = ["temperature", "pressure"], location = (50.0, 50.0); -- And a reading for the sensor CREATE reading:[sensor:one, time::now()] SET pressure = 600, -- JSON object sourced from somewhere else, `weather` field is a schemaless object so can be any object format weather = { "temperature": 17.4, "humidity": 52.0, "wind_speed": 12.8 }; -- Set up event to generate alerts DEFINE EVENT alert_from_create ON reading WHEN $event = "CREATE" THEN { LET $source = $after.id[0]; LET $time = $after.id[1]; -- Select everything over the past 15 minutes up to but not including the present reading LET $recents_average = math::mean(SELECT VALUE pressure FROM reading:[$source, $time - 15m]..[$source, $time]); LET $drop = $recents_average - $after.pressure; IF $drop > 15 { CREATE alert SET equipment = $source, severity = "high", message = "Pressure drop over 15 PSI: drop of " + $drop, triggered_at = time::now(); }; }; -- Some readings with good values FOR $_ IN 0..10 { -- Sleep to keep timestamp in IDs unique, consider a ULID instead if timestamps may not be unique sleep(10ns); CREATE reading:[sensor:one, time::now()] SET pressure = 600; }; -- Pressure has suddenly dropped CREATE reading:[sensor:one, time::now()] SET pressure = 500; -- See the alert SELECT * FROM alert; -- Or use a LIVE SELECT for alerts: https://surrealdb.com/docs/reference/query-language/statements/live-select LIVE SELECT * FROM alert; ``` ### Risk management Project risk assessment and mitigation tracking schema. Features temporal risk modeling with active/inactive periods, probability-impact calculations, and automated risk scoring using [futures](/docs/reference/query-language/statements/define/field.md#futures). Demonstrates [unique constraints](/docs/reference/query-language/statements/define/indexes.md#unique-indexes) and complex mathematical aggregations across related records. ```surql DEFINE TABLE risk SCHEMAFULL; DEFINE FIELD project ON risk TYPE record REFERENCE; DEFINE FIELD description ON risk TYPE string; DEFINE FIELD category ON risk TYPE string; -- e.g. "technical", "commercial", "regulatory" DEFINE FIELD likelihood ON risk TYPE float ASSERT $value IN 0.0..=1.0; DEFINE FIELD maximum_impact ON risk TYPE int; -- in dollars, etc. DEFINE FIELD start ON risk TYPE datetime; DEFINE FIELD end ON risk TYPE datetime; -- Use a computed field to calculate value on each SELECT DEFINE FIELD active ON risk COMPUTED time::now() IN start..=end; -- Ensure no duplicate `risk` records exist for each project DEFINE INDEX risk_name ON risk FIELDS project, description UNIQUE; -- See all total_impact DEFINE FIELD total_risk_impact ON project COMPUTED math::sum(<~risk.map(|$risk| $risk.maximum_impact * $risk.likelihood)); -- See risks at the current date DEFINE FIELD current_risk_impact ON project COMPUTED math::sum(<~risk.filter(|$r| $r.active).map(|$risk| $risk.maximum_impact * $risk.likelihood)); CREATE project:one; CREATE risk SET project = project:one, description = "Migratory elk", category = "regulatory", likelihood = 0.9, start = d'2025-10-01', end = d'2025-12-15', maximum_impact = 1000000; CREATE risk SET project = project:one, description = "Wildfires", category = "technical", likelihood = 0.5, start = d'2025-06-01', end =d'2025-10-15', maximum_impact = 10000000; SELECT * FROM project; ``` ### Supply chain and contract management Vendor relationship and contract lifecycle management schema. Covers contract value tracking with change orders, deliverable management, and automated total commitment calculations using [futures](/docs/reference/query-language/statements/define/field.md#futures). Demonstrates complex financial calculations and status tracking across multiple related entities. ```surql -- Vendors who supply goods or services DEFINE TABLE vendor SCHEMAFULL; DEFINE FIELD name ON vendor TYPE string; -- Contracts awarded under a project DEFINE TABLE contract SCHEMAFULL; DEFINE FIELD project ON contract TYPE record; DEFINE FIELD vendor ON contract TYPE record; DEFINE FIELD title ON contract TYPE string; DEFINE FIELD original_value ON contract TYPE int; DEFINE FIELD total_value ON contract COMPUTED original_value + math::sum(SELECT VALUE amount FROM change_order WHERE contract = $parent.id); DEFINE FIELD currency ON contract TYPE "dollars" | "euro"; DEFINE FIELD start ON contract TYPE datetime; DEFINE FIELD end ON contract TYPE datetime; -- Deliverables expected under a contract DEFINE TABLE deliverable SCHEMAFULL; DEFINE FIELD contract ON deliverable TYPE record; DEFINE FIELD description ON deliverable TYPE string; DEFINE FIELD due_date ON deliverable TYPE datetime; DEFINE FIELD received ON deliverable TYPE option; DEFINE FIELD status ON deliverable COMPUTED IF $parent.received { "complete" } ELSE { "pending" }; -- Change orders during a project DEFINE TABLE change_order SCHEMAFULL; DEFINE FIELD contract ON change_order TYPE record; DEFINE FIELD amount ON change_order TYPE int; DEFINE FIELD description ON change_order TYPE string; DEFINE FIELD signed_on ON change_order TYPE option; -- Total committed value of a project (sum of all contract values) DEFINE FIELD total_commitment ON project COMPUTED math::sum((SELECT VALUE value FROM contract WHERE project = $parent.id)); CREATE project:one; CREATE vendor:one SET name = "Good vendor"; CREATE contract:one SET project = project:one, currency = "euro", start = d'2025-12-01', end = d'2026-01-01', original_value = 1000, title = "Services for so-and-so project", vendor = vendor:one; CREATE change_order SET contract = contract:one, amount = 500, description = "Highway wasn't set up yet"; SELECT * FROM contract; ``` ### Procure-to-pay (SAP-aligned starter) Oil-and-gas-style **procure-to-pay** flow: WBS cost objects, purchase orders, vendor invoices, and payment terms (due on receipt through Net 45 and beyond). Each business object carries an **`sap_*`** external key so rows synced from **SAP PS / MM / FI-AP** can upsert without replacing SAP as the system of record. Continues the **`project`** and **`vendor`** tables from [Supply chain and contract management](#supply-chain-and-contract-management) above, or run this snippet on its own. SAP payment terms are master data (arbitrary net days, cash discounts, calendar rules). Here you can store a normalised **`net_due`** [`duration`](/docs/reference/query-language/language-primitives/data-types/durations.md) for queries, and treat **`code`** as the authoritative key from SAP. Document-number **`ASSERT`** patterns below are deliberately generic (ten-digit numeric). Assertions on prefixes (for example that PO ranges must start with `45`) can be tightened once your connector documents a specific number range. Demonstrates [literal unions](/docs/reference/query-language/language-primitives/data-types/literals.md), [`duration`](/docs/reference/query-language/language-primitives/data-types/durations.md), [graph relations](/docs/reference/query-language/statements/relate.md) with [`ENFORCED`](/docs/reference/query-language/statements/define/table.md#using-enforced-to-ensure-that-related-records-exist) endpoints, [record references](/docs/reference/query-language/language-primitives/record-references.md), [field assertions](/docs/reference/query-language/statements/define/field.md#asserting-rules-on-fields) on relation `in` / `out`, and [events](/docs/reference/query-language/statements/define/event.md) for lifecycle rules. ```surql DEFINE TABLE project SCHEMAFULL; DEFINE FIELD name ON project TYPE string ASSERT string::len($value) > 3; DEFINE TABLE vendor SCHEMAFULL; DEFINE FIELD name ON vendor TYPE string ASSERT string::len($value) > 3; DEFINE TABLE payment_terms SCHEMAFULL; DEFINE FIELD code ON payment_terms TYPE string ASSERT $value.len() IN 1..=10; DEFINE FIELD net_due ON payment_terms TYPE duration; DEFINE FIELD description ON payment_terms TYPE string; DEFINE INDEX payment_terms_code ON payment_terms FIELDS code UNIQUE; DEFINE FIELD sap_vendor_number ON vendor TYPE option ASSERT $value = /[0-9]{10}/; DEFINE FIELD default_payment_terms ON vendor TYPE option>; DEFINE TABLE wbs_element SCHEMAFULL; DEFINE FIELD sap_id ON wbs_element TYPE string ASSERT $value.len() IN 1..=24; DEFINE FIELD project ON wbs_element TYPE record REFERENCE ON DELETE REJECT; DEFINE FIELD parent ON wbs_element TYPE option>; DEFINE FIELD name ON wbs_element TYPE string ASSERT $value.len() > 0; DEFINE INDEX wbs_sap_id ON wbs_element FIELDS sap_id UNIQUE; -- Graph: WBS hierarchy and project membership (dual-write with record fields above) DEFINE TABLE wbs_of SCHEMAFULL TYPE RELATION IN wbs_element OUT project ENFORCED; ALTER FIELD in ON wbs_of ASSERT in.project = out; ALTER FIELD out ON wbs_of ASSERT in.project = out; DEFINE TABLE wbs_child_of SCHEMAFULL TYPE RELATION IN wbs_element OUT wbs_element ENFORCED; ALTER FIELD in ON wbs_child_of ASSERT in != out AND in.project = out.project AND in.parent = out; ALTER FIELD out ON wbs_child_of ASSERT in != out AND in.project = out.project AND in.parent = out; DEFINE TABLE purchase_order SCHEMAFULL; -- Ten-digit numeric document numbers are common; add a prefix (e.g. /^45/) per deployment. DEFINE FIELD sap_po_number ON purchase_order TYPE string ASSERT $value = /[0-9]{10}/; DEFINE FIELD vendor ON purchase_order TYPE record REFERENCE ON DELETE REJECT; DEFINE FIELD currency ON purchase_order TYPE "CAD" | "USD" | "EUR"; DEFINE FIELD status ON purchase_order TYPE "open" | "closed" | "cancelled" DEFAULT "open"; DEFINE FIELD issued_at ON purchase_order TYPE datetime; DEFINE INDEX po_sap_number ON purchase_order FIELDS sap_po_number UNIQUE; DEFINE TABLE ordered_from SCHEMAFULL TYPE RELATION IN purchase_order OUT vendor ENFORCED; ALTER FIELD in ON ordered_from ASSERT in.vendor = out; ALTER FIELD out ON ordered_from ASSERT in.vendor = out; DEFINE TABLE po_line SCHEMAFULL; DEFINE FIELD purchase_order ON po_line TYPE record REFERENCE ON DELETE CASCADE; DEFINE FIELD line_number ON po_line TYPE int ASSERT $value IN 0..=9990; DEFINE FIELD description ON po_line TYPE string ASSERT $value.len() > 0; DEFINE FIELD amount_cents ON po_line TYPE int ASSERT $value >= 100; DEFINE FIELD wbs ON po_line TYPE record REFERENCE ON DELETE REJECT; DEFINE INDEX po_line_key ON po_line FIELDS purchase_order, line_number UNIQUE; DEFINE TABLE charges SCHEMAFULL TYPE RELATION IN po_line OUT wbs_element ENFORCED; ALTER FIELD in ON charges ASSERT in.wbs = out; ALTER FIELD out ON charges ASSERT in.wbs = out; DEFINE TABLE vendor_invoice SCHEMAFULL; DEFINE FIELD sap_doc_number ON vendor_invoice TYPE string ASSERT $value = /[0-9]{10}/; DEFINE FIELD purchase_order ON vendor_invoice TYPE record REFERENCE ON DELETE REJECT; DEFINE FIELD vendor ON vendor_invoice TYPE record ASSERT vendor = purchase_order.vendor; DEFINE FIELD amount_cents ON vendor_invoice TYPE int ASSERT $value >= 100; DEFINE FIELD payment_terms ON vendor_invoice TYPE record; DEFINE FIELD baseline_date ON vendor_invoice TYPE datetime; DEFINE FIELD due_date ON vendor_invoice TYPE datetime ASSERT $value >= baseline_date; DEFINE FIELD status ON vendor_invoice TYPE "parked" | "approved" | "paid" | "blocked" DEFAULT "parked"; DEFINE INDEX invoice_sap_number ON vendor_invoice FIELDS sap_doc_number UNIQUE; DEFINE TABLE invoices SCHEMAFULL TYPE RELATION IN vendor_invoice OUT purchase_order ENFORCED; ALTER FIELD in ON invoices ASSERT in.purchase_order = out AND in.vendor = out.vendor; ALTER FIELD out ON invoices ASSERT in.purchase_order = out AND in.vendor = out.vendor; DEFINE EVENT po_must_have_lines ON purchase_order WHEN $event = "UPDATE" AND $before.status != "closed" AND $after.status = "closed" THEN { LET $lines = ( SELECT id FROM po_line WHERE purchase_order = $after.id ); IF $lines.len() = 0 { THROW "Cannot close a purchase order with no lines"; }; }; CREATE payment_terms:net30 SET code = "NT30", net_due = 30d, description = "Net 30 days"; CREATE payment_terms:net27 SET code = "Z027", net_due = 27d, description = "Net 27 days"; CREATE payment_terms:payrcpt SET code = "Z000", net_due = 0d, description = "Due on receipt"; CREATE project:pad3 SET name = "Pad 3 expansion"; CREATE wbs_element:pad3 SET sap_id = "EP-24-PAD3", project = project:pad3, name = "Pad 3 package"; CREATE wbs_element:pad3_civil SET sap_id = "EP-24-PAD3-CIVIL", project = project:pad3, parent = wbs_element:pad3, name = "Civil & earthworks"; RELATE wbs_element:pad3 -> wbs_of -> project:pad3; RELATE wbs_element:pad3_civil -> wbs_of -> project:pad3; RELATE wbs_element:pad3_civil -> wbs_child_of -> wbs_element:pad3; CREATE vendor:welding SET name = "Northern Weld Services Ltd", sap_vendor_number = "0000100042", default_payment_terms = payment_terms:net30; CREATE vendor:small SET name = "Camp Catering Co-op", sap_vendor_number = "0000100891", default_payment_terms = payment_terms:payrcpt; CREATE purchase_order:450001 SET sap_po_number = "4500000123", vendor = vendor:welding, currency = "CAD", issued_at = d'2026-05-15'; RELATE purchase_order:450001 -> ordered_from -> vendor:welding; CREATE po_line:line10 SET purchase_order = purchase_order:450001, line_number = 10, description = "Structural welding - pipe rack", amount_cents = 18500000, wbs = wbs_element:pad3_civil; RELATE po_line:line10 -> charges -> wbs_element:pad3_civil; CREATE vendor_invoice:510001 SET sap_doc_number = "5100000456", purchase_order = purchase_order:450001, vendor = vendor:welding, amount_cents = 9250000, payment_terms = payment_terms:net30, baseline_date = d'2026-06-01', due_date = d'2026-07-01', status = "approved"; RELATE vendor_invoice:510001 -> invoices -> purchase_order:450001; -- Record fields (connector / tabular queries) SELECT *, purchase_order.{sap_po_number, status, vendor: vendor.name} AS po FROM po_line WHERE wbs = wbs_element:pad3_civil; -- Graph: lines charging a WBS, then their POs and vendor SELECT name, sap_id, <-charges<-po_line.{ line_number, amount_cents, sap_po_number: purchase_order.sap_po_number } AS lines, ->wbs_child_of->wbs_element.{ name, sap_id } AS parent_wbs FROM wbs_element:pad3_civil; SELECT sap_doc_number, amount_cents / 100 AS amount_dollars, due_date, status FROM vendor_invoice WHERE status IN ["parked", "approved"] ORDER BY due_date; ``` ### HSSE (health, safety, security, environment) incidents Incident reporting and investigation schema using [graph relationships](/docs/learn/data-models/graph/overview.md). Models safety events as edges between employees and projects with severity classification and role identification. Demonstrates graph-style data modeling for complex incident tracking and analysis. ```surql -- Projects and employees (nodes) DEFINE TABLE project SCHEMAFULL; DEFINE FIELD name ON project TYPE string; DEFINE TABLE employee SCHEMAFULL; DEFINE FIELD name ON employee TYPE string; -- Edges: incident links employee → project DEFINE TABLE incident SCHEMAFULL TYPE RELATION IN employee OUT project; DEFINE FIELD severity ON incident TYPE string ASSERT $value IN ["minor", "moderate", "major", "fatal"]; DEFINE FIELD type ON incident TYPE string ASSERT $value IN ["safety", "environment", "security", "health"]; DEFINE FIELD description ON incident TYPE string; DEFINE FIELD occurred_at ON incident TYPE datetime; DEFINE FIELD role ON incident TYPE string ASSERT $value IN ["witness", "injured", "involved"]; -- Create nodes CREATE employee:one SET name = "John Doe"; CREATE employee:two SET name = "Sally Lee"; CREATE project:one SET name = "Pad 3 Expansion"; -- Create incidents as edges with properties RELATE employee:one->incident->project:one SET severity = "moderate", type = "safety", description = "Pinched hand during pipe fitting", occurred_at = time::now() - 5d, role = "injured"; RELATE employee:two->incident->project:one SET severity = "moderate", type = "safety", description = "Pinched hand during pipe fitting", occurred_at = time::now() - 5d, role = "witness"; SELECT id, <-incident[WHERE severity = "moderate"]<-employee FROM project; ``` ## Finance ### General bank schema (graph schema) Multi-currency banking system using [graph relationships](/docs/learn/data-models/graph/overview.md). Demonstrates polymorphic account types (JPY, EUR, CAD, USD) with different field structures, customer-bank relationships, and [unique constraint enforcement](/docs/reference/query-language/statements/define/indexes.md#unique-indexes). Shows how to model complex financial relationships with type-specific behaviors. ```surql DEFINE TABLE bank SCHEMAFULL; DEFINE FIELD name ON bank TYPE string; DEFINE TABLE customer SCHEMAFULL; DEFINE FIELD name ON customer TYPE string; DEFINE TABLE customer_of TYPE RELATION IN customer OUT bank; DEFINE FIELD since ON customer_of TYPE datetime VALUE time::now() READONLY; DEFINE TABLE jpy SCHEMAFULL; DEFINE FIELD amount ON jpy TYPE int DEFAULT 0; DEFINE FIELD cent ON jpy TYPE int READONLY VALUE 0; DEFINE TABLE eur SCHEMAFULL; DEFINE FIELD amount ON eur TYPE int DEFAULT 0; DEFINE FIELD cent ON eur TYPE int ASSERT $value IN 0..=99; DEFINE FIELD total ON eur VALUE amount + (cent / 100); DEFINE TABLE cad SCHEMAFULL; DEFINE FIELD amount ON cad TYPE int DEFAULT 0; DEFINE FIELD cent ON cad TYPE int ASSERT $value IN 0..=99; DEFINE FIELD total ON cad VALUE amount + (cent / 100); DEFINE TABLE usd SCHEMAFULL; DEFINE FIELD amount ON usd TYPE int DEFAULT 0; DEFINE FIELD cent ON usd TYPE int ASSERT $value IN 0..=99; DEFINE FIELD total ON usd VALUE amount + (cent / 100); DEFINE TABLE account TYPE RELATION IN customer OUT jpy|eur|cad|usd; DEFINE FIELD since ON account TYPE datetime VALUE time::now() READONLY; -- stop the same customer opening two wallets in the same currency DEFINE INDEX unique_wallet ON account FIELDS in, out UNIQUE; CREATE bank:one SET name = "Central Bank"; CREATE customer:one SET name = "Billy"; RELATE customer:one->customer_of->bank:one; RELATE customer:one->account->(CREATE ONLY eur SET amount = 100, cent = 50); RELATE customer:one->account->(CREATE ONLY jpy SET amount = 10000); SELECT ->account->eur.total FROM customer:one; ``` ### Other bank-customer schema Traditional bank-customer schema with advanced features including [record references](/docs/reference/query-language/language-primitives/record-links.md#record-references), automated cent handling through [events](/docs/reference/query-language/statements/define/event.md), and historical interest rate tracking. Demonstrates event-driven data validation, [parameter usage](/docs/reference/query-language/statements/define/param.md), and complex relationship management with reference fields. ```surql DEFINE PARAM $CURRENCIES VALUE ["EUR", "JPY", "USD", "CAD"]; DEFINE TABLE account SCHEMAFULL; DEFINE FIELD customer ON account TYPE record REFERENCE; DEFINE FIELD currency ON account TYPE string ASSERT $value IN $CURRENCIES; DEFINE FIELD amount ON account TYPE int; DEFINE FIELD cent ON account TYPE option; DEFINE TABLE customer SCHEMAFULL; DEFINE FIELD name ON customer TYPE string; DEFINE FIELD bank ON customer TYPE record REFERENCE; DEFINE TABLE bank SCHEMAFULL; DEFINE FIELD name ON bank TYPE string; DEFINE FIELD code ON bank TYPE string; -- e.g., BIC or internal short code DEFINE FIELD swift ON bank TYPE option; DEFINE FIELD supported_currencies ON bank TYPE set ASSERT $value ALLINSIDE $CURRENCIES; DEFINE FIELD interest_rate ON bank TYPE float DEFAULT 0.0; DEFINE FIELD historical_interest_rates ON bank TYPE array<{ rate: float, set_at: datetime }> DEFAULT []; DEFINE FIELD customers ON bank COMPUTED <~customer; -- No assert for cent field, but event to update when > 100 or < 0 DEFINE EVENT update_cents ON account WHEN $event = "UPDATE" THEN { IF cent > 99 { UPDATE $after SET cent -= 100, amount += 1; } ELSE IF cent < 0 { UPDATE $after SET cent += 100, amount -= 1; } }; -- No assert for cent field, but event to update when > 100 or < 0 DEFINE EVENT update_interest_rate ON bank WHEN $event = "UPDATE" THEN { IF $before.interest_rate != $after.interest_rate { UPDATE $this SET historical_interest_rates += { rate: $after.interest_rate, set_at: time::now() }; } }; CREATE bank:one SET name = "Bank of One", code = "ONEBANK", supported_currencies = ["EUR", "JPY"]; UPDATE bank:one SET interest_rate = 5.0; CREATE customer:one SET bank = bank:one, name = "Galen Pathwarden"; CREATE account:one SET customer = customer:one, currency = "JPY", amount = 10000; SELECT *, customers.{id, name} FROM bank; ``` ### Customers and money transfers Secure money transfer system with credit-based limits and transaction logging. Features [custom functions](/docs/reference/query-language/statements/define/function.md) for atomic transfers, credit level enforcement, and comprehensive audit trails. Demonstrates transaction safety, business rule enforcement, and financial data integrity. ```surql DEFINE TABLE customer SCHEMAFULL; -- trusted customers can have greater negative amounts DEFINE FIELD amount ON customer ASSERT $value >= -1000 * credit_level; DEFINE FIELD credit_level ON customer TYPE int ASSERT $value IN 0..=5; -- Logs for money transfers DEFINE TABLE transfer SCHEMAFULL; DEFINE FIELD from ON transfer TYPE record; DEFINE FIELD to ON transfer TYPE record; DEFINE FIELD amount ON transfer TYPE int; DEFINE FIELD ts ON transfer TYPE datetime DEFAULT time::now(); DEFINE FUNCTION fn::send_money($from: record, $to: record, $amount: int) -> record { -- Use manual transaction for all statements so all changes are rolled back -- if something is wrong BEGIN; IF $amount < 1 { THROW "Can't send less than 1 "; }; UPDATE $from SET amount -= $amount; UPDATE $to SET amount += $amount; LET $tx = CREATE ONLY transfer SET from = $from, to = $to, amount = $amount; COMMIT; -- Return the transfer record as a receipt the caller can use $tx.id }; CREATE customer:one SET amount = 100, credit_level = 0; CREATE customer:two SET amount = 500, credit_level = 5; -- customer:one has bad credit, can't be negative fn::send_money(customer:one, customer:two, 500); -- but customer:two can fn::send_money(customer:two, customer:one, 1000); SELECT * FROM customer; SELECT * FROM transfer; ``` ### Loans and repayments Loan management system with automated interest calculations and repayment scheduling. Features [parameterised loan terms](/docs/reference/query-language/statements/define/param.md), mathematical payment calculations using [custom functions](/docs/reference/query-language/statements/define/function.md), and status tracking. Demonstrates complex financial formulas, temporal data management, and regulatory compliance constraints. ```surql -- Some government-set maximum term for loans DEFINE PARAM $MAX_TERM VALUE 84; DEFINE TABLE loan SCHEMAFULL; DEFINE FIELD customer ON loan TYPE record; DEFINE FIELD principal ON loan TYPE int; -- Total borrowed, in cents DEFINE FIELD interest_rate ON loan TYPE float; -- e.g., 5.5 for 5.5% DEFINE FIELD issued_at ON loan TYPE datetime; -- loans issuable at units of 6 months each DEFINE FIELD term_months ON loan TYPE int ASSERT $value % 6 = 0 AND $value <= $MAX_TERM; DEFINE FIELD balance ON loan TYPE int; -- Remaining amount to repay DEFINE FIELD status ON loan TYPE string ASSERT $value IN ["active", "paid", "defaulted"]; DEFINE TABLE repayment SCHEMAFULL; DEFINE FIELD loan ON repayment TYPE record; DEFINE FIELD due_date ON repayment TYPE datetime; DEFINE FIELD amount ON repayment TYPE int; DEFINE FIELD paid ON repayment TYPE bool DEFAULT false; DEFINE FIELD paid_at ON repayment TYPE option; DEFINE FIELD method ON repayment TYPE option; -- e.g., "auto", "manual" FOR $loan IN SELECT * FROM loan { LET $update_rate = 1 + ($loan.interest_rate / 365); UPDATE $loan SET balance = math::round(balance * $update_rate); }; DEFINE FUNCTION fn::repayment_amount($loan: record) -> float { LET $P = $loan.principal; LET $annual = $loan.interest_rate / 100; LET $r = $annual / 12; -- Monthly interest rate LET $n = $loan.term_months; math::round( $P * $r / (1 - math::pow(1 + $r, -$n)) ); }; CREATE customer:one; CREATE loan:one SET customer = customer:one, principal = 5000000, interest_rate = 5.0, issued_at = time::now(), term_months = 12, balance = 5000000, status = "active"; UPDATE loan:one SET balance -= fn::repayment_amount(loan:one); ``` ### Fraud prevention patterns Anti-fraud detection system using [events](/docs/reference/query-language/statements/define/event.md) and temporal analysis. Implements velocity checks, new account restrictions, and suspicious transaction pattern detection. Demonstrates real-time fraud prevention, temporal constraints, and complex business rule enforcement through database events. ```surql DEFINE FIELD created_at ON account VALUE time::now() READONLY; DEFINE EVENT cancel_high_volume ON TABLE sends WHEN $event = "CREATE" THEN { IF $after.amount > 1000 AND time::now() - $after.in.created_at < 1d { THROW "New accounts can only send up to $1000 per transaction"; } }; DEFINE FIELD sent_at ON TABLE sends VALUE time::now() READONLY; DEFINE EVENT cancel_rapid_transactions ON TABLE sends WHEN $event = "CREATE" THEN { LET $sender = $after.in; LET $receiver = $after.out; -- Disallow more than two transactions within a 5 minute period LET $recents = $sender->sends[WHERE out = $receiver] .filter(|$tx| time::now() - $tx.sent_at < 5m); IF $recents.len() > 2 { THROW "Can't send that many times within a short period of time"; }; }; ``` ### Using SurrealDB Studio's graph visualisation to see fraudulent activities For more on these queries and their visual output, see [this dedicated blog post](/blog/fraud-detection-with-surrealdb). Star pattern: one card used to pay large number of accounts: ```surql DEFINE FIELD paid_at ON pays DEFAULT time::now(); -- sketchy cards FOR $card IN CREATE |card:10| { FOR $_ IN 0..rand::int(5, 15) { LET $payee = UPSERT ONLY account; RELATE $card->pays->$payee SET amount = rand::int(100, 1000); }; }; -- regular card CREATE card:normal; FOR $_ IN 0..rand::int(5, 15) { LET $payee = UPSERT ONLY account; RELATE card:normal->pays->$payee SET amount = rand::int(100, 1000), paid_at = time::now() - rand::duration(1d, 100d); }; SELECT id, ->pays.filter(|$payment| time::now() - $payment.paid_at < 1d).out FROM card; ``` Tight communities that interact mostly among themselves: ```surql -- Regular community of 200 CREATE |account:200|; -- Smaller community that interacts among itself CREATE |account:5| SET is_sketchy = true; -- The sketchy community interacts only between itself -- the regular community has more general interactions -- and sometimes sends money to the sketchy accounts FOR $account IN SELECT * FROM account { FOR $_ IN 0..10 { LET $counterpart = IF $account.is_sketchy { rand::enum(SELECT * FROM account WHERE is_sketchy) } ELSE { rand::enum(SELECT * FROM account) }; RELATE $account->sends_to->$counterpart SET amount = rand::int(100, 1000); } }; SELECT id, ->sends_to->account FROM account; ``` Circles showing loops of money returning to its origin: ```surql CREATE |account:50|; CREATE |account:1..16| SET is_sketchy = true; FOR $sketchy IN SELECT * FROM account WHERE is_sketchy { LET $counterpart = rand::enum(SELECT * FROM account WHERE is_sketchy AND !<-sent); RELATE $sketchy->sent->$counterpart SET amount = rand::int(100, 1000); }; LET $normal = SELECT * FROM account WHERE !is_sketchy; FOR $account IN SELECT * FROM account WHERE !is_sketchy { LET $counterpart = rand::enum(SELECT * FROM $normal); RELATE $account->sent->$counterpart SET amount = rand::int(100, 1000); }; SELECT id, ->sent->account FROM account; ``` ## Gaming ### Characters and quests RPG game system with character progression, inventory management, and quest tracking. Features polymorphic item effects, character statistics, and complex game state management. Demonstrates flexible data modeling for gaming applications with rich object structures and [relationship tracking](/docs/learn/data-models/graph/overview.md). ```surql -- Characters controlled by players DEFINE TABLE character SCHEMAFULL; DEFINE FIELD name ON character TYPE string; DEFINE FIELD level ON character TYPE int DEFAULT 1; DEFINE FIELD xp ON character TYPE int DEFAULT 0; DEFINE FIELD class ON character TYPE string ASSERT $value IN ["warrior", "mage", "rogue"]; DEFINE FIELD stats ON character TYPE { str: int, dex: int, int: int }; -- Items in the game world DEFINE TABLE item SCHEMAFULL; DEFINE FIELD name ON item TYPE string; DEFINE FIELD type ON item TYPE string ASSERT $value IN ["weapon", "armor", "potion"]; DEFINE FIELD rarity ON item TYPE string ASSERT $value IN ["common", "rare", "epic", "legendary"]; DEFINE FIELD effects ON item TYPE array<{ str: int } | { int: int } | { heal: int }>; // etc. -- Items possessed by characters DEFINE TABLE owns TYPE RELATION IN character OUT item; DEFINE FIELD equipped ON owns TYPE bool DEFAULT false; -- Quests available in the world DEFINE TABLE quest SCHEMAFULL; DEFINE FIELD name ON quest TYPE string; DEFINE FIELD required_level ON quest TYPE int DEFAULT 1; DEFINE FIELD rewards ON quest TYPE { exp: int, items: array> }; -- Character quest progress DEFINE TABLE quest_log TYPE RELATION IN character OUT quest; DEFINE FIELD status ON quest_log TYPE string ASSERT $value IN ["active", "completed"]; DEFINE FIELD started_at ON quest_log TYPE datetime DEFAULT time::now(); DEFINE FIELD completed_at ON quest_log TYPE option; -- Events DEFINE TABLE character_event SCHEMAFULL; DEFINE FIELD character ON character_event TYPE record; DEFINE FIELD details ON character_event TYPE { type: "combat", exp: int, against: string, summary: string } | { type: "item_used", item: record, summary: string } | { type: "quest_update", summary: string }; DEFINE FIELD ts ON character_event TYPE datetime DEFAULT time::now(); -- Create a new character CREATE character:aria SET name = "Aria", class = "mage", stats = { str: 4, dex: 6, int: 12 }; -- Give Aria an item RELATE character:aria->owns->(CREATE ONLY item SET name = "Wand of Sparks", type = "weapon", rarity = "rare", effects = [{ int: 2 }]); -- Start a quest RELATE character:aria->quest_log->quest:slime_hunt SET status = "active"; ``` ## Aerospace and astronomy ### Telescopes and observations Astronomical observation tracking system with instrument management and data collection. Features geospatial telescope locations, flexible observation metadata, and scientific data URL management. Demonstrates [point data types](/docs/reference/query-language/language-primitives/data-types/geometries.md#point), complex temporal relationships, and scientific data organisation patterns. ```surql -- Telescopes (instruments) DEFINE TABLE telescope SCHEMAFULL; DEFINE FIELD name ON telescope TYPE string; DEFINE FIELD location ON telescope TYPE point; DEFINE FIELD aperture_mm ON telescope TYPE int; -- e.g. 200 for 8" scope -- Astronomical targets DEFINE TABLE target SCHEMAFULL; DEFINE FIELD name ON target TYPE string; DEFINE FIELD type ON target TYPE string ASSERT $value IN ["star", "planet", "nebula", "galaxy", "asteroid"]; -- Observation logs DEFINE TABLE observed SCHEMAFULL TYPE RELATION IN telescope OUT target; DEFINE FIELD observer ON observed TYPE record; DEFINE FIELD observed_at ON observed TYPE datetime; DEFINE FIELD observed_until ON observed TYPE option; DEFINE FIELD exposure_length ON observed VALUE IF observed_until { observed_until - observed_at } ELSE { 0ns }; DEFINE FIELD seeing ON observed TYPE option; -- arcseconds DEFINE FIELD notes ON observed TYPE option; DEFINE FIELD filter ON observed TYPE option ASSERT $value IN ["B", "V", "R", "I", "H-alpha", "OIII", "IR"]; DEFINE FIELD sky_conditions ON observed TYPE option ASSERT $value IN ["clear", "thin cloud", "hazy", "overcast"]; DEFINE FIELD data_url ON observed TYPE option; -- e.g. to FITS file, rendered image, or DOI CREATE telescope:one SET name = "The one telescope", location = (-68.44, -29.14), aperture_mm = 200; CREATE target:venus SET type = "planet", name = "Venus"; CREATE person:one; RELATE telescope:one->observed->target:venus SET observer = person:one, observed_at = time::now(), observed_until = time::now() + 1h, filter = "R", seeing = 0.7, sky_conditions = "clear", data_url = "https://astro.example.org/data/venus-2025.fits"; ``` ### Launch telemetry Space launch monitoring system with real-time telemetry data collection. Features component-level tracking, [time-series data management](/docs/learn/data-models/time-series/overview.md) with composite keys, and launch lifecycle status tracking. Demonstrates high-frequency data ingestion, temporal range queries, and [live data streaming](/docs/reference/query-language/statements/live-select.md). ```surql -- A specific launch instance (e.g., Falcon 9 Flight 100) DEFINE TABLE launch SCHEMAFULL; DEFINE FIELD name ON launch TYPE string; DEFINE FIELD vehicle_name ON launch TYPE option; DEFINE FIELD scheduled_at ON launch TYPE datetime; DEFINE FIELD liftoff_at ON launch TYPE option; DEFINE FIELD status ON launch TYPE string ASSERT $value IN ["scheduled", "launched", "scrubbed", "failed", "success"] DEFAULT "scheduled"; DEFINE FIELD completed ON launch TYPE option; -- Components involved in the launch DEFINE TABLE component SCHEMAFULL; DEFINE FIELD launch ON component TYPE record; DEFINE FIELD name ON component TYPE string; -- e.g., "first_stage", "engine_1" DEFINE FIELD type ON component TYPE string ASSERT $value IN ["stage", "engine", "payload", "fairing"]; -- Time-series telemetry linked to a component DEFINE TABLE telemetry SCHEMAFULL; DEFINE FIELD id ON telemetry TYPE [record, datetime]; -- [component, ulid] DEFINE FIELD altitude_m ON telemetry TYPE option; DEFINE FIELD velocity_mps ON telemetry TYPE option; DEFINE FIELD thrust_kN ON telemetry TYPE option; DEFINE FIELD pressure_kPa ON telemetry TYPE option; DEFINE FIELD temperature_C ON telemetry TYPE option; DEFINE FIELD status ON telemetry TYPE option; CREATE launch:one SET name = "Launch 1", vehicle_name = "Fire rocket", scheduled_at = time::now() - 5s, liftoff_at = time::now() - 1s; CREATE component:one SET launch = launch:one, name = "Engine 1", type = "engine"; CREATE component:two SET launch = launch:one, name = "Engine 2", type = "engine"; -- Add durations to all datetimes below to simulate passage of time CREATE telemetry:[component:one, time::now()] SET temperature_C = 30.5, status = "good"; CREATE telemetry:[component:one, time::now() + 1s] SET temperature_C = 30.7, status = "good"; CREATE telemetry:[component:one, time::now() + 2s] SET temperature_C = 30.9, status = "good"; CREATE telemetry:[component:one, time::now() + 3s] SET temperature_C = 35.0, status = "good"; CREATE telemetry:[component:two, time::now()] SET temperature_C = 30.5, status = "good"; CREATE telemetry:[component:two, time::now() + 1s] SET temperature_C = 30.7, status = "good"; CREATE telemetry:[component:two, time::now() + 2s] SET temperature_C = 30.9, status = "good"; CREATE telemetry:[component:two, time::now() + 3s] SET temperature_C = 35.0, status = "good"; UPDATE launch:one SET completed = time::now() + 5s; -- Get all telemetry for component:two during launch:one SELECT * FROM telemetry:[component:two, launch:one.liftoff_at]..=[component:two, launch:one.completed]; -- Or LIVE SELECT during the flight LIVE SELECT * FROM telemetry WHERE id[0] = component:two; ``` ## Defense / mission operations ### Missions and tasks Military mission management system with unit tracking and operational logging. Features hierarchical command structure, real-time status updates, and comprehensive audit trails. Demonstrates complex organisational modeling, [geospatial tracking](/docs/reference/query-language/language-primitives/data-types/geometries.md#point), and mission-critical data management patterns. ```surql -- Mission-level directive DEFINE TABLE operation SCHEMAFULL; DEFINE FIELD name ON operation TYPE string; DEFINE FIELD status ON operation TYPE string ASSERT $value IN ["planned", "active", "complete", "aborted"] DEFAULT "planned"; DEFINE FIELD commander ON operation TYPE option>; DEFINE FIELD start_time ON operation TYPE option; DEFINE FIELD end_time ON operation TYPE option; DEFINE TABLE unit SCHEMAFULL; DEFINE FIELD members ON unit TYPE array>; DEFINE FIELD operation ON unit TYPE record; DEFINE FIELD name ON unit TYPE string; -- e.g., "drone-2", "squad-a" DEFINE FIELD type ON unit TYPE string ASSERT $value IN ["drone", "vehicle", "infantry", "support"]; DEFINE FIELD status ON unit TYPE string ASSERT $value IN ["ready", "deployed", "engaged", "inactive"]; -- Time-stamped unit log (e.g., movement, engagement, report) DEFINE TABLE log SCHEMAFULL; DEFINE FIELD id ON log TYPE [record, datetime]; -- [unit, timestamp] DEFINE FIELD message ON log TYPE string; DEFINE FIELD status ON log TYPE option; -- e.g., "engaged", "moving", "waiting" DEFINE FIELD lonlat ON log TYPE option; DEFINE FIELD visibility ON log TYPE option ASSERT $value IN ["clear", "obscured", "night"]; -- Tasks assigned within a mission DEFINE TABLE task SCHEMAFULL; DEFINE FIELD operation ON task TYPE record; DEFINE FIELD name ON task TYPE string; DEFINE FIELD objective ON task TYPE string; DEFINE FIELD assigned_to ON task TYPE option>>; DEFINE FIELD priority ON task TYPE string ASSERT $value IN ["high", "medium", "low"]; DEFINE FIELD completed ON task TYPE bool DEFAULT false; CREATE operation:alpha SET name = "Operation Alpha", commander = person:one, start_time = time::now(); CREATE unit:squad1 SET operation = operation:alpha, name = "squad-1", type = "infantry", status = "deployed", members = [person:one, person:two]; CREATE unit:drone1 SET operation = operation:alpha, name = "drone-1", type = "drone", status = "ready", members = [person:three, person:four]; CREATE task SET operation = operation:alpha, name = "Secure Ridge", objective = "Clear hilltop sector", assigned_to = [unit:squad1], priority = "high"; -- Log messages (simulate time with + durations) CREATE log:[unit:squad1, time::now()] SET message = "Entered zone", status = "moving", lonlat = (44.2, 6.3); CREATE log:[unit:squad1, time::now() + 3m] SET message = "Engaged hostiles", status = "engaged", visibility = "clear"; CREATE log:[unit:drone1, time::now()] SET message = "Recon sweep complete", status = "waiting", lonlat = (44.3, 6.2); ``` ## Retail ### People, products and commerce E-commerce platform schema with customer profiles, product catalog, and shopping cart management. Features flexible address storage, multi-currency support, and comprehensive timestamp tracking. Demonstrates modern e-commerce data modeling with [flexible object fields](/docs/reference/query-language/language-primitives/data-types/objects.md#flexible-objects) and relationship management. ```surql -- Person / customer profile DEFINE TABLE person SCHEMAFULL; DEFINE FIELD name ON person TYPE string; DEFINE FIELD email ON person TYPE string ASSERT string::is_email($value); DEFINE FIELD address ON person TYPE object; DEFINE FIELD time ON person TYPE object; DEFINE FIELD time.created_at ON person TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON person TYPE datetime VALUE time::now(); -- Payment method linked to person DEFINE TABLE payment_details SCHEMAFULL; DEFINE FIELD person ON payment_details TYPE record; DEFINE FIELD stored_cards ON payment_details TYPE array; DEFINE FIELD time ON payment_details TYPE object; DEFINE FIELD time.created_at ON payment_details TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON payment_details TYPE datetime VALUE time::now(); -- Seller profile DEFINE TABLE seller SCHEMAFULL; DEFINE FIELD name ON seller TYPE string; DEFINE FIELD email ON seller TYPE string ASSERT string::is_email($value); DEFINE FIELD time ON seller TYPE object; DEFINE FIELD time.created_at ON seller TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON seller TYPE datetime VALUE time::now(); -- Product listings DEFINE TABLE product SCHEMAFULL; DEFINE FIELD name ON product TYPE string; DEFINE FIELD price ON product TYPE number; DEFINE FIELD currency ON product TYPE string ASSERT $value IN ["USD", "GBP", "CAD"]; DEFINE FIELD category ON product TYPE string; DEFINE FIELD seller ON product TYPE record; DEFINE FIELD time ON product TYPE object; DEFINE FIELD time.created_at ON product TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON product TYPE datetime VALUE time::now(); -- Wishlist links (person -> product) DEFINE TABLE wishlist TYPE RELATION FROM person TO product SCHEMAFULL; DEFINE FIELD colour ON wishlist TYPE string; DEFINE FIELD size ON wishlist TYPE string; DEFINE FIELD time ON wishlist TYPE object; DEFINE FIELD time.created_at ON wishlist TYPE datetime DEFAULT time::now(); DEFINE FIELD time.deleted_at ON wishlist TYPE option; -- Cart links (person -> product) DEFINE TABLE cart TYPE RELATION FROM person TO product SCHEMAFULL; DEFINE FIELD quantity ON cart TYPE number; DEFINE FIELD price ON cart TYPE number; DEFINE FIELD currency ON cart TYPE string ASSERT $value IN ["CAD", "EUR", "USD"]; DEFINE FIELD time ON cart TYPE object; DEFINE FIELD time.created_at ON cart TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON cart TYPE datetime VALUE time::now(); ``` ### Orders, reviews, reports Order processing and analytics system with review management and business intelligence. Features order lifecycle tracking, automated analytics tables, and [full-text search](/docs/reference/query-language/statements/define/analyzer.md) capabilities. Demonstrates complex aggregations, [materialized views](/docs/reference/query-language/statements/define/table.md#pre-computed-table-views), and search optimization for e-commerce applications. ```surql -- Orders placed (person -> product) DEFINE TABLE order TYPE RELATION FROM person TO product SCHEMAFULL; DEFINE FIELD quantity ON order TYPE number; DEFINE FIELD price ON order TYPE number; DEFINE FIELD currency ON order TYPE string; DEFINE FIELD order_status ON order TYPE string ASSERT $value IN ["pending", "processed", "shipped", "cancelled"]; DEFINE FIELD shipping_address ON order TYPE object FLEXIBLE; DEFINE FIELD payment_method ON order TYPE string; DEFINE FIELD time ON order TYPE object; DEFINE FIELD time.created_at ON order TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON order TYPE datetime VALUE time::now(); DEFINE FIELD time.processed_at ON order TYPE option; DEFINE FIELD time.shipped_at ON order TYPE option; -- Product reviews DEFINE TABLE review TYPE RELATION FROM person TO product SCHEMAFULL; DEFINE FIELD rating ON review TYPE number ASSERT $value IN 0..=5; DEFINE FIELD review_text ON review TYPE string; DEFINE FIELD time ON review TYPE object; DEFINE FIELD time.created_at ON review TYPE datetime DEFAULT time::now(); DEFINE FIELD time.updated_at ON review TYPE datetime VALUE time::now(); -- Indexes and analytics DEFINE FUNCTION fn::number_of_unfulfilled_orders() -> array<{ count: int }> { RETURN (SELECT count() FROM order WHERE order_status NOTINSIDE ["processed", "shipped"] GROUP ALL); }; -- Monthly order summary DEFINE TABLE monthly_sales TYPE NORMAL SCHEMAFULL AS SELECT count() AS number_of_orders, time::format(time.created_at, '%Y-%m') AS month, math::sum(price * quantity) AS sum_sales, currency FROM order GROUP BY month, currency; -- Average product rating DEFINE TABLE avg_product_review TYPE NORMAL SCHEMAFULL AS SELECT count() AS number_of_reviews, math::mean( rating) AS avg_review, ->product.id AS product_id, ->product.name AS product_name FROM review GROUP BY product_id, product_name; -- Full-text search DEFINE ANALYZER blank_snowball TOKENIZERS blank FILTERS lowercase, snowball(english); DEFINE INDEX review_content ON review FIELDS review_text FULLTEXT ANALYZER blank_snowball BM25 HIGHLIGHTS; ``` ## Medical ### Patient records and encounters Healthcare management system with patient records, encounter tracking, and clinical data management. Features vital signs [time-series data](/docs/learn/data-models/time-series/overview.md), medication tracking, and automated encounter lifecycle management using [events](/docs/reference/query-language/statements/define/event.md). Demonstrates healthcare data modeling with temporal data, clinical workflows, and medical record compliance patterns. ```surql -- Patient record DEFINE TABLE patient SCHEMAFULL; DEFINE FIELD name ON patient TYPE string; DEFINE FIELD dob ON patient TYPE datetime; DEFINE FIELD gender ON patient TYPE string ASSERT $value IN ["male", "female", "other", "uncertain"]; DEFINE FIELD email ON patient TYPE option ASSERT string::is_email($value); DEFINE FIELD created_at ON patient TYPE datetime DEFAULT time::now(); -- One healthcare visit DEFINE TABLE encounter SCHEMAFULL; DEFINE FIELD patient ON encounter TYPE record; DEFINE FIELD occurred_at ON encounter TYPE datetime DEFAULT time::now(); DEFINE FIELD type ON encounter TYPE string ASSERT $value IN ["checkup", "emergency", "followup", "consult"]; DEFINE FIELD reason ON encounter TYPE option; DEFINE FIELD location ON encounter TYPE option; DEFINE FIELD ongoing ON encounter TYPE bool DEFAULT true; DEFINE FIELD ended_at ON encounter TYPE option; DEFINE EVENT close_encounter ON encounter WHEN $event = "UPDATE" THEN { IF $before.ongoing = true AND $after.ongoing = false { UPDATE $this SET ended_at = time::now(); } }; -- Vital signs time-series (per encounter) DEFINE TABLE vital_signs SCHEMAFULL; DEFINE FIELD id ON vital_signs TYPE [record, datetime]; DEFINE FIELD heart_rate ON vital_signs TYPE option ASSERT $value IN 20..=300; DEFINE FIELD bp_systolic ON vital_signs TYPE option ASSERT $value IN 40..=300; DEFINE FIELD bp_diastolic ON vital_signs TYPE option ASSERT $value IN 20..=200; DEFINE FIELD temp_c ON vital_signs TYPE option ASSERT $value IN 25.0..=45.0; DEFINE FIELD notes ON vital_signs TYPE option; -- Diagnoses made during encounter DEFINE TABLE diagnosis SCHEMAFULL; DEFINE FIELD encounter ON diagnosis TYPE record; DEFINE FIELD code ON diagnosis TYPE string; -- e.g., ICD-10 DEFINE FIELD label ON diagnosis TYPE string; DEFINE FIELD confirmed ON diagnosis TYPE bool DEFAULT true; -- Medications prescribed DEFINE TABLE medication SCHEMAFULL; DEFINE FIELD encounter ON medication TYPE record; DEFINE FIELD name ON medication TYPE string; DEFINE FIELD dose_mg ON medication TYPE float; DEFINE FIELD frequency ON medication TYPE string; -- e.g., "2x daily" DEFINE FIELD duration ON medication TYPE string; -- e.g., "7 days" DEFINE FIELD prn ON medication TYPE bool DEFAULT false; -- "as needed" -- Notes written by practitioner DEFINE TABLE note SCHEMAFULL; DEFINE FIELD id ON note TYPE [record, datetime]; DEFINE FIELD author ON note TYPE string; DEFINE FIELD content ON note TYPE string; DEFINE FIELD tags ON note TYPE option>; CREATE patient:one SET name = "Alex Quinn", dob = d'1988-06-12', gender = "male"; CREATE encounter:one SET patient = patient:one, type = "checkup", reason = "Routine annual"; -- Vital signs log CREATE vital_signs:[encounter:one, time::now()] SET heart_rate = 72, bp_systolic = 120, bp_diastolic = 80, temp_c = 36.8; -- Diagnosis CREATE diagnosis SET encounter = encounter:one, code = "E66.9", label = "Obesity, unspecified"; -- Medication CREATE medication SET encounter = encounter:one, name = "Metformin", dose_mg = 500, frequency = "2x daily", duration = "30 days"; -- Progress note CREATE note:[encounter:one, time::now()] SET author = "Dr. Leung", content = "Patient reports improved energy since last visit."; ``` ## Related SurrealQL statements - [SurrealKit schema migration](/docs/manage/schema-migration.md) - official CLI for versioning and applying schema from `.surql` files - [DEFINE TABLE](/docs/reference/query-language/statements/define/table.md) - [DEFINE FIELD](/docs/reference/query-language/statements/define/field.md) - [RELATE](/docs/reference/query-language/statements/relate.md) - [DEFINE INDEX](/docs/reference/query-language/statements/define/indexes.md) - [DEFINE FUNCTION](/docs/reference/query-language/statements/define/function.md) - [DEFINE EVENT](/docs/reference/query-language/statements/define/event.md) - [DEFINE PARAM](/docs/reference/query-language/statements/define/param.md) - [DEFINE ANALYZER](/docs/reference/query-language/statements/define/analyzer.md) --- Source: https://surrealdb.com/docs/learn/schema-management/schema-design/schema-best-practices # Schema best practices Best practices for creating schemas in SurrealDB. With SurrealDB, you can create a schema that is as simple or as complex as you need it to be. This page contains a number of best practices for creating schemas that are both easy to understand and easy to maintain. When those definitions live in source control and need to roll out safely to shared databases, follow [SurrealKit schema migration](/docs/manage/schema-migration.md) in the **Manage** section for sync, rollouts, testing in CI, and related tooling. ## Define arrays and sets with a type and maximum size In addition to a type, both arrays and sets can have a required number of items built into the type definition itself. The definition below pairs this with an assertion using the [`array::all()`](/docs/reference/query-language/functions/database-functions/array.md#arrayall) function to also ensure that every item in the `small_bytes` field is between 0 and 255. ```surql DEFINE FIELD small_bytes ON data TYPE array ASSERT $value.all(|$int| $int IN 0..=255); ``` Learn more about [database functions](/docs/reference/query-language/functions/database-functions.md). ## Define individual indexes of an array Even the individual indexes of an array can be defined. This is useful for data types like RGB colours that must be exactly three items in length. ```surql DEFINE FIELD rgb ON colour TYPE ; DEFINE FIELD rgb[0] ON colour TYPE int ASSERT $value IN 0..=255; DEFINE FIELD rgb[1] ON colour TYPE int ASSERT $value IN 0..=255; DEFINE FIELD rgb[2] ON colour TYPE int ASSERT $value IN 0..=255; CREATE colour SET rgb = [0, 2, 30]; -- Fails: must have three items CREATE colour SET rgb = [0, 2, 30]; -- Fails: must be between 0 and 255 CREATE colour SET rgb = [0, 2, 400]; ``` [Learn more about assertions in `DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md#assertions) ## How to work with objects inside `SCHEMALESS` and `SCHEMAFULL` tables The behaviour of an object as a field of a table depends on whether the table is `SCHEMALESS` (the default) or `SCHEMAFULL`. Inside a schemaless table, the only time an object will be schemafull is if the structure of the object is indicated using the [literal](/docs/reference/query-language/language-primitives/data-types/literals.md) syntax. ```surql -- This table is schemaless DEFINE TABLE some_table; -- So an object defined on it will be schemaless DEFINE FIELD some_object ON some_table TYPE object; -- Same for an array of objects DEFINE FIELD some_objects ON some_table TYPE array; -- But this is schemafull because it has a set structure DEFINE FIELD some_specific_objects ON some_table TYPE array<{ a: string }>; ``` Inside a schemafull table, a field typed as `object` or `array` (or any other definition that contains an `object`) is schemafull by default. Add the `FLEXIBLE` field clause after `TYPE` to allow arbitrary keys on every object in that field's type. ```surql DEFINE TABLE some_table SCHEMAFULL; -- Schemafull object and array of objects DEFINE FIELD some_object ON some_table TYPE object; DEFINE FIELD some_objects ON some_table TYPE array; -- Field clause makes objects inside these fields schemaless DEFINE FIELD flexible_object ON some_table TYPE object FLEXIBLE; DEFINE FIELD flexible_objects ON some_table TYPE array FLEXIBLE; ``` Inside a schemafull table, any input value that does not match the defined schema will cause an error. ```surql DEFINE TABLE user SCHEMAFULL; DEFINE FIELD name ON user TYPE string; DEFINE FIELD metadata ON user TYPE object; DEFINE FIELD metadata.created_at ON user TYPE datetime; DEFINE FIELD metadata.age ON user TYPE int; CREATE user SET name = "Billy", metadata = { created_at: time::now(), age: 5, wrong_field: "WRONG DATA" }; ``` ```surql title="Output" "Found field 'metadata.wrong_field', but no such field exists for table 'user'" ``` If you have data that includes a non-defined field in such a table, you can use the destructuring operator to access the current structure and only pass on the necessary fields for the operation. ```surql -- This object has too much info and capitalization doesn't match LET $chaotic_content = { name: "Billy", unneeded_number: 10, metadata: { CREATED_AT: time::now(), age: 5, wrong_field: "WRONG DATA" } }; -- Pass on the needed fields and rename CREATED_AT to lowercase CREATE user CONTENT $chaotic_content.{ name, metadata.{ created_at: CREATED_AT, age } }; ``` Defining a specific function to return the expected structure can be a nice convenience in this case. ```surql DEFINE FUNCTION fn::filter_for_user($obj: object) -> object { $obj.{ name, metadata.{ created_at: CREATED_AT, age }}; }; LET $chaotic_content = { name: "Billy", unneeded_number: 10, metadata: { CREATED_AT: time::now(), age: 5, wrong_field: "WRONG DATA" } }; CREATE user CONTENT fn::filter_for_user($chaotic_content); ``` ## Use `THROW` to add more detailed error messages to `ASSERT` clauses A `DEFINE FIELD` statement allows an `ASSERT` clause to be added in order to ensure that the value, which here is represented as the parameter `$value`, meets certain expectations. A simple example here makes sure that the `name` field on the `person` table is under 20 characters in length. ```surql DEFINE FIELD name ON person TYPE string ASSERT $value.len() < 20; CREATE person SET name = "Mr. Longname who has much too long a name"; ``` In this case, the default error message is pretty good. ```surql "Found 'Mr. Longname who has much too long a name' for field `name`, with record `person:2gpvut914k1qfysqs3lc`, but field must conform to: $value.len() < 20" ``` However, `ASSERT` only expects a truthy value at the end and otherwise isn't concerned at all with what happens before. This means that you can outright customise the logic, including a custom error message. Let's give this a try. ```surql DEFINE FIELD name ON person TYPE string ASSERT { IF $value.len() >= 20 { THROW "`" + $value + "` too long, must be under 20 characters. Up to `" + $value.slice(0,19) + "` is acceptable"; } ELSE { RETURN true; } }; CREATE person SET name = "Mr. Longname who has much too long a name"; ``` Not bad! ```surql 'An error occurred: `Mr. Longname who has much too long a name` too long, must be under 20 characters. Up to `Mr. Longname who ha` is acceptable' ``` ## Use formatters on internal datetimes for strings with alternative formats A lot of legacy systems require datetimes to be displayed in a format that doesn't quite match a `datetime`. That doesn't mean that you have to give up the precision of a `datetime` though. By using the [`time::format()`](/docs/reference/query-language/language-primitives/formatters.md) function, you can keep the actual stored date as a precise SurrealQL `datetime` and then use that to output a string in any format you like. ```surql DEFINE FIELD created_at ON user VALUE time::now() READONLY; DEFINE FIELD since ON user VALUE time::format(created_at, "%Y-%m-%d"); CREATE user RETURN id, since; ``` ```surql title="Output" [ { id: user:50s2riya8fm3cdbrhwpe, since: '2026-02-12' } ] ``` ## Use `!!$value` in `DEFINE` statements As the `!` operator reverses the truthiness of a value, using it twice in a row as `!!` returns a value's truthiness. As empty and default values (such as 0 for numbers) are considered to be non-truthy, this operator is handy if you want to ensure that a value is both present and not empty. ```surql DEFINE FIELD name ON character TYPE string; DEFINE FIELD metadata ON character TYPE object; -- Works because "" is of type string CREATE character SET name = "", metadata = {}; DEFINE FIELD OVERWRITE name ON character TYPE string ASSERT !!$value; -- Now returns an error because "" and {} are non-truthy CREATE character SET name = "", metadata = {}; ``` ## Use `DEFINE PARAM` for clarity If you find that parts of your table- or field-specific code are getting a bit long, it might be time to think about moving parts of it to a [database-wide parameter](/docs/reference/query-language/statements/define/param.md). ```surql DEFINE FIELD month_published ON book TYPE string ASSERT $value IN ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; ``` Doing so not only makes the code cleaner, but makes it easy to reuse in other parts of the schema as well. ```surql DEFINE PARAM $MONTHS VALUE ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]; DEFINE FIELD month_published ON book TYPE string ASSERT $value IN $MONTHS; DEFINE FUNCTION fn::do_something_with_month($input: string) { IF !($input IN $MONTHS) { THROW "Some error about wrong input"; } ELSE { -- do something with months here } }; ``` ## Use literals to return rich error output Error types in programming languages often take the form of a long list of possible things that could go wrong. SurrealQL's [literal](/docs/reference/query-language/language-primitives/data-types/literals.md) type allows you to specify a list of all possible forms it could take, making it the perfect type for error logic. ```surql DEFINE PARAM $ERROR_CODES VALUE [200, 300, 400, 500]; DEFINE FUNCTION fn::return_response($input: { type: "internal_error", message: string } | { type: "bad_request", message: string } | { type: "invalid_date", got: any, expected: "YYYY-MM-DD" } | int) -> object | int { IF $input.is_int() { IF $input IN $ERROR_CODES { $input } ELSE { THROW "Input must be one of " + $ERROR_CODES; } } ELSE { $input } }; fn::return_response(500); fn::return_response(999999); fn::return_response({ type: "internal_error", message: "You can't do that"}); fn::return_response(a:wrong_argument); ``` ```surql title="Output" -------- Query -------- 500 -------- Query -------- 'An error occurred: Input must be one of [200, 300, 400, 500]' -------- Query -------- { message: "You can't do that", type: 'internal_error' } -------- Query -------- "Expected `{ message: string, type: 'internal_error' } | { message: string, type: 'bad_request' } | { expected: 'YYYY-MM-DD', got: any, type: 'invalid_date' } | int` but found `a:wrong_argument`" ``` ## Use graph queries in the schema While graph queries are usually seen in `SELECT` statements in the documentation, they can live inside your database schema just like any other datatype or expression. In the schema below for a family tree, any inserted record must either have a parent (via the `<-parent_of<-person` path) or be `first_generation`. ```surql DEFINE FIELD parents ON person ASSERT <-parent_of<-person OR first_generation; -- Is first_generation, doesn't need to indicate parents CREATE person:one SET first_generation = true; CREATE person:two; //- Error: //- 'Found NONE for field `parents`, with record `person:two`, //- but field must conform to: <-parent_of<-person OR first_generation' -- Give person:two a parent RELATE person:one->parent_of->person:two; -- CREATE now works CREATE person:two; ``` By the way, this pattern is possible because `RELATE` statements can be used before the records to relate exist. To disallow this, you can add the [`ENFORCED`](/docs/reference/query-language/statements/define/table.md#using-enforced-to-ensure-that-related-records-exist) clause to a `DEFINE TABLE table_name TYPE RECORD` definition. --- Source: https://surrealdb.com/docs/learn/schema-management/schema-design/schema-evolution # Schema evolution Changing schema over time with ALTER and related patterns. Production databases rarely stay still, and in SurrealDB they tend to move from less to more strict over time as you become more aware of the expected behaviour of your database. Some examples of schema changes are tightening a table from schemaless to schemafull, changing permissions, or preparing an index for removal. SurrealDB's `ALTER` statement is the lever for mutating definitions in place without always rewriting a full `DEFINE` block. When you are ready to version and apply those changes across environments (for example from CI or with reviewed rollouts), use [SurrealKit schema migration](/docs/manage/schema-migration.md) - the official `.surql`-based workflow documented in the **Manage** section. For a wholesale replacement, you can always use `DEFINE` with the `OVERWRITE` clause. ## Tightening a table from schemaless to schemafull A common pattern is to start with a completely schemaless table, adding defined fields one at a time to ensure that they are present in all records of a certain table. If you are certain that no other fields should be present, you can move the entire table from `SCHEMALESS` to `SCHEMAFULL` to ensure that only defined fields are present. ```surql DEFINE TABLE user SCHEMALESS; DEFINE FIELD name ON TABLE user TYPE string; CREATE user SET name = "LordofSalty"; -- Now make it schemafull so undeclared fields are rejected ALTER TABLE user SCHEMAFULL; ``` ## Adjusting permissions without restating the whole table `ALTER` only needs the parts you are changing. Here the table stays schemafull; only create permission moves from the default to FULL. ```surql DEFINE TABLE user SCHEMAFULL; ALTER TABLE user PERMISSIONS FOR create FULL; ``` ## Idempotent alterations In scripts, `IF EXISTS` avoids failing when the resource name was never created in that environment. ```surql ALTER TABLE IF EXISTS user SCHEMAFULL; DEFINE TABLE IF EXISTS writer; ``` ## Where to read more * [`ALTER` overview](/docs/reference/query-language/statements/alter/overview.md) * [Schema best practices](/docs/learn/schema-management/schema-design/schema-best-practices.md) * [SurrealKit schema migration](/docs/manage/schema-migration.md) - sync and rollout workflows for schema files --- Source: https://surrealdb.com/docs/learn/schema-management/tables-and-fields/fields-and-validation # Fields and validation Defining fields: types, defaults, VALUE, ASSERT, permissions, and references. Fields are where you can spell out what each table column means by using certain clauses, such as its type (`TYPE`), optional default values (`DEFAULT`), how writes are normalised (`VALUE`), what counts as valid data (`ASSERT`), or who may see or change it (`PERMISSIONS`). A [`COMMENT`](/docs/reference/query-language/statements/define/overview.md#comments-on-definitions) is a good place for comparison rules and invariants that are not fully captured by the type alone. You need database-level access and an active [`USE`](/docs/reference/query-language/statements/use.md) scope, same as for tables. Computed fields (derived on every read) have their own learn page: [Computed fields](/docs/learn/schema-management/computed-data/computed-fields.md). ## Example usage The following expression shows the simplest way to use the `DEFINE FIELD` statement. ```surql -- Declare the name of a field. DEFINE FIELD email ON TABLE user; ``` The fields of an object and the items in an array can be defined individually using the `.` operator for objects, or the indexing operator for arrays. ```surql -- Define nested object property types DEFINE FIELD emails.address ON TABLE user TYPE string; DEFINE FIELD emails.primary ON TABLE user TYPE bool; -- Define individual fields on an array DEFINE FIELD metadata[0] ON person TYPE datetime; DEFINE FIELD metadata[1] ON person TYPE int; ``` ### Simple data types ```surql -- Set a field to have the string data type DEFINE FIELD email ON TABLE user TYPE string; -- Set a field to have the datetime data type DEFINE FIELD created ON TABLE user TYPE datetime; -- Set a field to have the bool data type DEFINE FIELD locked ON TABLE user TYPE bool; -- Set a field to have the number data type DEFINE FIELD login_attempts ON TABLE user TYPE number; ``` A `|` vertical bar can be used to allow a field to be one of a set of types. The following example shows a field that can be a [`UUID`](/docs/reference/query-language/language-primitives/data-types/uuids.md) or an [`int`](/docs/reference/query-language/language-primitives/data-types/numbers.md#integer-numbers), perhaps for `user` records that have varying data due to two diffent legacy ID types. ```surql -- Set a field to have either the uuid or int type DEFINE FIELD user_id ON TABLE user TYPE uuid|int; ``` ### Array type You can also set a field to have the array data type. The array data type can be used to store a list of values. You can also set the data type of the array's contents, as well as the required number of items that it must hold. ```surql -- Set a field to have the array data type DEFINE FIELD roles ON TABLE user TYPE array; -- Set a field to have the array data type, equivalent to `array` DEFINE FIELD posts ON TABLE user TYPE array; -- Set a field to have the array object data type DEFINE FIELD emails ON TABLE user TYPE array; -- Set a field that holds exactly 640 bytes DEFINE FIELD bytes ON TABLE data TYPE array ASSERT $value.all(|$val| $val IN 0..=255); -- Field for a block in a game showing the possible distinct directions a character can move next. -- The array can contain no more than four directions DEFINE FIELD next_paths ON TABLE block TYPE array<"north" | "east" | "south" | "west"> VALUE $value.distinct() ASSERT $value.len() <= 4; ``` ### Making a field optional You can make a field optional by wrapping the inner type in an `option`, which allows you to store `NONE` values in the field. ```surql -- A user may enter a biography, but it is not required. -- By using the option type you also allow for NONE values. DEFINE FIELD biography ON TABLE user TYPE option; ``` The example below shows how to define a field `user` on a `POST` table. The field is of type [record](/docs/reference/query-language/language-primitives/record-links.md). This means that the field can store a `record` or `NONE`. ```surql DEFINE FIELD user ON TABLE post TYPE option>; ``` ### Flexible objects in schemafull tables On a `SCHEMAFULL` table, objects are schemafull by default. The `FLEXIBLE` field clause (written after `TYPE`) marks a field so that every `object` in its type accepts arbitrary keys. This includes objects nested in arrays, options, and unions - not only a top-level `TYPE object` field. ```surql DEFINE TABLE user SCHEMAFULL; DEFINE FIELD name ON TABLE user TYPE string; DEFINE FIELD metadata ON TABLE user TYPE object FLEXIBLE; DEFINE FIELD metadata.user_id ON TABLE user TYPE int; ``` Taking the following `CREATE` statement: ```surql CREATE ONLY user SET name = "User1", metadata = { user_id: 8876687, country_code: "ee", time_zone: "EEST", age: 25 }; ``` Without `FLEXIBLE`, the `metadata` field is a schemafull object and only declared subfields such as `metadata.user_id` are accepted. With `FLEXIBLE`, the field accepts any extra keys on `metadata` while still requiring `name` and a valid `metadata.user_id`. ```surql title="Output" { id: user:lsdk473e279oik1k484b, metadata: { age: 25, country_code: 'ee', time_zone: 'EEST', user_id: 8876687 }, name: 'User1' } ``` ### Using the `DEFAULT` clause to set a default value You can set a default value for a field using the `DEFAULT` clause. The default value will be used if no value is provided for the field. ```surql -- A user is not locked by default. DEFINE FIELD locked ON TABLE user TYPE bool -- Set a default value if empty DEFAULT false; ``` ### Using the `DEFAULT ALWAYS` clause In addition to the `DEFAULT` clause, you can use the `DEFAULT ALWAYS` clause to set a default value for a field. The `ALWAYS` keyword indicates that the `DEFAULT` clause is used not only on `CREATE`, but also on `UPDATE` if the value is empty (NONE). ```surql DEFINE TABLE product SCHEMAFULL; -- Set a default value of 123.456 for the primary field DEFINE FIELD primary ON product TYPE number DEFAULT ALWAYS 123.456; ``` With the above definition, the `primary` field will be set to `123.456` when a new `product` is created without a value for the `primary` field or with a value of `NONE`, and when an existing `product` is updated if the value is specified the result will be the new value. In the case of `NULL` or a mismatching type, an error will be returned. ```surql -- This will return an error CREATE product:test SET primary = NULL; -- result "Couldn't coerce value for field `primary` of `product:test`: Expected `number` but found `NULL`" ``` On the other hand, if a valid number is provided during creation or update, that number will be used instead of the default value. In this case, `123.456`. ```surql -- This will set the value of the `primary` field to `123.456` CREATE product:test; -- This will set the value of the `primary` field to `463.456` UPSERT product:test SET primary = 463.456; -- This will set the value of the `primary` field to `123.456` UPSERT product:test SET primary = NONE; ``` ### Using the `VALUE` clause to set a field's value The `VALUE` clause differs from `DEFAULT` in that a default value is calculated if no other is indicated, otherwise accepting the value given in a query. ```surql DEFINE FIELD updated ON TABLE user DEFAULT time::now(); -- Set `updated` to the year 1900 CREATE user SET updated = d"1900-01-01"; -- Then set to the year 1910 UPDATE user SET updated = d"1910-01-01"; ``` A `VALUE` clause, on the other hand, will ignore attempts to set the field to any other value. ```surql DEFINE FIELD updated ON TABLE user VALUE time::now(); -- Ignores 1900 date, sets `updated` to current time CREATE user SET updated = d"1900-01-01"; -- Ignores again, updates to current time UPDATE user SET updated = d"1900-01-01"; ``` As the example above shows, a `VALUE` clause sets the value every time a record is modified (created or updated). However, the value will not be recalculated in a `SELECT` statement, which simply accesses the current set value. ```surql DEFINE FIELD updated ON TABLE user VALUE time::now(); CREATE user:one; SELECT * FROM ONLY user:one; -- Sleep for one second SLEEP 1s; -- `updated` is still the same SELECT * FROM ONLY user:one; ``` To create a field that is calculated each time it is accessed, a [`computed field`](/docs/reference/query-language/statements/define/field.md#restrictions-on-computed-fields) can be used. ```surql DEFINE FIELD accessed_at ON TABLE user COMPUTED time::now(); CREATE user:one; SELECT * FROM ONLY user:one; -- Sleep for one second SLEEP 1s; -- `accessed_at` is a different value now SELECT * FROM ONLY user:one; ``` ### Altering a passed value You can alter a passed value using the `VALUE` clause. This is useful for altering the value of a field before it is stored in the database. In the example below, the `VALUE` clause is used to ensure that the email address is always stored in lowercase characters by using the [`string::lowercase`](/docs/reference/query-language/functions/database-functions/string.md#stringlowercase) function. ```surql -- Ensure that an email address is always stored in lowercase characters DEFINE FIELD email ON TABLE user TYPE string VALUE string::lowercase($value); ``` ## Asserting rules on fields You can take your field definitions even further by using asserts. Assert can be used to ensure that your data remains consistent. For example you can use asserts to ensure that a field is always a valid email address, or that a number is always positive. ```surql -- Give the user table an email field. Store it in a string DEFINE FIELD email ON TABLE user TYPE string -- Check if the value is a properly formatted email address ASSERT string::is_email($value); ``` As the `ASSERT` clause expects an expression that returns a boolean, an assertion with a custom message can be manually created by returning `true` in one case and using a [`THROW`](/docs/reference/query-language/statements/throw.md) clause otherwise. ```surql DEFINE FIELD num ON data TYPE int ASSERT { IF $input % 2 = 0 { RETURN true } ELSE { THROW "Tried to make a " + $this + " but `num` field requires an even number" } }; CREATE data:one SET num = 11; ``` ```surql title="Error output" 'An error occurred: Tried to make a { id: data:one, num: 11 } but `num` field requires an even number' ``` ### Making a field `READONLY` The `READONLY` clause can be used to prevent any updates to a field. This is useful for fields that are automatically updated by the system. To make a field `READONLY`, add the `READONLY` clause to the `DEFINE FIELD` statement. As seen in the example below, the `created` field is set to `READONLY`. ```surql DEFINE FIELD created ON resource VALUE time::now() READONLY; ``` ## Setting permissions on fields By default, the permissions on a field are set to `FULL` unless otherwise specified. That means once a [table](/docs/learn/schema-management/tables-and-fields/tables.md#defining-permissions) allows an operation for a record user, the field is included unless you narrow it with your own `PERMISSIONS` clause. Tables default the other way: omitting table `PERMISSIONS` stores `PERMISSIONS NONE`. ```surql DEFINE FIELD some_info ON TABLE some_table TYPE string; INFO FOR TABLE some_table; ``` ```surql title="Output" { events: {}, fields: { info: 'DEFINE FIELD info ON some_table TYPE string PERMISSIONS FULL' }, indexes: {}, lives: {}, tables: {} } ``` You can set permissions on fields to control who can perform operations on them using the `PERMISSIONS` clause. The `PERMISSIONS` clause can be used to set permissions for `SELECT`, `CREATE`, and `UPDATE` operations. The `DELETE` operation only relates to records and, as such, is not available for fields. Like table permissions, field permissions apply to [record users](/docs/learn/security/authentication/users.md#record-users) (and guests when enabled), not to system users. ```surql /[test] [[test.results]] value = "NONE" */ -- Set permissions for the email field DEFINE FIELD email ON TABLE user PERMISSIONS FOR select WHERE published=true OR user=$auth.id FOR update WHERE user=$auth.id OR $auth.role="admin"; ``` ## Order of operations when setting a field's value As `DEFINE FIELD` statements are computed in alphabetical order, be sure to keep this in mind when using fields that rely on the values of others. The following example is identical to the above except that `full_name` has been chosen for the previous field `name`. The `full_name` field will be calculated after `first_name`, but before `last_name`. ```surql DEFINE TABLE person SCHEMAFULL; DEFINE FIELD first_name ON TABLE person TYPE string VALUE string::lowercase($value); DEFINE FIELD last_name ON TABLE person TYPE string VALUE string::lowercase($value); DEFINE FIELD full_name ON TABLE person VALUE first_name + ' ' + last_name; -- Creates a `person` with `full_name` of "bob BOBSON", not "bob bobson" CREATE person SET first_name = "Bob", last_name = "BOBSON"; ``` A good rule of thumb is to organise your `DEFINE FIELD` statements in alphabetical order so that the field definitions show up in the same order as that in which they are computed. ## Defining a literal on a field A field can also be defined as a [literal type](/docs/reference/query-language/language-primitives/data-types/literals.md), by specifying one or more possible values and/or permitted types. ```surql DEFINE FIELD coffee ON TABLE order TYPE "regular" | "large" | { special_order: string }; CREATE order:good SET coffee = { special_order: "Venti Quadruple Ristretto Half-Decaf Soy Latte with 4 pumps of sugar-free vanilla syrup" }; CREATE order:bad SET coffee = "small"; ``` ```surql title="Output" -------- Query -------- [ { coffee: { special_order: 'Venti Quadruple Ristretto Half-Decaf Soy Latte with 4 pumps of sugar-free vanilla syrup' }, id: order:good } ] -------- Query -------- "Found 'small' for field `coffee`, with record `order:bad`, but expected a 'regular' | 'large' | { special_order: string }" ``` --- Source: https://surrealdb.com/docs/learn/schema-management/tables-and-fields/record-id-best-practices # Record ID best practices How best to make a decision on what kind of record ID format to use in your database. A record ID is chosen once and queried forever, so its format decides how easily records can be looked up, ranged over and kept unique. This page collects the tips and trade-offs for choosing one. ## Tips and best practices for record IDs This page contains a number of tips and best practices when working with record IDs in SurrealDB. ### Why choose the right record ID format Choosing an apt record ID format is especially important because record IDs in SurrealQL are immutable. Take the following `user` records for example: ```surql FOR $i IN 0..5 { CREATE user SET user_num = $i, name = "User number " + user_num; }; ``` Each of these `user` records will have a random ID, such as `user:wvjqjc5ebqvfg3aw7g61`. If a decision is made to move away from random IDs to some other form, such as an incrementing number, this will have to be done manually. ```surql FOR $user IN SELECT * FROM user { -- Use type::record to make a record ID -- from the user_num field CREATE type::record("user", $user.user_num); -- Then delete the old user DELETE $user; }; SELECT * FROM user; ``` The final query returning just the IDs shows that they have been recreated with new IDs. ```surql title="Output" [ { id: user:0, name: 'User number 0' }, { id: user:1, name: 'User number 1' }, { id: user:2, name: 'User number 2' }, { id: user:3, name: 'User number 3' }, { id: user:4, name: 'User number 4' } ] ``` However, record IDs are also used as [record links](/docs/reference/query-language/language-primitives/record-links.md) and to create [graph relations](/docs/reference/query-language/statements/relate.md). If this is the case, more work will have to be done in order to recreate the former state. The following example shows five `user` records, which each have a 50% chance of liking each of the other users. ```surql FOR $i IN 0..5 { CREATE user SET user_num = $i, name = "User number " + user_num; }; LET $users = SELECT * FROM user; FOR $user IN $users { LET $others = array::complement($users, [$user.id]); FOR $counterpart IN $others { IF rand::bool() { RELATE $user->likes->$counterpart; } } }; ``` Finding out the current relational state can be done with a query like the following which shows all of the graph tables in which a record is located at the `in` or `out` point. The `?` is a wildcard operator, returning any and all tables found at this point of the graph query. ```surql SELECT id, ->?->? AS did, <-?<-? AS done_to FROM user; ``` ```surql title="Output" [ { did: [ user:zwfnk4by9gmopf6eeqm0 ], done_to: [ user:d6bx6sch5li8qmhq3ljl, user:ekovipptanvmgr8f48v6 ], id: user:6ycb63zr0k3cpzwel1ga }, { did: [ user:ekovipptanvmgr8f48v6, user:6ycb63zr0k3cpzwel1ga, user:zk7tpaduzaiuswll58sg ], done_to: [], id: user:d6bx6sch5li8qmhq3ljl } -- and so on.. ] ``` SurrealDB Studio's [graph visualisation view](/blog/whats-new-in-surrealist-3-2#graph-visualisation) can help as well. ![SurrealDB Studio's graph view showing possible output from the previous randomized query in which each of the five user records may or may not like another user. In this case, the output resembles a rhombus with an extra line jutting out from the top left.](~/assets/img/surrealql/datamodel/graph_view.png) With this in mind, here are some of the items to keep in mind when deciding what sort of record ID format to use. ### Meaningful sortable IDs are faster to query Records are returned in ascending record ID order by default. As the following query shows, a `SELECT` statement on a large number of `user` records with random IDs will show those with record identifiers starting with a large number of zeroes. While the IDs are sortable, the IDs themselves are completely random. ```surql CREATE |user:200000| RETURN NONE; SELECT VALUE id FROM user LIMIT 4; ``` ```surql title="Output" [ user:0001th0nnywnczi7mrvk, user:000t5r3y7u8stqtecvht, user:000tjk1nbi1it1bedplc, user:001dfral92ltbdznypcd ] ``` For a large number of records, pagination can be used to retrieve a certain amount of records at a time. ```surql -- Returns the same four records as above SELECT VALUE id FROM user START 0 LIMIT 2; SELECT VALUE id FROM user START 2 LIMIT 2; ``` ```surql title="Output" -------- Query -------- [ user:0001th0nnywnczi7mrvk, user:000t5r3y7u8stqtecvht ] -------- Query -------- [ user:001dfral92ltbdznypcd, user:001hv9g1uzh32nophrpo ] ``` As record ranges are very performant, consider moving any fields that may be used in a `WHERE` clause into the ID itself. In the following example, a number of `user` records are created using the default random ID, plus a `num` field that tracks in which order the user was created. ```surql FOR $num IN 0..100 { CREATE user SET num = $num; sleep(1ms); -- Simulate a bit of time between user creation }; SELECT * FROM user WHERE num IN 50..=51; SELECT * FROM user START 50 LIMIT 2; ``` As the output from the `SELECT` statements show, a `WHERE` clause is needed to find two users starting at a `num` of 50, as `START 50` starts based on the user of the record ID, which is entirely random. ```surql -------- Query -------- [ { id: user:pqpeg0edt8kpda907o01, num: 50 }, { id: user:ty6qr7zyob5dh882it08, num: 51 } ] -------- Query -------- [ { id: user:hvfp5m5ty7n2k95dbamv, num: 70 }, { id: user:hvfumcmmveuolg4e2h26, num: 36 } ] ``` Using a ULID in this case will allow the IDs to remain random, but still sorted by date of creation. ```surql FOR $num IN 0..100 { CREATE user:ulid() SET num = $num; sleep(1ms); }; SELECT * FROM user WHERE num IN 50..=51; SELECT * FROM user START 50 LIMIT 2; ``` Not only is the `START 50 LIMIT 2` query more performant, but the entire `num` field could be removed if its only use is to return records by order of creation. ```surql title="Same record IDs for both queries this time" -------- Query -------- [ { id: user:01JM1AHN7DDN7XM5KZ2RR2YM1S, num: 50 }, { id: user:01JM1AHN7FS4A3B6RNFCF64H90, num: 51 } ] -------- Query -------- [ { id: user:01JM1AHN7DDN7XM5KZ2RR2YM1S, num: 50 }, { id: user:01JM1AHN7FS4A3B6RNFCF64H90, num: 51 } ] ``` ### Move exact matches in array-based record IDs to the front Take the following `event` records which can be queried as a perfomant record range. ```surql CREATE event:[d'2025-05-05T08:00:00Z', user:one, "debug"] SET info = "Logged in"; CREATE event:[d'2025-05-05T08:10:00Z', user:one, "debug"] SET info = "Logged out"; CREATE event:[d'2025-05-05T08:01:00Z', user:two, "debug"] SET info = "Logged in"; ``` The ordering of the ID in this case is likely not ideal, because the first item in the array, a `datetime`, will be the first to be evaluated in a range scan. A query such as the one below on a range of dates will effectively ignore the second and third parts of the ID. ```surql SELECT * FROM event:[d'2025-05-05', user:one, "debug"]..[d'2025-05-06', user:one, "debug"]; -- Same result! user name and "debug" are irrelevant -- SELECT * FROM event:[d'2025-05-05']..[d'2025-05-06']; ``` ```surql title="Output" [ { id: event:[ d'2025-05-05T08:00:00Z', user:one, 'debug' ], info: 'Logged in' }, { id: event:[ d'2025-05-05T08:01:00Z', user:two, 'debug' ], info: 'Logged in' }, { id: event:[ d'2025-05-05T08:10:00Z', user:one, 'debug' ], info: 'Logged out' } ] ``` Instead, the parts of the array that are more likely to be exactly matched (such as `user:one` and `"debug"`) should be moved to the front. ```surql CREATE event:[user:one, "debug", d'2025-05-05T08:00:00Z'] SET info = "Logged in"; CREATE event:[user:one, "debug", d'2025-05-05T08:10:00Z'] SET info = "Logged out"; CREATE event:[user:two, "debug", d'2025-05-05T08:01:00Z'] SET info = "Logged in"; ``` Using this format, queries can now be performed for a certain user and logging level, over a range of datetimes. ```surql -- Only returns events for user:one and "debug" SELECT * FROM event:[user:one, "debug", d'2025-05-05']..[user:one, "debug", d'2025-05-06']; ``` ```surql title="Output" [ { id: event:[ user:one, 'debug', d'2025-05-05T08:00:00Z' ], info: 'Logged in' }, { id: event:[ user:one, 'debug', d'2025-05-05T08:10:00Z' ], info: 'Logged out' } ] ``` ### Auto-incrementing IDs While SurrealDB does not use auto-incrementing IDs by default, this behaviour can be achieved in a number of ways. One is to use the [`record::id()`](/docs/reference/query-language/functions/database-functions/record.md#recordid) function on the latest record, which returns the latter part of a record ID (the '1' in the record ID `person:1`). This can then be followed up with the [`type::record()`](/docs/reference/query-language/functions/database-functions/type.md#typerecord) function to create a new record ID. ```surql -- Create records from person:1 to person:10 CREATE |person:1..11|; LET $latest = SELECT VALUE id FROM ONLY person ORDER BY id DESC LIMIT 1; CREATE type::record("person", $latest.id() + 1); ``` ```surql title="Output" [ { id: person:11 } ] ``` When dealing with a large number of records, a more performant option is to use a separate record that holds a single value representing the latest ID. An [`UPSERT`](/docs/reference/query-language/statements/upsert.md) statement is best here, which will allow the counter to be initialised if it does not yet exist, and updated otherwise. This is best done [inside a manual transaction](/docs/reference/query-language/statements/begin.md) so that the latest ID will be rolled back if any failures occur when creating the next record. ```surql BEGIN TRANSACTION; UPSERT person_id:counter SET num += 1; -- Creates a person:1 CREATE type::record("person", person_id:counter.num); COMMIT TRANSACTION; BEGIN TRANSACTION; -- Latest ID is now 2 UPSERT person_id:counter SET num += 1; -- Whoops, invalid datetime format -- Transaction fails and all changes are rolled back CREATE type::record("person", person_id:counter.num) SET created_at = '2025_01+01'; COMMIT TRANSACTION; -- Latest ID is still 1 RETURN person_id:counter.num; ``` ### Record IDs are record links As a record ID is a pointer to all of the data of a record, a single record ID is enough to access all of a record's fields. This behaviour is the key to the convenience of [record links](/docs/reference/query-language/language-primitives/record-links.md) in SurrealDB, as holding a record ID is all that is needed for one record to have a link to another. When using a standalone record ID as a record pointer, be sure to use the record ID itself. ```surql CREATE person:1 SET data = { some: "demo", data: "for", demonstration: "purposes" }; LET $record = SELECT id FROM person:1; SELECT * FROM $record; ``` The output of the above query is just the `id` field on its own, as the `$record` parameter is an object with an `id` field, not the `id` field (the pointer) itself. ```surql title="Output" [ { id: person:1 } ] ``` To rectify this, `id.*` can be used to follow the pointer to the entire data for the record. ```surql SELECT id.* FROM $record; ``` ```surql title="Output" [ { id: { data: { data: 'for', demonstration: 'purposes', some: 'demo' }, id: person:1 } } ] ``` --- Source: https://surrealdb.com/docs/learn/schema-management/tables-and-fields/record-ids-and-addressing # Record IDs and addressing How record IDs work, ranges, and practical choices for addressing data. SurrealDB record IDs are composed of a table name and a record identifier separated by a `:` in between, allowing for a simple and consistent way to reference records across the database. Record IDs are used to uniquely identify records within a table, to [query](/docs/reference/query-language/statements/select.md), [update](/docs/reference/query-language/statements/update.md), and [delete](/docs/reference/query-language/statements/delete.md) records, and serve as [links](/docs/reference/query-language/language-primitives/record-links.md) from one record to another. Record IDs can be constructed from a number of ways, including [alphanumeric text](/docs/reference/query-language/language-primitives/data-types/record-ids.md#text-record-ids), complex Unicode text and symbols, [numbers](/docs/reference/query-language/language-primitives/data-types/record-ids.md#numeric-record-ids), arrays, objects, [built-in ID generation functions](/docs/reference/query-language/language-primitives/data-types/record-ids.md#random-ids), and [a function to generate an ID from values](/docs/reference/query-language/functions/database-functions/type.md#typerecord). All of the following are examples of valid record IDs in SurrealQL. ```surql company:surrealdb company:w6xb3izpgvz4n0gow6q7 reaction:`🤪` weather:['London', d'2025-02-14T01:52:50.375Z'] ``` As all record IDs are unique, trying to create a new record with an existing record ID will return an error. To create a record or modify it if the ID already exists, use an [`UPSERT`](/docs/reference/query-language/statements/upsert.md) statement or an [`INSERT`](/docs/reference/query-language/statements/insert.md#example-usage) statement with an `ON DUPLICATE KEY UPDATE` clause. ## Types of record IDs ### Random IDs When you [create a record](/docs/reference/query-language/statements/create.md) without specifying the full ID, a random identifier is assigned after the table name. ```surql CREATE company; ``` ```surql title="Output" [ { id: company:ezs644u19mae2p68404j } ] ``` Record IDs can be generated with a number of built-in ID generation functions, which are cryptographically secure and suitable for dispersion across a distributed datastore. These include a 20 digit alphanumeric ID (the default), sequentially incrementing and temporally sortable ULID Record identifiers, and UUID version 7 Record identifiers. ```surql -- Generate a random record ID 20 characters in length -- Charset: `abcdefghijklmnopqrstuvwxyz0123456789` CREATE temperature:rand() SET time = time::now(), celsius = 37.5; -- Identical to the above CREATE statement, because -- :rand() is the default random ID format CREATE temperature SET time = time::now(), celsius = 37.5; -- Generate a ULID-based record ID CREATE temperature:ulid() SET time = time::now(), celsius = 37.5; -- Generate a UUIDv7-based record ID CREATE temperature:uuid() SET time = time::now(), celsius = 37.5; ``` ### Text record IDs Text record IDs can contain letters, numbers and `_` characters. ```surql CREATE company:surrealdb SET name = 'SurrealDB'; CREATE user_version_2025 SET name = 'Alucard'; ``` To create a record ID with complex characters, use ` (backticks) around the table name and/or record identifier. ```surql CREATE article:`8424486b-85b3-4448-ac8d-5d51083391c7` SET time = time::now(), author = person:tobie; CREATE `Artykuł`:100 SET author = person:`Lech_Wałęsa`; ``` ### Numeric record IDs If you create a record ID with a number as a string, it will be stored with ` backticks to differentiate it from a number. ```surql CREATE article SET id = 10; CREATE article SET id = "10"; CREATE article SET id = "article10"; SELECT VALUE id FROM article; ``` As the record ID `article:10` is different from ```article:`10` ```, no errors are returned when creating and both records turn up in the output of the `SELECT` statement. Meanwhile, the article with the identifier `article10` does not use backticks as there is no `article10` number to differentiate it from. ```surql title="Output" [ article:10, article:`10`, article:article10 ] ``` If a numeric value is specified without any decimal point suffix and is within the range `-9223372036854775808` to `9223372036854775807` then the value will be parsed, stored, and treated as a 64-bit signed integer. Any numeric numbers outside of the range of a signed 64-bit integer will be stored as a string. ```surql CREATE temperature:17493 SET time = time::now(), celsius = 37.5; CREATE year:29878977097987987979232 SET events = [ "Galactic senate convenes", "Mr. Bean still waits in a field" ]; ``` ```surql title="Output" -------- Query -------- [ { celsius: 37.5f, id: temperature:17493, time: d'2025-02-17T06:21:08.911Z' } ] -------- Query -------- [ { events: [ 'Galactic senate convenes', 'Mr. Bean still waits in a field' ], id: year:`29878977097987987979232` } ] ``` ### Array-based record IDs Record IDs can be constructed out of arrays and even objects. This sort of record ID is most used when you have a field or two that will be used to look up records inside a [record range](/docs/reference/query-language/language-primitives/data-types/record-ids.md#record-ranges), which is extremely performant. This is in contrast to using a `WHERE` clause to filter, which involves a table scan. Records in SurrealDB can store arrays of values, including other nested arrays or objects within them. Different types of values can be stored within the same array, unless defined otherwise. ```surql CREATE weather:['London', d'2025-02-13T05:00:00Z'] SET temperature = 5.7, conditions = "cloudy"; ``` ### Why record ranges are performant The main reason why record ranges are so performant is simply because the database knows ahead of time in which area to look for records in a query, and therefore has a smaller "surface area" to work in. This can be demonstrated by seeing what happens when a single record range query encompasses all of the records in a database. The example below creates 10,000 `player` records that have an array-based record ID that begins with `'mage'`, allowing them to be used in a record range query, as well as a field called `class` that is also `'mage'`, which will be used in a `WHERE` clause to compare performance. Interestingly, in this case a record range query is only somewhat more performant. This is because both queries end up iterating over 10,000 records, with the only difference being that the query with a `WHERE` clause also checks to see if the value of the `class` field is equal to `'mage'`. ```surql FOR $_ IN 0..10000 { CREATE player:['mage', rand::id()] SET class = 'mage'; }; LET $_ = SELECT * FROM player:['mage', NONE]..['mage', ..]; LET $_ = SELECT * FROM player WHERE class = 'mage'; ``` If the number of `player` records is extended to a larger number of classes, however, the difference in performance will be much larger. In this case the record range query is still only iterating a relatively small surface area of 10,000 records, while the second one has ten times this number to go through in addition to the `WHERE` clause on top. ```surql FOR $_ IN 0..10000 { CREATE player:['mage', rand::id()] SET class = 'mage'; CREATE player:['barbarian', rand::id()] SET class = 'barbarian'; CREATE player:['rogue', rand::id()] SET class = 'rogue'; CREATE player:['bard', rand::id()] SET class = 'bard'; CREATE player:['sage', rand::id()] SET class = 'sage'; CREATE player:['psionic', rand::id()] SET class = 'psionic'; CREATE player:['thief', rand::id()] SET class = 'thief'; CREATE player:['paladin', rand::id()] SET class = 'paladin'; CREATE player:['ranger', rand::id()] SET class = 'ranger'; CREATE player:['cleric', rand::id()] SET class = 'cleric'; }; LET $_ = SELECT * FROM player:['mage', NONE]..['mage', ..]; LET $_ = SELECT * FROM player WHERE class = 'mage'; ``` ## Defining record IDs in a schema The type name of a record ID is `record`, which by default allows any sort of record. This type can be set inside a [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md) statement. ```surql DEFINE FIELD possessions ON TABLE person TYPE option>; DEFINE FIELD friends ON TABLE person TYPE option>>; ``` --- Source: https://surrealdb.com/docs/learn/schema-management/tables-and-fields/tables # Tables Declaring tables: schemaless vs schemafull, relations, views, changefeeds, and permissions. A table is the first structural unit most people define: it is a named home for records. Even a `SELECT` statement requires a table to be defined before it will work. ```surql SELECT * FROM doesnt_exist; -- Error: "The table 'doesnt_exist' does not exist" DEFINE TABLE doesnt_exist; SELECT * FROM doesnt_exist; ``` As a convenience, the creation of a record in a non-strict database will [define the table](/docs/reference/query-language/statements/define/database.md#defining-a-strict-database) for you. The following example does not return any errors because the table definition will exist once `CREATE` is executed. ```surql CREATE doesnt_exist; SELECT * FROM doesnt_exist; ``` `DEFINE TABLE` can also set high-level rules, such as whether new fields are allowed without a definition, whether records are normal documents or graph edges, whether the table is a pre-computed view, and who may select / create / update / delete. A [`COMMENT`](/docs/reference/query-language/statements/define/overview.md#comments-on-definitions) is optional but worth adding when the table's role, record-ID convention, or graph edges would otherwise be unclear to readers or to agents that load the schema via [`INFO`](/docs/reference/query-language/statements/info.md) or MCP. Individual columns still use [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md); the table statement does not replace that. You need appropriate auth (root, namespace, or database owner/editor) and an active [`USE`](/docs/reference/query-language/statements/use.md) for namespace and database. Full grammar and every clause live under [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) in the reference. ## Defining a changefeed on a table The following expression shows how you can define a `CHANGEFEED` for a table. After creating, updating, and deleting records in the table as usual, using `SHOW CHANGES FOR` returns the mutations recorded in that window. If an entry reflects an update to an existing record and the feed stores differences (`INCLUDE ORIGINAL`), the diff is a **reverse diff**, namely the operations needed to reach the state immediately before that write. See [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md#example-usage) for full examples and response shapes. ```surql -- Define the changefeed and its duration -- Optionally, append INCLUDE ORIGINAL to include info -- on the current record before a change took place DEFINE TABLE reading CHANGEFEED 3d; -- Create some records in the reading table CREATE reading SET story = "Once upon a time"; CREATE reading SET story = "there was a database"; -- Replay changes to the reading table since a certain date -- Must be after the timestamp at which the changefeed began SHOW CHANGES FOR TABLE reading SINCE d"2025-09-07T01:23:52Z" LIMIT 10; -- Alternatively, show the changes for the table since a version number SHOW CHANGES FOR TABLE reading SINCE 0 LIMIT 10; ``` ## Schemafull tables The following example demonstrates the `SCHEMAFULL` portion of the `DEFINE TABLE` statement. When a table is defined as schemafull, the database strictly enforces any schema definitions that are specified using the `DEFINE TABLE` statement. New fields can not be added to a `SCHEMAFULL` table unless they are defined via the [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md) statement. ```surql -- Create schemafull user table. DEFINE TABLE user SCHEMAFULL; -- Define some fields. DEFINE FIELD firstName ON TABLE user TYPE string; DEFINE FIELD lastName ON TABLE user TYPE string; DEFINE FIELD email ON TABLE user TYPE string ASSERT string::is_email($value); -- Statement succeeds as all defined fields are present CREATE user CONTENT { firstName: 'Tobie', lastName: 'Hitchcock', email: 'Tobie.Hitchcock@surrealdb.com' }; ``` ## Schemaless tables The following example demonstrates the `SCHEMALESS` portion of the `DEFINE TABLE` statement. This allows you to explicitly state that the specified table has no schema. ```surql -- Create schemaless user table. DEFINE TABLE user SCHEMALESS; -- Define some fields. DEFINE FIELD firstName ON TABLE user TYPE string; DEFINE FIELD lastName ON TABLE user TYPE string; DEFINE FIELD email ON TABLE user TYPE string ASSERT string::is_email($value); -- Statement succeeds even with extra `photoURI` field, as table is schemaless CREATE user:tobie SET firstName = 'Tobie', lastName = 'Hitchcock', email = 'Tobie.Hitchcock@surrealdb.com', photoURI = 'photo/yxCFi22Jw2.webp'; -- Statement fails because `email` does not pass validation CREATE user:jaime SET firstName = 'Jamie', lastName = 'Hitchcock', email = 'Jamie.Hitchcock', photoURI = 'photo/yxCFi22Jw2.webp'; ``` ## `DROP` tables and pre-computed table views In SurrealDB, like in other databases, you can create views. The way you create views is using the `DEFINE TABLE` statement like you would for any other table, then adding the `AS` clause at the end with your `SELECT` query. `DROP` tables are useful in combination with events or foreign (view) tables, as you can compute a record and drop the input. ```surql DEFINE TABLE review DROP; -- Define a table as a view which aggregates data from the review table DEFINE TABLE avg_product_review TYPE NORMAL AS SELECT count() AS number_of_reviews, math::mean( rating) AS avg_review, ->product.id AS product_id, ->product.name AS product_name FROM review GROUP BY product_id, product_name; -- Query the projection SELECT * FROM avg_product_review; ``` ## Defining permissions Table `PERMISSIONS` control what [record users](/docs/learn/security/authentication/users.md#record-users) (and [guests](/docs/learn/security/authorization/capabilities.md#guest-access), when guest access is enabled) may do with records in that table. They do not restrict [system users](/docs/learn/security/authentication/users.md#system-users) at the root, namespace, or database level, as those users are governed by roles instead. If you omit the clause, SurrealDB will default to `PERMISSIONS NONE` which denies `SELECT`, `CREATE`, `UPDATE`, and `DELETE` for record users until you grant access explicitly. The opposite shorthand is `PERMISSIONS FULL`, which allows all four operations. ```surql CREATE some_table; DEFINE TABLE some_other_table; INFO FOR DB; ``` ```surql title="Output" { analyzers: {}, functions: {}, models: {}, params: {}, scopes: {}, tables: { some_other_table: 'DEFINE TABLE some_other_table TYPE ANY SCHEMALESS PERMISSIONS NONE', some_table: 'DEFINE TABLE some_table TYPE ANY SCHEMALESS PERMISSIONS NONE' }, tokens: {}, users: {} } ``` You can also set independent rules for selecting, creating, updating, and deleting data. Each `FOR` clause is a SurrealQL expression evaluated in the context of the current authentication (often using the [`$auth`](/docs/learn/security/authorization/permissions-and-row-level-security.md) parameter which is set when a record user authenticates). ```surql -- Specify access permissions for the 'post' table DEFINE TABLE post SCHEMALESS PERMISSIONS FOR select -- Published posts can be selected WHERE published = true -- A user can select all their own posts OR user = $auth.id FOR create, update -- A user can create or update their own posts WHERE user = $auth.id FOR delete -- A user can delete their own posts WHERE user = $auth.id -- Or an admin can delete any posts OR $auth.admin = true ; ``` Field permissions work the same way but default to `PERMISSIONS FULL` instead of `NONE`. The table is the main access gate, while field permissions only narrow further when you need to (for example hiding a password). With `FULL`, a field follows the table's rules without adding its own. See [Setting permissions on fields](/docs/learn/schema-management/tables-and-fields/fields-and-validation.md#setting-permissions-on-fields) and [Permissions & row-level security](/docs/learn/security/authorization/permissions-and-row-level-security.md). ## Table with specialized `TYPE` clause When defining a table in SurrealDB, you can specify the type of data that can be stored in the table. This can be done using the `TYPE` clause, followed by either `ANY`, `NORMAL`, or `RELATION`. With `TYPE ANY`, you can specify a table to store any type of data, whether it's a normal record or a relational record. With `TYPE NORMAL`, you can specify a table to only store "normal" records, and not relations. When a table is defined as `TYPE NORMAL`, it will not be able to store relations this can be useful when you want to restrict the type of data that can be stored in a table in schemafull mode. Finally, with `TYPE RELATION`, you can specify a table to only store relational type content. This can be useful when you want to restrict the type of data that can be stored in a table. ```surql DEFINE TABLE person TYPE ANY; DEFINE TABLE person; ``` With `TYPE NORMAL`, you can specify a table to only store "normal" records, and not relations. ```surql -- Since it's default, we can also omit the TYPE clause DEFINE TABLE person TYPE NORMAL; ``` With `TYPE RELATION`, you can specify a table to only store relational type content, and restrict what kind of relations can be stored. ```surql -- Just a RELATION table, no constraints on the type of table DEFINE TABLE likes TYPE RELATION; -- Define a relation table, and constrain the type of relation which can be stored DEFINE TABLE likes TYPE RELATION FROM user TO post; -- OR use IN and OUT alternatively to FROM and TO DEFINE TABLE likes TYPE RELATION IN user OUT post; -- To allow a link to one of a possible set of record types, use the | operator DEFINE TABLE likes TYPE RELATION FROM user TO post|video; DEFINE TABLE likes TYPE RELATION IN user OUT post|video; ``` ## Using ENFORCED to ensure that related records exist As relations are represented by standalone tables, they can be constructed before any linked records exist. ```surql RELATE city:one->road_to->city:two SET distance = 12.4, slope = 5.4; ``` As such, a query on the relation will return nothing until the records it has been defined upon are created. ```surql SELECT ->road_to->city FROM city; CREATE city:one, city:two; SELECT ->road_to->city FROM city; ``` ```surql title="Output" -------- Query -------- [] -------- Query -------- [ { "->road_to": { "->city": [ city:two ] } }, { "->road_to": { "->city": [] } } ] ``` If this behaviour is not desirable, the `ENFORCED` clause can be used on a table of `TYPE RELATION` to disallow a `RELATE` statement from working unless it points to existing data. ```surql DEFINE TABLE road_to TYPE RELATION IN city OUT city ENFORCED; RELATE city:one->road_to->city:three SET distance = 5.5, slope = 30.0; ``` ```surql title="Output" "The record 'city:one' does not exist" ``` --- Source: https://surrealdb.com/docs/learn/security # Security Secure a SurrealDB deployment with authentication and authorisation. Plus security best practices. SurrealDB provides a layered security model that covers how users and systems prove their identity, what they are allowed to do once authenticated, and the operational practices that keep a deployment safe. This section is organised into three areas: - [Authentication](/docs/learn/security/authentication/overview.md) - signing in with credentials, record-based access, and third-party identity providers. - [Authorization](/docs/learn/security/authorization/permissions-and-row-level-security.md) - controlling access at the table, field, and row level with the `PERMISSIONS` clause and JWTs. - [Best practices](/docs/learn/security/best-practices/security-best-practices.md) - guidance on secure configuration, token handling, network exposure, and common pitfalls. ## Authorization - [Tokens & JWTs](/docs/learn/security/authorization/tokens-and-jwts.md) - how a token carries identity and what the database checks ## Troubleshooting - [Troubleshooting](/docs/learn/security/best-practices/troubleshooting.md) - what a rejected token, an expired session or a denied permission looks like, and what to check --- Source: https://surrealdb.com/docs/learn/security/authentication/overview # Authentication methods The four ways to authenticate with SurrealDB - system users, record users, JWT access and bearer access - what each is for, and how to sign in over HTTP or from an SDK. SurrealDB has four authentication methods. Which one you want depends on who is signing in: a person administering the instance, an end user of your application, a client an external identity provider has already authenticated, or another system. Each method is defined in SurrealQL and then used the same way from every interface, so the choice below decides the rest of the page. ## Choosing a method | Method | Who it is for | Defined with | Signed in with | | ------ | ------------- | ------------ | -------------- | | [System users](/docs/learn/security/authentication/users.md#system-users) | Operators and services administering an instance | [`DEFINE USER`](/docs/reference/query-language/statements/define/user.md) at root, namespace or database level | A username and password | | [Record users](/docs/learn/security/authentication/users.md#record-users) | End users of your application, one record each | [`DEFINE ACCESS ... TYPE RECORD`](/docs/reference/query-language/statements/define/access/record.md), whose `SIGNUP` and `SIGNIN` clauses are queries you write | Whatever fields your `SIGNIN` query reads | | [JWT access](/docs/reference/query-language/statements/define/access/jwt.md) | Clients an external provider has already authenticated | [`DEFINE ACCESS ... TYPE JWT`](/docs/reference/query-language/statements/define/access/jwt.md), holding the issuer's public key or shared secret | A token the provider issued | | [Bearer access](/docs/reference/query-language/statements/define/access/bearer.md) | Other systems and software | [`DEFINE ACCESS ... TYPE BEARER`](/docs/reference/query-language/statements/define/access/bearer.md), plus an [`ACCESS ... GRANT`](/docs/reference/query-language/statements/access.md#grant) per client | The key from a grant | System users and record users answer to credentials SurrealDB holds. JWT access holds no end user credential: it checks the signature on a token an external provider issued, then trusts the claims inside it, which is how an OpenID Connect or OAuth provider is brought in. It does hold the material for that check, and which kind matters - a [public key](/docs/reference/query-language/statements/define/access/jwt.md#public-key-cryptography) or a [JWKS URL](/docs/reference/query-language/statements/define/access/jwt.md#json-web-key-set-jwks) can only verify, while an [HMAC](/docs/reference/query-language/statements/define/access/jwt.md#hash-based-message-authentication-code-hmac) key is symmetric and can also sign. Bearer access sits between the two, issuing a key per client that you can [audit](/docs/reference/query-language/statements/access.md#show) and [revoke](/docs/reference/query-language/statements/access.md#revoke) without touching the user it acts as. > [!WARNING] > `TYPE JWT` uses `HS256` when no algorithm is given, and the HMAC algorithms (`HS256`, `HS384`, `HS512`) take one secret that both signs and verifies. Anyone holding it can issue tokens with any claims they like, and SurrealDB will trust them. Protect that key on the SurrealDB side as well as at the issuer, or define the access method with a public key or a JWKS URL, neither of which can sign. > [!NOTE] > A record user is scoped to one database and restricted by table and field [permissions](/docs/learn/security/authorization/permissions-and-row-level-security.md). A root system user is not restricted by permissions at all, so it is the wrong credential to put in an application. ## Signing in Every interface signs in with the same credentials, so an example written for one carries over to the others. **SurrealQL, HTTP and the SDKs** are covered end to end on [Users](/docs/learn/security/authentication/users.md), which defines a user of each kind and then signs in as it. **Over HTTP**, post the credentials to [`POST /signin`](/docs/reference/rest-api/http-protocol.md#signin), or create a record user with [`POST /signup`](/docs/reference/rest-api/http-protocol.md#signup). Both return a token for later requests: ```bash curl -X POST \ -H "Accept: application/json" \ -d '{"user":"root", "pass":"secret"}' \ http://localhost:8000/signin ``` **From an SDK**, each language documents its own `signin` call and the shape of the credentials it takes: [Rust](/docs/reference/rust/methods/signin.md) · [JavaScript](/docs/reference/javascript/concepts/authentication.md) · [Python](/docs/reference/python/concepts/authentication.md) · [Go](/docs/reference/golang/concepts/authentication.md) · [Java](/docs/reference/java/concepts/authentication.md) · [Kotlin](/docs/reference/kotlin/concepts/authentication.md) · [.NET](/docs/reference/dotnet/methods/signin.md) · [PHP](/docs/reference/php/v1/methods/signin.md) · [Swift](/docs/reference/swift/methods/signin.md) · [Mojo](/docs/reference/mojo/methods/signin.md) ## After signing in - [Sessions](/docs/learn/security/authentication/users.md#sessions) - what the connection carries once authenticated, and how tokens and sessions expire. - [Permissions and row-level security](/docs/learn/security/authorization/permissions-and-row-level-security.md) - what the authenticated user is then allowed to read and write. - [Tokens and JWTs](/docs/learn/security/authorization/tokens-and-jwts.md) - the claims SurrealDB reads from a token, and the parameters they populate. - [Security best practices](/docs/learn/security/best-practices/security-best-practices.md) - choosing expiry, storing secrets, and what to expose to a browser. --- Source: https://surrealdb.com/docs/learn/security/authentication/summary # Summary This page summarizes some of the security features offered by SurrealDB as well as some security elements of its development process with the goal of providing a starting point to both new and experienced users who wish to know more about the security of SurrealDB. This page is a starting point for the security of SurrealDB: the features the product offers, and the security practices behind how it is built. Each section links to the fuller treatment elsewhere in the documentation. > [!NOTE] > This page is intended to direct the reader to other more specific and comprehensive resources. Some information shown in this page may be simplified or omit information that could be relevant in a particular scenario. When available, we recommend that you consult the provided references. ## Product ### Capabilities SurrealDB offers the ability to limit its functionality to what is strictly required to reduce its attack surface. Most capabilities (e.g. scripting, networking…) are disabled by default. Even when enabled, capabilities can be restricted to specific targets such as functions that can be executed or network addresses that outbound connections can be made to. - [Security: Capabilities](/docs/learn/security/authorization/capabilities.md) - [Security Best Practices: Capabilities](/docs/learn/security/best-practices/security-best-practices.md#capabilities) ### System users SurrealDB is managed by system users. Such users can be defined at the root, namespace and database level, which they can sign into with username and password. The password for those users is stored hashed and salted using the [Argon2id](https://datatracker.ietf.org/doc/html/rfc9106#name-recommendations) algorithm with default parameters, which ensures robust resistance against modern attacks. Passwords can also be provided to SurrealDB already hashed in the form of a passhash, ensuring that the SurrealDB server never has knowledge of the original password. When defining a user, the maximum duration for authentication tokens and authenticated sessions can be explicitly defined to mitigate the impact of compromised credentials. - [Statement: DEFINE USER](/docs/reference/query-language/statements/define/user.md) - [Security Best Practices: Expiration](/docs/learn/security/best-practices/security-best-practices.md#expiration) #### Roles SurrealDB implements Role-Based Access Control (RBAC) for system users at any level. This means that even if a person or system needs to authenticate with a SurrealDB user at the root, namespace or database level, its access can be restricted within that level by the owner, editor and viewer roles to minimize the impact of an incident involving the user. - [Statement: DEFINE USER (Roles)](/docs/reference/query-language/statements/define/user.md#roles) - [Security: Authentication (System Users)](/docs/learn/security/authentication/users.md#system-users) - [Security Best Practices: Least Privilege](/docs/learn/security/best-practices/security-best-practices.md#least-privilege) ### Record users SurrealDB allows the creation of users that can be easily signed up but with no access to the database aside from specifically defined permissions. This allows clients like single-page applications or mobile applications to directly connect to the database and access certain data or even run arbitrary queries. When accessing the database as a record user, users will be restricted by table and field permissions, which deny all operations by default. End users can independently sign up and sign in to use surreal following custom logic that can be defined with SurrealQL. - [Statement: DEFINE ACCESS ... TYPE RECORD](/docs/reference/query-language/statements/define/access/record.md) - [Security: Authentication (Record Users)](/docs/learn/security/authentication/users.md#record-users) #### Permissions SurrealDB enforces table and field permissions for record users. Those permissions ensure that record users can only perform explicitly defined actions over explicitly defined data. Permissions are specified when defining a table or a field and use SurrealQL syntax to establish the conditions under which the table or the field can be queried with SELECT, UPDATE, CREATE and DELETE operations individually. Tables default to `PERMISSIONS NONE`, so a record user cannot query any data unless you grant access; fields default to `PERMISSIONS FULL`. - [Statement: DEFINE TABLE](/docs/reference/query-language/statements/define/table.md#defining-permissions) - [Statement: DEFINE FIELD](/docs/reference/query-language/statements/define/field.md#setting-permissions-on-fields) - [Permissions & row-level security](/docs/learn/security/authorization/permissions-and-row-level-security.md) ### JSON web tokens SurrealDB internally uses JWTs to perform and manage authentication for both system and record users. It also supports accepting tokens issued by third party authentication providers in order to authenticate as a system user on any level as well as a record user for an application. This ensures that advanced authentication features not present in SurrealDB can be integrated through a third party provider. This integration is simple and reliable thanks to JSON Web Key Set (JWKS) support implemented by SurrealDB. - [Statement: DEFINE ACCESS ... TYPE JWT](/docs/reference/query-language/statements/define/access/jwt.md) - [Statement: DEFINE ACCESS ... TYPE RECORD ... WITH JWT](/docs/reference/query-language/statements/define/access/record.md#with-json-web-token) - [Tutorial: Integrate Auth0 as an authentication provider](/docs/explore/tutorials/tutorials/auth0-integration.md) - [Tutorial: Integrate AWS Cognito as an authentication provider](/docs/explore/tutorials/tutorials/aws-cognito-integration.md) - [Security Best Practices: JSON Web Tokens](/docs/learn/security/best-practices/security-best-practices.md#json-web-tokens) - [Security Best Practices: Expiration](/docs/learn/security/best-practices/security-best-practices.md#expiration) ### Custom authentication SurrealDB allows record users to authenticate using a token that can be issued by a third party or SurrealDB itself after successful authentication. When verifying these tokens, custom logic can be implemented using SurrealQL to abort authentication while returning a custom error if certain conditions are not met. This logic can be used to implement various kinds of token auditing and revocation mechanisms. Additionally, tokens issued by SurrealDB can be customised to be signed with specific keys or using a specific algorithms so that other services can rely on the authentication provided by SurrealDB. - [Statement: DEFINE ACCESS ... TYPE RECORD ... AUTHENTICATE](/docs/reference/query-language/statements/define/access/record.md#with-authenticate-clause) - [Statement: DEFINE ACCESS ... TYPE RECORD ... WITH ISSUER](/docs/reference/query-language/statements/define/access/record.md#with-issuer) - [Security Best Practices: Expiration](/docs/learn/security/best-practices/security-best-practices.md#expiration) ### Parametrized queries SurrealDB is usually queried through [multiple SDKs](/docs/languages.md) and a powerful [RPC interface](/docs/reference/rest-api/rpc-protocol.md). The default query method for both of those interfaces is designed to accept query logic and variables separately to prevent query injection attacks like SQL injection. This separation ensures that user-controlled inputs are not mixed with any business logic defined in SurrealQL. - [Interfaces: RPC (Query Method)](/docs/reference/rest-api/rpc-protocol.md#query) - [Interfaces: Rust SDK (Query Method)](/docs/reference/rust/methods/query.md) - [Security Best Practices: Query Safety](/docs/learn/security/best-practices/security-best-practices.md#query-safety) ### Sessions SurrealDB accepts persistent connections through its RPC interface in the form of sessions. Sessions will usually be associated with an authentication token that represents a system user or a record user. Sessions and tokens can be configured to have different expiration times. Thanks to this, tokens can be issued to last the minimum time required to mitigate the impact of an attacker stealing the token while ensuring that sessions can last as long as required for the service or application. - [Security: Sessions](/docs/learn/security/authentication/users.md#sessions) - [Statements: DEFINE USER (Duration)](/docs/reference/query-language/statements/define/user.md#duration) - [Statements: DEFINE ACCESS (Duration)](/docs/reference/query-language/statements/define/access.md#duration) - [Security Best Practices: Expiration](/docs/learn/security/best-practices/security-best-practices.md#expiration) - [Interfaces: RPC (Authenticate)](/docs/reference/rest-api/rpc-protocol.md#authenticate) ### Cryptographic functions SurrealDB provides a series of cryptographic functions that can be called from within SurrealQL in order to implement modern and robust security practices in your application. This includes state of the art password hashing algorithms such as Argon2, Bcrypt, Scrypt and PBKDF2. Traditional hashing algorithms like SHA-256 and SHA-512 are also provided for other applications such as integrity verification. - [Cryptographic functions](/docs/reference/query-language/functions/database-functions/crypto.md) - [Security Best Practices: Passwords](/docs/learn/security/best-practices/security-best-practices.md#passwords) ### Other security functions SurrealDB also provides other functions that support developers with building secure applications. An example of this are functions which encode and sanitize HTML content that is stored in the database to prevent code injection (which can lead to cross-site scripting, clickjacking or content injection) when displaying the content in an HTML page. - [string::html::encode() function](/docs/reference/query-language/functions/database-functions/string.md#stringhtmlencode) - [string::html::sanitize() function](/docs/reference/query-language/functions/database-functions/string.md#stringhtmlsanitize) - [Security Best Practices: Content Safety](/docs/learn/security/best-practices/security-best-practices.md#content-safety) ### Transport layer security SurrealDB has the ability to provide a secure TLS connection to its HTTP server without the need for a reverse proxy or load balancer. A certificate and a private key can be provided when starting the server in order to provide a secure connection. - [CLI Start command](/docs/reference/cli/surrealdb-cli/commands/start.md) - [Security Best Practices: Encryption in Transit](/docs/learn/security/best-practices/security-best-practices.md#encryption-in-transit) ## Process ### Open source security SurrealDB has an open source security policy extending its security process to the wider community. This policy is made available through GitHub and a “security.txt” file, and allows SurrealDB to benefit from the security expertise and resources of its community. In turn, it provides the community with safe and responsible avenues to contribute to the security of a source available product that they rely on. - [GitHub: Security Policy](https://github.com/surrealdb/surrealdb/security/policy) - [Website: Well-Known Security File](/.well-known/security.txt) ### Responsible disclosure SurrealDB encourages external contributors to report security vulnerabilities following a small set of practices described in the open source security policy. SurrealDB commits to address all legitimate reports within three days, work on resolving the issue while keeping the reporter updated and crediting the reporter when an advisory is eventually published. The responsible disclosure process protects legitimate security reporters from legal repercussions and promotes an open discussion around the security of SurrealDB. - [GitHub: Security Policy (Reporting a Vulnerability)](https://github.com/surrealdb/surrealdb/security/policy#reporting-a-vulnerability) ### Security advisory SurrealDB releases security advisories whenever a significant security issue has been resolved in the product. These advisories provide details about the vulnerability, its potential impact, affected versions and possible workarounds. The publication of advisories assists both humans and automations in identifying existing risks early as well as being aware of how to immediately mitigate or resolve them. - [GitHub: Security Policy (Security Advisories)](https://github.com/surrealdb/surrealdb/security/policy#security-advisories) - [GitHub: Security Advisories](https://github.com/surrealdb/surrealdb/security/advisories) ### Software composition analysis SurrealDB includes SCA in its development process by using both the Cargo Deny binary crate for Rust code as well as Dependabot in its CI/CD pipelines. The latter ensures that changes including dependencies with known vulnerabilities cannot be merged unless those vulnerabilities are explicitly acknowledged in a public file; this usually requires either updating or replacing the affected dependency. The former provides notification of emerging vulnerabilities in dependencies that are currently being used by SurrealDB so that they can be updated or replaced. - [GitHub: Security Policy (Dependencies)](https://github.com/surrealdb/surrealdb/security/policy#dependencies) - [GitHub: Cargo Deny Configuration](https://github.com/surrealdb/surrealdb/blob/main/deny.toml) ### Fuzzing SurrealDB automatically identifies crashes and other security-relevant bugs using automated coverage-based fuzzing. This process is hosted by the Google OSS-Fuzz project and continuously runs SurrealDB with a huge range of valid and invalid inputs mutated with the help of artificial intelligence in order to cover every part of the code. This enables the early detection of edge cases which may result in security issues. The OSS-Fuzz project automatically publicly discloses bugs that have not been fixed in 90 days. - [GitHub: Security Policy (Fuzzing)](https://github.com/surrealdb/surrealdb/security/policy#fuzzing) ### Supply chain security SurrealDB aims to reduce the risk and impact of compromised dependencies in its supply chain by implementing processes and automation to ensure that new and updated dependencies are consciously evaluated, especially when they have not yet been audited by trusted organisations or they extend their access to Rust standard library interfaces. This is accomplished in Rust code by using the Cargo ACL and Cargo Vet binary crates, both of which will need to pass in CI/CD before any changes to dependencies can be merged. - [GitHub: Supply Chain Security](https://github.com/surrealdb/surrealdb/blob/main/supply-chain/README.md) --- Source: https://surrealdb.com/docs/learn/security/authentication/users # Users There are multiple forms of authentication built into SurrealDB, supporting server-side and client-side authentication. SurrealDB has two kinds of user, and they exist for different purposes: one administers the database, the other belongs to the application built on it. - [System users](#system-users): Created by the SurrealDB administrator and used for managing and consuming the database. - [Record users](#record-users): Used for consuming the database within permissions logic, they allow custom signup and signin. ## System users System users are users defined directly on SurrealDB by an administrator. Users may belong to different levels (root, namespace or database) and have different roles assigned to limit what they can do to the system. Users are defined with the [DEFINE USER](/docs/reference/query-language/statements/define/user.md) statement. SurrealDB implements RBAC (Role Based Access Control) to define what a user can do. Each user is assigned one or more roles (currently limited to the built-in `OWNER`, `EDITOR` and `VIEWER` roles) and will be allowed to perform an action over a resource as long as at least one of their roles allow it. Go to [DEFINE USER](/docs/reference/query-language/statements/define/user.md) for more information. ### Example: Define a root-level user Root-level users have visibility into all namespaces and databases, which means that their permissions apply to all of those levels. In this example we will create a root-level user `john` with a password and the `OWNER` role: ```surql DEFINE USER john ON ROOT PASSWORD "VerySecurePassword!" ROLES OWNER; ``` Note that even a root-level user can be given a limited role such as `VIEWER`. This can be useful for automated services that need to monitor each namespace and database, or coworkers high up the org chart that are not particularly tech-savvy. ```surql DEFINE USER birthday_bot ON ROOT PASSWORD "botpassword9!" ROLES VIEWER; DEFINE USER clumsy_ceo ON ROOT PASSWORD "password" ROLES VIEWER COMMENT "Don't let the CEO have more than VIEWER access"; ``` This explainer video covers the user groups SurrealDB authenticates as, and the roles a system user can hold: [Watch on YouTube](https://www.youtube.com/watch?v=cGAxH9FezUY) ### Sign in using the new user Examples using the JavaScript SDK or a raw HTTP request. #### JavaScript SDK ```javascript const db = new Surreal(); db.connect('ws://localhost:8000/rpc', { namespace: 'main', database: 'main', }); db.signin({ username: 'john', password: 'VerySecurePassword!', }); ``` #### HTTP request ```bash curl -X POST \ -H "Accept: application/json" \ -d '{"user":"john", "pass":"VerySecurePassword!"}' \ http://localhost:8000/signin ``` ### Example: Define a database-level user Database-level users have visibility into all resources that belong to the database where the user is defined. In this example we will create a database-level user `mary` with a password and the `EDITOR` role: ```surql DEFINE USER mary ON DATABASE PASSWORD "VerySecurePassword!" ROLES EDITOR; ``` ### Sign in using the new user Examples using the JavaScript SDK or a raw HTTP request. Notice how we need to pass along `NS` and `DB` properties here, to let SurrealDB know where the user is defined. #### JavaScript SDK ```javascript const db = new Surreal(); db.connect('ws://localhost:8000/rpc', { namespace: 'main', database: 'main', }); db.signin({ // Because we are signin in a database user, we need to let SurrealDB know on which database this user is located. namespace: 'main', database: 'main', username: 'mary', password: 'VerySecurePassword!', }); ``` #### HTTP request ```bash curl -X POST \ -H "Accept: application/json" \ -d '{"NS":"main", "DB":"main", "user":"mary", "pass":"VerySecurePassword!"}' \ http://localhost:8000/signin ``` ## Record users Record users represent users that are defined as a record in a database instead of through the `DEFINE USER` statement. Since these users exist as regular database records, they can have associated fields containing any information required for authentication and authorization. Thanks to this, SurrealDB is able to offer mechanisms to define your own signin and signup logic as well as custom table and field permissions for record users. This feature contributes to making SurrealDB an all-in-one BaaS (Backend-as-a-Service). Record users are defined with the [DEFINE ACCESS](/docs/reference/query-language/statements/define/access.md) statement of `TYPE RECORD`. A record access is configured with the following specific clauses: - `SIGNUP`: Defines the logic for when a user signs up as a record user. Usually creates a new record in a table. - `SIGNIN`: Defines the logic for when a user signs in as a record user. Usually checks credentials against table records. By default, record users have no permissions. They don't use the Role-Based Access Control (RBAC) system and can only access data if allowed by a `PERMISSIONS` clause, which is defined on every data resource (i.e. tables and fields) and defaults to `NONE`. To learn more about creating a record user, refer to the [DEFINE ACCESS ... TYPE RECORD](/docs/reference/query-language/statements/define/access/record.md) documentation. ### Example: Setup record authentication We will go over one of the many ways you can set up record authentication. Given you can define your own logic, there is not a single way to do it. Feel free to modify where needed! #### Define the user table and fields Typically, you would define a user table where new records are created every time a user signs up. In the following code snippet we will define the `user` table and a few `fields` that enforce the following: - An authenticated user can select, update and delete its own user record. - Asserts that the email provided by the user is actually an email address. - Forbid users to use an email that is already in use by another user. We do this by creating a unique index for the email field. ```surql title="Define tables and fields" DEFINE TABLE user SCHEMAFULL PERMISSIONS FOR select, update, delete WHERE id = $auth.id; DEFINE FIELD name ON user TYPE string; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); DEFINE FIELD password ON user TYPE string; DEFINE INDEX email ON user FIELDS email UNIQUE; ``` ##### Define the user record access Defining the `user` record access allows users to sign in and sign up by using the table and fields defined in the previous step. We will configure the record access like this: - The signin logic needs the `email` and `password` parameters to be provided by the user. In the query, we can use them as `$email` and `$password`. - The signup logic needs the `name`, `email` and `password` parameters to be provided by the user. In the query, we can use them as `$name`, `$email` and `$password`. ```surql title="Scope definition" DEFINE ACCESS user ON DATABASE TYPE RECORD SIGNIN ( SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(password, $password) ) SIGNUP ( CREATE user CONTENT { name: $name, email: $email, password: crypto::argon2::generate($password) } ); ``` ### Sign up as a user record Now that the record access is defined, we can start using it. Examples using the JavaScript SDK or a raw HTTP request. #### JavaScript SDK ```js const db = new Surreal(); db.connect('ws://localhost:8000/rpc', { namespace: 'main', database: 'main', }); db.signup({ namespace: 'main', database: 'main', // Provide the name of the access method access: 'user', // Provide the variables used by the signup query variables: { name: 'John Doe', email: 'john.doe@example.com', password: 'VerySecurePassword!', } }); ``` #### HTTP request ```bash curl -X POST \ -H "Accept: application/json" \ -d '{"NS":"main", "DB":"main", "AC":"user", "name":"John Doe", "email":"john.doe@example.com", "password":"VerySecurePassword!"}' \ http://localhost:8000/signup ``` ### Sign in as a record user Once a user has signed up, it can now sign in when needed. Examples using the JavaScript SDK or a raw HTTP request. #### JavaScript SDK ```js const db = new Surreal(); db.connect('ws://localhost:8000/rpc', { namespace: 'main', database: 'main', }); db.signin({ namespace: 'main', database: 'main', // Provide the name of the access method access: 'user', // Provide the variables used by the signin query variables: { email: 'john.doe@example.com', password: 'VerySecurePassword!', } }); ``` #### HTTP request ```bash curl -X POST \ -H "Accept: application/json" \ -d '{"NS":"main", "DB":"main", "AC":"user", "email":"john.doe@example.com", "password":"VerySecurePassword!"}' \ http://localhost:8000/signin ``` ## Sessions Whenever authentication is performed with any kind of user against SurrealDB, a session is established between the client and the SurrealDB server with which the connection was established. These sessions exist only in memory on the server for the duration of the connection, whether it is a single request through the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) or through multiple requests in the same connection using the [WebSocket API](/docs/reference/rest-api/rpc-protocol.md) and any of the [SDKs](/docs/languages.md) that leverage it. ### Parameters Certain security-related parameters are automatically set by SurrealDB in the context of a session. These parameters can be referenced in SurrealQL during authentication and authorization. #### Session The `$session` parameter contains information about the current session. This parameter is set in every SurrealDB session. In the following example, you can see the result of the `SELECT * FROM $session` query in an authenticated session for a record user using the [HTTP REST API](/docs/reference/rest-api/http-protocol.md): ```json { "ac": "user", "db": "main", "exp": null, "id": "example-client", "ip": "127.0.0.1", "ns": "main", "or": "http://www.example.com", "rd": "user:example", "tk": { "AC": "user", "DB": "main", "ID": "user:example", "NS": "main", "exp": 1723118226, "iat": 1723114626, "iss": "SurrealDB", "jti": "3b3fe74a-955c-46d7-9400-363848912292", "nbf": 1723114626 } } ``` On the root of the object, you will find the following fields: - `ip`: The IP address that established the connection with SurrealDB. - `exp`: The time at which the session will expire. - Will be `NONE` or `null` when the session does not expire. - `ns`: The name of the currently selected namespace. - Will be `NONE` or `null` when no namespace is selected. - `db`: The name of the currently selected database. - Will be `NONE` or `null` when no database is selected. - `rd`: The record identifier of the currently authenticated record user. - Will be `NONE` or `null` when unauthenticated or authenticated as a system user. - `ac`: The name of the access method that was used to authenticate. - Will be `NONE` or `null` when not authenticated with an access method. - `or`: The value of the `Origin` header of the HTTP request. - This header is usually set by browsers to identify the site that originated the request. - `id`: The value of the `surreal-id` header of the HTTP request. - This value can be set by clients to identify their individual sessions to the server. - `tk`: An object containing the claims present in the authentication token used to establish the session. - Will be `NONE` or `null` when not authenticated. The values stored in the session parameter can be accessed through the [`session::*` family of functions](/docs/reference/query-language/functions/database-functions/session.md). #### Token The `$token` parameter contains the claims contained in the token used to authenticate the current session. This parameter is set in every authenticated SurrealDB session. In this example, you can see the result of the `SELECT * FROM $token` query in an authenticated session for a record user: ```js { "AC": "user", "DB": "main", "ID": "user:example", "NS": "main", "exp": 1723118226, "iat": 1723114626, "iss": "SurrealDB", "jti": "3b3fe74a-955c-46d7-9400-363848912292", "nbf": 1723114626 } ``` Whenever authentication is done directly using a token, this object will contain the claims contained in that token. When that token is issued by SurrealDB after successful authentication, the object will usually contain the following fields: - `NS`: The name of the namespace the user is authenticated in. - Will be `NONE` or `null` when the user is authenticated at the root level. - `DB`: The name of the database the user is authenticated in. - Will be `NONE` or `null` when the user is authenticated at the root or namespace level. - `ID`: The record identifier of the currently authenticated record user. - Will not be present when not authenticated as a record user. - `AC`: The name of the access method that is used to authenticate. - Will be `NONE` or `null` when authenticating without an access method. The following fields correspond to claims defined in the standard JWT implementation described in [RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519): - `iat`: The time at which the token was issued. - `nbf`: The time before which the token will not be accepted to establish new authenticated sessions. - `exp`: The time after which the token will not be accepted to establish new authenticated sessions. - `jti`: A unique identifier used to reference the token. - `iss`: A string identifying the entity which issued the token. #### Auth The `$auth` parameter points to the record belonging to the current authenticated record user. This parameter is set only when the session is authenticated with an existing record user. When the `$auth` parameter is set, you can access any of the fields of the record corresponding to the authenticated user (e.g. `$auth.name` or `$auth.email`) via that parameter. ### Expiration Authenticated sessions remain valid for a certain duration. This duration is `NONE` by default, meaning that sessions will not expire unless otherwise specified. This duration can be customised on both the `DEFINE USER` and `DEFINE ACCESS` statements to any specific value defining the maximum duration of an authenticated session associated with that user or access method. After the defined duration, the authenticated session will expire. For example, the `DEFINE USER example DURATION FOR SESSION 1d` clause will ensure that any authenticated sessions for the `example` user will expire after a day. Expired sessions can no longer be used to call authenticated methods and doing so will result in a specific error indicating that the session has expired. SurrealQL can be used to check the expiration of the active session by running `SELECT exp FROM $session`, which show the Unix time when the session will expire or `NONE` in the case that the session does not expire. Clients can reuse a connection with an expired session to refresh the session using valid credentials. This includes calling the [`signin`](/docs/reference/rest-api/rpc-protocol.md#signin) method to obtain a new token with credentials and reauthenticate the session using that token or calling the [`authenticate`](/docs/reference/rest-api/rpc-protocol.md#authenticate) method to reauthenticate the session with an existing token. It is important to note that token duration and session duration are independent concepts. Token duration, which can be similarly customised, refers to the validity of the token (i.e. the value of its `exp` claim), during which it can be used to establish an authenticated session. Tokens issued by SurrealDB have a default duration of one hour. Token expiration is used to limit the time during which a token can be compromised resulting in unauthorized access; tokens are often stored in the client and could be stolen with attacks like cross-site scripting. Session expiration can be used to ensure that users are required to reauthenticate in order to prove that they still have access to valid credentials as well as to limit the impact of a compromised client with an established session; sessions are stored in memory in the server and cannot be similarly stolen. It is recommended that tokens are configured to last for as little as necessary before being exchanged for an authenticated session, whereas sessions are recommended to last for as little as necessary to allow for the typical client to complete a set of authenticated actions. --- Source: https://surrealdb.com/docs/learn/security/authorization/capabilities # Capabilities SurrealDB is secure by default and is suitable for all database use cases. It offers powerful features like scripting, functions or network access from within your SurrealQL queries. Most powerful features - scripting, functions, network access - are disabled by default and must be explicitly enabled by an administrator per use case. > [!IMPORTANT] > Capabilities are configured per instance from **Instance settings** in [SurrealDB Studio](https://studio.surrealdb.com). See [Configure an instance](/docs/manage/instances/configure.md) for the available settings. When a query wants to use a capability that is not allowed, SurrealDB will reject it. ```surql title="Rejected Query" http::get("https://www.surrealdb.com"); ["Access to network target 'www.surrealdb.com:443' is not allowed"] ``` This rejection will also be logged in the SurrealDB server. ```bash title="Rejected Query Logging" WARN surrealdb_core::ctx::context: Capabilities denied outgoing network connection attempt, target: 'www.surrealdb.com:443' ``` In production deployments, we recommend denying all capabilities by default and specifically allowing only those required. ```bash surreal start --deny-all --allow-funcs "array, string, crypto::argon2, http::get" --allow-net api.example.com:443 ``` You can learn more about best practices when using capabilities in our [Security Best Practices](/docs/learn/security/best-practices/security-best-practices.md#capabilities) guide. ## Priority By default, all capabilities are denied unless allowed. Some few capabilities (e.g. functions) are allowed by default. Capabilities can be configured globally (e.g. `--allow-all`, `--deny-all`), generally (e.g. `--allow-net`, `--deny-funcs`) or specifically (e.g. `--deny-net 192.168.1.1`, `--allow-funcs string::len`). When capabilities are configured, the more specific capabilities prevail over the less specific. At the same level of specificity, denies always prevail over allows. ### Examples Capabilities configured generally prevail over those defined globally: - Running with `--deny-all --allow-scripting` will deny all capabilities except for scripting. - Running with `--allow-all --deny-net` will allow all capabilities except for network. Capabilities configured specifically prevail over those defined globally or generally: - Running with `--deny-all --allow-net example.com` will deny all capabilities except network connections to `example.com`. - Running with `--allow-all --deny-funcs http` will allow all capabilities except for calling functions of the `http` family. - Running with `--deny-funcs --allow-funcs string::len` will deny all functions except for `string::len`. - Running with `--allow-net --deny-net 10.0.0.0/8` will allow all network connections except to the `10.0.0.0/8` block. Capabilities denied specifically prevail over those allowed specifically: - Running with `--deny-funcs crypto --allow-funcs md5` will deny all functions of the `crypto` including `crypto::md5`. - Running with `--allow-funcs crypto --deny-funcs md5` will allow all functions of the `crypto` family except for `crypto::md5`. ## Server versus client configuration Capability flags on [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) configure the **running database instance**. That is what enforces permissions when clients connect over HTTP, WebSocket, RPC, or [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md) to a remote endpoint. [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md) accepts the same flags, but they apply differently: | How you connect | Configure capabilities on | | --- | --- | | Remote server (`ws://`, `http://`, …) | **`surreal start`** (or server env vars) | | Embedded storage (`memory`, `rocksdb://…`, …) | **`surreal sql`** | When using a remote server, flags on `surreal sql` do **not** enable or disable execution-time checks such as [`eval::*`](/docs/reference/query-language/functions/database-functions/eval.md), arbitrary-query gates, or experimental features. They can still affect **REPL line parsing** before a query is sent. See [Capabilities and remote connections](/docs/reference/cli/surrealdb-cli/commands/sql.md#capabilities-and-remote-connections) for detail. ## List List of options for allowing capabilities:
Option Description Default
-A, --allow-all Allow all capabilities except for those more specifically denied like experimental features False
-A, --allow-arbitrary-query Denies arbitrary queries to be used by user groups. Possible user groups are: 'guest', 'record', and 'system'. False
--allow-experimental Allow the usage of one or more experimental features. Possible values are `files` and `surrealism`, separated by a comma. See experimental capabilities for which tag enables each feature. (The legacy tag gql is still accepted for compatibility but is unused from 3.3.0 - GQL is on by default.) None
--allow-eval-query [<target>,...] Allow the eval::surql and eval::gql functions for certain user groups (guest, record, system). Denied for everyone by default, even under --allow-all. Cannot bypass --deny-arbitrary-query for the same subject. See eval queries. None
--allow-funcs [<target>,...] Allow execution of all functions except for functions that are specifically denied. Alternatively, you can provide a comma-separated list of function names to allow None
--allow-guests Allow non-authenticated users to execute queries when authentication is enabled False
--allow-net [<target>,...] Allow all outbound network access except for network targets that are specifically denied. Alternatively, you can provide a comma-separated list of network targets to allow None
--allow-scripting Allow execution of embedded scripting functions False
List of options for denying capabilities:
Option Description Default
-D, --deny-all Deny all capabilities except for those more specifically allowed False
-D, --deny-arbitrary-query Denies arbitrary queries from being used by user groups. Possible user groups are: 'guest', 'record', and 'system' False
--deny-eval-query [<target>,...] Deny the eval::surql and eval::gql functions for certain user groups (guest, record, system). Specifically denied groups prevail over allowed groups. See eval queries. None
--deny-funcs [<target>,...] Deny execution of all functions except for functions that are specifically allowed. Alternatively, you can provide a comma-separated list of function names to deny None
--deny-guests Deny non-authenticated users to execute queries when authentication is enabled False
--deny-net [<target>,...] Deny all outbound network access except for network targets that are specifically allowed. Alternatively, you can provide a comma-separated list of network targets to deny None
--deny-scripting Deny execution of embedded scripting functions False
## Guest access Guest access is used when you want to expose certain parts of a database to non-authenticated users. It's useful when you want to serve datasets publicly and still require authentication for the rest of the system. Even when this capability is allowed, a guest user can only execute functions or data operations like SELECT, CREATE, etc, and only if the `PERMISSIONS` clause for the resource being used in the query allows it. ```surql -- Prepare tables with custom PERMISSIONS test/test> DEFINE TABLE protected PERMISSIONS NONE; test/test> DEFINE TABLE public PERMISSIONS FULL; -- When guest access is allowed $ surreal start --allow-guests test/test> CREATE public; [{ id: public:uy0qzy31v4xox8vivrd4 }] test/test> SELECT * FROM public; [{ id: public:uy0qzy31v4xox8vivrd4 }] test/test> CREATE protected; [] test/test> SELECT * FROM protected; [] -- When guest access is denied $ surreal start --deny-guests test/test> CREATE public; There was a problem with the database: There was a problem with the database: IAM error: Not enough permissions to perform this action test/test> SELECT * FROM public; There was a problem with the database: There was a problem with the database: IAM error: Not enough permissions to perform this action test/test> CREATE protected; There was a problem with the database: There was a problem with the database: IAM error: Not enough permissions to perform this action test/test> SELECT * FROM protected; There was a problem with the database: There was a problem with the database: IAM error: Not enough permissions to perform this action ``` ## Functions SurrealDB offers [built-in functions](/docs/reference/query-language/functions/database-functions/array.md) to perform common operations like string manipulation, math, etc. Users can also define [their own functions](/docs/reference/query-language/statements/define/function.md) with custom logic. In certain environments, you may not want users to use specific functions (i.e. `http::*`) or execute any custom function at all. You can use the allow/deny lists to configure what functions are allowed and what functions are denied. ```bash // Allow all functions except the http family and crypto::md5() surreal start --allow-funcs --deny-funcs "http","crypto::md5" // Allow certain custom functions only (all custom functions start with "fn::") surreal start --allow-funcs "fn::shared_fn" ``` ## Network SurrealDB can make outbound network connections from [`http::*`](/docs/reference/query-language/functions/database-functions/http.md) functions and from JWKS fetches used by [JWT access methods](/docs/reference/query-language/statements/define/access/jwt.md#json-web-key-set-jwks). Use `--allow-net` and `--deny-net` to control which targets those connections may reach. ```bash # Deny network access to localhost and private IPv4 ranges surreal start --allow-net --deny-net "127.0.0.1","localhost","10.0.0.0/8","192.168.0.0/16","172.16.0.0/12" # Allow access to an internal system on port 443 only surreal start --allow-net internal.example.com:443 # Allow access to some private networks but not to others surreal start --allow-net 10.0.0.0/16 --deny-net 10.10.0.0/24 ``` SurrealDB resolves hostnames with DNS, then checks every resulting IP address against `--allow-net` and `--deny-net`. A hostname that resolves to an address listed in `--deny-net` is blocked. Note that allowlisting a hostname alone is not enough when that name resolves to a private or special-use address (loopback, link-local, RFC1918 ranges, cloud metadata endpoints, etc.). SurrealDB blocks those addresses unless you also list the IP or CIDR in `--allow-net`, or you pass `--allow-net` with no target filter. This reduces SSRF risk from DNS rebinding or HTTP redirects into the local network. Example for an in-cluster JWKS issuer whose service name resolves inside `10.0.0.0/8`: ```bash surreal start --deny-all --allow-net idp.namespace.svc,10.0.0.0/8 ``` > [!WARNING] > SurrealDB does not perform reverse DNS lookups. A client can still reach an IP address directly even when a hostname that resolves to that IP is listed in `--deny-net`. This matters when network access is allowed by default (for example `--allow-net --deny-net www.example.com`) or when an IP is allowlisted while its hostname is denylisted (for example `--allow-net 203.0.113.10 --deny-net www.example.com`). Deny by default: list only the `--allow-net` targets you need, and keep additional network controls in your infrastructure. ## Arbitrary queries _(since v2.2.0)_ The `--allow-arbitrary-query` and `--deny-arbitrary-query` allows database administrators to allow or deny arbitrary quering by either guest, record or system users, or a combination of those. This capability settings affects the following: [/sql endpoint](/docs/reference/rest-api/http-protocol.md#sql), [/key endpoints](/docs/reference/rest-api/http-protocol.md#get-table), [/graphql endpoint](/docs/reference/rest-api/http-protocol.md#graphql), [/gql endpoint](/docs/reference/rest-api/http-protocol.md#gql), the [Postgres wire protocol](/docs/reference/rest-api/postgres-protocol.md) listener, [RPC methods](/docs/reference/rest-api/rpc-protocol.md) `use`, `select`, `create`, `update`, `merge`, `patch`, `delete`, `relate`, `insert`, `insert_relation`, `query`, `gql`, and `graphql`, and the [`eval::*`](/docs/reference/query-language/functions/database-functions/eval.md) functions. Endpoints that do not accept arbitrary queries such as [`/version`](/docs/reference/rest-api/http-protocol.md#version) and [authentication endpoints](/docs/reference/rest-api/http-protocol.md#signin) are not affected. The `--deny-arbitrary-query` flag is often preceded with a [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) statement to set up certain endpoints that users can use to access database resources in place of arbitrary queries. ## Eval queries _(since v3.2.0)_ The [`eval::surql`](/docs/reference/query-language/functions/database-functions/eval.md) and [`eval::gql`](/docs/reference/query-language/functions/database-functions/eval.md) functions evaluate query strings at runtime. They are controlled by **`--allow-eval-query`** and **`--deny-eval-query`**, with subject groups `guest`, `record`, and `system`. Unlike most capabilities, **eval is denied for every subject by default**, including when you pass `--allow-all`. You must opt in explicitly: ```bash surreal start --allow-eval-query ``` Arbitrary queries are **allowed by default** on the server, so you do not need `--allow-arbitrary-query` for eval unless you have restricted arbitrary queries (for example with `--deny-arbitrary-query` or `--deny-all`). If a subject is denied arbitrary queries - a common pattern alongside [`DEFINE API`](/docs/reference/query-language/statements/define/api.md) - **`eval` is denied for that subject too**, even when `--allow-eval-query` includes them. `eval` cannot bypass arbitrary-query lockdown. Deny rules at the same specificity prevail over allow rules. A record user calling an owner-defined function that invokes `eval` is still checked as `record` - auth limiting never escalates the subject. Configure `--allow-eval-query` on **`surreal start`** when clients connect to a remote instance. It is not required on `surreal sql` for remote connections - only for [embedded REPL sessions](/docs/reference/cli/surrealdb-cli/commands/sql.md#capabilities-and-remote-connections). `eval::gql` does not need an experimental capability from **3.3.0** (on **3.2.x**, also allow `gql`). See [Eval functions](/docs/reference/query-language/functions/database-functions/eval.md) and [Representations and codecs](/docs/learn/querying/concepts-and-guides/representations-and-codecs.md). --- Source: https://surrealdb.com/docs/learn/security/authorization/permissions-and-row-level-security # Permissions & row-level security How SurrealDB's PERMISSIONS clause on tables and fields controls create, select, update, and delete, including per-record and field-level rules using $auth. SurrealDB lets you declare permissions alongside your schema so access is enforced in the database, not only in application code. You attach a `PERMISSIONS` clause to [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md) and [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md) to describe which operations are allowed under which conditions. These clauses apply to [record users](/docs/learn/security/authentication/users.md#record-users) and to [guests](/docs/learn/security/authorization/capabilities.md#guest-access) when guest access is enabled. [System users](/docs/learn/security/authentication/users.md#system-users) are not restricted by table or field `PERMISSIONS`, using roles instead. ## Defaults: `NONE` and `FULL` On a table, omitting `PERMISSIONS` results in a `PERMISSIONS NONE` in the actual statement passed to the database: record users cannot `SELECT`, `CREATE`, `UPDATE`, or `DELETE` records in that table until you grant access. `PERMISSIONS FULL` allows all four operations. Field permissions default the other way. A field without an explicit clause is stored as `PERMISSIONS FULL`. The table is the main access gate, while field permissions only narrow further when you need to (for example hiding a password). With `FULL`, a field follows the table's rules without adding its own. ## Record and field rules For each table, you set independent rules for **create**, **select**, **update**, and **delete**. Each clause is a SurrealQL expression that must succeed for that operation to proceed; if it fails, the operation is rejected for the affected records. **Field-level permissions** refine this: you can constrain **select**, **create**, and **update** on individual fields - useful for sensitive data that should not be readable or writable under the same rules as the rest of the record. What the wider industry calls **row-level security** is implemented here by writing expressions that depend on the authenticated context. The **`$auth`** variable holds the current identity and claims after sign-in, so you can compare it to fields on the record (for example `owner = $auth.id`) and ensure each user only sees or changes their own data. ### Example: users read only their own records ```surql DEFINE TABLE order SCHEMALESS PERMISSIONS FOR select WHERE customer = $auth.id FOR create WHERE customer = $auth.id FOR update, delete WHERE customer = $auth.id ; ``` ### Example: a field only administrators may update ```surql DEFINE FIELD internal_note ON order TYPE string PERMISSIONS FOR select FULL FOR update WHERE $auth.role = 'admin' ; ``` Together, table- and field-level `PERMISSIONS` give you flexible authorisation without duplicating policy in every client. For full syntax and options, see [`DEFINE TABLE`](/docs/reference/query-language/statements/define/table.md#defining-permissions) and [`DEFINE FIELD`](/docs/reference/query-language/statements/define/field.md#setting-permissions-on-fields). --- Source: https://surrealdb.com/docs/learn/security/authorization/tokens-and-jwts # Tokens & JWTs How SurrealDB authenticates clients with JWTs, DEFINE ACCESS TYPE JWT, symmetric and asymmetric verification, claims as session variables, and RECORD access for database-managed sign-in. SurrealDB can authenticate clients using **JSON Web Tokens (JWTs)**. Your application or identity provider issues a signed JWT; SurrealDB verifies the signature and builds a **session** from the token’s claims so subsequent queries run with the correct identity and scope. The same mechanism applies whether the client connects over HTTP, WebSocket, or other supported interfaces: the token establishes **who** is calling before **authorisation** runs. You register JWT-based access with **`DEFINE ACCESS … TYPE JWT`** (see [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md)). SurrealDB validates the token’s signature and standard claims (such as issuer and audience) according to your definition. Verification supports **symmetric** algorithms such as **HS256** with a shared secret, and **asymmetric** algorithms such as **RS256** and **ES256** using a configured public key or JWKS - matching how major providers sign tokens at scale. Choose the model that matches your key management: shared secrets for internal services, public-key verification when keys are rotated by an external IdP. A typical JWT access definition names: - The signing method (for example HS256 with a secret, or RS256/ES256 with a public key or JWKS URL). - Expected **issuer** and **audience** values so only tokens minted for your deployment are accepted. - How claim names map into session data (so **`$auth`** exposes fields your [`PERMISSIONS`](/docs/learn/security/authorization/permissions-and-row-level-security.md) rules can use). **Claims** from the JWT are mapped into SurrealDB **session variables** (including **`$auth`**) so [permissions](/docs/learn/security/authorization/permissions-and-row-level-security.md) and queries see a consistent view of the user whether authentication came from SurrealDB or from elsewhere. Typical mappings include subject, roles, and custom namespaces your application relies on for **authorisation**. The exact claim names depend on your issuer; configure the mapping in `DEFINE ACCESS` so your policies read stable fields regardless of whether the upstream token uses flat or nested JSON. That design works well with **third-party identity providers** - Auth0, AWS Cognito, Okta, Azure AD, and similar systems that issue OIDC/OAuth2 JWTs. You configure issuer, audience, signing keys, and claim bindings once; users authenticate with the provider, receive a JWT, and present it to SurrealDB without storing end-user passwords in your database. Keep token lifetimes and key rotation in mind: short-lived access tokens and explicit issuer/audience checks reduce the impact of a leaked JWT, and asymmetric verification lets SurrealDB trust keys published via JWKS when your provider rotates them. When you need **database-managed authentication** (for example sign-up and credential checks implemented as part of your data model), use **`DEFINE ACCESS … TYPE RECORD`**. That path issues and validates access in coordination with records in your database while still producing a session compatible with the same permission model as JWT access. Many deployments combine both: system operators use **root** or scoped database users, end users authenticate via **RECORD** or brokered **JWT** access, and **`PERMISSIONS`** enforce row- and field-level rules for everyone. In practice, teams often start with **HS256** and a single secret for development or private networks, then move to **RS256** or **ES256** with JWKS in production so verification does not depend on long-lived shared secrets crossing service boundaries. The [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md) reference lists the exact clauses for each approach. For how authentication fits the wider security model, read the [authentication overview](/docs/learn/security/authentication/overview.md). For every option on JWT and record access, use the reference for [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access.md). --- Source: https://surrealdb.com/docs/learn/security/best-practices/security-best-practices # Security best practices This guide outlines some key security best practices for using SurrealDB 3.0. While SurrealDB offers powerful and flexible features to support you in meeting your desired security standards, the use that you make of those features will ultimately determine whether or not you meet them. The following is a non-exhaustive list of security best practices you should consider when building services and applications with SurrealDB to help you address common security challenges while preventing frequent pitfalls. ## Capabilities When running a SurrealDB server, you can configure the [capabilities](/docs/learn/security/authorization/capabilities.md) for your SurrealQL queries. Most of these capabilities are disabled by default to expose as little attack surface as possible to malicious actors. For the strongest security, we recommend denying all capabilities by default and only allowing the specific capabilities necessary for your service, following an allowlisting approach. We strongly discourage running SurrealDB with all capabilities allowed. ### Example: Deny all capabilities with some exceptions ```bash # Allow SurrealDB to call any functions from the array and string families, generate and compare Argon2 hashes # and make HTTP GET requests over HTTPS to the address of a specific API. surreal start --deny-all --allow-funcs \ "array, string, crypto::argon2, http::get" --allow-net \ api.example.com:443 ``` When you need to enable a capability, we recommend doing it specifically instead of generally. For example, suppose you know that your queries need to be able to parse emails using functions. In that case, we recommend you run SurrealDB with the `--allow-funcs "parse::email::*"` flag instead of allowing all functions with `--allow-funcs` without arguments. Doing this can help mitigate the performance impact that users can have when using certain functions and ensures that your SurrealDB instance will not be affected by vulnerabilities in the code of any other functions that a malicious actor could leverage to attack SurrealDB. In the case where it is absolutely necessary to generally allow a capability, we recommend carefully reviewing the scope of that capability and denying any specific instances where it may introduce unacceptable risks. This is especially important in the case of the network capability, which allows SurrealDB to perform network requests such as those required by the [`http::*`](/docs/reference/query-language/functions/database-functions/http.md) functions and by [JWKS](/docs/reference/query-language/statements/define/access/jwt.md#json-web-key-set-jwks) downloads. Allowing untrusted users to perform network requests from your SurrealDB instance can allow them access to its local network or services that specifically allow network access from the SurrealDB server. When you allow a hostname, also allow any private or special-use address that hostname resolves to, or use `--allow-net` without a filter. Hostname-only allowlists are not enough for in-cluster or RFC1918 targets. See [Network capabilities](/docs/learn/security/authorization/capabilities.md#network). ### Anti-pattern example: Allowing all outgoing network connections with some exceptions ```bash // highlight-next-line # Avoid doing this: # Allow SurrealDB to make outgoing HTTP GET and POST request to any address except to some known private CIDR blocks. surreal start --deny-all --allow-funcs "http::get, http::post" \ --allow-net --deny-net "10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16" ``` Following a denylisting approach as described above should only be used as a last resort, since it is common to miss some risky cases (e.g. [169.254.169.254](https://book.hacktricks.xyz/pentesting-web/ssrf-server-side-request-forgery/cloud-ssrf)), which in that case would become allowed by default. Additionally, SurrealDB currently does not perform reverse DNS lookups to prevent http functions directly accessing an IP address, even when the hostname that resolves to that IP address is listed within `--deny-net`. This is an issue when SurrealDB is configured with allow network access by default e.g `--allow-net --deny-net www.google.com`. It is **strongly recommended** that you deny by default by defining specific `--allow-net` targets and using additional layers of network security within your infrastructure. ## Passwords If you require storing passwords for your users, do not rely on table or field permissions to keep them private. In the event that your application or database is compromised, these passwords would become known by the attacker. Instead, use the [password hashing functions](/docs/reference/query-language/functions/database-functions/crypto.md) provided by SurrealDB such as `crypto::argon2::*`, `crypto::bcrypt::*`, `crypto::pbkdf2::*` and `crypto::scrypt::*` . These functions ensure that irreversible cryptographic hashes are stored instead of the original passwords, so that the passwords from your users remain safe even in the event of a compromise. Do not use other cryptographic hash functions (e.g. `crypto::md5`, `crypto::sha1`, `crypto::sha512`) for hashing passwords, even if you do use an additional salt. These functions are designed to be efficient in computing, which will benefit an attacker that sets out to crack any hashes that they may have obtained from the compromise of your application. Hash functions intended for password hashing already incorporate a salt as well as other mechanisms to prevent hash cracking by making the computation of such hashes less efficient. This mitigates password cracking at scale at the small cost of adding a few milliseconds delay while checking credentials for legitimate users. Even if you only store password hashes, it is a good practice to additionally use field permissions to prevent unauthorised access to the password hashes, which could allow an attacker to perform inefficient but potentially effective attacks such as testing candidate passwords against a specific hash. For even better security, you may store passwords in a separate table and use table permissions to disallow all access to that table. Due to their internal implementation, table permissions provide additional security compared to field permissions. ### Example: Securely hash user passwords ```surql DEFINE TABLE user SCHEMAFULL -- Only allow users to query their own record, including their password. PERMISSIONS FOR select, update, delete WHERE id = $auth.id; DEFINE FIELD name ON user TYPE string; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); DEFINE FIELD password ON user TYPE string; DEFINE INDEX email ON user FIELDS email UNIQUE; DEFINE ACCESS user ON DATABASE TYPE RECORD SIGNUP ( CREATE user CONTENT { name: $name, email: $email, password: crypto::argon2::generate($password) -- Use Argon2 to generate the hash. } ) SIGNIN ( SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(password, $password) -- Use Argon2 to compare the hashes. ); ``` ## Expiration When defining [users](/docs/reference/query-language/statements/define/user.md) and [access methods](/docs/reference/query-language/statements/define/access.md), ensure that you set a specific [session and token duration](/docs/learn/security/authentication/users.md#expiration) whenever possible using the `DURATION` clause. Default values provided by SurrealDB are intended to support cases where SurrealDB is used as a traditional backend database, which is why sessions do not expire by default. Suppose you build an application where your end users directly connect with SurrealDB. In that case, we strongly encourage setting a session expiration that is as short as possible (typically a few hours) to provide a good experience to your users without compromising security. Expiring user sessions ensures that a user cannot remain authenticated long after their access has been revoked. This cannot be done on demand, as user sessions do not persist in the database. However, unlike tokens, user sessions are not typically susceptible to being stolen, as they exist only in the context of an established WebSocket connection. ### Example: Set a session duration ```surql DEFINE USER username ON DATABASE PASSWORD 'CHANGE_THIS' DURATION FOR SESSION 5d; ``` ```surql DEFINE ACCESS account ON DATABASE TYPE RECORD SIGNUP ( CREATE user SET email = $email, pass = crypto::argon2::generate($pass) ) SIGNIN ( SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(pass, $pass) ) DURATION FOR SESSION 12h ; ``` Tokens, however, are usually stored in the client (e.g. a web browser) and may be stolen by client-side attacks such as a cross-site scripting vulnerability in your application. For this reason, we strongly recommend reducing the token duration from the default one hour to the minimum amount of time that your use case can tolerate. Ideally, a token should only be valid for as long as the client needs to use it to establish a session, which can be as little as a few seconds. ### Example: Set a token duration ```surql DEFINE USER username ON DATABASE PASSWORD 'CHANGE_THIS' DURATION FOR TOKEN 15m; ``` ```surql DEFINE ACCESS account ON DATABASE TYPE RECORD SIGNUP ( CREATE user SET email = $email, pass = crypto::argon2::generate($pass) ) SIGNIN ( SELECT * FROM user WHERE email = $email AND crypto::argon2::compare(pass, $pass) ) DURATION FOR TOKEN 5s ; ``` ## Query safety When using SurrealDB as a traditional backend database, your application will usually build SurrealQL queries that may need to contain some untrusted input, such as that provided by the users of your application. To do so, SurrealDB offers [`bind`](/docs/reference/rust/methods/query.md) as a method to `query` (implemented in other SDKs as [the `vars` argument to `query`](/docs/reference/javascript/concepts/executing-queries.md)), which should always be used when including untrusted input into queries. Otherwise, SurrealDB will be unable to separate the actual query syntax from the user input, resulting in the well-known [SQL injection](https://en.wikipedia.org/w/index.php?title=SQL_injection&oldid=1234729055) vulnerabilities. This practice is known as [prepared statements or parameterised queries](https://en.wikipedia.org/w/index.php?title=Prepared_statement&oldid=1195122133). Binding parameters ensure that untrusted data is passed to SurrealDB as SurrealQL parameters, which are independent of the query syntax, preventing SQL injection attacks. ### Example: Bind parameters in the provided sdks **Rust** ```rust // Do this: let name = "tobie"; // User-controlled input. let mut result = db .query("CREATE person CONTENT name = $name;") .bind(("name", name)) .await?; ``` ```rust // highlight-next-line // Do NOT do this: let name = "tobie"; // User-controlled input. let mut result = db .query(format!("CREATE person CONTENT name = {name};")) .await?; ``` **JavaScript** ```jsx // Do this: const name = "tobie"; // User-controlled input. const result = await db.query( 'CREATE person CONTENT name = $name;', { name } ); ``` ```jsx // highlight-next-line // Do NOT do this: const name = "tobie"; // User-controlled input. const result = await db.query(`CREATE person CONTENT name = \ "${name}";`); ``` **.NET (C#)** ```csharp // Do this: string name = "tobie"; // User-controlled input. var result = await db.Query($"CREATE person CONTENT name = {name};"); // Translated as "CREATE person CONTENT name = $p0;" // with the parameter $p0 having the value "tobie" ``` ```csharp // highlight-next-line // Do NOT do this: string name = "tobie"; // User-controlled input. var result = await db.RawQuery($"CREATE person CONTENT name = "{name}";"); ``` ### Example: Bind parameters in the HTTP REST API ```bash title="Request" curl -X POST -u "root:secret" -H "surreal-ns: main" -H \ "surreal-db: main" -H "Accept: application/json" \ -d 'SELECT * FROM person WHERE age > $age' \ http://localhost:8000/sql?age=18 ``` ## Content safety Content generated by users and other untrusted parties will often be stored in SurrealDB and later rendered in an HTML page to be displayed. Regardless of SurrealDB, rendering untrusted content is the source of some dangerous pitfalls which can lead to [cross-site scripting](https://en.wikipedia.org/w/index.php?title=Cross-site_scripting&oldid=1232455342) attacks and other client-side code injection issues like [site defacement](https://en.wikipedia.org/w/index.php?title=Website_defacement&oldid=1231310592) or [clickjacking](https://en.wikipedia.org/w/index.php?title=Clickjacking&oldid=1227193298). When retrieving content that may be rendered in an HTML document, we strongly recommend that you use the [`string::html::encode`](/docs/reference/query-language/functions/database-functions/string.md#stringhtmlencode) function, which will encode any characters that have special meaning in HTML syntax (e.g. `<`, `>`, `&`...) into HTML entities (e.g. `<`, `>`, `&`...) that will be rendered as the actual original character instead of interpreted as HTML syntax. ### Example: Encode HTML content ```surql string::html::encode("

Safe Title

Safe paragraph. Not safe event.

"); ['<h1>Safe Title</h1><script>alert('XSS')</script><p>Safe paragraph. Not safe <span onload='logout()'>event</span>.</p>'] ``` If you absolutely require user-generated content to be rendered as HTML but still want to prevent users from injecting dangerous HTML into your page, you can use the [`string::html::sanitize`](/docs/reference/query-language/functions/database-functions/string.md#stringhtmlsanitize) function instead, which will keep all characters intact, so that the content can be interpreted as HTML syntax, while removing the specific syntax that is deemed dangerous. It is important to note that, although the set of accepted syntax is very conservative, sanitization is less safe that encoding and could potentially be bypassed due to a flaw in the function. ### Example: Sanitize HTML content ```surql string::html::sanitize("

Safe Title

Safe paragraph. Not safe event.

"); ['

Safe Title

Safe paragraph. Not safe event.

'] ``` ## JSON Web Tokens When configuring how [JSON Web Tokens](https://datatracker.ietf.org/doc/html/rfc7519) are verified before authenticating a [system](/docs/learn/security/authentication/users.md#system-users) or [record](/docs/learn/security/authentication/users.md#record-users) user with [`DEFINE ACCESS ... TYPE JWT`](/docs/reference/query-language/statements/define/access/jwt.md) or [`DEFINE ACCESS ... TYPE RECORD ... WITH JWT`](/docs/reference/query-language/statements/define/access/record.md#with-json-web-token), we recommend using an asymmetric algorithm (i.e. `PSXXX`, `RSXXX`, `ECXXX`) when only a mechanism for token verification is being defined. This ensures that the only key stored by SurrealDB is a public key that does not represent a threat in the event of a compromise. On the other hand, symmetric algorithms (i.e., HSXXX) use the same key for signature and verification, which the attacker could use to issue tokens that SurrealDB would trust. ### Example: Define a JWT access method ```surql DEFINE ACCESS token ON DATABASE TYPE RECORD WITH JWT ALGORITHM RS256 KEY "-----BEGIN PUBLIC KEY----- MUO52Me9HEB4ZyU+7xmDpnixzA/CUE7kyUuE0b7t38oCh+sQouREqIjLwgHhFdhh3cQAwr6GH07D ThioYrZL8xATJ3Youyj8C45QnZcGUif5PkpWXDi0HJSoMFekbW6Pr4xuqIqb2LGxGDVJcLZwJ2AS Gtu2UAfPXbBD3ffiad393M22g1iHM80YaNi+xgswG7qtXE4lR/Lt4s0MeKKX7stdWI1VIsoB+y3i r/OWUvJPjjDNbAsyy8tQmxydv+FUnLEP9TNT4AhN4DXcJ+XsDtW7OWt4EdSVDeKpGbIMvIrh1Pe+ Nilj8UHNyNDHa2AjK3seMo6CMvaIQJKj5o4xGFblFGwvvPD03SbuQLs1FdRjsZCeWLdYeQ3JDHE9 sFG7DCXlpMJcaYT1mf4XHJ0gPekNLQyewTY3Vxf7FgV3GCNjV20kcDFgJA2+iVW2wSrb+txD1ycE kbi8jh0pedWwE40VQWaTh/8eAvX7IHWya/AEro25mq+m6vktNZLbvLphhp586kJK3Tdt3YjpkPre M3nkFWOWurIyKbtIV9JemfwCgt89sNV45dTlnEDEZFFGnIgDnWgx3CUo4XmhICEQU8+tklw9jJYx iCTjhbIDEBHySSSc/pQ4ftHQmhToTlQeOdEy4LYiaEIgl1X+hzRH1hBYvWlNKe4EY1nMCKcjgt0= -----END PUBLIC KEY-----"; ``` Additionally, we recommend using [JSON Web Key Sets](https://datatracker.ietf.org/doc/html/rfc7517) to configure the verification algorithm and key from a remote authoritative source using the `URL` clause instead of providing them directly to SurrealDB using the `ALGORITHM` and `KEY` clauses. This ensures that the original token issuer will be able to rotate keys in the event of a compromise to prevent potentially compromised tokens to be used with your application without affecting the availability of your service. ### Example: Define a JWT access method with JWKS ```surql DEFINE ACCESS token ON DATABASE TYPE RECORD WITH JWT URL "https://example.com/.well-known/jwks.json"; ``` ## Network exposure When deploying SurrealDB, we recommend limiting the attack surface as much as possible in order to minimise the risk of attacks or information gathering from unauthorised parties. If your database should only be available to other internal services, we suggest that you expose SurrealDB exclusively to the internal network instead of deploying the service with a publicly addressable network interface that is accessible from the internet, regardless of whether or not allowlisting has been applied at the networking or application level. If you must publish SurrealDB to the internet (e.g. if your users directly connect to SurrealDB), you may want to monitor and prevent unwanted connections using tools such as a network [intrusion prevention system](https://en.wikipedia.org/w/index.php?title=Intrusion_detection_system&oldid=1223972754#Intrusion_prevention) or a [web application firewall](https://en.wikipedia.org/w/index.php?title=Web_application_firewall&oldid=1234730173). If you do so, ensure that these systems are appropriately tuned and do not interfere with the regular use of SurrealDB. In cases where SurrealDB is publicly exposed in environments where any sort of information leakage is unacceptable, the `--no-identification-headers` flag can be enabled, which will result in the SurrealDB server no longer responding to HTTP requests with headers that identify the product or its current version to prevent passive fingerprinting and metadata indexing. Note that this will not prevent active fingerprinting such as directly querying the `/version` endpoint if available or directly attempting to exploit a known security vulnerability without regard for compatibility. On the other hand, consider whether or not enabling this feature is compatible with your clients, which may rely on these headers in order to identify the version of SurrealDB running on the server. ### Example: Start SurrealDB with identification headers ```bash $ surreal start & $ curl -vvv "127.0.0.1:8000" * Trying 127.0.0.1:8000... * Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:8000 > User-Agent: curl/7.81.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 307 Temporary Redirect < location: https://surrealdb.com/app < access-control-allow-origin: * < vary: origin < vary: access-control-request-method < vary: access-control-request-headers # highlight-start < surreal-version: surrealdb-2.0.0+20240612.2184e80f < server: SurrealDB # highlight-end < x-request-id: 157413ce-7cc4-41a1-a93b-0940bf87874c < content-length: 0 < date: Mon, 17 Jun 2024 15:47:29 GMT < * Connection #0 to host 127.0.0.1 left intact ``` ### Example: Start SurrealDB without identification headers ```bash $ surreal start --no-identification-headers & $ curl -vvv "127.0.0.1:8000" * Trying 127.0.0.1:8000... * Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:8000 > User-Agent: curl/7.81.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 307 Temporary Redirect < location: https://surrealdb.com/app < access-control-allow-origin: * < vary: origin < vary: access-control-request-method < vary: access-control-request-headers < x-request-id: deec3301-e930-4389-a0da-b2a336bd2631 < content-length: 0 < date: Mon, 17 Jun 2024 15:49:43 GMT < * Connection #0 to host 127.0.0.1 left intact ``` ## Least privilege When defining [system users](/docs/learn/security/authentication/users.md#system-users) in SurrealDB, you may assign them [roles](/docs/reference/query-language/statements/define/user.md#roles) that will limit the actions they can perform inside the level where they are defined. Ensure that you employ the principle of least privilege and create users at the lowest level possible and with the minimum role in order to be able to perform their duties inside of SurrealDB. This will mitigate some of the risk in the case where credentials for that user are ever compromised. ### Example: Define users with specific roles A user who only needs to query the database: ```surql DEFINE USER db_viewer ON DATABASE PASSWORD 'CHANGE_THIS' ROLES VIEWER; ``` A user who only needs to manage content in any databases on the same namespace: ```surql DEFINE USER ns_editor ON NAMESPACE PASSWORD 'CHANGE_THIS' ROLES EDITOR; ``` ## Encryption in transit Encryption in transit is recommended, especially when deploying SurrealDB on a server in a different network than its clients. This mitigates the impact of man-in-the-middle attacks and provides confidentiality and integrity guarantees with regard to the data being exchanged. Encryption in transit can be achieved by using the SurrealDB server to serve its interfaces through HTTPS by providing the `--web-crt` and `--web-key` arguments when calling [the `start` subcommand in the CLI](/docs/reference/cli/surrealdb-cli/commands/start.md#command-help). For production deployments, we recommend that TLS termination be performed by a load balancer or reverse proxy, which will often provide additional guarantees to the process. ### Example: Start SurrealDB with TLS ```bash # If you want to serve TLS directly with SurrealDB: surreal start --web-crt "cert.pem" --web-key "key.pem" ``` ## Encryption at rest Encryption at rest is recommended especially when storing sensitive data in a location where you cannot guarantee the security of the storage media. If encryption at rest is not used, physical access to the storage media may result in the complete compromise of the data stored. It is important to note that most kinds of encryption at rest will not prevent logical attacks from resulting in compromise of the data, as such attacks will often access the data using the compromised system as a [confused deputy](https://en.wikipedia.org/w/index.php?title=Confused_deputy_problem&oldid=1230222963) in order to leverage its ability to access data after it is already decrypted. Encryption at rest can be achieved by ensuring that the data is stored encrypted using a disk encryption solution such as [LUKS](https://en.wikipedia.org/w/index.php?title=Linux_Unified_Key_Setup&oldid=1225491340) or [BitLocker](https://en.wikipedia.org/w/index.php?title=BitLocker&oldid=1232782210) in Linux and Windows systems respectively and, in the case where you are hosting SurrealDB in a cloud provider, by leveraging their storage encryption solutions in the volume or disk that will store your data. You might consider additional encryption for your datastore in some specific scenarios. This can provide increased security when your database servers, storage media and their corresponding encryption keys are managed in different security contexts, where the storage media and its keys may be compromised without also compromising the datastore servers. Encryption at rest at the datastore level can be achieved by using a datastore backend that offers transparent encryption. This encryption is independent from SurrealDB. It is important to note that, even in this scenario, physical or logical access to the SurrealDB server will result in access to the data, as SurrealDB must receive decrypted data from the datastore in order to perform any sort of queries. ## Untrusted queries Due to the powerful SurrealQL language and the addition of functions, scripting and network capabilities, running untrusted queries in SurrealDB as a [system user](/docs/learn/security/authentication/users.md#system-users) should be treated similarly to running untrusted software in any system. When copying queries or importing datasets from sources that you do not trust, make sure to review their contents to ensure that they do not contain any malicious code intended to perform unauthorized changes, computations or network requests. ## Session isolation One of the interfaces to SurrealDB is [RPC through WebSockets](/docs/reference/rest-api/rpc-protocol.md). This interface is usually used by the official [SDKs](/docs/languages.md) and offers performance benefits over the [HTTP REST API](/docs/reference/rest-api/http-protocol.md), which requires establishing a new connection for every operation. The RPC interface can either be directly exposed to end users or used internally from your backend to communicate with SurrealDB. In the later scenario, some developers may choose to still authenticate each user individually (e.g. using [`signin`](/docs/reference/rest-api/rpc-protocol.md#signin) or [`authenticate`](/docs/reference/rest-api/rpc-protocol.md#authenticate) in the WebSockets [session](/docs/learn/security/authentication/users.md#sessions) as opposed to using a single service user for their backend. This could be done by calling [`invalidate`](/docs/reference/rest-api/rpc-protocol.md#invalidate) or just authenticating a new user in the same connection and may provide some performance benefits over establishing a new WebSocket connection for each user. However, we recommend using separate WebSocket sessions or connections for different users. Consider terminating the connection and establishing a new one for every individual user. WebSocket connections offer an additional degree of isolation between users that may become relevant in the event where some session information for previous users who were using the same connection was not properly cleared. Additionally, even if successfully isolated from the security perspective, some resources associated with users are freed by SurrealDB only when the connection is terminated. Sharing the same WebSockets connection between several users may cause these unused resources to grow indefinitely. ## Token storage In some instances, applications may need to store some of the authentication tokens issued by SurrealDB. Even when token expiration has been configured to be as low as possible, tokens may potentially be stolen as a result of attacks against the application. To mitigate this risk, it is important to take steps to protect tokens in storage from being stolen as a result of these attacks. This is specially relevant in web applications, which usually expose additional attack vectors compared to other client applications. The best way to protect tokens against stealing is to not store them at all. If your use case supports it, use the token in memory to [`authenticate`](/docs/reference/javascript/api/core/surreal-session.md) a persistent session using the WebSocket protocol and destroy the token from memory after the session is established. When the session expires, ask your users to sign in again with their credentials and establish a new authenticated session with SurrealDB. Your use case may even support not using a token at all by directly authenticating the session with user credentials using [`signin`](/docs/reference/javascript/api/core/surreal-session.md#signin). However, if you must store the authentication token (e.g. you want authentication to persist across browser tabs or restarts), our recommendation for most use cases is that you store tokens using browser storage primitives such as local storage and that you take steps to protect your web application from script injection attacks by taking measures including the following: - Encode or at least sanitize all [untrusted input](#content-safety) before showing it on the page. - Implement a [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to prevent unauthorized scripts from executing. - Implement [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) to verify authorised external scripts. - Use modern frontend frameworks that are designed to prevent content injection. Understand that an attacker who is ultimately able to inject scripts into your web application or compromise the devices of your users will still be able to steal their tokens. These recommendations are intended to prevent this script injection from taking place. There is very little you can do to protect your users if you application is vulnerable to script injection attacks regardless of storage method. The impact of this actually happening can be mitigated by ensuring that token expiration is short to minimize the chance of an attacker capturing a valid token and reduce the window of oportunity to exploit it otherwise. ### Why not cookies? SurrealDB does not support authenticating via cookies. Although cookies with the `secure` and `HttpOnly` flags are often cited as the superior choice for token storage, this is [not always the case](https://portswigger.net/research/web-storage-the-lesser-evil-for-session-tokens). This is specially not true in the case of generic backend services such as SurrealDB, where protecting against [Cross-Site Request Forgery (CRSF)](https://owasp.org/www-community/attacks/csrf) attacks is not trivial without additional control of the frontend application. These attacks are possible because of how cookies work and would allow attackers to force users to make unauthorized requests to SurrealDB using their own valid cookies. Additionally, cookies are limited to a 4KB size, making them unsuitable for storing certain JWT payloads. The proposed benefits of using cookies would be that [Cross-Site Scripting (XSS)](https://owasp.org/www-community/attacks/xss/) attacks could not be used to directly read the contents of the token as long as cookies were configure with the `HttpOnly` flag. Although this is true, XSS attacks could still be used to take control of the browser session and impersonate the user using their own cookies to perform any authenticated actions that the token could be used for. This is essentially as bad as the token being stolen because an attacker is not interested in the token itself but rather in what the token can be used for. Most modern attacks against cookies with `HttpOnly` will lead to essentially the same results as those against cookies without it. In our opinion, the CSRF attacks made possible by cookies would be an unmitigated threat to applications built on SurrealDB. Additionally, XSS attacks are still a threat when using cookies with the `HttpOnly` flag. On the other hand, significant advances have been made by modern browsers and frontend frameworks to prevent XSS attacks, whereas CSRF attacks are not possible to mitigate without the frontend and backend services working together in a way that would not be trivial to implement between SurrealDB and self-developed frontend applications. ## Vulnerabilities When SurrealDB is part of your service or application, vulnerabilities that affect SurrealDB may also impact your environment. Due to this fact, we highly recommend that you track [vulnerabilities published for SurrealDB](https://github.com/surrealdb/surrealdb/security/advisories) so that you become aware of any updates that address vulnerabilities that you may be affected by. This can be done most effectively by leveraging automation tools that will consume the [GitHub Advisory Database](https://github.com/advisories?query=surrealdb). These automations will usually also warn of vulnerabilities in dependencies used by SurrealDB, which may also have an impact in your environment. Keeping up to date with the latest releases of SurrealDB is, in general, a good practice. If you identify a vulnerability in SurrealDB that has not been published yet, we encourage you to [create a security advisory report](https://github.com/surrealdb/surrealdb/security/advisories/new) on GitHub so that the SurrealDB team can privately look into it in order to identify and work on a solution that can benefit you as well as the rest of the users. --- Source: https://surrealdb.com/docs/learn/security/best-practices/troubleshooting # Troubleshooting This page provides some troubleshooting advice to support users in addressing issues either caused by or involved in the usage of specific security features provided by SurrealDB. This page collects the errors that come up most often when using SurrealDB's security features, with the cause of each and how to resolve it. ## Authentication ### Invalid authentication error The most common error that users may receive when authenticating to SurrealDB is `InvalidAuth`, which results in a generic message such as `There was a problem with authentication`. The reason that this message is so vague and yet so common is that it is a placeholder for other more specific internal errors related to authentication. These errors are not returned to the client due to their potential to leak internal information about the database (e.g. whether or not a user or access method exists, whether a token failed to verify because of its signature or a specific claim...) to unauthenticated users. Although the internal cause for these errors is not revealed to clients, it can be identified by the SurrealDB administrator through its server logs. These logs will usually be prefixed by `surrealdb::core::iam`. Most helpful messages will be displayed when [starting SurrealDB with `--log debug`](/docs/reference/cli/surrealdb-cli/commands/start.md#command-help). If those logs are not enough to diagnose the problem, starting the server with `--log trace` will provide additional messages describing the authentication process. In situations where debugging must be performed on the client, starting the SurrealDB server with the environment variable `SURREAL_INSECURE_FORWARD_ACCESS_ERRORS` set to `true` will forward errors resulting from [`SIGNIN`, `SIGNUP`](/docs/reference/query-language/statements/define/access/record.md#example-usage) and [`AUTHENTICATE`](/docs/reference/query-language/statements/define/access/record.md#with-authenticate-clause) clauses directly to clients. Since these errors may expose the internal logic of such clauses, this variable should only be used for debugging in controlled environments with trusted clients. #### Common causes The following are the most common causes for a generic authentication error in SurrealDB. ##### Incorrect user credentials The simplest reason why authentication may fail. Ensure that the credentials that you are using to authenticate (e.g. username and password) match the ones that you have defined. For system users, ensure that any special characters (e.g. quotes) have been interpreted correctly as part of the username or password rather than as SurrealQL syntax. Use the `INFO` statements to display the defined users and ensure that their password hash matches the hash of the password that you are providing. You can use the included [compare hash functions in SurrealQL](/docs/reference/query-language/functions/database-functions/crypto.md#cryptoargon2compare) to perform this verification. For record users, ensure that the `SIGNIN` and `SIGNUP` queries will return a user. You can do that by running those queries by themselves in the SurrealQL interpreter (`surreal sql`) or in SurrealDB Studio. You will just need to replace any parameters from those queries by the parameters that you are providing during authentication. Ensure that you are in fact providing all the expected parameters during authentication in your actual code. ##### The referenced user or access method does not exist Ensure that the user and access method that you are authenticating with exists on the level that you are trying to authenticate. Check that you are authenticating against the namespace or database where the resource is defined and that you have correctly spelled the resource everywhere. When authenticating with the SurrealDB CLI or with the HTTP REST API, ensure that you specify the namespace and database that your user exists in (i.e. `--auth-level ns` and `--auth-level db` or `surreal-auth-ns` and `surreal-auth-db`), in addition to the namespace and database that you want to use for the connection (i.e. `--namespace` and `--database` or `surreal-ns` and `surreal-db`). Otherwise, SurrealDB will default to authenticating at the root level. ##### The token could not be cryptographically verified When using a token to authenticate, SurrealDB will reject tokens that fail the verification of its cryptographic signature. If the token has been issued by SurrealDB, this is most likely not the cause for your issue. If the token has not been issued by SurrealDB, this usually means that the provided token is signed using a different algorithm than defined or that the key used to sign the token is incorrect. For tokens that are [verified from a URL hosting a JWKS object](/docs/reference/query-language/statements/define/access/jwt.md#json-web-key-set-jwks), failure to verify the token may be caused by the URL being incorrect, unreachable by the SurrealDB server, or outbound connections to the address not being allowed by the [network capability](/docs/learn/security/authorization/capabilities.md#network). If the JWKS hostname resolves to a private or special-use address, allow that IP or CIDR in addition to the hostname. Relevant logs appear on the SurrealDB server when you run with `--log debug` or `--log trace`. ##### The token is missing a required claim Tokens that are missing any of the claims required by SurrealDB will be rejected with a generic error. Ensure that tokens issued outside of SurrealDB contain all the required claims. Depending on whether the token is for a system user, a record access method or a JWT access method, different claims will be expected. In all cases, the `exp` claim is required. Consult the [Using Tokens](/docs/reference/query-language/statements/define/access/jwt.md#using-tokens) section of the `DEFINE ACCESS ... TYPE JWT` documentation for more information about the required claims for system and record users. ### Other authentication errors The following is a non-comprehensive list of authentication errors which may sometimes be challenging to understand. #### Unexpected authentication error Like `InvalidAuth`, `UnexpectedAuth` is an error that acts as a placeholder for a different group of errors. This error results in a generic message such as `There was an unexpected error while performing authentication`. Unlike errors specific to authentication, this error is returned when the original error is not directly related with authentication, but rather with a failure in some of the components necessary for authentication. This distinction is made because, unlike `InvalidAuth` errors, `UnexpectedAuth` errors may be retried by clients, as it is possible that the internal failure that resulted in the error is ephemeral. Clients may want to handle these two errors differently; for example, by returning `InvalidAuth` errors to user but internally attempting to retry on `UnexpectedAuth` errors, only returning an internal error to the user if the error persists. A specific example for this error is the `AUTHENTICATE`, `SIGNIN` or `SIGNUP` clauses failing due to a transaction error, such as those which can arise from a write conflict during the transaction. These conflicts may appear for some datastores in any query when two transactions simultaneously access the same document. #### Token expired error This error is returned when the token that is being used to authenticate a session has [expired](/docs/learn/security/authentication/users.md#expiration). If this token was issued by SurrealDB, this expiration defaults to an hour and can be changed using the `DURATION FOR TOKEN` clause in [`DEFINE ACCESS`](/docs/reference/query-language/statements/define/access/record.md) and [`DEFINE USER`](/docs/reference/query-language/statements/define/user.md). If the token was not issued by SurrealDB, this expiration will be set by the `exp` claim. Note that this error will only appear when trying to authenticate the [session](/docs/learn/security/authentication/users.md#sessions). After authentication, the session will not expire when the token does, but rather after the independent session duration that has been defined in the `DURATION FOR SESSION` clause. By default, sessions will not expire. When using the HTTP REST API, a persistent session is not established as each request will be individually authenticated, as a consequence, requests made using an expired token will be rejected with this error. Although tokens accepted by SurrealDB must have some expiration, you can configure any amount of time that fits your security and usability requirements with the `DURATION FOR TOKEN` clause or by configuring the `exp` claim if you are using an external token issuer. To address token expiration errors you will either have to ask your end user to authenticate again with credentials to obtain a new token, rely on a persistent authenticated WebSocket session, rely on an external identity provider like [Auth0](/docs/explore/tutorials/tutorials/auth0-integration.md) or [AWS Cognito](/docs/explore/tutorials/tutorials/aws-cognito-integration.md) or, for development purposes, test the [expimental refresh token feature](/docs/reference/query-language/statements/define/access/record.md#with-refresh-token). ### Other authentication issues #### Slow / expensive authentication In SurrealDB, clients can authenticate using different methods, such as credentials (e.g. username and password), a token (i.e. JWT) and other access methods like bearer keys. These access methods provide different benefits and have different performance implications. Different options are provided precisely to allow flexibility for different use cases. If you observe that authentication requests are slow or computationally (e.g. CPU, memory...) expensive, consider switching from authentication with credentials to token authentication. By default, SurrealDB verifies passwords for system users with [Argon2id](https://en.wikipedia.org/wiki/Argon2), which is a password hashing algorithm that ensures that the password verification process is slow and expensive to prevent password cracking. If you are using the [HTTP REST API](/docs/reference/rest-api/http-protocol.md) to authenticate using credentials (i.e. with `Authorization: Basic ...` rather than `Authorization: Bearer ...`), this password verification process will be performed for every request. Instead, consider [authenticating once with credentials](/docs/reference/rest-api/http-protocol.md#signin) to obtain a token and performing subsequent requests using that token in the `Authorization` header prefixed by `Bearer`. Token verification is a different cryptographic process that is designed to be orders of magnitude faster and cheaper. When using credentials to authenticate record users, you are able to choose the password hashing algorithm that you use in your `SIGNIN` query. If you must repeatedly authenticate using credentials and the performance impact of using Argon2id is unacceptable to your use case, you may consider using different password hashing algorithms (e.g. [Bcrypt](/docs/reference/query-language/functions/database-functions/crypto.md#cryptobcryptcompare), [Scrypt](/docs/reference/query-language/functions/database-functions/crypto.md#cryptoscryptcompare) or [PBKDF2](/docs/reference/query-language/functions/database-functions/crypto.md#cryptopbkdf2compare)) that may be more efficient at the cost of diminished security against certain types of password cracking. Authentication logic that is implemented by users via the `SIGNIN`, `SIGNUP` and `AUTHENTICATION` clauses may also be costly depending on the logic implemented therein. If you are using any custom authentication logic, verify that it is not the cause for the performance cost by manually executing the query in the SurrealQL interpreter (`surreal sql`) or in SurrealDB Studio and ensuring that the time that it takes to execute is acceptable. ### Requesting authentication support If you are not able to diagnose a specific authentication issue with the information provided on this page, you may consider asking support from the SurrealDB community. If you do so, this section describes some information that you should provide in order to help others help you with your issue. Please, only ask support directly from the developers of SurrealDB if you believe your issue may be related to a bug in SurrealDB. Before sharing any information with other SurrealDB users or the SurrealDB team, please ensure that it does not contain any sensitive data, including secrets which may be used to access sensitive data. We recommend setting up a separate SurrealDB environment with dummy data for the purposes of debugging. In that environment, you can sprinkle [`.expect()`](/docs/reference/query-language/functions/database-functions/value.md#valueexpect) (_(since v3.1.0)_) on intermediate results in a SurrealQL reproduction to see which step first diverges from what you expect, without changing schema `ASSERT` rules or production `THROW` handlers. When requesting support with authentication, we recommend that you provide the following information: - The version of SurrealDB that you are using, which you can obtain with the `surreal version` command. - The command line arguments that you are using to start the SurrealDB server with the `surreal start` command. - The error message that your client receives when attempting to authenticate against SurrealDB. - The logs that are displayed on the server, running with `--log debug`, when authentication fails. - A description of the resources you are using to authenticate. These resources will most likely be included in the output of `INFO FOR ROOT`, `INFO FOR NAMESPACE` and `INFO FOR DATABASE`. If your authentication logic relies on data from specific records through `SIGNIN`, `SIGNUP` or `AUTHENTICATE` clauses, include the contents of those records as well. - A simplified proof of concept that reproduces your issue. Please, do not share the complete code that you are using to authenticate with SurrealDB. Instead, provide a minimal example that only contains the relevant elements and that does not require a specific SDK to reproduce. If possible, provide a simple request that uses the [HTTP REST API](/docs/reference/rest-api/http-protocol.md#signin) (e.g. with `curl` or similar) to reproduce the error that you are encountering. - If you are using a token to authenticate, provide an example of a token that triggers the issue. In the case of a JWT, please include the full encoded contents of the token, not just the values of its claims. --- Source: https://surrealdb.com/docs/manage/enterprise # Enterprise Edition What Enterprise Edition adds over Community: distributed live queries. S3-backed file storage, audit logging, FIPS-validated cryptography, trusted execution, and contractual support. Enterprise Edition is the same server with a licence key applied. The binary differs, so the capabilities below exist only in an Enterprise build and are no-ops on a Community one; nothing about your schema, queries or SDK code changes when you move across. ## What the edition adds | Capability | What it is | Documented | | --- | --- | --- | | **Audit logging** | An identity-bound record of every authenticated action - statements, queries, transactions, RPC calls, sign-ins, sessions and HTTP requests - written to a durable NDJSON sink, with optional SHA-256 hash chaining and PII redaction. | [Audit logging](/docs/manage/observability/audit-logging.md) | | **Distributed live queries** | Live queries that resolve across every node in a cluster rather than the node holding the subscription. | Contact sales | | **File storage** | S3-compatible backends for file and blob objects, addressed through `DEFINE BUCKET`. | [`DEFINE BUCKET`](/docs/reference/query-language/statements/define/bucket.md) | | **FIPS mode** | FIPS 140-2 validated cryptographic modules, and the switch that enforces their use. | Contact sales | | **Trusted execution** | Running the server inside a trusted execution environment, with attestation. | Contact sales | | **Contractual support** | Support tiers with defined response times by severity, and uptime commitments. | Contact sales | ## Observability is where most of it is written down The Enterprise observability surface is documented in full alongside the Community one, with the Enterprise-only parts marked. Start there rather than here: - [Audit logging](/docs/manage/observability/audit-logging.md) - events captured, record shape, rotation, hash chaining, redaction and overflow semantics. - [Configuration](/docs/manage/observability/configuration.md#audit-log-knobs) - every audit-log environment variable, and the compliance checklist for a tamper-evident deployment. - [Enterprise observability](/docs/manage/observability/enterprise-observability.md) - the pipelines that exist only in an Enterprise build. - [Slow-query logging](/docs/manage/observability/slow-query-logging.md) - the sister pipeline for the long tail of slow queries. ## Licensing and getting in touch Licence models, pricing, support tiers and the specifics of FIPS, trusted execution and distributed live queries are handled by the SurrealDB team rather than published here. [Contact us](https://surrealdb.com/contact) to discuss an evaluation, a licence, or an upgrade from an existing Community cluster. > [!NOTE] > Moving an existing deployment to Enterprise does not migrate data. The same storage backend and the same schema are read by the Enterprise binary; the licence key is what changes. --- Source: https://surrealdb.com/docs/manage/instances # Instances What a SurrealDB Cloud instance is. How the Start and Scale plans differ, and where each operational task is documented. An instance is a SurrealDB deployment that SurrealDB runs for you. That covers provisioning, high availability, patching, backups, and version upgrades. You choose a plan, a size, and a region, then connect your application. Every instance belongs to an [organisation](/docs/manage/organisations.md), which also holds members, usage, and billing. You operate instances from [SurrealDB Studio](https://studio.surrealdb.com), or from a terminal with [`surrealctl`](/docs/manage/surrealctl/instances.md). To run and operate the server yourself, see [Self-hosted](/docs/manage/self-hosted.md). [Open SurrealDB Studio](https://studio.surrealdb.com) ## Plans Two plans are available. The plan you choose at deploy time fixes the topology of the instance. | Plan | Topology | How it grows | Storage | Suited to | | --- | --- | --- | --- | --- | | **Start** | A single node | Vertically, by moving to a larger instance type | A dedicated disk per instance | Development, staging, and workloads that fit on one node | | **Scale** | A cluster of three nodes or more | Vertically and horizontally, by using larger nodes or more nodes | Distributed storage shared across the cluster | Production that must survive the loss of a node | Start offers three families of instance type. **Free** is for trying SurrealDB out, **Burstable** is for low-traffic and intermittent workloads, and **General purpose** is for steady production traffic. Scale uses General purpose nodes. The plan also gates some options. Configurable [backup frequency](/docs/manage/instances/backups.md) is available on Scale instances only. Current capacity ceilings and prices are on the [pricing page](https://surrealdb.com/pricing). The plan cards in the deploy flow show what applies to your organisation. For the topology behind each plan, see [Architecture](/docs/manage/instances/architecture.md). ## Where instances live The **Instances** section of SurrealDB Studio lists every instance in the organisation you are viewing. Each card shows the SurrealDB version and the region. Use the search box and the **Version** and **Type** filters to narrow a long list. ![The Instances page of SurrealDB Studio for the Acme Corp organisation, listing three instance cards: api-production on SurrealDB 3.2.4, and api-staging and analytics-eu on SurrealDB 3.2.1. All three are in AWS Europe (Ireland). The page also has a search box, Version and Type filters, and a Deploy new instance button.](~/assets/img/surrealdb/manage/instances-list.webp) Selecting an instance opens its own workspace. That workspace holds a [dashboard](/docs/manage/instances/monitoring.md) of resource use, the schema and query views, and **Settings**. Settings is where [configuration](/docs/manage/instances/configure.md), [capabilities](/docs/manage/instances/configure.md#capabilities), [versions](/docs/manage/instances/versions-and-upgrades.md), compute, and [backups](/docs/manage/instances/backups.md) live. `surrealctl instance list` prints the same list in a terminal, and `--json` makes it scriptable. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Lifecycle You control the lifecycle of an instance with four actions. | Action | Effect | Where | | --- | --- | --- | | **Deploy** | Provisions the instance on the chosen plan, type, and region | [Create an instance](/docs/manage/instances/create.md) | | **Resize** | Changes the instance type or the storage capacity in place | [Scaling](/docs/manage/instances/scaling.md) | | **Pause** | Stops compute and usage billing. Data and configuration are kept, and the instance is unreachable until you resume it | [Configure an instance](/docs/manage/instances/configure.md#pause-an-instance) | | **Delete** | Destroys the instance and everything stored in it, with no way to recover it | [Configure an instance](/docs/manage/instances/configure.md#delete-an-instance) | You cannot change an instance name after deployment. Pick a name a colleague will still recognise in six months, such as `api-production` rather than `db1`. ## Topics - **[Create an instance](/docs/manage/instances/create.md):** plan, instance type, region, version, name, and starting data. - **[Configure an instance](/docs/manage/instances/configure.md):** capabilities, compute, storage, pausing, and deletion. - **[Connect to an instance](/docs/manage/instances/connect.md):** SurrealDB Studio, the CLI, SDKs, and HTTP. - **[Architecture](/docs/manage/instances/architecture.md):** single-node and multi-node topologies. - **[Scaling](/docs/manage/instances/scaling.md):** matching compute and storage to the workload. - **[High availability](/docs/manage/instances/high-availability.md):** what survives a node failure on each plan. - **[Backups and recovery](/docs/manage/instances/backups.md):** automated snapshots, retention, and restore. - **[Import and export](/docs/manage/instances/import-and-export.md):** moving data with `surreal export` and `surreal import`. - **[Versions and upgrades](/docs/manage/instances/versions-and-upgrades.md):** changing the SurrealDB version. - **[Monitoring](/docs/manage/instances/monitoring.md):** the instance dashboard, metrics, and logs. - **[Network access](/docs/manage/instances/network-access.md):** which outbound destinations queries may reach. - **[Private connectivity](/docs/manage/instances/private-connectivity.md):** AWS PrivateLink and instance access modes. ## Related sections - **[Organisations](/docs/manage/organisations.md):** accounts, members, roles, usage, billing, and support. - **[surrealctl](/docs/manage/surrealctl.md):** the management-plane command-line tool. - **[Observability](/docs/manage/observability.md):** the full metric, audit-log, and slow-query reference. - **[Self-hosted](/docs/manage/self-hosted.md):** running SurrealDB on your own infrastructure instead. --- Source: https://surrealdb.com/docs/manage/instances/architecture # Architecture The topologies behind the Start and Scale plans. Single-node and multi-node, and what each means for growth and recovery. This page explains the two topologies an instance can run on, and what each one means for growth and recovery. The plan you choose when you [create an instance](/docs/manage/instances/create.md) selects the topology. Instances build on [the layered architecture of SurrealDB](/docs/learn/data-models/architecture.md), which separates storage from compute. That separation lets an instance grow, replicate, and recover without you sharding or resharding the database yourself. ## Start: single node The **Start** plan provides one compute node backed by a dedicated disk. Compute and storage are sized together and resized together, and the node holds the only copy of the working data. Diagram of the single-node topology: one SurrealDB compute node paired with its own dedicated disk-based storage volume. This topology suits development, staging, and production workloads that fit on one node and can tolerate a short recovery window. It scales vertically: as load grows, you move to a larger instance type. See [Scaling](/docs/manage/instances/scaling.md). Because there is a single node, there is no failover inside the instance. [Backups](/docs/manage/instances/backups.md) are the recovery mechanism. See [High availability](/docs/manage/instances/high-availability.md) for what that means in practice. ## Scale: multi-node cluster The **Scale** plan runs a cluster of three compute nodes or more over distributed storage. Each node runs in a different availability zone within the region, backed by its own disk. Diagram of the multi-node topology: three SurrealDB compute nodes, each in its own availability zone and backed by its own storage disk, split into a compute layer and a storage layer. Data is replicated continuously between nodes, and every node serves reads, writes, and queries. The storage layer handles replication, quorum consensus, and distributed transactions. A query therefore does not need to know which node holds which data. The practical consequences are: - **Node loss is survivable.** A quorum of the remaining nodes keeps serving. Three nodes is the minimum, because three nodes leave a majority when one node is out for a failure, an upgrade, or maintenance. - **Query capacity scales horizontally.** Adding nodes adds throughput. Vertical scaling is bounded by the largest available instance type. - **Self-hosting the same topology is more work.** Running it yourself means operating Kubernetes, storage replication, patching, backups, and upgrade orchestration. Scale covers that work for you. Compare [Self-hosted deployment models](/docs/manage/self-hosted/deployment-models.md). > [!NOTE] > Several Scale capabilities are still on the roadmap: object-storage backing, cross-region replication, automatic sharding, and read replicas. Object-storage backing keeps hot data on disk and cold data in object storage, which in turn enables instant database branching. See [Pricing](https://surrealdb.com/pricing) and [Roadmap](https://surrealdb.com/roadmap) for current availability. ## Related pages - **[Scaling](/docs/manage/instances/scaling.md):** resizing nodes and adding them. - **[High availability](/docs/manage/instances/high-availability.md):** what each topology survives. - **[Self-hosted deployment models](/docs/manage/self-hosted/deployment-models.md):** running the same topologies on your own infrastructure. --- Source: https://surrealdb.com/docs/manage/instances/backups # Backups and recovery Automated snapshots and retention tiers. Restoring a snapshot into a new instance. Instances are backed up for you: snapshots run on a schedule and are kept under a retention policy. There is nothing to install or script, and the same mechanism applies on Start and Scale. A restore creates a new instance from a chosen snapshot. Snapshots live inside the platform. They are not files you can download, copy to your own object storage, or open locally. For a portable copy, see [Import and export](/docs/manage/instances/import-and-export.md). ## Where backups are managed Open the instance in [SurrealDB Studio](https://studio.surrealdb.com), then go to **Instance settings → Backups**. ![The Backups tab of instance settings in SurrealDB Studio, listing one automatic backup taken on 17 August 2026 at 5:58 PM with a Create from selected button, a Backup frequency section set to once a day with a note that configurable frequency is available on Scale instances, and a Retention policy section listing daily, weekly, and monthly snapshot tiers.](~/assets/img/surrealdb/manage/instance-backups.webp) The tab has three parts: - **Available backups:** the list of snapshots you can restore from. - **Backup frequency:** how often a new snapshot is taken. - **Retention policy:** how long each snapshot is kept. ## Frequency and retention Automated snapshots run daily by default. Configurable backup frequency is available on Scale instances. On other instance types the schedule is fixed. Retention is tiered, and each tier is kept for its own period. | Tier | When the snapshot is taken | Retention setting | | --- | --- | --- | | **Daily** | Every day | The number of days of daily snapshots kept | | **Weekly** | Each Sunday | The number of weeks of weekly snapshots kept | | **Monthly** | The first of each month | The number of months of monthly snapshots kept | Which tiers you can change depends on the instance type. Tiers that are fixed on your plan are shown but not editable. Longer retention lets you recover from further back and increases storage cost. Set it against your actual recovery objectives rather than taking the maximum. If you need more retention than the plan allows, [contact support](/docs/manage/organisations/support.md). ## Taking a backup before a risky change Take an on-demand snapshot before anything you might want to undo, such as a major version upgrade, a schema migration, or a bulk delete. On-demand snapshots appear in the same list as scheduled ones and follow the same retention. `surrealctl instance backup` triggers one from a terminal, which is what you want in a deployment pipeline that runs a migration. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Restoring from a snapshot A restore produces a **new instance**. The original is left untouched, and there is no in-place restore. 1. Open **Instance settings → Backups** on the source instance. 2. Select the snapshot you want. 3. Select **Create from selected**. That opens the [deploy flow](/docs/manage/instances/create.md) with the snapshot already chosen as the starting data. You can reach the same result from the other direction: start a new deploy and pick **Restore from backup** as the data source. Because the result is a new instance, cutting over is an application change. The new instance has its own name and endpoint. Verify the data before you repoint traffic, then [delete](/docs/manage/instances/configure.md#delete-an-instance) or [pause](/docs/manage/instances/configure.md#pause-an-instance) the old instance. Three constraints apply to a restore: - The new instance must be in the **same region** as the source. - Its **storage size** must be at least as large as the size of the source. - Its **SurrealDB version** must be compatible with the snapshot. The deploy form only offers versions that are. ## What backups do not cover - **They are not portable.** You cannot download a snapshot. It is not an archival format, not a way to seed a local development database, and not a route to another provider. Use [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) for those cases. See [Import and export](/docs/manage/instances/import-and-export.md). - **They are not cross-region disaster recovery.** A snapshot restores into the region it was taken in. Recovering into a different region needs a logical export and import. - **They are not a substitute for testing a restore.** Restore into a throwaway instance periodically and check that the data is what you expect. That is the only way to know the recovery path works. ## Related pages - **[Import and export](/docs/manage/instances/import-and-export.md):** portable data with `surreal export` and `surreal import`. - **[Versions and upgrades](/docs/manage/instances/versions-and-upgrades.md):** snapshot first, then upgrade. - **[High availability](/docs/manage/instances/high-availability.md):** what redundancy covers and what backups cover. --- Source: https://surrealdb.com/docs/manage/instances/configure # Configure an instance Instance settings in SurrealDB Studio. Capabilities, compute and storage, pausing, and deletion. Change the settings of a running instance from **Instance settings** in [SurrealDB Studio](https://studio.surrealdb.com). To open the settings, select the instance, then select **Instance settings** on the dashboard or **Settings** in the sidebar. Settings is organised into tabs. | Tab | What it controls | | --- | --- | | **General** | The instance id, and the [pause](#pause-an-instance) and [delete](#delete-an-instance) actions | | **Capabilities** | [What the instance permits at runtime](#capabilities): scripting, endpoints, functions, and network access | | **Version** | The SurrealDB release the instance runs. See [Versions and upgrades](/docs/manage/instances/versions-and-upgrades.md) | | **Compute** | [The instance type and the storage capacity](#compute-and-storage) | | **Backups** | Snapshot frequency, retention, and restore. See [Backups and recovery](/docs/manage/instances/backups.md) | The **Databases** and **Data** tabs work on the contents of the instance rather than on the instance itself. ## Capabilities Capabilities decide what the database engine will do at runtime. They are the main security control on an instance: a capability that is switched off cannot be re-enabled by a query, a user, or an access method. The underlying model is described in [Capabilities](/docs/learn/security/authorization/capabilities.md). ![The Capabilities tab of instance settings in SurrealDB Studio, subtitled What this instance permits at runtime, with import and export buttons for the capability configuration as a JSON file, and a granular configuration list showing Scripting and Guest Access switched off, RPC methods, HTTP endpoints, Functions, and Arbitrary queries set to Allowed, Network access set to Denied, and Insecure storable closures switched off.](~/assets/img/surrealdb/manage/instance-capabilities.webp) Two kinds of control appear in the list. **Switches** turn a single capability on or off: - **Scripting:** whether embedded JavaScript functions run. - **Guest access:** whether unauthenticated clients can perform any operation. - **Insecure storable closures:** whether closures can be stored in records. **Allow and deny lists** apply to capabilities with many members: RPC methods, HTTP endpoints, functions, and [network access](/docs/manage/instances/network-access.md). Each list starts from a default and takes exceptions: - **Allowed by default:** everything is permitted, and you list what to deny. - **Denied by default:** nothing is permitted, and you list what to allow. For a production instance, deny by default and allow only what the application uses. The cost is that you must extend the list whenever the application changes. > [!IMPORTANT] > **Network access** governs outbound HTTP requests made from inside queries, and is denied by default. It does not control who can reach the instance. See [Network access](/docs/manage/instances/network-access.md) for the pattern syntax, and [Private connectivity](/docs/manage/instances/private-connectivity.md) for inbound access modes. The **Import** and **Export** buttons read and write the whole capability configuration as a JSON file. Use the export to keep staging and production aligned, or to hold the configuration in version control. `surrealctl instance capabilities` does the same from a terminal. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Compute and storage The **Compute** tab holds the instance type and the storage capacity. The active type is marked, and the Free, Burstable, and General purpose families are on separate tabs. ![The Compute tab of instance settings in SurrealDB Studio, showing the Free instance type marked as active with 0.25 core vCPU and 512 MB memory, and a Storage capacity section explaining that storage expansion is unavailable for free instances with an Upgrade instance type button.](~/assets/img/surrealdb/manage/instance-compute.webp) To resize an instance: 1. Open **Instance settings → Compute**. 2. Select the instance type you want. 3. Set the value under **Storage capacity** if you also need more disk. 4. Select **Save changes**. Changing the instance type moves the instance to a larger or smaller size and rebills accordingly. Storage can grow but never shrink, and you can increase the disk size once every six hours. > [!IMPORTANT] > Storage expansion is unavailable on free instances. Move to a paid instance type to change storage capacity. Expect a brief reconnect while the instance restarts on the new size. For when to resize and what to watch afterwards, see [Scaling](/docs/manage/instances/scaling.md). ## Pause an instance Pausing stops compute without losing data. A paused instance is unreachable until you resume it, its data and configuration are kept, and usage is not billed while it is paused. To pause an instance: 1. Open **Instance settings → General**. 2. Select **Pause instance**. 3. Confirm the action. Pausing suits environments with predictable idle periods, such as a staging instance overnight or a demo between engagements. Before you pause, check that nothing depends on the instance being reachable. Connection pools, scheduled jobs, and health checks fail against a paused instance rather than queue against it. Unused free instances are paused automatically after seven days. ![The General tab of instance settings in SurrealDB Studio, showing the instance id with a copy button, a Pause instance panel explaining that a paused instance is entirely unreachable until resumed while its data is kept and usage is not billed, and a Delete instance panel warning that permanently removing an instance cannot be undone.](~/assets/img/surrealdb/manage/instance-settings.webp) Resuming brings the instance back on the same configuration and the same endpoint. ## Delete an instance Deleting destroys the instance and everything stored in it. You cannot undo it, and support cannot recover a deleted instance. > [!WARNING] > Do not treat the snapshots of an instance as the safety net for deleting it. If you might need the data later, [restore a backup into a new instance](/docs/manage/instances/backups.md) or take a [logical export](/docs/manage/instances/import-and-export.md) first. Confirm the export loads cleanly before you delete. To delete an instance: 1. Open **Instance settings → General**. 2. Select **Delete instance**. 3. Confirm the action. `surrealctl instance pause`, `surrealctl instance resume`, and `surrealctl instance delete` are the command-line equivalents. ## The instance id The **General** tab also shows the instance id. SurrealDB support may ask for it when investigating an issue. The id identifies the instance unambiguously where the name does not, because names are only unique within an organisation. ## Next steps - **[Connect to an instance](/docs/manage/instances/connect.md):** SurrealDB Studio, the CLI, SDKs, and HTTP. - **[Monitoring](/docs/manage/instances/monitoring.md):** confirm a configuration change had the effect you expected. - **[Backups and recovery](/docs/manage/instances/backups.md):** snapshot frequency, retention, and restore. --- Source: https://surrealdb.com/docs/manage/instances/connect # Connect to an instance The four routes to a SurrealDB Cloud instance. SurrealDB Studio, the CLI, a client SDK, and the HTTP API, plus where to find the connection details. An instance exposes one endpoint, and every client reaches it the same way. What differs is the tool you use and how you authenticate. | Route | Use it for | | --- | --- | | [SurrealDB Studio](/docs/manage/instances/connect/via-studio.md) | Running queries and browsing data in a graphical interface. | | [CLI](/docs/manage/instances/connect/via-cli.md) | An interactive SurrealQL shell, and one-off queries from a terminal or a script. | | [SDK](/docs/manage/instances/connect/via-sdk.md) | Application code in Rust, JavaScript, Python, .NET, PHP, and the other supported languages. | | [HTTP](/docs/manage/instances/connect/via-http.md) | cURL, Postman, and any HTTP client, including bulk import. | ## Before you begin You need: - An account. See [Accounts and sign-in](/docs/manage/organisations/sign-in.md). - A running instance. See [Create an instance](/docs/manage/instances/create.md). - A **namespace** and a **database** to work in. SurrealDB needs both to know where a query runs. See [system structure](/docs/learn/data-models/architecture.md#system-structure). - Credentials, unless you connect from SurrealDB Studio, which authenticates with your own session. ## Finding the connection details The **Connect** menu of the instance in [SurrealDB Studio](https://studio.surrealdb.com) holds the endpoint. It also produces a ready-made command or snippet for each route, with the details of your instance already filled in. Copying from there avoids transcription mistakes in the hostname. `surrealctl instance endpoint` prints the same endpoint, and `surrealctl instance token` issues a token for it. Both are useful in scripts and in CI, where opening a browser is not an option. See [surrealctl instances](/docs/manage/surrealctl/instances.md). --- Source: https://surrealdb.com/docs/manage/instances/connect/via-cli # Via CLI Open an interactive SurrealQL shell against an instance with surreal sql, and where to get the token. Open an interactive SurrealQL shell against an instance with the [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md). Use the shell for ad-hoc queries, for checking a schema definition, and for scripting a one-off change. ## Prerequisites Install the CLI. See the [installation guide](/docs/running/installation.md). It is a single executable and does not need a local server. ## Open a shell [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md) connects to the endpoint and opens a prompt: ```bash title="Connect to an instance" surreal sql \ --endpoint wss:// \ --ns main --db main \ --token ``` The **Connect** menu of the instance in [SurrealDB Studio](https://studio.surrealdb.com) generates this command with your endpoint and token filled in. ![The Connect menu in SurrealDB Studio showing the ready-made surreal sql command for an instance, with the endpoint, namespace, database, and token already populated.](~/assets/img/image/cloud/open-in-cli.png) The `--token` value is a JSON Web Token that authenticates the session. Treat it as a credential, because it grants whatever the authenticating user or access method grants. Keep it out of shell history and out of committed scripts. Where a script needs a token, mint one at runtime with `surrealctl instance token` rather than store it. You can authenticate with `--user` and `--pass` instead if the instance has a system user defined. See [DEFINE USER](/docs/reference/query-language/statements/define/user.md). ## Next steps - **[SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md):** every command and flag. - **[surrealctl](/docs/manage/surrealctl/instances.md):** the management-plane CLI. `surrealctl instance sql` opens the same shell against a named instance without you supplying an endpoint, and calls `surreal` to do it. --- Source: https://surrealdb.com/docs/manage/instances/connect/via-http # Via HTTP Query an instance with cURL or any HTTP client, and the request size limits that apply per endpoint. Instances serve the [HTTP API](/docs/reference/rest-api.md), so anything that speaks HTTP can query them. That includes cURL, Postman, a serverless function, and a language without an SDK. To get the URL, open the instance in [SurrealDB Studio](https://studio.surrealdb.com), select **Connect**, then select **HTTP cURL**. ![The Connect menu in SurrealDB Studio with HTTP cURL selected, showing a generated cURL command containing the instance endpoint and its namespace and database headers.](~/assets/img/image/cloud/open-in-http.png) > [!NOTE] > The generated command is cURL, but the URL and headers work in any HTTP client. Paste them into Postman or your own code unchanged. ## Send a query Post SurrealQL to `/sql`. Name the namespace and database in headers, and authenticate with a bearer token: ```bash title="Run a query over HTTP" curl -X POST "https:///sql" \ -H "Surreal-NS: main" \ -H "Surreal-DB: main" \ -H "Authorization: Bearer " \ -H "Accept: application/json" \ -d "SELECT * FROM person LIMIT 10;" ``` The response is a JSON array with one result object per statement in the request. ## Request size limits Instances run the SurrealDB defaults for request size, and those defaults differ per endpoint. | Endpoint | Limit | | --- | --- | | `/sql` | 1 MiB | | `/rpc` | 4 MiB | | `/import` | 4 GiB | | WebSocket message | 128 MiB | A request over the cap is rejected with `413 Payload Too Large`. The 1 MiB cap on `/sql` is the one large queries tend to reach first. If a payload does not fit, you have three options: - **Use an SDK.** Most, including [Rust](/docs/reference/rust.md) and [JavaScript](/docs/reference/javascript.md), default to the WebSocket engine and its 128 MiB per message. - **Stay on HTTP but change endpoint.** Send the query through [`POST /rpc`](/docs/reference/rest-api/rpc-protocol.md), which accepts 4 MiB. - **Load bulk data through import.** [`POST /import`](/docs/reference/rest-api/http-protocol.md#import) accepts up to 4 GiB per request. See [Import and export](/docs/manage/instances/import-and-export.md#size-limits). The full table is in [request size limits](/docs/reference/rest-api/http-protocol.md#request-size-limits). A self-hosted server sets these caps with environment variables. Instances run the defaults. --- Source: https://surrealdb.com/docs/manage/instances/connect/via-sdk # Via SDK Connect to an instance from application code, including the credentials each SDK needs and how to create them. Reach an instance from application code through one of the [SurrealDB SDKs](/docs/languages.md). This page covers what an SDK needs to connect and how to create it. The SDK reference for your language covers the API itself. To get a snippet with the endpoint already filled in, select **Connect** on the instance in [SurrealDB Studio](https://studio.surrealdb.com), then select your language. ![The Connect menu in SurrealDB Studio with the SDK option selected, showing a generated connection snippet containing the instance endpoint.](~/assets/img/image/cloud/open-in-sdk.png) ## What an SDK needs An SDK needs three things: - **The endpoint** of the instance, from the **Connect** menu. - **A namespace and a database**, which tell SurrealDB where the query runs. A prompt to create them appears at the top of the dashboard if the instance has none. See [system structure](/docs/learn/data-models/architecture.md#system-structure). - **Credentials**, unless the connection is anonymous. Studio authenticates with your own session, but application code needs a user or an access method defined on the instance. ## Create credentials > [!NOTE] > This step applies when the SDK calls `signin` on connection. Skip it if the application authenticates some other way. See [access methods](/docs/reference/query-language/statements/define/access.md) and [system users](/docs/reference/query-language/statements/define/user.md). 1. Open the **Authentication** panel of the instance in SurrealDB Studio. 2. Select **+** in the **Root Authentication** section. 3. Choose the kind of credential you need. 4. Set the token duration and the session duration. Two kinds of credential are offered: - **New system user:** a username, a password, and a role that determines what the user may do. This is what a backend service typically uses. - **New access method:** a named method whose type determines how clients authenticate through it. This is what record-level and end-user authentication uses. Keep both durations short enough that a leaked token expires on its own. ![The Root Authentication dialog in SurrealDB Studio, offering a new system user with a username, password, and role, or a new access method with a name and type, each with configurable token and session durations.](~/assets/img/image/cloud/create-root-user.png) Root credentials reach everything on the instance. For namespace-scoped or database-scoped authentication, and for record-level access, create the namespace and database first. Then define the user or the access method at that level. ## Connect Every SDK follows the same shape. `connect` takes the endpoint, then you select the namespace and database, then you sign in. The examples below use root credentials. > [!NOTE] > With a non-root user, the sign-in call also needs the `access` details for the access method you defined. The [SDK reference](/docs/languages.md) for your language shows the exact call. **Rust** ```rust use serde::{Deserialize, Serialize}; use surrealdb::engine::any; use surrealdb::opt::auth::Root; use tokio; use chrono::{DateTime, Utc}; #[derive(Serialize, Deserialize)] struct Project { name: String, description: String, status: String, priority: String, tags: Vec, created_at: DateTime, } // Open a connection let db = any::connect("wss://").await?; // Select namespace and database db.use_ns("DEMO namespace").use_db("DEMO database").await?; // Authenticate db.signin(Root { username: "", password: "", }).await?; // Create a record let project = Project { name: "SurrealDB Dashboard".to_string(), description: "Admin interface for SurrealDB".to_string(), status: "in_progress".to_string(), priority: "high".to_string(), tags: vec!["typescript".to_string(), "react".to_string(), "database".to_string()], created_at: Utc::now(), }; db.create("project").content(project).await?; ``` **JavaScript** ```js import { Surreal, Table } from "surrealdb"; const db = new Surreal(); // Open a connection and authenticate await db.connect("wss://", { namespace: "DEMO namespace", database: "DEMO database", authentication: { username: "", password: "", } }); // Create record await db.create(new Table("project"), { name: "SurrealDB Dashboard", description: "Admin interface for SurrealDB", status: "in_progress", priority: "high", tags: ["typescript", "react", "database"], created_at: new Date(), }); // Select all records in project table console.log(await db.select(new Table("project"))); await db.close(); ``` **Python** ```py from surrealdb import Surreal from datetime import datetime, timezone from surrealdb import RecordID # Open a connection with Surreal("wss://") as db: # Select namespace and database db.use("DEMO namespace", "DEMO database") # Authenticate db.signin({ "username": "", "password": "", }) # Create a record db.create(RecordID("project", "1"), { "name": "SurrealDB Dashboard", "description": "Admin interface for SurrealDB", "status": "in_progress", "priority": "high", "tags": ["typescript", "react", "database"], "created_at": datetime.now(timezone.utc), }) # Select a specific record print(db.select(RecordID("project", "1"))) ``` **.NET** ```csharp using SurrealDb.Net; using SurrealDb.Net.Models.Auth; const string TABLE = "project"; using var db = new SurrealDbClient("wss:///rpc"); // Select namespace and database await db.Use("DEMO namespace", "DEMO database"); // Authenticate await db.SignIn(new RootAuth { Username = "", Password = "" }); // Create record var project = new Project { Name = "SurrealDB Dashboard", Description = "Admin interface for SurrealDB", Status = "in_progress", Priority = "high", Tags = new[] { "typescript", "react", "database" }, CreatedAt = DateTime.UtcNow, }; await db.Create(TABLE, project); ``` **PHP** ```php $db = new \Surreal\Surreal(); // Open a connection $db->connect("wss:///rpc", [ "namespace" => "DEMO namespace", "database" => "DEMO database", ]); // Authenticate $db->signin([ "username" => "", "password" => "", ]); // Create a record $db->create("project", [ "name" => "SurrealDB Dashboard", "description" => "Admin interface for SurrealDB", "status" => "in_progress", "priority" => "high", "tags" => ["typescript", "react", "database"], "created_at" => new DateTime(), ]); ``` ## Next steps - **[SDK reference](/docs/languages.md):** the full API for each language, including live queries and transactions. - **[Connect via HTTP](/docs/manage/instances/connect/via-http.md):** for languages without an SDK, and the request size limits that apply. - **[Authentication](/docs/learn/security/authentication/summary.md):** choosing between system users, access methods, and record-level access. --- Source: https://surrealdb.com/docs/manage/instances/connect/via-studio # Via SurrealDB Studio Select a namespace and database, then query an instance from the SurrealDB Studio query view. Query an instance from [SurrealDB Studio](https://studio.surrealdb.com) without setting up a client. Studio is already connected to your instances, so signing in is the whole authentication step. Use it to explore data, write a query, and check the effect of a schema change. ## Select a namespace and database Queries run against a specific database inside a specific namespace, so select both before you run anything. 1. Open the instance and go to the query view. 2. Select the **Namespace** control and choose a namespace. If the instance has none, the same control offers to create one. 3. Select the **Database** control and choose a database, or create one. ![The namespace and database selectors in the SurrealDB Studio query view, each showing a dropdown of the namespaces and databases available on the instance.](~/assets/img/image/cloud/namespace-database.png) An instance created with the **Empty** starting-data option has neither, so this is the first thing to do after you deploy one. ## Run a query With both selected, the editor runs SurrealQL against that database. Results appear beneath the query, and records created by a statement show up immediately in the explorer. ![The SurrealDB Studio query view with a SurrealQL statement in the editor and its result set displayed in the panel below.](~/assets/img/image/cloud/querying-instance.png) ## Next steps - **[SurrealDB Studio](/docs/explore/studio.md):** the rest of the interface, including the schema designer, the explorer, and keyboard shortcuts. - **[SurrealQL](/docs/reference/query-language.md):** the query language reference. - **[Connect via SDK](/docs/manage/instances/connect/via-sdk.md):** moving from exploration to application code. --- Source: https://surrealdb.com/docs/manage/instances/create # Create an instance Deploy an instance from SurrealDB Studio. Choose a plan, instance type, region, version, name, starting data, and storage. Deploy an instance from [SurrealDB Studio](https://studio.surrealdb.com) in seven steps. Deployment takes a few minutes. You can change everything afterwards except the instance name and the region. ## Before you begin You need: - An account. See [Accounts and sign-in](/docs/manage/organisations/sign-in.md). - An organisation to hold the instance. See [Organisations](/docs/manage/organisations.md). ## 1. Select a plan 1. Open the organisation that will hold the instance. 2. Go to **Instances**. 3. Select **Deploy new instance**. The first step of the flow asks for a plan. | Plan | Topology | How it scales | | --- | --- | --- | | **Start** | A single node | Vertically only | | **Scale** | A fault-tolerant cluster of three nodes or more | Vertically and horizontally | Each card lists the capacity ceilings and the starting price for that plan. ![Step one of the deploy flow in SurrealDB Studio, titled Select a plan, showing two cards side by side: Start, designed for applications that require vertical scalability, with a single node, up to 512 GB storage, 16 vCPU, and 64 GB memory; and Scale, designed for applications that require fault tolerance and horizontal scalability, with multiple nodes, up to 1 PB cluster storage, 64 vCPU per node, and 256 GB memory per node.](~/assets/img/surrealdb/manage/instance-select-plan.webp) The plan fixes the topology of the instance. To move a workload from Start to Scale, deploy a Scale instance, then [restore a backup](/docs/manage/instances/backups.md) or [import an export](/docs/manage/instances/import-and-export.md) into it. Choose **Scale** if the workload must survive the loss of a node. See [High availability](/docs/manage/instances/high-availability.md). Select **Configure instance** on the plan you want. ## 2. Choose an instance type Instance types set the vCPU, the memory, and the baseline storage for each node. Three families are available. | Family | Behaviour | Suited to | | --- | --- | --- | | **Free** | A fixed, no-cost instance with a small storage allowance | Trying SurrealDB out | | **Burstable** | Full CPU in short bursts, throttled under sustained load | Testing, starter projects, and low-traffic applications | | **General purpose** | Sustained CPU and memory with no throttling | Production traffic and workloads at scale | Select **View more configurations** to expand the shortlist to every size available on the plan. ![Step two of the deploy flow in SurrealDB Studio, titled Configure your single-node instance, showing three instance type cards: Burstable small with 0.5 core and 1 GB memory, General purpose medium with 1 core and 4 GB memory, and General purpose xlarge with 4 cores and 16 GB memory. Below the cards is an Instance details form with the name api-production, the region AWS Europe (Ireland), and the version SurrealDB 3.2.4.](~/assets/img/surrealdb/manage/instance-configure.webp) ## 3. Enter the instance details Complete the three fields in the **Instance details** form. **Name** identifies the instance in the organisation and in every connection string. You cannot change it later, so use a name that stays meaningful: `api-production`, `api-staging`, or `analytics-eu`. **Region** determines the latency to your clients and where the data resides, and it sets which cloud provider runs the instance. You cannot change it later either. Deploy into the same region as your application stack where you can. Instances are currently available in: | Provider | Region | | --- | --- | | AWS | US East (N. Virginia) | | AWS | US West (Oregon) | | AWS | Europe (Ireland) | | AWS | AP South (Mumbai) | | Azure | Europe (Germany) | | Azure | US East 2 (Virginia) | | Azure | South America (Brazil) | The region selector shows what is available to your organisation, and that list is the authoritative one. [AWS PrivateLink](/docs/manage/instances/private-connectivity.md) is offered in a subset of AWS regions. **Version** sets the SurrealDB release the instance runs. Take the latest stable release unless you have a reason not to. You can upgrade an older release later from **Settings → Version**. See [Versions and upgrades](/docs/manage/instances/versions-and-upgrades.md). ## 4. Choose the starting data A new instance can start empty or with data already in place. | Option | Result | | --- | --- | | **Empty** | An instance with no namespaces or databases | | **Demo dataset** | A sample dataset and example queries to explore | | **Upload from file** | Data loaded from a file you supply | | **Restore from backup** | A copy of an existing instance, taken from one of its [backup snapshots](/docs/manage/instances/backups.md) | **Restore from backup** is how a restore works. The snapshot becomes a new instance and never overwrites the original. The backup must come from the same region, and the storage size must be at least as large as the source. ## 5. Set the storage capacity Storage is provisioned per instance, and the available range depends on the plan and the instance type. You can increase storage later but never decrease it. Start close to what you need rather than over-provisioning. See [Scaling](/docs/manage/instances/scaling.md). ## 6. Review and deploy 1. Select **Continue to checkout** to show the order for review. 2. Enter payment details if the organisation does not have them yet. See [Billing](/docs/manage/organisations/billing.md). 3. Confirm the order. The instance is provisioned within a few minutes. ## 7. Verify the instance 1. Go to **Instances** in the organisation. 2. Confirm the new instance appears in the list. 3. Open the instance and check that the [dashboard](/docs/manage/instances/monitoring.md) reports the version, region, and instance type you chose. 4. [Connect to the instance](/docs/manage/instances/connect.md) and run a query. If the instance started empty, it has no namespace or database yet. Create both before you run a query. See [Connect via SurrealDB Studio](/docs/manage/instances/connect/via-studio.md). ## Deploying from the command line `surrealctl instance create` performs the same deploy, which is what you want in CI or in a provisioning script. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Next steps - **[Connect to an instance](/docs/manage/instances/connect.md):** SurrealDB Studio, the CLI, SDKs, and HTTP. - **[Configure an instance](/docs/manage/instances/configure.md):** capabilities, compute, and storage. - **[Monitoring](/docs/manage/instances/monitoring.md):** confirm the instance is healthy. --- Source: https://surrealdb.com/docs/manage/instances/high-availability # High availability What a Start instance and a Scale cluster each survive. How to plan for the failures they do not. This page explains what each instance topology survives, and how to plan for the failures it does not. High availability means the database stays reachable and consistent when a component fails or during planned maintenance. How much of that you get depends on the topology, which the plan sets when you [create the instance](/docs/manage/instances/create.md). ## Start: single node A **Start** instance runs on one compute node with a dedicated disk. That is simple and inexpensive, and it means there is no failover inside the instance. If the node, the disk, or the host underneath fails, connections drop until the instance recovers. What you get instead is managed recovery. Patching, snapshots, and host replacement are handled for you, and [backups](/docs/manage/instances/backups.md) are the mechanism for recovering from data loss or a corrupted state. Plan around that: - **Treat backups and a rehearsed restore as the recovery path**, not in-instance redundancy. A backup you have never restored is an assumption, not a plan. - **Give clients retry and reconnect logic.** Resizes, version upgrades, and host recovery all produce a short window where connections fail. - **Accept a recovery window** in the service level you promise upstream, or move to Scale. ## Scale: multi-node cluster A **Scale** cluster runs at least three SurrealDB nodes over distributed storage, with each node in a different availability zone. Availability is a property of the architecture rather than something added on top: - **Node loss is survivable.** The storage layer coordinates replication and quorum consensus, so the cluster keeps serving while a node is unavailable. Three separate single-node instances would not do this, because they would be three independent databases. - **Maintenance is not an outage.** Version upgrades roll through nodes one at a time. Query capacity dips while each node restarts, but the cluster stays reachable. See [Versions and upgrades](/docs/manage/instances/versions-and-upgrades.md). - **Three nodes are the minimum for a reason.** Three nodes leave a majority when one is out for a failure, an upgrade, or maintenance. Two nodes would not. Scale still needs backups. Quorum protects against infrastructure failure, not against a mistaken `DELETE` or a bad migration. Those replicate to every node exactly as intended. ## Comparing the two | Requirement | Start | Scale | | --- | --- | --- | | Tolerate a recovery window during host failure | Yes | Yes | | Survive the loss of a node without manual failover | No | Yes | | Stay reachable through a version upgrade | No. Expect a reconnect | Yes, through a rolling upgrade | | Scale query throughput horizontally | No. Vertical resize only | Yes, by adding nodes | | Recover from an application-level mistake | Restore a backup | Restore a backup | ## Beyond a single region Both topologies run within one region. Node-level redundancy does not cover an outage that affects a whole region. If your recovery objectives extend that far, plan for it at the application layer and rehearse a restore into a second region. A [backup restores into the region it came from](/docs/manage/instances/backups.md), so cross-region recovery uses a [logical export](/docs/manage/instances/import-and-export.md). Cross-region replication is on the roadmap. See [Architecture](/docs/manage/instances/architecture.md). ## Related pages - **[Architecture](/docs/manage/instances/architecture.md):** the topology behind each plan. - **[Scaling](/docs/manage/instances/scaling.md):** moving between sizes and node counts. - **[Backups and recovery](/docs/manage/instances/backups.md):** snapshots, retention, and restore. --- Source: https://surrealdb.com/docs/manage/instances/import-and-export # Import and export Move data in and out with surreal export and surreal import. Covers size limits and partial-import behaviour. Export and import move data as SurrealQL text, so you get a portable copy of a database. Use them for four cases: - Migrating from a self-hosted server. - Seeding a local development database. - Moving data between instances or regions. - Keeping an archive outside the platform. [Backups](/docs/manage/instances/backups.md) cover the recover-this-instance case instead, and you cannot download them. Both commands come from the [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md), which you install once and point at any instance. The connection URL, namespace, and database are on the **Connect** menu of the instance in [SurrealDB Studio](https://studio.surrealdb.com). ## Exporting [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) writes the contents of one database to a `.surql` file: ```bash title="Export a database to a file" surreal export \ --conn wss:// \ --user root --pass \ --ns main --db main \ ./api-production.surql ``` The file contains the SurrealQL statements needed to rebuild the schema and the records. It is therefore readable, diffable, and safe to store in an artefact repository. An export covers a single namespace and database. Export each one you need separately. ## Importing [`surreal import`](/docs/reference/cli/surrealdb-cli/commands/import.md) replays that file into a target database: ```bash title="Import a file into an instance" surreal import \ --conn wss:// \ --user root --pass \ --ns main --db main \ ./api-production.surql ``` The target namespace and database do not have to match the source. That is how you promote a staging dataset into a differently named database. ## Size limits An import is accepted up to **4 GiB per request**. The limit applies to the whole request rather than to each statement, so you must split a larger file and import the parts in sequence. The other endpoints have lower caps: 1 MiB on `/sql` and 4 MiB on `/rpc`. That is why bulk loading goes through import rather than through a large query. See [request size limits](/docs/reference/rest-api/http-protocol.md#request-size-limits) for the full table. A self-hosted server sets these caps with environment variables. Instances run the defaults. ## Partial imports An import is applied statement by statement as the file is parsed, and each statement commits as it goes. No single transaction wraps the file, so **a failure partway through leaves the statements that already ran in place**. That matters for retries. Rerunning the same file against a half-populated database can produce duplicate records or conflicting definitions. Import into a fresh namespace or database, verify the result, then switch the application over. If something fails, drop the target and start again from a known-empty state. ## Related pages - **[Backups and recovery](/docs/manage/instances/backups.md):** snapshots and in-platform restore. - **[Migrating to SurrealDB](/docs/build/migrating.md):** moving from another database. - **[surrealctl instances](/docs/manage/surrealctl/instances.md):** `surrealctl instance import` and `surrealctl instance export` wrap the same operations against an instance. --- Source: https://surrealdb.com/docs/manage/instances/monitoring # Monitoring The instance dashboard, metrics groups, and log filters. How each relates to the full observability surface in SurrealDB Studio. Every instance reports its own resource use and activity, with no agent to install and no scrape target to configure. The at-a-glance view is the instance **Dashboard**. The detail is under **Metrics & logs** in the instance sidebar of [SurrealDB Studio](https://studio.surrealdb.com). ## The instance dashboard Opening an instance lands on its dashboard, which states what the instance is and what it is currently doing. The header reports the version, region, instance type, and organisation. ![The dashboard for the api-production instance in SurrealDB Studio, showing it running SurrealDB 3.2.4 in aws-euw1 on the Free type in the Acme Corp organisation, with cards for 512 MB memory, 0.25 vCPUs of single-node compute, 21.64 MB of 1 GB storage used, and version v3.2.4, above Activity charts plotting memory usage, compute usage, and network ingress and egress over the last hour.](~/assets/img/surrealdb/manage/instance-dashboard.webp) The four cards across the top are the numbers to check first. **Storage** is the one to watch over time. It shows consumption against the provisioned limit, and a disk that reaches its limit stops accepting writes. See [Scaling](/docs/manage/instances/scaling.md) for raising it. Below the cards, **Activity** plots resource use over a period you select. The default is one hour. Longer windows show trends rather than spikes. ## Metrics **Metrics & logs** groups the signals into three. | Group | What it reports | | --- | --- | | **System** | Compute and memory use of the instance | | **Connections** | HTTP and RPC request volume | | **Network traffic** | Network ingress and egress | Each group takes a time range: the last hour, twelve hours, day, week, or month. Short ranges show what is happening now. Longer ranges are what you need before you decide to resize, because a resize should answer a trend rather than a single spike. Read the groups together rather than separately. Rising connection counts with flat compute point at connection handling in the application. Rising compute with flat connections points at the queries themselves. ## Logs The **Logs** view lists instance activity: startups and shutdowns, imports, version changes, and other lifecycle events. Filters narrow the list by: - **Level:** the severity of the entry. - **Source:** the component that emitted it. - **Message:** free text within the entry. Log entries stay queryable for a retention period. To keep them longer, ship them to your own log store rather than rely on the console view. That is what compliance requirements and correlation with other systems usually need. > [!NOTE] > These are activity logs for instance-level events. Audit logs, which record who ran which statement against which resource, are not yet available for instances. The record shape and the pipeline are documented under [audit logging](/docs/manage/observability/audit-logging.md) for deployments that have them. ## What to watch These signals most often warrant action: - **Storage** climbing towards the provisioned limit. - **Compute or memory** sustained near the instance-type ceiling under ordinary traffic. - **Connection counts** saturating, or clients being refused. - **Error-rate spikes** in the logs that correlate with a deployment or a schema change. Sustained pressure on the first three is a [scaling](/docs/manage/instances/scaling.md) signal. A spike that coincides with a change you made is usually a query or schema problem, and a resize hides it rather than fixes it. ## From the command line `surrealctl instance metrics` and `surrealctl instance logs` print the same data for scripting or for feeding an external monitor. `surrealctl instance watch` follows the state of an instance during a change. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## The wider observability surface This view presents the same telemetry a self-hosted deployment scrapes directly. When you need the full vocabulary, see the [Observability](/docs/manage/observability.md) section, and the [metrics reference](/docs/manage/observability/metrics.md) in particular. Those pages cover every metric family and label, the environment variables that control them, the audit-log record shape, and the slow-query pipeline. --- Source: https://surrealdb.com/docs/manage/instances/network-access # Network access Control which outbound destinations queries may reach. Uses the network access capability and its allow and deny patterns. Network access controls which external destinations an instance may reach **from inside a query**. The functions it governs are the [HTTP functions](/docs/reference/query-language/functions/database-functions/http.md) that call an API, fetch a document, or post to a webhook as part of a statement. > [!IMPORTANT] > This is outbound access. It has nothing to do with who can reach the instance. For inbound connectivity, see [Private connectivity](/docs/manage/instances/private-connectivity.md) and its `public`, `private`, and `dual` access modes. The capability is **denied by default**. Until you allow something, `http::get` and its siblings fail rather than reach the network. That default matters. An instance with unrestricted outbound access lets a query author send data anywhere. Keep the allowed set to the shortest list that makes the application work. > [!NOTE] > Configurable network access requires SurrealDB `>=2.1.8, <2.2.0`, `>=2.2.6, <2.3.0`, or `>=2.3.6`. On an earlier release, [upgrade the instance](/docs/manage/instances/versions-and-upgrades.md) before you configure the rules. ## Configuring the rules Network access is one of the [instance capabilities](/docs/manage/instances/configure.md#capabilities), so it lives with the rest of them. 1. Open the instance in [SurrealDB Studio](https://studio.surrealdb.com). 2. Go to **Instance settings → Capabilities**. 3. Find **Network access** in the granular configuration list. 4. Choose its default: **Allowed by default** or **Denied by default**. 5. Add the exceptions. Under a denied default, list the domains, IP addresses, and ranges to permit. Under an allowed default, list what to block. 6. Save the change to apply it to the instance. ![The Capabilities tab of instance settings in SurrealDB Studio, with Network access set to Denied in the granular configuration list, alongside the Scripting, Guest Access, RPC methods, HTTP endpoints, Functions, and Arbitrary queries controls.](~/assets/img/surrealdb/manage/instance-capabilities.webp) At the same specificity, a deny rule prevails over an allow rule. A host you have denied therefore stays unreachable even when a broader allow rule would otherwise cover it. That is how you permit the public endpoints of an API while keeping an internal hostname on the same domain out of reach. ## Choosing the patterns Prefer named destinations over ranges. `api.stripe.com` documents its own intent. A wide CIDR block does not, and it grants reachability to whatever moves into that block later. Review the list when the application changes. Rules outlive the feature that needed them, and an allowance nobody remembers adding is the one worth removing. You can export both the default and the exception lists as JSON from the same tab, which is how you keep staging and production consistent. `surrealctl instance capabilities` reads and writes the same configuration from a terminal. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Additional resources - **[Capabilities](/docs/learn/security/authorization/capabilities.md):** the full capability model and every flag it covers. - **[Configure an instance](/docs/manage/instances/configure.md#capabilities):** the other capabilities on the same tab. - **[HTTP functions](/docs/reference/query-language/functions/database-functions/http.md):** the functions this capability governs. --- Source: https://surrealdb.com/docs/manage/instances/private-connectivity # Private connectivity Reach an instance from your AWS VPC over AWS PrivateLink. Set the public, private, or dual access mode. [AWS PrivateLink](https://aws.amazon.com/privatelink/) gives your AWS VPC a private network path to an instance that never crosses the public internet. By default an instance is reachable over the public internet, protected by the authentication and authorisation defined inside it. This page covers inbound connectivity, which is how clients reach the instance. For outbound requests made from queries, see [Network access](/docs/manage/instances/network-access.md). > [!NOTE] > PrivateLink is an enterprise feature. Onboarding is a manual process handled by the SurrealDB team and needs coordinated setup on both sides, so you cannot enable it from SurrealDB Studio. [Contact us](/contact) to start. ## Prerequisites You need: - An AWS-hosted instance in a supported region. PrivateLink is an AWS service, so it does not apply to instances deployed on Azure. - An AWS account with a VPC. - AWS IAM credentials, as a user, role, or policy, permitted to create interface VPC endpoints. ## Supported regions PrivateLink is currently available in **AWS US West (Oregon)** only. More regions will be added over time. ## How it works After your organisation is onboarded, you create an [interface VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html) in your AWS account. The endpoint receives a private IP address inside your VPC. Traffic between your application and the instance then stays on the AWS private network, with no internet gateway, no NAT gateway, and no public hop. Public and private paths use separate DNS namespaces and separate infrastructure. | Path | Hostname | Reachable from | | --- | --- | --- | | Public | `*.aws-usw2.surreal.cloud` | The internet | | PrivateLink | `*.privatelink.aws-usw2.surreal.cloud` | Inside your VPC only | The two paths are isolated at the network level rather than filtered, so a misconfiguration cannot expose a private instance publicly. The PrivateLink hostname does not resolve outside the VPC. ## Access modes Each instance in a PrivateLink-enabled region has an access mode. | Mode | Behaviour | | --- | --- | | `public` | Reachable over the public internet only. The default for every instance. | | `private` | Reachable over PrivateLink only. No public hostname is assigned. | | `dual` | Reachable over both paths. | Use `dual` while you migrate: bring the private path up, move clients across, confirm that nothing still connects publicly, then switch to `private`. > [!NOTE] > Existing instances are `public`. The `private` and `dual` modes only appear after your organisation has been onboarded for a region. Choose the access mode when you [create an instance](/docs/manage/instances/create.md), after you select a supported region. To change it afterwards, open the instance in SurrealDB Studio and use the network settings in **Instance settings**. ## Onboarding Enabling PrivateLink involves both your team and SurrealDB. 1. **Request access** for your organisation and region through [support](/docs/manage/organisations/support.md). 2. **SurrealDB provisions the service side** and shares the details you need. 3. **You create the interface VPC endpoint** in your AWS account with those details. 4. **SurrealDB completes the setup** and enables PrivateLink for your organisation and region. After that, you manage access modes per instance. ## Limitations - AWS US West (Oregon) only. - The SurrealDB team must enable PrivateLink for your organisation before any instance can use it. - PrivateLink hostnames resolve only inside your VPC, so you cannot use them to connect from outside AWS. Keep `dual` mode, or a separate public instance, if you also need access from developer machines or from CI outside the VPC. ## Related pages - **[Connect to an instance](/docs/manage/instances/connect.md):** the connection routes themselves. - **[Network access](/docs/manage/instances/network-access.md):** outbound requests from queries. - **[Configure an instance](/docs/manage/instances/configure.md):** the rest of instance settings. --- Source: https://surrealdb.com/docs/manage/instances/scaling # Scaling When to resize an instance, and what scales on each plan. How to confirm a resize solved the bottleneck. This page explains when to change the resources behind an instance, and what to expect from each kind of change. What you can change depends on the plan. **Start** instances scale vertically on one node. **Scale** clusters scale vertically and horizontally. The steps for making a change are in [Configure an instance](/docs/manage/instances/configure.md#compute-and-storage). ## What scales on each plan | | **Start** | **Scale** | | --- | --- | --- | | Compute up | A larger instance type on one node | A larger node size, or more nodes | | Compute down | A smaller instance type | A smaller node size, or fewer nodes | | Storage | Increase only | Increase only | | Fault tolerance while resizing | A brief reconnect while the node restarts | The cluster stays available. Capacity dips as each node restarts | Storage can be increased but never decreased on either plan, and you can increase the disk size once every six hours. Over-provisioning storage is therefore a one-way decision. Start close to what the dataset needs and grow it as the dataset does. ## When to scale The signals worth acting on are: - **CPU pegged:** the instance sits near its instance-type limit under normal traffic, not only at peaks. - **Memory pressure:** the working set no longer fits, so the engine reads from disk more often. - **Storage growth:** consumption trends towards the provisioned limit. Act well before the limit, because a full disk stops writes. - **Connection saturation:** clients queue or are refused. - **Latency objectives breached:** queries that used to meet them no longer do. Read those signals from the [instance dashboard, metrics, and logs](/docs/manage/instances/monitoring.md). Rule out query-level causes first. A missing index or a full-table scan is cheaper to fix than a larger instance, and a resize hides the problem rather than removing it. Schedule resizes outside peak traffic where you can. Some changes cause a brief reconnect, so clients need retry logic in any case. ## Scaling a Start instance A Start instance has one node, so growth means a larger instance type and a larger disk. A larger type gives more vCPU, more memory, and more I/O. Move to **Scale** instead when a single node is no longer the right shape: - The workload must survive the loss of a node. - Query throughput needs to spread across nodes. - You have reached the largest available Start instance type. Moving from Start to Scale is not an in-place upgrade. Deploy a Scale instance, then [restore a backup](/docs/manage/instances/backups.md) or [import an export](/docs/manage/instances/import-and-export.md) into it. ## Scaling a Scale cluster A Scale cluster grows in two directions. **Vertically**, each node moves to a larger size. **Horizontally**, you add nodes, so a three-node cluster becomes four, then five, and so on. The storage layer handles replication, consensus, and distributed transactions across whatever the node count becomes. > [!NOTE] > An even node count does not improve fault tolerance. A cluster of N nodes survives the same number of failures as one of N−1 nodes. The fast commit path also needs every node to agree, so one slow node forces the slower path. Odd sizes of three, five, or seven give the best ratio of resilience to cost. After you scale out, check two things. Confirm that **replication lag** stays within what the application tolerates. Then check **per-node balance**, because an unbalanced cluster means the added nodes are not taking their share of the load. See [Architecture](/docs/manage/instances/architecture.md) for the topologies, and [High availability](/docs/manage/instances/high-availability.md) for what each one survives. ## Confirming the resize worked Compare the same metrics before and after the change. A resize that shifts the bottleneck rather than removing it is common: CPU headroom appears, and disk I/O becomes the new limit. If utilisation stays low after a large resize, step back down. Instance type is billed on what is provisioned, not on what is used, so an oversized instance is a standing cost with no benefit. `surrealctl instance metrics` and `surrealctl instance update` let you read the signals and apply the change from a terminal. See [surrealctl instances](/docs/manage/surrealctl/instances.md). --- Source: https://surrealdb.com/docs/manage/instances/versions-and-upgrades # Versions and upgrades Change the SurrealDB release an instance runs. What happens to the instance during the upgrade. Change the SurrealDB release an instance runs from **Instance settings → Version** in [SurrealDB Studio](https://studio.surrealdb.com). You pick the release when you [create the instance](/docs/manage/instances/create.md). The running image is then replaced for you, so there are no binaries to swap and no rolling restart to orchestrate yourself. ## Changing the version 1. Open the instance in [SurrealDB Studio](https://studio.surrealdb.com). 2. Go to **Instance settings → Version**. 3. Select the release you want. Each entry links to its release notes. 4. Apply the change. Only upgrades are offered. The flow will not move an instance to an older release. On-disk data written by a newer version is not readable by an older one. On paid instance types a snapshot is taken before the change, and it appears in the [backup list](/docs/manage/instances/backups.md) of the instance. That snapshot is the way back if the new version behaves differently than you expected. Restoring it produces a new instance on a compatible version. > [!NOTE] > Pre-release builds, such as nightly or beta tags, are only offered when they have been enabled for your organisation. Keep production on stable releases unless you are deliberately testing a preview. ## What happens during the upgrade | Plan | Behaviour | | --- | --- | | **Start** | The single node restarts on the new image. Expect a short window where connections fail and clients reconnect. | | **Scale** | The new image rolls through nodes one at a time while the storage layer keeps data consistent. The cluster stays reachable, and query capacity dips while each node restarts. | Timing depends on the instance size, the dataset size, and the cluster layout. Watch [metrics and logs](/docs/manage/instances/monitoring.md) during and after the change. Confirm that query latency and error rates return to their previous level. ## Planning an upgrade 1. **Read the [release notes](https://surrealdb.com/releases)** for the target version and for every version between it and your current one. Look for breaking changes, changed defaults, and removed options. 2. **Test against a non-production instance** on the target version, with your own application and SDK versions. SDK and server versions are released independently, so check both. 3. **Confirm you have a recent snapshot.** For a copy that lives outside the platform, take a [logical export](/docs/manage/instances/import-and-export.md) instead. 4. **Apply the change**, then run your smoke tests. Cover health checks, the queries your critical paths depend on, and anything that touches features named in the release notes. For a major-version change that migrates data on disk, work through [Migrating from older SurrealDB versions](/docs/build/migrating/from-old-surrealdb-versions/overview.md) against a staging instance before you touch production. `surrealctl instance update` sets the version from a terminal, and `surrealctl instance status` reports what an instance is running. The second command is useful for checking a fleet at once. See [surrealctl instances](/docs/manage/surrealctl/instances.md). ## Compared with self-hosting Self-hosted operators replace the `surreal` binary, run `surreal fix` where a release requires it, and coordinate the restart order across nodes themselves. On an instance, the control plane does that work. The underlying mechanics, and the release-specific migration notes, are in [Self-hosted upgrades and patching](/docs/manage/self-hosted/upgrades-and-patching.md). --- Source: https://surrealdb.com/docs/manage/observability # Observability Production visibility for SurrealDB Community and Enterprise. Logging, Prometheus pull, OTLP push, Enterprise pipelines, metric catalogues, and Tokio console. SurrealDB exposes **logging**, **metrics**, **traces**, and (with **SurrealDB Enterprise**) durable **audit** and **slow-query** record pipelines through one OpenTelemetry-oriented surface. Signals can leave the process in two complementary ways: - **Pull** - scrapers call `GET /metrics` (Prometheus text exposition). - **Push** - the server exports metrics, logs, and traces over **OTLP** to a collector when `SURREAL_TELEMETRY_PROVIDER=otlp`. Both paths can run together or independently; each is controlled with environment variables documented on the [configuration reference](/docs/manage/observability/configuration.md). **Suggested reading order** 1. [Logging](/docs/manage/observability/logging.md) - stderr, JSON, files, sockets, and line-based slow-query logging (no metrics stack required). 2. [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md) - `GET /metrics`, naming, migration from pre-3.1 series, the public allowlist, and behaviour common to **Community and Enterprise**. 3. [Telemetry (OTLP)](/docs/manage/observability/telemetry.md) - push export, intervals, and backward-compatible instruments on the wire. 4. [Enterprise observability](/docs/manage/observability/enterprise-observability.md) - cluster metrics, audit and slow-query file pipelines, pipeline self-metrics, and OTLP log export opt-ins. 5. [Metrics reference](/docs/manage/observability/metrics.md) - full catalogue, labels, alert hints, and the 3.0 → 3.1 migration table. 6. [Configuration](/docs/manage/observability/configuration.md) - every telemetry, audit, and slow-query environment variable. 7. [Audit logging](/docs/manage/observability/audit-logging.md) and [Slow-query logging](/docs/manage/observability/slow-query-logging.md) - Enterprise pipeline references. 8. [Tokio console](/docs/manage/observability/tokio-console.md) - optional async-runtime debugging alongside metrics and traces. **Pull** means scrapers call your server. **Push** means the server opens an export connection to a collector. From SurrealDB 3.1 onward, both paths share the same instruments. Canonical prose also ships in-tree: [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) and [`doc/TELEMETRY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/TELEMETRY.md); the Enterprise distribution extends `doc/OBSERVABILITY.md` with **[C]** / **[E]** catalogue markers. _(since v3.1.0)_ The `surrealdb.*` metric namespace, the `PUBLIC_METRICS` allowlist, the audit-log pipeline, and the slow-query log pipeline are new in SurrealDB 3.1. Operators upgrading from 3.0 should read the [Migration from 3.0](/docs/manage/observability/metrics.md#migration-from-30) section in the metrics reference. ## Editions at a glance The Community server publishes the full set of primary signal families. The Enterprise composer adds the distributed-storage cluster metrics, the audit and slow-query log pipelines (with their own self-metrics), and the reserved per-tenant rollup scope.
Capability Community Enterprise
Prometheus /metrics endpoint _(Community)_ _(Enterprise)_
OTLP push (metrics, logs, traces) _(Community)_ _(Enterprise)_
Primary signal families (statement, query, transaction, RPC, auth, session, network, HTTP, live query, slow-query counter, GraphQL, MCP, storage) _(Community)_ _(Enterprise)_
Public metrics allowlist for anonymous scrapers _(Community)_ _(Enterprise)_
Cluster metrics (surrealdb.ds.*) - _(Enterprise)_ (when the distributed storage runtime is deployed)
Audit log records (file sink + optional OTel logs) - _(Enterprise)_
Slow-query log records (file sink + optional OTel logs) - _(Enterprise)_
Hash-chained, tamper-evident records - _(Enterprise)_
Three-pass redaction (literal, identifier, regex) - _(Enterprise)_
Per-tenant rollup scope (surrealdb.tenant) (reserved) - _(Enterprise)_
Edition is also carried on the OTel `Resource` via the `service.edition` attribute (`community` or `enterprise`), so dashboards can group or filter on it without hard-coding metric scopes. ## Quickstart **Prometheus pull** The `/metrics` endpoint is mounted by default. Anonymous scrapers receive only the six metrics on the [public allowlist](/docs/manage/observability/metrics.md#public-metrics-allowlist); root credentials unlock the full surface. ```bash # Start the server (Community example) surreal start --user root --pass secret # Anonymous scrape - public allowlist only curl http://127.0.0.1:8000/metrics # Operator scrape - full surface, including labelled families curl -u root:secret http://127.0.0.1:8000/metrics ``` To turn the endpoint off entirely: ```bash SURREAL_METRICS_ENABLED=false surreal start ... # /metrics now returns 404 ``` **OTLP push** OTLP push targets any OpenTelemetry collector. SurrealDB exports metrics, logs (including audit and slow-query records when opted in) and traces over gRPC, using native-TLS roots when the endpoint is HTTPS. ```bash SURREAL_TELEMETRY_PROVIDER=otlp \ OTEL_EXPORTER_OTLP_ENDPOINT="http://my-collector.monitoring.svc.cluster.local:4317" \ surreal start ... ``` To selectively disable parts of the OTLP pipeline: ```bash SURREAL_TELEMETRY_DISABLE_METRICS=true # OTLP metrics off; logs + traces still push SURREAL_TELEMETRY_DISABLE_TRACING=true # OTLP traces off; metrics + logs still push ``` Audit and slow-query records flow over OTLP **only when explicitly opted in** per pipeline via `SURREAL_AUDIT_OTEL_EXPORT=true` / `SURREAL_SLOW_QUERY_OTEL_EXPORT=true`. The local file sink remains the primary path for compliance ingestion. ## What is new in 3.1 _(since v3.1.0)_ - A reworked metric namespace - every instrument is now `surrealdb.*`, grouped by signal family (statement, query, transaction, RPC, …). Names from 3.0 are mapped in the [migration table](/docs/manage/observability/metrics.md#migration-from-30). - **Dual access paths.** Prometheus pull on `/metrics` and OTLP push run side-by-side. Both pipelines can be toggled independently. - **Public metrics allowlist (`PUBLIC_METRICS`).** Six low-sensitivity gauges are safe to expose anonymously; the rest require root credentials. - **Audit log pipeline.** _(Enterprise)_ Durable NDJSON file sink with size-based rotation, tunable fsync cadence, optional hash chaining for tamper-evidence, and three-pass redaction. - **Slow-query log pipeline.** _(Enterprise)_ Mirrors the audit pipeline, with a configurable duration threshold and the slow-query counter metric. - **Cluster metrics (`surrealdb.ds.*`).** _(Enterprise)_ Around thirty instruments covering network, consensus, view changes, recovery and garbage collection when distributed storage is deployed (SurrealDB Cloud Scale or self-hosted Enterprise). - **OpenTelemetry alignment.** Standard semantic-convention attribute keys (`http.request.method`, `http.route`, `http.response.status_code`, `db.namespace`, `db.user`) replace ad-hoc keys. ## Where to go next - [Logging](/docs/manage/observability/logging.md) - Log level, text or JSON format, files, sockets, and line-based slow-query logging. - [Metrics and Prometheus](/docs/manage/observability/observability.md) - Pull scraping, naming, multi-tenant guidance, and version-specific tabs before and after 3.1. - [Telemetry (OTLP)](/docs/manage/observability/telemetry.md) - Push export, intervals, process gauges, and legacy instruments on the wire. - [Enterprise observability](/docs/manage/observability/enterprise-observability.md) - Cluster metrics, audit and slow-query pipelines, and OTLP log export opt-ins. - [Metrics reference](/docs/manage/observability/metrics.md) - Access paths, label catalogue, every metric grouped by signal family, and the 3.0 → 3.1 migration table. - [Configuration reference](/docs/manage/observability/configuration.md) - All telemetry, audit log and slow-query log environment variables, plus recommended configurations for local, production and multi-tenant deployments. - [Audit logging](/docs/manage/observability/audit-logging.md) - Enterprise audit log pipeline: events captured, record shape, rotation, hash chaining and redaction. - [Slow-query logging](/docs/manage/observability/slow-query-logging.md) - Enterprise slow-query log pipeline: how a query qualifies, record shape and pipeline self-metrics. - [Tokio console](/docs/manage/observability/tokio-console.md) - Optional Tokio runtime debugging - tasks, poll times, and scheduling alongside metrics and traces. - [Instance monitoring](/docs/manage/instances/monitoring.md) - The built-in monitoring dashboard, log retention and metrics views for managed instances. - [Self-hosted monitoring](/docs/manage/self-hosted/monitoring-and-observability.md) - Pairing the `/health` endpoint, Prometheus and Grafana with the observability surface for a self-hosted deployment. ## Structured logs versus audit and slow-query records Server **structured logs** (levels, format, files, sockets, and the `--slow-log-*` line-based slow-query helpers) are configured on [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) and in the [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md) catalogue - see [Logging](/docs/manage/observability/logging.md). That stream is separate from the **Enterprise audit** and **slow-query** NDJSON pipelines, which have their own sinks and optional OTel log export - see [Audit logging](/docs/manage/observability/audit-logging.md) and [Slow-query logging](/docs/manage/observability/slow-query-logging.md). For async runtime introspection (tasks, poll histograms), use the [Tokio console](/docs/manage/observability/tokio-console.md) on trusted hosts only; it complements OTLP and `/metrics`, it does not replace them. --- Source: https://surrealdb.com/docs/manage/observability/audit-logging # Audit logging The Enterprise audit log pipeline. Events captured, record shape, rotation, hash chaining, redaction, and pipeline self-metrics. _(Enterprise)_ _(since v3.1.0)_ The audit log pipeline records authoritative, identity-bound events that operators and auditors need to reconstruct who did what, when, and with what outcome. It is part of SurrealDB Enterprise and is **off by default** - set `SURREAL_AUDIT_SINK=file` and `SURREAL_AUDIT_FILE_PATH` to enable it. > [!WARNING] > The `SURREAL_AUDIT_*` variables are registered only by the Enterprise binary. As Community builds never reads them, the server starts normally, nothing is written to the configured path, and no audit file is created. The loud startup failures described below apply to Enterprise builds only, so do not rely on these settings for compliance on a Community build. Records flow through two parallel paths: 1. **Durable file sink.** A bounded queue feeds a background worker that appends each record to an NDJSON file. Optional SHA-256 hash chaining provides tamper-evidence; size-based rotation and a tunable fsync cadence keep the file manageable. This is the primary path for compliance and SIEM ingestion. 2. **OpenTelemetry logs.** The same record can also be emitted as an OTel `LogRecord` on the SDK logger provider. **Off by default**; opt in per pipeline with `SURREAL_AUDIT_OTEL_EXPORT=true`. Compliance-sensitive deployments typically keep this off and rely on the file sink. The observer hot path never blocks on I/O. [Redaction](#redaction) runs synchronously on the executor thread before the record reaches the queue, so the worker can write raw bytes straight to the sink and the OTel emit cannot leak unredacted content. The full set of configuration variables lives on the [configuration reference](/docs/manage/observability/configuration.md#audit-log-knobs). ## Events captured Each event surfaces as an OTel `LogRecord` with a specific event name and a severity that depends on the outcome:
Event name Severities Captures
surrealdb.audit.statement Info / Error Completion of an individual statement.
surrealdb.audit.query Info / Error Completion of a multi-statement query.
surrealdb.audit.transaction Info / Error Commit or rollback of a transaction.
surrealdb.audit.rpc Info / Error Completion of an RPC call.
surrealdb.audit.auth Info (success) / Warn (any non-success) / Error (outcome=error) Sign-in, sign-up and authentication outcomes.
surrealdb.audit.session Info Session connect and disconnect events.
surrealdb.audit.http Info / Error (typically 5xx responses) Completion of an HTTP request.
The OTel `LogRecord` body is a short human-readable string. The structured fields live on attributes (`db.namespace`, `db.name`, `db.user`, `db.statement`, `session.id`, `client.address`, `surrealdb.statement_type`, `surrealdb.outcome`, `surrealdb.duration_ms`, `surrealdb.error_class`). ## Record shape Audit and slow-query files contain one JSON record per line, each terminated by a newline. Audit and slow-query records share a common envelope; audit records additionally carry the `event_type` that distinguishes the seven event variants above. Audit `event_type` values: `statement`, `query`, `transaction`, `rpc`, `auth`, `session`, `http`. Envelope fields (common to all records): - `ts` - RFC 3339 timestamp. - `event_type` - see above. - `outcome` - `success`, `error`, `cancelled`, or (for auth events) `denied` / `failed`. - `duration_ms` - wall-clock duration of the captured operation. - Identity context - `namespace`, `database`, `user` resolved from the session. - `sql` - captured statement text when `SURREAL_AUDIT_INCLUDE_SQL=true`; absent otherwise. - `prev_hash` / `hash` - SHA-256 chain fields, present when [hash chaining](#hash-chaining) is enabled. A captured `statement` event with hash chaining enabled looks like: ```json { "ts": "2026-03-04T10:23:11.482Z", "event_type": "statement", "outcome": "success", "duration_ms": 14, "namespace": "acme", "database": "prod", "user": "svc_orders", "sql": "UPDATE orders:abc SET status = 'shipped'", "prev_hash": "f1a3…", "hash": "c92e…" } ``` ## Rotation and durability - File mode `0600` on Unix. The parent directory must exist; the server refuses to start otherwise. - Size-based rotation at `SURREAL_AUDIT_FILE_ROTATE_BYTES` (default 256 MiB). - The oldest rotation is dropped once `SURREAL_AUDIT_FILE_ROTATE_KEEP` (default `8`) generations are present. - Mid-stream fsync cadence is governed by `SURREAL_AUDIT_FSYNC_EVERY`. Rotation and graceful shutdown always flush and `sync_data` regardless of cadence. ## Hash chaining When `SURREAL_AUDIT_HASH_CHAIN=true` every record carries: - `prev_hash` - SHA-256 of the previous record in the same file. Absent on the genesis record at the start of a new file. - `hash` - SHA-256 of this record's canonical serialisation (including `prev_hash`). Rotation closes a chain and starts a new one with a fresh genesis record. A detector verifies the chain by recomputing each hash sequentially and comparing against the stored `hash`. > **Hash chaining requires `SURREAL_AUDIT_FSYNC_EVERY=1`.** Without per-record fsync, the chain pointer could advance for records that are not durably on disk, leaving on-disk gaps the chain still references and silently weakening the guarantee. Startup fails loudly when the two knobs disagree. ## Redaction Redaction is applied **synchronously on the executor thread** before the record reaches the queue, so the worker, the file sink, and the OTel logger all see the same already-scrubbed text. Three layered passes run in order: 1. **Literal pass** - when `SURREAL_AUDIT_REDACT_LITERALS=true` every single- or double-quoted span in the SQL is replaced with `'***'` / `"***"`. 2. **Identifier-token pass** - `SURREAL_AUDIT_REDACT_TABLES=secrets,pii` performs a case-insensitive replacement of each identifier token with `***`. 3. **Regex pass** - `SURREAL_AUDIT_REDACT_REGEX=";"` (note: **semicolon-separated**) compiles each pattern at startup. An invalid pattern fails startup; valid patterns run in order against the SQL text. The slow-query log pipeline supports the same three passes under `SURREAL_SLOW_QUERY_REDACT_LITERALS`, `SURREAL_SLOW_QUERY_REDACT_TABLES` and `SURREAL_SLOW_QUERY_REDACT_REGEX`. ## Overflow semantics Neither overflow policy offers a lossless guarantee: - `drop` - single non-blocking `try_send`. On `Full` or `Closed` the record is dropped and the `surrealdb_audit_dropped` gauge increments. - `block` - bounded busy-yield loop (200 retries with `std::thread::yield_now`). `yield_now` does **not** park the OS thread, so on a multi-threaded runtime the drain task can make progress between yields and short bursts may be absorbed without drops. On a `current_thread` runtime the producer holds the only worker and the policy degrades to immediate drop. There is no wall-clock time-bound on the loop - the budget caps iterations only. The audit pipeline defaults to `block` because audit records are compliance-sensitive; the slow-query pipeline defaults to `drop` because slow-query records are triage data. > Whichever policy is configured, alert on `rate(surrealdb_audit_dropped[5m]) > 0` and `rate(surrealdb_audit_append_errors[5m]) > 0`. Both indicate records were lost. ## Pipeline self-metrics Five observable gauges expose the live state of the pipeline. Each is read at scrape time from atomic counters on the worker, so the cost is zero when nothing consumes the metric. | Metric | Notes | | --- | --- | | `surrealdb_audit_records` | Cumulative records successfully enqueued. | | `surrealdb_audit_dropped` | Cumulative records dropped (overflow or queue closed). **Alert on any non-zero rate.** | | `surrealdb_audit_queue_depth` | Records currently buffered between observer and worker. Sustained depth above ~50% of `SURREAL_AUDIT_QUEUE_CAPACITY` indicates a slow sink. | | `surrealdb_audit_appended` | Cumulative records the worker wrote to the sink. The gap to `surrealdb_audit_records` is queue depth plus append errors. | | `surrealdb_audit_append_errors` | Cumulative sink-write failures. **Alert on any non-zero rate.** | The slow-query pipeline exposes the same shape under the `surrealdb_slow_query_*` prefix - see [slow-query logging](/docs/manage/observability/slow-query-logging.md#pipeline-self-metrics). ## Related references - [Configuration → Audit log knobs](/docs/manage/observability/configuration.md#audit-log-knobs) - every audit-log environment variable. - [Configuration → Compliance checklist](/docs/manage/observability/configuration.md#compliance-checklist) - the minimum tamper-evident configuration. - [Slow-query logging](/docs/manage/observability/slow-query-logging.md) - the sister pipeline for triage data. - [Metrics reference → Audit log pipeline self-metrics](/docs/manage/observability/metrics.md#audit-log-pipeline-self-metrics) - the five gauges in the metric catalogue. --- Source: https://surrealdb.com/docs/manage/observability/configuration # Configuration reference Every observability, audit log and slow-query log variable. Recommended configurations for local, production and multi-tenant deployments. Every environment variable that controls the observability surface. Variables marked _(Enterprise)_ exist only when the Enterprise binary is running; everything else is available in all editions. _(since v3.1.0)_ The audit log, slow-query log and cluster configuration surfaces are new in SurrealDB 3.1. ## How to read this page - **Default** is the value used when the variable is unset. Cells marked ` - ` are required when the surrounding feature is enabled. - **Edition** identifies which builds register the variable. Variables marked _(Enterprise)_ are no-ops on a Community binary. - All variables can be set via environment, a `.env` file loaded by the deployment, or the orchestrator's secret store. ## Core knobs These knobs control the two access paths (Prometheus pull and OTLP push) and the global telemetry switches. They are available in every edition.
Variable Default Edition Purpose
SURREAL_METRICS_ENABLED true _(Community)_ Mount the /metrics endpoint. When false the route returns 404.
SURREAL_TELEMETRY_PROVIDER unset _(Community)_ Set to otlp to enable the OTLP push pipeline (metrics, logs and traces). Any other value leaves OTLP off.
SURREAL_TELEMETRY_DISABLE_METRICS false _(Community)_ Skip the OTLP metrics reader specifically, leaving logs and traces unaffected.
SURREAL_TELEMETRY_DISABLE_TRACING false _(Community)_ Skip the OTLP trace exporter specifically, leaving metrics and logs unaffected.
SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS 1000 _(Community)_ Threshold above which a completed statement is counted on surrealdb_slow_query_total. Set to 0 to disable the counter.
SURREAL_PROCESS_METRICS_REFRESH_INTERVAL 5 _(Community)_ Cadence in seconds for refreshing the cached process snapshot that backs surrealdb_process_memory_bytes and surrealdb_process_cpu_percent. Floored at 1.
OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4317 _(Community)_ OTLP gRPC endpoint. Standard OpenTelemetry variable.
OTEL_METRIC_EXPORT_INTERVAL 60000 (ms) _(Community)_ OTLP push cadence. Standard OpenTelemetry variable.
## Audit log knobs _(Enterprise)_ Controls the [Enterprise audit log pipeline](/docs/manage/observability/audit-logging.md). Setting `SURREAL_AUDIT_SINK=none` (the default) leaves the pipeline disabled entirely; no observer is registered and the cost is zero.
Variable Default Purpose
SURREAL_AUDIT_SINK none One of none, file. syslog and table are reserved but rejected at startup.
SURREAL_AUDIT_FILE_PATH - Required when SURREAL_AUDIT_SINK=file. Parent directory must exist; startup fails loudly if it doesn't. File is opened with mode 0600 on Unix.
SURREAL_AUDIT_FILE_ROTATE_BYTES 268435456 (256 MiB) Size threshold that triggers rotation.
SURREAL_AUDIT_FILE_ROTATE_KEEP 8 Number of rotated files to retain.
SURREAL_AUDIT_FSYNC_EVERY 0 Mid-stream fsync cadence. 0 never fsyncs mid-stream; 1 fsyncs every record; N>1 fsyncs every Nth record. Rotation and graceful shutdown always fsync regardless.
SURREAL_AUDIT_HASH_CHAIN false Adds prev_hash / hash SHA-256 fields to every record for tamper-evidence. Requires SURREAL_AUDIT_FSYNC_EVERY=1 - startup fails otherwise.
SURREAL_AUDIT_INCLUDE_SQL false Include the full SQL text on statement records. Off by default; the identity and action context is still emitted.
SURREAL_AUDIT_QUEUE_CAPACITY 4096 Records the bounded observer-to-worker queue can hold.
SURREAL_AUDIT_OVERFLOW block drop (single non-blocking try_send) or block (bounded busy-yield retry). See overflow semantics. Neither offers a lossless guarantee.
SURREAL_AUDIT_REDACT_TABLES - Comma-separated identifier tokens replaced with *** in captured SQL.
SURREAL_AUDIT_REDACT_REGEX - Semicolon-separated regex patterns applied to captured SQL. Each pattern is compiled at startup; an invalid pattern fails startup.
SURREAL_AUDIT_REDACT_LITERALS false Replace every quoted literal with ***. Off by default.
SURREAL_AUDIT_OTEL_EXPORT false Also emit each audit record as an OTel LogRecord. Off by default so compliance-sensitive records stay on the local file sink.
## Slow-query log knobs _(Enterprise)_ Controls the [Enterprise slow-query log pipeline](/docs/manage/observability/slow-query-logging.md). Setting `SURREAL_SLOW_QUERY_SINK=none` (the default) leaves the pipeline disabled entirely.
Variable Default Purpose
SURREAL_SLOW_QUERY_SINK none Same selector set as SURREAL_AUDIT_SINK.
SURREAL_SLOW_QUERY_FILE_PATH - Required when the sink is file.
SURREAL_SLOW_QUERY_FILE_ROTATE_BYTES 268435456 (256 MiB) Rotation threshold.
SURREAL_SLOW_QUERY_FILE_ROTATE_KEEP 8 Retained rotation generations.
SURREAL_SLOW_QUERY_FSYNC_EVERY 0 Same semantics as the audit equivalent.
SURREAL_SLOW_QUERY_HASH_CHAIN false Tamper-evident chain. Requires SURREAL_SLOW_QUERY_FSYNC_EVERY=1.
SURREAL_SLOW_QUERY_THRESHOLD_MS - Duration threshold above which a statement is captured. Required when the sink is enabled.
SURREAL_SLOW_QUERY_INCLUDE_SQL true Capture SQL text in slow-query records. Setting to false lets throughput-sensitive deployments skip per-statement to_sql() rendering at the cost of losing SQL context on captures.
SURREAL_SLOW_QUERY_QUEUE_CAPACITY 4096 Records the queue can hold.
SURREAL_SLOW_QUERY_OVERFLOW drop Default differs from audit - slow-query records are triage data, so dropping is preferred over busy-yielding the executor.
SURREAL_SLOW_QUERY_REDACT_TABLES - Same syntax as the audit equivalent.
SURREAL_SLOW_QUERY_REDACT_REGEX - Same syntax as the audit equivalent.
SURREAL_SLOW_QUERY_REDACT_LITERALS false Same semantics as the audit equivalent.
SURREAL_SLOW_QUERY_OTEL_EXPORT false Emit each slow-query record as an OTel LogRecord at WARN severity.
## Cluster networking, consensus and storage _(Enterprise)_ > [!NOTE] > The `SURREAL_DS_*` variables below apply to multi-node clusters, which use distributed storage with replication and consensus - on [SurrealDB Cloud Scale](https://surrealdb.com/pricing/scale) or in self-hosted Enterprise installations. A short list of the cluster networking, consensus and storage-memory knobs that operators routinely tune in response to a metric signal. The complete `SURREAL_DS_*` reference is part of the Enterprise Kubernetes deployment guide.
Variable Default Tune in response to
SURREAL_DS_NTW_INBOUND_BYTES 3 * MAX_MSG_BYTES (768 MiB) per MessageClass QUIC inbound bytes ... saturation warning log.
SURREAL_DS_NTW_INFLIGHT_PROCESSING_CAP max(384, num_cpus*32) QUIC inbound handlers ... saturation warning log.
SURREAL_DS_MAX_READ_OPERATIONS 192 Steady-state RSS pressure or scan-heavy workloads.
SURREAL_DS_MAX_WRITE_OPERATIONS 96 Write operations limit saturation warning log.
SURREAL_DS_CONSENSUS_FAST_QUORUM_TIMEOUT_MS implementation default Rising surrealdb_ds_consensus_fast_quorum_timeouts_total.
SURREAL_DS_RETRY_BASE_MS / SURREAL_DS_RETRY_MAX_MS 500 / 5000 Rising surrealdb_ds_finalize_prepare_retries_total.
SURREAL_DS_ROCKSDB_BLOCK_CACHE_SIZE_(since v3.1.1)_ max(memory/2 - 1 GiB, 16 MiB) Steady-state RSS pressure. The largest single consumer in a node using the RocksDB durable backend. Full-text index reads are served through it entirely; vector index reads hit it on load and on cache misses. The default is derived from detected memory, so set it explicitly on memory-constrained pods.
SURREAL_DS_ROCKSDB_MAX_WRITE_BUFFER_NUMBER_(since v3.1.1)_ Derived from detected memory (2-32) RSS pressure during sustained write phases. The memtable ceiling is this value multiplied by SURREAL_DS_ROCKSDB_WRITE_BUFFER_SIZE and by the four heavy column families, so it is the knob to cap first when headroom is tight.
> [!NOTE] > `SURREAL_DS_ROCKSDB_*` applies only when a node's store path selects the RocksDB durable backend. Memtable and block-cache memory are accounted together, and an undersized write budget presents as write latency rather than errors - but the operative limits are the per-column-family memtable count and the L0 compaction trigger, not the combined ceiling. `SURREAL_DS_ROCKSDB_WRITE_BUFFER_SIZE` sets the per-column-family memtable size; the complete reference is in the Enterprise Kubernetes deployment guide. ## Recommended configurations **Local development** The minimum needed to scrape metrics from a local server during development: ```bash surreal start --user root --pass secret # Anonymous: only the public allowlist curl http://127.0.0.1:8000/metrics # Authenticated: the full surface curl -u root:secret http://127.0.0.1:8000/metrics ``` To push to a local OpenTelemetry collector (for example a Grafana / Tempo / Loki / Prometheus docker-compose stack): ```bash SURREAL_TELEMETRY_PROVIDER=otlp \ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \ surreal start --user root --pass secret ``` To disable the endpoint entirely (useful when running with a third-party agent that scrapes via OTLP only): ```bash SURREAL_METRICS_ENABLED=false surreal start ... # /metrics → 404 ``` **Self-hosted production** For a self-hosted production deployment that ships metrics over OTLP and writes audit and slow-query records to a local sink for SIEM ingestion: ```bash # Telemetry SURREAL_METRICS_ENABLED=true SURREAL_TELEMETRY_PROVIDER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.internal:4317 OTEL_METRIC_EXPORT_INTERVAL=30000 # Audit log (Enterprise) - durable, hash-chained SURREAL_AUDIT_SINK=file SURREAL_AUDIT_FILE_PATH=/var/log/surrealdb/audit.log SURREAL_AUDIT_FSYNC_EVERY=1 SURREAL_AUDIT_HASH_CHAIN=true SURREAL_AUDIT_INCLUDE_SQL=true SURREAL_AUDIT_REDACT_LITERALS=true SURREAL_AUDIT_OVERFLOW=block # Slow-query log (Enterprise) - best-effort triage data SURREAL_SLOW_QUERY_SINK=file SURREAL_SLOW_QUERY_FILE_PATH=/var/log/surrealdb/slow-query.log SURREAL_SLOW_QUERY_THRESHOLD_MS=250 ``` Pair this with the alert hints in the [metrics reference](/docs/manage/observability/metrics.md#alert-hints) and the alerting rules on `surrealdb_audit_dropped`, `surrealdb_audit_append_errors` and `surrealdb_audit_queue_depth` listed in the [Compliance checklist](#compliance-checklist) below. **Multi-tenant / cloud-facing** For pods that serve customer traffic directly, the default posture is **deny scraping at the pod boundary** and serve metrics from an internal sidecar instead. Customer-visible metrics should be filtered at a proxy. ```bash # Customer-facing pod SURREAL_METRICS_ENABLED=false # Internal sidecar pod (same workload, internal-only interface) SURREAL_METRICS_ENABLED=true SURREAL_TELEMETRY_PROVIDER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.internal:4317 ``` Filter on `namespace` and `database` at the proxy and strip the `user` label before exposing to a customer. See the [Recommended cloud whitelist](/docs/manage/observability/metrics.md#recommended-cloud-whitelist) for the suggested customer-visible subset. ## Compliance checklist _(Enterprise)_ For deployments that require tamper-evident, durable audit trails the recommended configuration is: ```bash SURREAL_AUDIT_SINK=file SURREAL_AUDIT_FILE_PATH=/var/log/surrealdb/audit.log SURREAL_AUDIT_FSYNC_EVERY=1 # required for hash chain SURREAL_AUDIT_HASH_CHAIN=true SURREAL_AUDIT_INCLUDE_SQL=true # if a regulator requires statement text SURREAL_AUDIT_REDACT_LITERALS=true # scrub embedded PII from quoted values SURREAL_AUDIT_OVERFLOW=block # already the default; explicit for clarity ``` Plus alerts on: - `surrealdb_audit_dropped` - any non-zero rate is a lost record. - `surrealdb_audit_append_errors` - any non-zero rate is a lost record. - `surrealdb_audit_queue_depth` sustained above ~50% of `SURREAL_AUDIT_QUEUE_CAPACITY` - the sink is falling behind. For the full pipeline details (record shape, rotation, hash chain, redaction) see the [Audit logging](/docs/manage/observability/audit-logging.md) reference. ## Troubleshooting | Symptom | Likely cause | Investigation | | --- | --- | --- | | `/metrics` returns `404` | `SURREAL_METRICS_ENABLED=false` | Re-enable, or scrape over OTLP instead. | | `/metrics` returns only six metrics | Anonymous scrape against the public allowlist | Configure basic-auth with root credentials. | | No `surrealdb_ds_*` metrics on a single-node setup | Enterprise composer did not start a metrics reader | Confirm the Enterprise binary is running and at least one of `SURREAL_METRICS_ENABLED=true` / `SURREAL_TELEMETRY_PROVIDER=otlp` is set. | | `surrealdb_audit_records` missing entirely | Audit sink set to `none` (the default) | Set `SURREAL_AUDIT_SINK=file` and `SURREAL_AUDIT_FILE_PATH`. | | Sustained `surrealdb_audit_dropped` rate | Worker can't keep up | Raise `SURREAL_AUDIT_QUEUE_CAPACITY`; check sink disk I/O; consider switching from `block` to `drop` if drops are acceptable. | | `QUIC inbound bytes ... saturation warning` log | Per-class inbound budget saturated | Raise `SURREAL_DS_NTW_INBOUND_BYTES`. | | OTLP collector not receiving metrics | `SURREAL_TELEMETRY_PROVIDER` unset, or `SURREAL_TELEMETRY_DISABLE_METRICS=true` | Confirm both, plus that `OTEL_EXPORTER_OTLP_ENDPOINT` is reachable. | | Hash chain rejected at startup | `SURREAL_AUDIT_HASH_CHAIN=true` without `SURREAL_AUDIT_FSYNC_EVERY=1` | Set both, or disable hash chaining. | --- Source: https://surrealdb.com/docs/manage/observability/enterprise-observability # Enterprise observability Metrics, logs, and traces in Enterprise versus Community. OTLP opt-ins, audit and slow-query pipelines, and cluster instruments. SurrealDB **Community** and **Enterprise** share one unified observability implementation in the server: the same labelled metric families, the same `GET /metrics` path, and the same OTLP push pipeline when enabled. **Enterprise** adds optional durability and cluster depth on top: - **Cluster instruments** (`surrealdb.ds.*`) - consensus, networking, recovery, and garbage collection when distributed storage is the active backend ([SurrealDB Cloud](https://surrealdb.com/pricing) Scale tiers and self-hosted Enterprise). - **Audit and slow-query pipelines** - bounded queues, file sinks with rotation and optional hash chaining, and pipeline self-metrics (`surrealdb.audit.*`, `surrealdb.slow_query.*`) when those sinks are configured. - **OpenTelemetry logs for audit and slow-query** - the same redacted records can be emitted as OTel log records, but **only when you opt in** with `SURREAL_AUDIT_OTEL_EXPORT=true` and/or `SURREAL_SLOW_QUERY_OTEL_EXPORT=true` (default `false` so sensitive payloads stay on the file sink unless you choose otherwise). Throughout this page, markers mean: - **[C]** - Registered in every build that exposes metrics (Community and Enterprise). - **[E]** - Registered only when running a **SurrealDB Enterprise** binary with the relevant feature enabled. For Prometheus pull behaviour, naming rules, the public `PUBLIC_METRICS` allowlist, and multi-tenant guidance common to both editions, see [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md). For OTLP push, intervals, and legacy HTTP/RPC instruments, see [Telemetry (OTLP)](/docs/manage/observability/telemetry.md). ## Pipeline overview ```text HTTP / WS / Executor → ExecutionObserver → FanOutObserver ├── MetricsObserver (labelled surrealdb.*) [C] ├── DsMetrics (surrealdb.ds.*) [E] ├── AuditObserver (audit → file + optional OTel) [E] └── SlowQueryObserver (slow-query → file + optional OTel) [E] │ SdkMeterProvider + SdkLoggerProvider │ ┌───────────────┼────────────────┐ ▼ ▼ ▼ Prometheus text OTLP metrics push OTLP logs / traces push exporter (/metrics) ``` `ExecutionObserver` short-circuits when no observer in the chain has work, so a process with neither `/metrics` nor OTLP configured avoids hot-path cost. Build flavour is carried on the OpenTelemetry **resource** as `service.edition` (`community` or `enterprise`), for example on `target_info` in Prometheus - filter dashboards on that attribute rather than on meter scope names. ## `/metrics` versus OTLP | Pipeline | Trigger | Transport | Authentication | What is exposed | | --- | --- | --- | --- | --- | | Prometheus pull | `SURREAL_METRICS_ENABLED=true` (default) | `GET /metrics` on the main HTTP port (`text/plain; version=0.0.4`) | Root credentials unlock the full surface. Anonymous scrapers receive only the **public allowlist** (aggregate process gauges). | Every instrument the running binary registers. | | OTLP push | `SURREAL_TELEMETRY_PROVIDER=otlp` | gRPC to `OTEL_EXPORTER_OTLP_ENDPOINT` (default `http://localhost:4317`). Push cadence: `OTEL_METRIC_EXPORT_INTERVAL` (ms, default `60000`). | TLS and client authentication are the collector’s responsibility. | Same metrics as an authenticated Prometheus view, plus traces and (when configured) audit and slow-query **log** records. | Disabling pieces: | Switch | Default | Effect | | --- | --- | --- | | `SURREAL_METRICS_ENABLED` | `true` | When `false`, `/metrics` is not mounted (404). | | `SURREAL_TELEMETRY_PROVIDER` | unset | Set to `otlp` to start OTLP exporters. | | `SURREAL_TELEMETRY_DISABLE_METRICS` | `false` | Skips only the OTLP **metrics** reader; logs and traces can still push. | | `SURREAL_TELEMETRY_DISABLE_TRACING` | `false` | Skips only the OTLP **trace** exporter. | ## Community versus Enterprise (at a glance) | Concern | Community | Enterprise | | --- | --- | --- | | Labelled `surrealdb.*` statement, query, transaction, RPC, HTTP, session, auth, network, live query, process, storage | Yes [C] | Yes [C] | | Slow-query **counter** (`surrealdb.slow_query.*`) vs threshold | `SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS` (default 1000 ms; `0` disables) | Same [C] | | `surrealdb.ds.*` cluster metrics | Not emitted | Emitted when the distributed storage runtime is active and a metrics reader exists [E] | | Durable **audit** / **slow-query** file sinks | Not available | `SURREAL_AUDIT_*`, `SURREAL_SLOW_QUERY_*` [E] | | Pipeline self-metrics (`surrealdb_audit_*`, `surrealdb_slow_query_*`) | Not registered | Registered when the matching pipeline is configured [E] | | OTel log export for audit / slow-query records | N/A (no enterprise pipelines) | Opt-in per pipeline: `SURREAL_AUDIT_OTEL_EXPORT`, `SURREAL_SLOW_QUERY_OTEL_EXPORT` [E] | | Per-tenant rollup scope `surrealdb.tenant` | Reserved for future low-cardinality rollups | Same; **no instruments are registered there yet**. Use proxy-side filtering on dimensional families for customer-visible views today. | ## Configuration highlights (Enterprise) ### Core switches ([C]) These apply to every build: | Variable | Default | Purpose | | --- | --- | --- | | `SURREAL_METRICS_ENABLED` | `true` | Mount `/metrics`. | | `SURREAL_TELEMETRY_PROVIDER` | unset | `otlp` enables OTLP push. | | `SURREAL_TELEMETRY_DISABLE_METRICS` | `false` | Skip OTLP metrics only. | | `SURREAL_TELEMETRY_DISABLE_TRACING` | `false` | Skip OTLP traces only. | | `SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS` | `1000` | Statements slower than this also increment the slow-query counter; `0` disables that counter. | | `SURREAL_PROCESS_METRICS_REFRESH_INTERVAL` | `5` | Seconds between process memory/CPU gauge refreshes (minimum 1). | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | OTLP gRPC endpoint. | | `OTEL_METRIC_EXPORT_INTERVAL` | `60000` | OTLP metric push interval (ms). | ### Audit log ([E]) | Variable | Default | Notes | | --- | --- | --- | | `SURREAL_AUDIT_SINK` | `none` | `none` or `file`. | | `SURREAL_AUDIT_FILE_PATH` | - | Required for `file`; parent directory must exist; file mode `0600` on Unix. | | `SURREAL_AUDIT_FSYNC_EVERY` | `0` | Mid-stream fsync cadence; rotation and graceful shutdown always flush. | | `SURREAL_AUDIT_HASH_CHAIN` | `false` | Tamper-evident chaining; **requires** `SURREAL_AUDIT_FSYNC_EVERY=1`. | | `SURREAL_AUDIT_INCLUDE_SQL` | `false` | Include full SQL on statement records. | | `SURREAL_AUDIT_OVERFLOW` | `block` | `drop` or `block` when the queue is full; compliance deployments usually keep `block`. | | `SURREAL_AUDIT_OTEL_EXPORT` | `false` | Also emit each record as an OTel log (opt-in). | ### Slow-query log ([E]) | Variable | Default | Notes | | --- | --- | --- | | `SURREAL_SLOW_QUERY_SINK` | `none` | Same pattern as audit. | | `SURREAL_SLOW_QUERY_THRESHOLD_MS` | - | Required when the sink is enabled. | | `SURREAL_SLOW_QUERY_INCLUDE_SQL` | `true` | SQL in slow-query records (turn off for throughput-sensitive deployments). | | `SURREAL_SLOW_QUERY_OVERFLOW` | `drop` | Defaults to **drop** (triage data); audit defaults to **block**. | | `SURREAL_SLOW_QUERY_OTEL_EXPORT` | `false` | OTel log export for slow-query records (opt-in). | Redaction (`SURREAL_AUDIT_REDACT_*` / `SURREAL_SLOW_QUERY_REDACT_*`) runs on the executor thread **before** the queue so the file sink and OTel path see the same scrubbed text. Full knob lists live with the rest of the server [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md). ### Cluster tuning ([E]) Large clusters expose saturation through `surrealdb.ds.*` and logs. Typical operator levers include inbound byte budgets, handler caps, read/write operation caps, and consensus timeouts. If you deploy on Kubernetes, pair metrics with the environment-variable reference in your platform runbook. ## Audit and slow-query records ([E]) Records flow in parallel: 1. **File sink** - optional NDJSON with rotation; optional hash chain for tamper evidence. 2. **OpenTelemetry logs** - only when the matching `SURREAL_*_OTEL_EXPORT` flag is `true`. Severity rules differ slightly between audit event kinds and slow-query (slow-query is always treated as a degraded path - typically **WARN**). Monitor **`surrealdb_audit_dropped`**, **`surrealdb_slow_query_dropped`**, **`surrealdb_audit_append_errors`**, and **`surrealdb_slow_query_append_errors`**: any sustained non-zero rate means **lost or failed** records. ## Recommended exposure tiers (operators) When you front `/metrics` with a proxy or expose metrics to customers: 1. **Always safe without authentication** - matches the built-in public allowlist (`surrealdb_build_info`, process gauges, `target_info`, `otel_scope_info`). 2. **Customer-visible** - only after the proxy **filters** dimensional series to that customer’s `namespace` / `database`. Strip `user` before showing outside trusted operator contexts. 3. **Operator-only** - `surrealdb.ds.*`, pipeline self-metrics, `surrealdb_auth_*`, storage backend detail, RPC method detail, and similar. Scrape these on a private interface with root credentials. ## Migration from older Prometheus names Legacy series names were collapsed into labelled families in SurrealDB 3.1. The same migration table applies to Community and Enterprise; see [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md) for the summary, [Telemetry (OTLP)](/docs/manage/observability/telemetry.md) for wire-format and deprecation notes, and the [Metrics reference](/docs/manage/observability/metrics.md#migration-from-30) for the full migration table. ## Local checks ```bash curl http://127.0.0.1:8000/metrics curl -u root:secret http://127.0.0.1:8000/metrics ``` Enterprise-only metrics (`surrealdb_ds_*`, audit/slow-query pipeline gauges) appear only in the authenticated view when those subsystems are active. ## Pipeline and configuration references - [Audit logging](/docs/manage/observability/audit-logging.md) - record shape, rotation, overflow semantics, hash chaining, redaction. - [Slow-query logging](/docs/manage/observability/slow-query-logging.md) - triage pipeline mirroring the audit observer. - [Configuration](/docs/manage/observability/configuration.md) - all `SURREAL_AUDIT_*` and `SURREAL_SLOW_QUERY_*` variables and the compliance checklist. --- Source: https://surrealdb.com/docs/manage/observability/logging # Logging Server log levels, text and JSON formats, file and socket output, slow-query logging, and OpenTelemetry log levels. SurrealDB writes server logs to stderr by default. You can set flags on [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) or the matching [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md) and ship lines to your collector or log store. ## Core options | Concern | CLI flag (examples) | Environment variable (examples) | | --- | --- | --- | | Level | `--log` | `SURREAL_LOG` | | Format | `--log-format` (`text` or `json`) | `SURREAL_LOG_FORMAT` | | Remote socket | `--log-socket` | `SURREAL_LOG_SOCKET` | | File logging | `--log-file-enabled`, `--log-file-path`, `--log-file-rotation`, … | `SURREAL_LOG_FILE_ENABLED`, `SURREAL_LOG_FILE_PATH`, `SURREAL_LOG_FILE_ROTATION`, … | | File level | `--log-file-level` | `SURREAL_LOG_FILE_LEVEL` | | Slow queries | `--slow-log-threshold`, `--slow-log-param-allow`, `--slow-log-param-deny` | `SURREAL_SLOW_QUERY_LOG_THRESHOLD`, `SURREAL_SLOW_QUERY_LOG_PARAM_ALLOW`, `SURREAL_SLOW_QUERY_LOG_PARAM_DENY` | Further socket and file options (`--log-socket-level`, `--log-file-name`, `--log-file-format`, and others) are documented on [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) and in [Environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md). Slow-query logging records statements that exceed your threshold. The `--slow-log-param-allow` / `--slow-log-param-deny` flags control whether bound parameters are expanded in the log line (for example turning `SELECT * FROM $table_name` into `SELECT * FROM person` when `table_name` is allowed), the use of which is recommended for any logs that may contain sensitive data. ## OpenTelemetry trace and log verbosity OpenTelemetry trace output can be tuned separately from ordinary server logs: * `--log-otel-level` or `SURREAL_LOG_OTEL_LEVEL` - verbosity of OTel-related tracing output. When OTLP is enabled, **SurrealDB Enterprise** can also emit audit and slow-query records as OpenTelemetry logs when you set `SURREAL_AUDIT_OTEL_EXPORT` / `SURREAL_SLOW_QUERY_OTEL_EXPORT`. Severity mapping and dual paths (file sink versus OTLP) are summarised on [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md), [Enterprise observability](/docs/manage/observability/enterprise-observability.md), and in the [Metrics reference](/docs/manage/observability/metrics.md). In-tree detail lives in [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) on GitHub. --- Source: https://surrealdb.com/docs/manage/observability/metrics # Metrics reference Access paths, label catalogue and the complete metric reference for SurrealDB Community and Enterprise, with a migration table from 3.0. This page is the canonical reference for every metric emitted by SurrealDB. It covers how operators access them, the labels and naming rules that govern the surface, the full catalogue grouped by signal family, the public allowlist for anonymous scrapers, and the migration table from 3.0 names. _(since v3.1.0)_ The instruments listed below were introduced as part of the 3.1 observability overhaul. The [Migration from 3.0](#migration-from-30) section at the bottom maps every legacy Prometheus name to its current replacement. ## Access paths Metrics can be exposed in two complementary ways. Both pipelines can run simultaneously. ### Prometheus pull | Property | Value | | --- | --- | | Trigger | `SURREAL_METRICS_ENABLED=true` (default) | | Transport | `GET /metrics` on the main HTTP port, content-type `text/plain; version=0.0.4` | | Authentication | Root credentials gate the full surface. Anonymous scrapers receive the [`PUBLIC_METRICS`](#public-metrics-allowlist) allowlist only. | | Surface | Every instrument the running build registers. | Disable the endpoint by setting `SURREAL_METRICS_ENABLED=false` - the route then returns `404`. ### OTLP push | Property | Value | | --- | --- | | Trigger | `SURREAL_TELEMETRY_PROVIDER=otlp` | | Transport | gRPC, default endpoint `http://localhost:4317`, push cadence from `OTEL_METRIC_EXPORT_INTERVAL` (milliseconds; default `60000`), cumulative temporality | | Authentication | The collector's responsibility. The exporter speaks gRPC and native-TLS but does not enforce client authentication. | | Surface | The same instruments as `/metrics`, plus the OTel logs surface (audit and slow-query records, when their OTel export is enabled) and `tracing` spans. | Selectively disable parts of the OTLP pipeline: - `SURREAL_TELEMETRY_DISABLE_METRICS=true` - suppress the OTLP metrics reader, leaving logs and traces unaffected. - `SURREAL_TELEMETRY_DISABLE_TRACING=true` - suppress the OTLP trace exporter, leaving metrics and logs unaffected. For the full environment variable reference, see [Configuration](/docs/manage/observability/configuration.md#core-knobs). ## Render rules OpenTelemetry instrument names use a dotted form (`surrealdb.statement.duration`). The Prometheus text exporter converts each instrument deterministically: - Counters get a `_total` suffix. - Histograms and counters with unit `s` get a `_seconds` suffix. - Histograms and counters with unit `By` get a `_bytes` suffix. - Up-down counters render as plain gauges with no suffix. - Attribute keys containing dots (`http.request.method`, `http.route`, `rpc.method`) are sanitised to underscore form at render time. A few worked examples:
OpenTelemetry name Unit Prometheus name
surrealdb.statement.duration s surrealdb_statement_duration_seconds
surrealdb.statement (counter) - surrealdb_statement_total
surrealdb.http.request.size By surrealdb_http_request_size_bytes_total
surrealdb.process.memory By surrealdb_process_memory_bytes
surrealdb.http.active_requests (up-down counter) - surrealdb_http_active_requests
## Common labels Every labelled family carries `outcome` plus the resolved tenant context (`namespace`, `database`, `user`) where applicable. Unresolved values render as the `"-"` sentinel; record-access principals render as the fixed `` sentinel.
Key Bounded values
outcome success, error, canceled
statement_type select, update, create, … (closed AST set)
namespace / database / user Free-form strings up to a user-configured length. "-" when unresolved; <record> for record-access principals.
protocol websocket, http
auth_action signin, signup, authenticate, …
auth_scope root, namespace, database, record, none
http_request_method get, post, put, delete, patch, head, options
http_route Matched router pattern, bounded by the router definitions. "-" for unmatched.
http_response_status_code Stringified status (200, 400, …)
rpc_method Bounded by the RPC dispatch table.
error_class Bounded; populated only when outcome != success.
The cluster family (`surrealdb.ds.*`) adds a small set of additional labels (`peer`, `kind`, `message_type`, `path`, `phase`, `role`, `result`, `source`, `reason`). See [Cluster labels](#cluster-labels) further down. ## Metric catalogue The catalogue is grouped by signal family. Each subsection lists the Prometheus name (the form an operator scrapes), the instrument type, the edition that exposes it, the labels it carries, and any notes. ### Process - `surrealdb.process` _(Community)_
Name Type Labels Notes
surrealdb_build_info gauge build_version Always 1. The value carries no signal; the label is the static compile-time version.
surrealdb_process_uptime_seconds gauge - Seconds since process start.
surrealdb_process_memory_bytes gauge - Resident set size of the process.
surrealdb_process_cpu_percent gauge - Aggregate process CPU %. May exceed 100 on multi-core hosts.
target_info gauge resource attributes Emitted by the OTel SDK from the process Resource. Carries service_edition.
otel_scope_info gauge scope attributes Emitted by the OTel SDK per instrumentation scope.
The cadence of the process snapshot can be tuned with `SURREAL_PROCESS_METRICS_REFRESH_INTERVAL` (seconds; default `5`, floored at 1). ### Statement - `surrealdb.statement` _(Community)_
Name Type Labels
surrealdb_statement_total counter statement_type, outcome, error_class, namespace, database, user
surrealdb_statement_duration_seconds histogram same as above
surrealdb_statement_rows_total counter same as above. Recorded only when result_rows > 0.
### Query - `surrealdb.query` _(Community)_
Name Type Labels
surrealdb_query_total counter outcome, error_class, namespace, database, user
surrealdb_query_duration_seconds histogram same as above
### Transaction - `surrealdb.transaction` _(Community)_
Name Type Labels
surrealdb_transaction_total counter write, outcome, error_class, namespace, database, user
surrealdb_transaction_duration_seconds histogram same as above
surrealdb_transaction_kv_ops_total counter op (get / scan / set / del / …)
surrealdb_transaction_keys_read_total counter outcome
surrealdb_transaction_keys_written_total counter outcome
surrealdb_transaction_key_bytes_read_total counter outcome
surrealdb_transaction_value_bytes_read_total counter outcome
surrealdb_transaction_key_bytes_written_total counter outcome
surrealdb_transaction_value_bytes_written_total counter outcome
surrealdb_transaction_conflicts_total counter namespace, database, user only. Doubles as the retry-pressure signal - alert on rate(...[5m]).
### RPC - `surrealdb.rpc` _(Community)_
Name Type Labels
surrealdb_rpc_total counter rpc_method, outcome, error_class, namespace, database, user
surrealdb_rpc_duration_seconds histogram same as above
### Auth - `surrealdb.auth` _(Community)_
Name Type Labels
surrealdb_auth_total counter auth_action, auth_scope, outcome, error_class, namespace, database, user
### Session - `surrealdb.session` _(Community)_
Name Type Labels
surrealdb_session_total counter session_action (connect / disconnect), protocol, service
surrealdb_session_active gauge protocol, service
surrealdb_session_duration_seconds histogram protocol, service
### Network - `surrealdb.network` _(Community)_
Name Type Labels
surrealdb_network_received_bytes_total counter protocol, namespace, database, user
surrealdb_network_sent_bytes_total counter same as above
### HTTP - `surrealdb.http` _(Community)_
Name Type Labels
surrealdb_http_request_total counter http_request_method, http_route, http_response_status_code, outcome, error_class, namespace, database, user
surrealdb_http_request_duration_seconds histogram same as above
surrealdb_http_request_size_bytes_total counter same as above
surrealdb_http_response_size_bytes_total counter same as above
surrealdb_http_active_requests gauge http_request_method, http_route (attribute-stripped to keep the gauge balanced)
### Live query - `surrealdb.live_query` _(Community)_ `surrealdb_live_query_orphaned_total` increments when a live-query registration cannot be torn down cleanly after a WebSocket disconnect, leaving a catalog row that still receives notifications. Alert on sustained growth.
Name Type Labels
surrealdb_live_query_active gauge -
surrealdb_live_query_notifications_total counter -
surrealdb_live_query_orphaned_total_(since v3.2.0)_ counter namespace, database
### Slow-query counter - `surrealdb.slow_query` _(Community)_
Name Type Labels
surrealdb_slow_query_total counter Same labels as surrealdb_statement_total. Increments when statement duration ≥ SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS (default 1000 ms; 0 disables the counter).
The slow-query counter is a Community-edition signal - it bumps for every statement that crosses the threshold. The [Enterprise slow-query log pipeline](/docs/manage/observability/slow-query-logging.md) produces full records on top of this counter. ### GraphQL - `surrealdb.graphql` _(Community)_
Name Type Labels
surrealdb_graphql_operation_total counter operation_type (query / mutation / subscription), outcome, error_class, namespace, database, user
surrealdb_graphql_operation_duration_seconds histogram same as above
### MCP - `surrealdb.mcp` _(Community)_
Name Type Labels
surrealdb_mcp_tool_invocation_total counter tool, transport, outcome, error_class, namespace, database, user
surrealdb_mcp_tool_duration_seconds histogram same as above
surrealdb_mcp_session_active gauge transport
### Storage backend - `surrealdb.storage.` _(Community)_ Each enabled backend publishes a manifest of `u64` values through the transaction layer. The server bridge registers each entry as an observable gauge under `surrealdb.storage..`; Prometheus renders them as `surrealdb_storage__`. The set is backend-specific (RocksDB, in-memory, SurrealKV) and may change between releases - the canonical list at runtime is whatever the running build exposes at `/metrics`. Treat these metrics as operator-only - they leak backend-specific implementation detail (compaction stats, block cache hits, etc.) that is rarely meaningful outside the operator persona. ### Cluster - `surrealdb.ds` _(Enterprise)_ _(since v3.1.0)_ > [!NOTE] > These instruments describe the distributed storage layer behind multi-node clusters, available on [SurrealDB Cloud Scale](https://surrealdb.com/pricing/scale) and in self-hosted Enterprise deployments. They apply when an Enterprise build runs that storage layer with at least one metrics reader configured. The cluster family is registered by the Enterprise composer when the runtime has at least one metrics reader configured. Every instrument is operator-only - the [`PUBLIC_METRICS`](#public-metrics-allowlist) allowlist excludes the family in its entirety. #### Network and transport | Name | Type | Labels | | --- | --- | --- | | `surrealdb_ds_network_bytes_sent_bytes_total` | counter (`By`) | `peer` | | `surrealdb_ds_network_bytes_received_bytes_total` | counter (`By`) | `peer` (omitted on reply-only variants) | | `surrealdb_ds_network_packets_sent_total` | counter | `peer` | | `surrealdb_ds_network_packets_received_total` | counter | `peer` (omitted on reply-only variants) | | `surrealdb_ds_network_send_errors_total` | counter | `peer, kind` | | `surrealdb_ds_messages_sent_total` | counter | `message_type, peer` | | `surrealdb_ds_messages_received_total` | counter | `message_type, peer` (omitted on reply-only variants) | | `surrealdb_ds_udp_reresolves_total` | counter | `reason` | #### Transactions and consensus | Name | Type | Labels | | --- | --- | --- | | `surrealdb_ds_transactions_committed_total` | counter | - | | `surrealdb_ds_transactions_aborted_total` | counter | `reason` | | `surrealdb_ds_occ_conflicts_total` | counter | - | | `surrealdb_ds_consensus_path_total` | counter | `path` | | `surrealdb_ds_consensus_fast_quorum_timeouts_total` | counter | - | | `surrealdb_ds_consensus_stale_view_aborts_total` | counter | - | | `surrealdb_ds_finalize_prepare_retries_total` | counter | - | | `surrealdb_ds_operation_timeouts_total` | counter | `phase` | | `surrealdb_ds_prepare_results_total` | counter | `role, result` | | `surrealdb_ds_begin_replies_total` | counter | `role, result` (currently coordinator-side only) | | `surrealdb_ds_reads_served_total` | counter | `source` | | `surrealdb_ds_reads_begin_backoffs_total` | counter | - | #### View management and recovery | Name | Type | Labels | | --- | --- | --- | | `surrealdb_ds_view_changes_total` | counter | `outcome` | | `surrealdb_ds_recovery_started_total` | counter | - | | `surrealdb_ds_recovery_completed_total` | counter | `outcome` | | `surrealdb_ds_recovery_transactions_applied_total` | counter | - | | `surrealdb_ds_prepare_probes_total` | counter | `outcome` | #### Garbage collection | Name | Type | Labels | | --- | --- | --- | | `surrealdb_ds_gc_supervisor_ticks_total` | counter | - | | `surrealdb_ds_gc_runs_total` | counter | `kind, outcome` | | `surrealdb_ds_gc_records_collected_total` | counter | `kind` | | `surrealdb_ds_gc_duration_seconds` | histogram | `kind, outcome` | #### Live state (observable gauges) Populated only after the replica is wired in via the enterprise observer install hook. | Name | Type | Notes | | --- | --- | --- | | `surrealdb_ds_replica_state` | gauge | Encoded `Normal=0`, `ViewChange=1`, `Recovering=2`, `RecoveringCompleted=3`. | | `surrealdb_ds_active_view_number` | gauge | Current TAPIR view number. | | `surrealdb_ds_prepared_list_len` | gauge | Length of the prepared list. | | `surrealdb_ds_records_len` | gauge | Length of the records list. | | `surrealdb_ds_pending_finalises_len` | gauge | Number of pending finalises. | | `surrealdb_ds_cluster_peers_unresolved` | gauge | Number of cluster peers currently unresolved. | #### Cluster labels | Key | Used on | Bounded values | | --- | --- | --- | | `peer` | `network_*`, `messages_*` | Stringified `NodeId` (interned). Cardinality bounded by cluster size. | | `kind` | `network_send_errors`, `gc_runs`, `gc_records_collected`, `gc_duration_seconds` | `send_failed`, `serialisation`, `channel_closed`, `unresolved`, `no_endpoint`, `write_failed`, `other` (send errors); `upgrade_tentative`, `gc_records`, `repair_tentative_consensus`, `gc_pending_finalises`, `gc_unlogged_inflight`, `gc_finalize_prepare_decisions`, `backup_recovery` (GC). | | `message_type` | `messages_sent`, `messages_received` | TAPIR message variants (closed set). | | `path` | `consensus_path` | `fast`, `slow` | | `phase` | `operation_timeouts` | `propose`, `finalize`, `broadcast`, `unlogged`, `recovery`, `inconsistent` | | `role` | `prepare_results`, `begin_replies` | `replica`, `coordinator` | | `result` | `prepare_results` | `ok`, `abort`, `abstain`, `retry` | | `result` | `begin_replies` | `eligible`, `anterior_prepared`, `higher_committed` | | `outcome` | `view_changes`, `recovery_completed`, `prepare_probes` | `triggered`, `completed`, `applied`, `retried`, `rebuild_failed` (view changes); `success`, `failure` (recovery); `committed`, `aborted`, `insufficient` (probes). | | `source` | `reads_served` | `local`, `remote_eligible`, `remote_higher_committed` | | `reason` | `transactions_aborted`, `udp_reresolves` | `conflict`, `abstain`, `other` (aborts); `unresolved`, `send_failure`, `recovery_view_change` (re-resolves). | ### Audit log pipeline self-metrics _(Enterprise)_ _(since v3.1.0)_ Observable gauges registered when the [audit log pipeline](/docs/manage/observability/audit-logging.md) is enabled. The values are pulled at scrape time from atomic counters on the pipeline, so the cost is zero when nothing consumes the metric. | Name | Type | Notes | | --- | --- | --- | | `surrealdb_audit_records` | gauge | Cumulative count of records successfully enqueued (observer → queue). | | `surrealdb_audit_dropped` | gauge | Cumulative count of records dropped (overflow or queue closed). **Alert on any non-zero rate** - every drop is a lost record. | | `surrealdb_audit_queue_depth` | gauge | Records currently buffered between observer and worker. Sustained depth above ~50% of `SURREAL_AUDIT_QUEUE_CAPACITY` indicates a slow sink. | | `surrealdb_audit_appended` | gauge | Cumulative records the worker wrote to the sink. The gap to `surrealdb_audit_records` is queue depth plus append errors. | | `surrealdb_audit_append_errors` | gauge | Cumulative sink-write failures. **Alert on any non-zero rate**. | > These are exposed as gauges, not counters - the values are pulled live from atomic counters on the pipeline. The Prometheus name therefore has no `_total` suffix even though the underlying values are monotonic. ### Slow-query log pipeline self-metrics _(Enterprise)_ _(since v3.1.0)_ The slow-query pipeline exposes the same shape as the audit pipeline under the `surrealdb.slow_query` scope: | Name | Type | Notes | | --- | --- | --- | | `surrealdb_slow_query_records` | gauge | Cumulative records enqueued. | | `surrealdb_slow_query_dropped` | gauge | Cumulative records dropped. **Alert on any non-zero rate**. | | `surrealdb_slow_query_queue_depth` | gauge | Current queue depth. | | `surrealdb_slow_query_appended` | gauge | Cumulative records written to the sink. | | `surrealdb_slow_query_append_errors` | gauge | Cumulative sink-write failures. **Alert on any non-zero rate**. | ### Per-tenant rollups - `surrealdb.tenant` _(Enterprise)_ The `surrealdb.tenant` meter scope is **reserved** for low-cardinality per-tenant rollups keyed on `(namespace, database)` only. No instruments are currently registered under it. Until rollups land, customer-tenant filtering must be done at a proxy by filtering the higher-cardinality labelled families on `namespace` / `database`. ## Public metrics allowlist Anonymous scrapers on `/metrics` see only the metrics on the public allowlist. Adding a metric to this list requires a security review. The six allowlisted metrics are: - `surrealdb_build_info` - `surrealdb_process_uptime_seconds` - `surrealdb_process_memory_bytes` - `surrealdb_process_cpu_percent` - `target_info` - `otel_scope_info` Root credentials unlock the full surface for an authenticated scrape. ### Recommended cloud whitelist For cloud and multi-tenant deployments the recommendation is a three-tier whitelist: - **Tier 1 - Always public.** The allowlisted six above. - **Tier 2 - Customer-visible (per-tenant, via proxy filter).** A subset of labelled families exposed to customers showing their own usage. Every metric in this tier carries `namespace` and `database` labels and the cloud proxy MUST filter on those before responding to a customer request - there is no server-side mechanism today that restricts a labelled family to a single tenant. Strip the `user` label before exposing. Suggested subset: `surrealdb_statement_*`, `surrealdb_transaction_total`, `surrealdb_transaction_duration_seconds`, `surrealdb_transaction_conflicts_total`, `surrealdb_query_*`, `surrealdb_http_request_total`, `surrealdb_network_received_bytes_total`, `surrealdb_network_sent_bytes_total`, `surrealdb_live_query_active`, `surrealdb_slow_query_total`. - **Tier 3 - Operator-only.** Everything else: all `surrealdb_ds_*`, all pipeline self-metrics, `surrealdb_rpc_*` (full method dimension), `surrealdb_auth_*` (failure rates can be probed by an attacker), `surrealdb_graphql_*`, `surrealdb_mcp_*`, `surrealdb_storage_*`, `surrealdb_http_active_requests` (real-time concurrency snapshot), and the KV-layer `surrealdb_transaction_*` counters. Reach these over an internal sidecar or private network with root credentials. ## Cardinality and sensitivity | Flag | Where it applies | Mitigation | | --- | --- | --- | | **Tenant-identifying** | `namespace`, `database`, `user` on every labelled family | Strip or filter at the proxy before exposing to a customer. Anonymous scrapers cannot see any family carrying these labels because the families are absent from the public allowlist. | | **High-cardinality** | `peer` (cluster family, bounded by cluster size), `http_route`, `rpc_method` | Fine internally. External exposure should be conditional on the route or method set already being public. | | **Workload inference** | All process gauges (uptime, memory, CPU) | Already on the allowlist, but in a single-tenant-per-pod deployment they correlate with tenant workload. Acceptable trade-off in most architectures. | | **PII risk** | Audit and slow-query *records* carry SQL when `*_INCLUDE_SQL=true`. The pipeline self-metrics are safe; the records themselves go to the file sink and (opt-in) OTLP logs, not to `/metrics`. | Run the [three-pass redactor](/docs/manage/observability/audit-logging.md#redaction); alert on `*_append_errors`. | | **Default-deny** | Multi-tenant deployments | Set `SURREAL_METRICS_ENABLED=false` on customer-facing pods; serve scrapes from an internal sidecar bound to a non-tenant-visible interface. | ## Alert hints A starter set for production: - `rate(surrealdb_statement_total{outcome="error"}[5m])` rising sharply against a baseline - application or operator error spike. - `rate(surrealdb_transaction_conflicts_total[5m])` rising - OCC pressure or retry storms. - `rate(surrealdb_ds_consensus_fast_quorum_timeouts_total[5m]) > 0` - consensus is falling through to the slow path. - `surrealdb_ds_replica_state != 0` for more than a few seconds - a replica is in view-change or recovery. - `surrealdb_ds_cluster_peers_unresolved > 0` for more than a startup window - DNS or connectivity gap. - `rate(surrealdb_audit_dropped[5m]) > 0` or `rate(surrealdb_audit_append_errors[5m]) > 0` - audit records being lost. - `surrealdb_audit_queue_depth` sustained above ~50% of `SURREAL_AUDIT_QUEUE_CAPACITY` - sink is falling behind. ## Migration from 3.0 The 3.1 release renamed the metric surface to a consistent `surrealdb.*` namespace. Legacy Prometheus names that operators may still see in old dashboards, and their current replacements:
3.0 name 3.1 replacement
surrealdb_statements_completed_total surrealdb_statement_total{outcome="success"}
surrealdb_statement_errors_total surrealdb_statement_total{outcome="error"}
surrealdb_queries_completed_total surrealdb_query_total{outcome="success"}
surrealdb_query_errors_total surrealdb_query_total{outcome="error"}
surrealdb_query_dim_duration_seconds surrealdb_query_duration_seconds
surrealdb_transactions_completed_total surrealdb_transaction_total{outcome="success"}
surrealdb_transaction_writes_total surrealdb_transaction_total{write="true",outcome="success"}
surrealdb_transaction_errors_total surrealdb_transaction_total{outcome="error"}
surrealdb_rpcs_completed_total surrealdb_rpc_total{outcome="success"}
surrealdb_rpc_errors_total surrealdb_rpc_total{outcome="error"}
surrealdb_auth_attempts_total surrealdb_auth_total
surrealdb_auth_failures_total surrealdb_auth_total{outcome!="success"}
surrealdb_http_requests_completed_total surrealdb_http_request_total{outcome="success"}
surrealdb_http_request_errors_total surrealdb_http_request_total{outcome="error"}
surrealdb_http_dim_active_requests surrealdb_http_active_requests
surrealdb_network_dim_received_bytes_total surrealdb_network_received_bytes_total
surrealdb_network_dim_sent_bytes_total surrealdb_network_sent_bytes_total
http_server_request_duration_milliseconds surrealdb_http_request_duration_seconds
http_server_request_count_total surrealdb_http_request_total
http_server_active_requests surrealdb_http_active_requests
rpc_server_request_duration_milliseconds surrealdb_rpc_duration_seconds
rpc_server_active_connections surrealdb_session_active{protocol="websocket"}
The `otel_scope_name` label also changes shape in 3.1: scopes are now signal-domain (`surrealdb.statement`, `surrealdb.query`, …) rather than edition-tier (`surrealdb.community`, `surrealdb.enterprise`). Dashboards that filtered by scope should switch to the metric name (which now carries the same information, since scopes mirror the family prefix) or to the `service.edition` resource attribute, which is the authoritative edition signal. --- Source: https://surrealdb.com/docs/manage/observability/observability # Observability (metrics and Prometheus) Unified OpenTelemetry metrics common to Community and Enterprise. The `GET /metrics` endpoint, scraper authentication, naming, and migration. For server log output (stderr, files, JSON, sockets, slow-query lines), see [Logging](/docs/manage/observability/logging.md). Use the tabs below depending on your server version. For OTLP (push), process intervals, and legacy SemConv instruments on the wire, see [Telemetry (OTLP)](/docs/manage/observability/telemetry.md). For cluster metrics, audit and slow-query file sinks, and OTLP log export opt-ins, see [Enterprise observability](/docs/manage/observability/enterprise-observability.md). **Before SurrealDB 3.1** Prior to SurrealDB 3.1, operators often saw overlapping metric paths: built-in Prometheus exposition could run alongside a separate OpenTelemetry pipeline oriented around HTTP/RPC semantic conventions. The same request could therefore be reflected more than once, under different scopes and label keys (for example `http.server.*` / `rpc.server.*` versus `surrealdb.*`-style families). Many deployments also emitted parallel families for the same logical signal - aggregate counters or histograms plus higher-cardinality “dimensional” variants (names with a `*_dim_*` segment, or separate `*_errors_total` series). Dashboards and alerts had to choose which family to query, and mixing them could double-count. Edition (`community` vs `enterprise`) could appear on meter scope identifiers (for example in `otel_scope_name` after export to Prometheus). That tied dashboards to build flavour at the scope level and made renames risky. Tenant context (namespace, database, user, and similar) could be attached to labels in more ad hoc ways across recording sites, so you should treat unauthenticated `/metrics` scrapes as potentially sensitive in multi-tenant layouts unless you had a deliberate proxy or allowlist in front of the endpoint. For the OTLP side of the same era (push interval, `http.server.*` / `rpc.server.*` instruments, deprecated variables), use the Before SurrealDB 3.1 tab on [Telemetry (OTLP)](/docs/manage/observability/telemetry.md). When you upgrade to 3.1 or later, replan dashboards and alerts using the migration material in [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) in the open-source repository (and the Enterprise companion, if you run SurrealDB Enterprise). **SurrealDB 3.1 and later** SurrealDB exposes a single, unified observability surface backed by OpenTelemetry. Every metric is recorded once and fanned out to whichever exporters you configure: - **Prometheus (pull)** - text exposition at `GET /metrics` (scrapers call your server). The usual path if you already run Prometheus. - **OTLP (push)** - when `SURREAL_TELEMETRY_PROVIDER=otlp` is set, the process pushes metrics, traces, and (when enabled) log records to a collector. **SurrealDB Enterprise** can additionally attach durable **audit** and **slow-query** pipelines and **cluster** metrics when distributed storage is the backend (Cloud Scale or self-hosted Enterprise); OTLP export of audit and slow-query **log** records is **opt-in** per pipeline. See [Enterprise observability](/docs/manage/observability/enterprise-observability.md). For OTLP configuration, push intervals, process gauges, and backward-compatible HTTP/RPC instruments, see [Telemetry (OTLP)](/docs/manage/observability/telemetry.md). To inspect async tasks and scheduling inside the Tokio runtime, see [Tokio console](/docs/manage/observability/tokio-console.md). ## Unified model (summary) ```text HTTP / WS / Executor → ExecutionObserver → FanOutObserver ├── MetricsObserver (labelled surrealdb.*) ├── DsMetrics (Enterprise, when active) ├── AuditObserver (Enterprise, when configured) └── SlowQueryObserver (Enterprise, when configured) │ ▼ SdkMeterProvider + SdkLoggerProvider │ ┌───────────────┼────────────────┐ ▼ ▼ ▼ Prometheus text OTLP metrics push OTLP logs / traces push exporter (/metrics) ``` Metrics are recorded once in labelled form. A single `surrealdb.statement` counter, for example, carries `statement_type`, `outcome`, `namespace`, `database`, and `user` in **Community and Enterprise**; unresolved tenant context appears as a `"-"` sentinel so series stay stable. Build flavour (`community` vs `enterprise`) is exposed on the OpenTelemetry **resource** as `service.edition` (for example on `target_info` in Prometheus) - use that attribute for edition-aware dashboards instead of filtering on meter scope alone. When no exporter is configured, the fan-out short-circuits so the hot path stays cheap. ## Naming (summary) Instruments use dotted OpenTelemetry-style names (for example `surrealdb.statement.duration`) under signal-domain scopes (`surrealdb.statement`, `surrealdb.query`, …). The Prometheus text exporter maps those to underscore names with conventional `_total`, `_seconds`, and `_bytes` suffixes. Attribute keys with dots are sanitised for Prometheus. Many historical Prometheus series names and label shapes changed in SurrealDB 3.1. Dashboards that pre-date that release should be updated using the migration table in [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) (and the Enterprise companion for **[E]**-only series). ## Metrics on or off | Switch | Default | Effect | | --- | --- | --- | | `SURREAL_METRICS_ENABLED` | `true` | When `false`, the `/metrics` route is not mounted | | `SURREAL_TELEMETRY_PROVIDER` | unset | Set to `otlp` to enable the OTLP push pipeline (metrics, logs, traces) | | `SURREAL_TELEMETRY_DISABLE_METRICS` | `false` | When `true`, suppresses the OTLP metrics reader (logs and traces unaffected) | | `SURREAL_TELEMETRY_DISABLE_TRACING` | `false` | When `true`, suppresses the OTLP trace exporter only | When metrics are disabled, `/metrics` returns 404. ## Collecting `/metrics` and security Anonymous scrapes only receive families on a public allowlist (aggregate process signals). Root credentials unlock the full Prometheus surface for that process; namespace and database users are not treated as authenticated for this endpoint. Dimensional families such as `surrealdb_statement_total` are intentionally excluded from anonymous output because they carry tenant labels - use OTLP or an authenticated scrape for those. See [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) for the `PUBLIC_METRICS` list and security discussion. ## Audit and slow-query (summary) **Community** builds do not ship the enterprise audit and slow-query file pipelines or their pipeline self-metrics on Prometheus. **Enterprise** can write tamper-aware NDJSON to disk **and** optionally emit the same redacted records as OpenTelemetry logs when `SURREAL_AUDIT_OTEL_EXPORT=true` / `SURREAL_SLOW_QUERY_OTEL_EXPORT=true` (both default `false`). Redaction runs on the executor thread before the queue so every sink sees the same scrubbed payload. See [Enterprise observability](/docs/manage/observability/enterprise-observability.md) and `SURREAL_AUDIT_*` / `SURREAL_SLOW_QUERY_*` in [Environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md). ## Multi-tenant guidance Treat `/metrics` as operator-only in multi-tenant deployments: avoid exposing the main listener to untrusted scrapers, prefer disabling metrics if you cannot isolate the endpoint, and never give tenants root credentials used for the authenticated metric view. ## Local development Compare anonymous and authenticated scrapes: ```bash curl http://127.0.0.1:8000/metrics curl -u root:secret http://127.0.0.1:8000/metrics ``` Disable metrics entirely: ```bash SURREAL_METRICS_ENABLED=false surreal start ... ``` Push metrics, logs, and traces to a local collector: ```bash SURREAL_TELEMETRY_PROVIDER=otlp \ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \ surreal start ... ``` For OTLP-focused setup and version tabs, see [Telemetry (OTLP)](/docs/manage/observability/telemetry.md). For catalogue-level detail, use the in-repository `doc/OBSERVABILITY.md` / `doc/TELEMETRY.md` files linked above (and the Enterprise `doc/OBSERVABILITY.md` where applicable). For every instrument name, label, and alert hint on this site, see the [Metrics reference](/docs/manage/observability/metrics.md). --- Source: https://surrealdb.com/docs/manage/observability/slow-query-logging # Slow-query logging The Enterprise slow-query log pipeline: how a query qualifies, record shape, rotation, hash chaining, redaction and pipeline self-metrics. _(Enterprise)_ _(since v3.1.0)_ The slow-query log captures the full statement text and execution context for queries that cross a configurable duration threshold. It is part of SurrealDB Enterprise and is **off by default** - set `SURREAL_SLOW_QUERY_SINK=file`, `SURREAL_SLOW_QUERY_FILE_PATH` and `SURREAL_SLOW_QUERY_THRESHOLD_MS` to enable it. The slow-query pipeline is architecturally identical to the [audit log pipeline](/docs/manage/observability/audit-logging.md) - same observer, same queue, same worker, same NDJSON file format, same optional OTel logs export, same hash chaining, same three-pass redaction. The intentional symmetry keeps operator runbooks short. The two pipelines differ in defaults and intent: - **Audit records are compliance data.** Default overflow policy is `block` (preserve at the cost of latency); typical configurations also enable hash chaining. - **Slow-query records are triage data.** Default overflow policy is `drop` (prefer dropping over busy-yielding the executor on a hot path). Hash chaining is rarely needed. The full set of configuration variables lives on the [configuration reference](/docs/manage/observability/configuration.md#slow-query-log-knobs). ## Relationship to the slow-query counter Two related instruments cover slow queries: - The `surrealdb_slow_query_total` counter (Community edition; see [metrics reference](/docs/manage/observability/metrics.md#slow-query-counter---surrealdbslow_query)) increments for every statement above `SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS` (default 1000 ms). It is cheap and always-on when metrics are enabled. - The slow-query log records on this page capture the full statement text and context for queries above `SURREAL_SLOW_QUERY_THRESHOLD_MS` (no default - required when the sink is enabled). This is opt-in and writes to a file sink. The two thresholds are independent. Most deployments set the metric threshold at a high value (the counter is a sanity-check signal) and the log threshold lower (to capture the long tail of triage candidates). ## How a query qualifies A statement is captured when its wall-clock duration is `≥ SURREAL_SLOW_QUERY_THRESHOLD_MS`. The pipeline only registers an observer when the sink is enabled, so the cost of slow-query capture is zero when `SURREAL_SLOW_QUERY_SINK=none` (the default). ## Record shape Slow-query records share the audit envelope (`ts`, `event_type`, `outcome`, `duration_ms`, identity context, optional `sql`, optional hash chain fields) and additionally carry: - `statement_type` - the statement category (`select`, `update`, `create`, …). - `threshold_ms` - the threshold the query crossed, for forensic context. - `result_rows` - number of rows produced (when applicable). The `event_type` field is always `slow_query`. A captured record looks like: ```json { "ts": "2026-03-04T10:23:11.482Z", "event_type": "slow_query", "outcome": "success", "statement_type": "select", "duration_ms": 512, "threshold_ms": 250, "result_rows": 1842, "namespace": "acme", "database": "prod", "user": "svc_orders", "sql": "SELECT * FROM orders WHERE status = 'shipped' FETCH customer" } ``` The `sql` field is included by default (`SURREAL_SLOW_QUERY_INCLUDE_SQL=true`). Throughput-sensitive deployments can set it to `false` to skip the per-statement `to_sql()` rendering, at the cost of losing SQL context on captures. ## OpenTelemetry logs export When `SURREAL_SLOW_QUERY_OTEL_EXPORT=true` each captured record is also emitted as an OTel `LogRecord` on the SDK logger provider: - **Event name:** `surrealdb.slow_query.statement`. - **Severity:** `WARN` (fixed - slow path is degraded by definition). - **Body:** a short human-readable string, for example `"slow select 500ms (threshold 100ms)"`. - **Attributes:** the structured fields from the record envelope (`db.namespace`, `db.name`, `db.user`, `db.statement`, `surrealdb.statement_type`, `surrealdb.outcome`, `surrealdb.duration_ms`, `surrealdb.threshold_ms`, `surrealdb.result_rows`). Like the audit pipeline, OTel export is off by default. The file sink is the primary path. ## Rotation and durability Same knobs as the audit pipeline, prefixed `SURREAL_SLOW_QUERY_*`: - File mode `0600` on Unix. The parent directory must exist or startup fails. - Size-based rotation at `SURREAL_SLOW_QUERY_FILE_ROTATE_BYTES` (default 256 MiB). - `SURREAL_SLOW_QUERY_FILE_ROTATE_KEEP` rotation generations retained (default `8`). - Mid-stream fsync cadence from `SURREAL_SLOW_QUERY_FSYNC_EVERY` (default `0`). Rotation and graceful shutdown always fsync regardless. ## Hash chaining Identical to the audit pipeline. `SURREAL_SLOW_QUERY_HASH_CHAIN=true` adds `prev_hash` / `hash` SHA-256 fields and requires `SURREAL_SLOW_QUERY_FSYNC_EVERY=1`. Most operators leave hash chaining off for slow-query data - the pipeline is triage tooling, not a compliance record, and the per-record fsync cost is rarely worth paying for a `drop`-default pipeline. See the audit log [Hash chaining](/docs/manage/observability/audit-logging.md#hash-chaining) section for the full semantics. ## Redaction Identical to the audit pipeline, under `SURREAL_SLOW_QUERY_REDACT_LITERALS`, `SURREAL_SLOW_QUERY_REDACT_TABLES` and `SURREAL_SLOW_QUERY_REDACT_REGEX`. Three layered passes run synchronously on the executor thread before the record reaches the queue. See [Redaction](/docs/manage/observability/audit-logging.md#redaction) on the audit log page for the full mechanics. ## Overflow semantics The slow-query pipeline defaults to `drop` - a record that cannot be enqueued is discarded and the `surrealdb_slow_query_dropped` gauge increments. The reasoning is that the executor is on a hot path during a slow query (by definition), and busy-yielding it to preserve a triage record is a poor trade-off. Operators who need lossless slow-query capture can set `SURREAL_SLOW_QUERY_OVERFLOW=block`. The trade-off is the same as on the audit pipeline - the policy uses a bounded busy-yield loop, not a wall-clock time-bound, and offers no lossless guarantee on a `current_thread` runtime. Whichever policy is configured, alert on `rate(surrealdb_slow_query_dropped[5m]) > 0` and `rate(surrealdb_slow_query_append_errors[5m]) > 0` - both indicate records were lost. ## Pipeline self-metrics Five observable gauges expose the live state of the pipeline, mirroring the audit equivalents: | Metric | Notes | | --- | --- | | `surrealdb_slow_query_records` | Cumulative records successfully enqueued. | | `surrealdb_slow_query_dropped` | Cumulative records dropped. **Alert on any non-zero rate.** | | `surrealdb_slow_query_queue_depth` | Records currently buffered between observer and worker. | | `surrealdb_slow_query_appended` | Cumulative records the worker wrote to the sink. | | `surrealdb_slow_query_append_errors` | Cumulative sink-write failures. **Alert on any non-zero rate.** | ## Related references - [Configuration → Slow-query log knobs](/docs/manage/observability/configuration.md#slow-query-log-knobs) - every slow-query environment variable. - [Audit logging](/docs/manage/observability/audit-logging.md) - the sister pipeline for compliance data. - [Metrics reference → Slow-query counter](/docs/manage/observability/metrics.md#slow-query-counter---surrealdbslow_query) - the Community-edition counter that bumps when a statement crosses `SURREAL_SLOW_QUERY_METRIC_THRESHOLD_MS`. - [Metrics reference → Slow-query pipeline self-metrics](/docs/manage/observability/metrics.md#slow-query-log-pipeline-self-metrics) - the five gauges in the metric catalogue. --- Source: https://surrealdb.com/docs/manage/observability/telemetry # Telemetry (OTLP) OTLP push export of metrics, logs, and traces. Intervals, backward-compatible instruments, and differences before and after SurrealDB 3.1. This page covers push export using the OpenTelemetry Protocol (OTLP): your server sends signals to a collector. For pull scraping of `GET /metrics` (Prometheus), see [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md). **Before SurrealDB 3.1** SurrealDB can be monitored by enabling built-in observability. ## Enable observability To enable observability, the `SURREAL_TELEMETRY_PROVIDER` environment variable has to be set to `otlp`. If set to anything else, no observability will be available. If enabled, SurrealDB sends metrics and/or traces to an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). Configuration of the collector follows the [OpenTelemetry environment variable conventions](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/). The most important variable is [`OTEL_EXPORTER_OTLP_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_endpoint). By default this points at localhost. Set it to the gRPC endpoint of your collector. For example, if your collector named `my-collector` runs in Kubernetes in the `monitoring` namespace: ```text OTEL_EXPORTER_OTLP_ENDPOINT="http://my-collector.monitoring.svc.cluster.local:4317" ``` Metrics can be disabled (even if `SURREAL_TELEMETRY_PROVIDER` is set to `otlp`) by setting `SURREAL_TELEMETRY_DISABLE_METRICS` to `true`. Similarly, traces can be disabled with `SURREAL_TELEMETRY_DISABLE_TRACING=true`. ## Metrics Metrics are gathered every minute and sent to the collector. The following metrics are present:
Name [Instrument](https://opentelemetry.io/docs/concepts/signals/metrics/#metric-instruments) Explanation
rpc.server.duration histogram Measures duration of inbound RPC requests in milliseconds
rpc.server.active_connections counter The number of active WebSocket connections
rpc.server.response.size histogram Measures the size of HTTP response messages
http.server.duration histogram The HTTP server duration in milliseconds
http.server.active_requests counter The number of active HTTP requests
http.server.request.size histogram Measures the size of HTTP request messages
http.server.response.size histogram Measures the size of HTTP response messages
The metrics are shown here in the form required by the [OpenTelemetry Metrics Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/general/metrics/) with a `.` separator. When ingested into Prometheus the `.` separator is [replaced](https://prometheus.io/blog/2024/03/14/commitment-to-opentelemetry/#support-utf-8-metric-and-label-names) with `_`. For example `rpc.server.active_connections` becomes `rpc_server_active_connections`. For async runtime introspection (tasks, polls, scheduling), you can use the [Tokio console](/docs/manage/observability/tokio-console.md) which is separate from OTLP. For server logging (levels, JSON, files, sockets, slow-query logging), see [Logging](/docs/manage/observability/logging.md). **SurrealDB 3.1 and later** From SurrealDB 3.1, OpenTelemetry is the single source of truth for metrics and, where configured, for log records emitted on the shared logger provider - including audit and slow-query events on **SurrealDB Enterprise** when you opt in with `SURREAL_AUDIT_OTEL_EXPORT` / `SURREAL_SLOW_QUERY_OTEL_EXPORT`. One meter provider and one logger provider route measurements to multiple exporters, including the Prometheus text exposition at [`GET /metrics`](/docs/manage/observability/observability.md). An OTLP subscriber receives the same **metric** surface that a fully authenticated Prometheus scrape sees, plus traces and any **log** signals you have enabled (subject to the `/metrics` allowlist only for the pull path, not for OTLP). For Prometheus pull - allowlists, naming, migration from older series, multi-tenant guidance, and Community versus Enterprise context - see [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md) and [Enterprise observability](/docs/manage/observability/enterprise-observability.md). ## What flows over OTLP | Signal | Source | Carries | | --- | --- | --- | | Metrics | Unified meter provider | Every instrument also surfaced on `/metrics`, plus the legacy `http.server.*` / `rpc.server.*` instruments kept for backward compatibility. | | Logs | Unified logger provider | **Enterprise:** audit (`surrealdb.audit`) and slow-query (`surrealdb.slow_query`) records when the file pipeline is enabled **and** the matching `SURREAL_*_OTEL_EXPORT` flag is `true`. Severity mapping is summarised on [Observability (metrics and Prometheus)](/docs/manage/observability/observability.md) and [Enterprise observability](/docs/manage/observability/enterprise-observability.md). | | Traces | `tracing` → OpenTelemetry bridge | Instrumented spans; enterprise builds may enrich spans with tenant attributes when enabled. | Configure the OTLP push pipeline with: ```text SURREAL_TELEMETRY_PROVIDER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 # gRPC by default ``` When `SURREAL_TELEMETRY_PROVIDER` is unset (or any value other than `otlp`), OTLP exporters are not built; metrics still flow to Prometheus when `/metrics` is enabled. **SurrealDB Enterprise** audit and slow-query **file** sinks are independent of OTLP - they keep writing whenever configured. When `SURREAL_TELEMETRY_DISABLE_METRICS=true`, the OTLP metrics reader is skipped while logs and traces continue to push. When `SURREAL_TELEMETRY_DISABLE_TRACING=true`, the OTLP trace exporter is skipped while metrics and logs continue to push. ## Push interval Metric push frequency follows the OpenTelemetry specification and is controlled by `OTEL_METRIC_EXPORT_INTERVAL` (milliseconds; default `60000`). The SDK reads this at startup and applies it to the periodic reader behind OTLP metrics. Sub-minute intervals (10-15 s) are common when you want responsive dashboards. ## Process metric freshness `surrealdb.process.memory` and `surrealdb.process.cpu_percent` are observable gauges backed by a process-wide cache. A background task refreshes the cache on a fixed cadence so values stay fresh whether scrapers hit `/metrics` or OTLP pushes on its own schedule. | Variable | Default | Notes | | --- | --- | --- | | `SURREAL_PROCESS_METRICS_REFRESH_INTERVAL` | `5` | Refresh interval in seconds. Tighter intervals reduce staleness but can make `cpu_percent` noisier (it is computed as a delta since the last refresh). Floored at one second. | The task runs only when at least one metrics reader is configured (Prometheus and/or OTLP). OTLP-only deployments get the same freshness guarantee as Prometheus scrapers. ## Histogram bucket views The meter provider applies three default views: - Instruments named `*.duration` with unit seconds use a quasi-exponential bucket family from 5 ms to 30 s. - Instruments named `*.duration` with unit milliseconds use a parallel millisecond-scale family for the legacy HTTP/RPC pipeline. - Instruments named `*.size` with unit bytes use a 1 KiB - 100 MiB byte family. Operators with custom views can override these before building the provider; refer to the OpenTelemetry SDK documentation. ## Backward-compatibility instrument set Legacy OpenTelemetry HTTP/RPC instruments are still recorded under separate meter scopes (`surrealdb.http`, `surrealdb.rpc`). They coexist with the new `surrealdb.*` families so existing OTLP dashboards that pivot on semantic-convention names keep working. | Instrument | Kind | Unit | Labels | | --- | --- | --- | --- | | `http.server.active_requests` | UpDownCounter\ | - | `http.request.method`, `http.route`, `network.protocol.{name,version}` | | `http.server.request.count` | Counter\ | - | as above + `http.response.status_code` | | `http.server.request.duration` | Histogram\ | `ms` | as `http.server.request.count` | | `http.server.request.size` | Histogram\ | `By` | as `http.server.request.count` | | `http.server.response.size` | Histogram\ | `By` | as `http.server.request.count` | | `rpc.server.active_connections` | UpDownCounter\ | - | `rpc.service` | | `rpc.server.connection.count` | Counter\ | - | `rpc.service` (incremented on connect only) | | `rpc.server.request.duration` | Histogram\ | `ms` | `rpc.service`, `rpc.method`, `rpc.error` | | `rpc.server.request.size` | Histogram\ | `By` | `rpc.service` (per WebSocket frame) | | `rpc.server.response.size` | Histogram\ | `By` | `rpc.service` (per WebSocket frame) | ## Deprecated environment variables The following variables are still parsed for backwards compatibility but are no longer applied. The server logs a deprecation warning at startup if either is set: - `SURREAL_TELEMETRY_NAMESPACE` - the `namespace` attribute was removed from telemetry metrics because it identifies tenants in multi-tenant deployments. - `SURREAL_TELEMETRY_RPC_LIVE_ID` - per-notification OTLP attribution by `rpc.live_id` was removed when WebSocket telemetry was unified into the execution observer pipeline. ## Local development From the repository root, start the observability stack under `dev/docker` (collector, Grafana, Prometheus, Tempo, and Loki): ```bash docker compose -f dev/docker/compose.yaml up -d SURREAL_TELEMETRY_PROVIDER=otlp OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" surreal start ``` Open `http://localhost:3000` in a browser; default Grafana credentials are `admin` / `admin`. For OpenTelemetry log and trace verbosity, file output, sockets, and other logging flags used alongside this stack, see [Logging](/docs/manage/observability/logging.md) and [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md). For async runtime debugging with the Tokio console (tasks, poll histograms), see [Tokio console](/docs/manage/observability/tokio-console.md). That workflow sits beside OTLP and is not version-split in the tabs above. For diagrams, histogram bucket rules, deprecated variables, and the full OTLP signal matrix in prose form, see [`doc/TELEMETRY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/TELEMETRY.md) in the open-source repository alongside [`doc/OBSERVABILITY.md`](https://github.com/surrealdb/surrealdb/blob/main/doc/OBSERVABILITY.md) for `/metrics`, allowlists, and metric catalogues. The Enterprise distribution ships an extended `doc/OBSERVABILITY.md` with **[C]** / **[E]** markers - summarised on [Enterprise observability](/docs/manage/observability/enterprise-observability.md). --- Source: https://surrealdb.com/docs/manage/observability/tokio-console # Tokio console Inspect SurrealDB’s async runtime with the Tokio console - tasks, polls, and bottlenecks separate from OpenTelemetry metrics and traces. _(since v3.0.0)_ [Telemetry (OTLP)](/docs/manage/observability/telemetry.md) and [`GET /metrics`](/docs/manage/observability/observability.md) (Prometheus pull) tell you what the database is doing from the outside (requests, statements, resources). The [Tokio console](https://github.com/tokio-rs/console) shows what the Tokio runtime is doing on the inside: which async tasks exist, how long they spend polled versus waiting, and where scheduling stalls appear. It is aimed at contributors and advanced operators debugging performance or deadlocks, not at replacing production metrics. SurrealDB embeds Tokio console support behind environment variables. It is disabled by default; enable it only on trusted hosts (for example local development or a dedicated staging instance). ## Install the console CLI The server exposes data to the console subscriber; you still need the Tokio console client on your machine. 1. [Install Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) if you do not already have them. 2. Install the CLI: `cargo install --locked tokio-console` 3. Start SurrealDB with console support enabled (see below), then run `tokio-console` in another terminal. By default it connects to `127.0.0.1:6669`. ## Enable console support on the server Set the following before starting `surreal`: * `SURREAL_TOKIO_CONSOLE_ENABLED` - set to `true` to turn on the subscriber. * `SURREAL_TOKIO_CONSOLE_SOCKET_ADDR` - optional; overrides the default listen address `127.0.0.1:6669`. * `SURREAL_TOKIO_CONSOLE_RETENTION` - optional; overrides the default `6s` (six seconds) for how long to [retain data for completed events](https://docs.rs/console-subscriber/0.4.1/console_subscriber/struct.Builder.html#method.retention). The socket listens only on the address you configure; keep it on loopback unless you fully understand the exposure (the stream describes internal task behaviour). ## What you will see ![A screenshot of the Tokio console showing active SurrealDB tasks.](~/assets/img/surrealdb/reference-guide/tokio_console_1.png) ![A screenshot of the Tokio console showing poll times by percentile.](~/assets/img/surrealdb/reference-guide/tokio_console_2.png) Typical uses for the tokio console are: spotting tasks that rarely get polled, comparing poll percentiles after a change, and correlating spikes in load with specific async workloads. --- Source: https://surrealdb.com/docs/manage/organisations # Organisations The container for instances, members, usage, and billing. What each section of the organisation view holds. An organisation groups instances, members, usage, and billing. Every instance, membership, and invoice belongs to exactly one organisation. Nobody in one organisation can see or change the resources of another. Create one organisation for each company, department, or environment you keep apart. Separate organisations for production and development each get their own member list and their own invoice. ## Create an organisation 1. Open the overview page in [SurrealDB Studio](https://studio.surrealdb.com). 2. Select **Create organisation**. 3. Enter a name for the organisation. 4. Confirm the name. You become the [owner](/docs/manage/organisations/members-and-roles.md), and Studio opens the organisation overview. The breadcrumb at the top of Studio switches between the organisations you belong to. To create an organisation from a terminal, run `surrealctl org create`. `surrealctl org use` sets which organisation later commands apply to. See [surrealctl organisations](/docs/manage/surrealctl/organisations.md). ## The organisation view The overview page shows everything the organisation holds. The sidebar leads into each area. ![The overview page for the Acme Corp organisation in SurrealDB Studio, listing the api-production, api-staging and analytics-eu instances with a Deploy new instance button, the support-agent and research-agent Agent Memory contexts below them, and a Resources panel linking to documentation, SurrealDB University, AI agents and community, with a sidebar holding Overview, Instances, Contexts, Connections, Team, Billing, Support, Usage and Settings.](~/assets/img/surrealdb/manage/studio-overview.webp) | Section | What it holds | | --- | --- | | **Instances** | Every [instance](/docs/manage/instances.md) in the organisation. Deploy, open, and filter them here. | | **Team** | [Members, invitations, and roles](/docs/manage/organisations/members-and-roles.md). | | **Billing** | [Billing details, payment method, discount codes, and invoices](/docs/manage/organisations/billing.md). | | **Usage** | [Compute, storage, and spend](/docs/manage/organisations/billing.md#usage-and-spend) for the current or previous month. | | **Support** | [Support plans and tickets](/docs/manage/organisations/support.md). | | **Settings** | The organisation name and its id. | **Contexts** and **Connections** hold [SurrealDB Agent Memory](/docs/agent-memory.md) resources rather than database instances. They use the same organisation for access and billing. Your role decides what you see. A member without billing permissions does not see the **Billing** section. See [Members and roles](/docs/manage/organisations/members-and-roles.md). ## Organisation settings **Settings** holds two values: the display name, which you can change at any time, and the organisation id. ![The Settings page for the Acme Corp organisation in SurrealDB Studio, with an editable Name field, a read-only Organisation ID field with a copy button noting the id may be requested by the SurrealDB support team, and a Save changes button.](~/assets/img/surrealdb/manage/organisation-settings.webp) Quote the organisation id when you raise a [support ticket](/docs/manage/organisations/support.md). The id names one organisation exactly, where a display name might not. Organisations cannot be deleted at present. Delete the instances inside an organisation you have finished with, and it stops accruing charges. ## Topics - **[Accounts and sign-in](/docs/manage/organisations/sign-in.md):** Create an account and sign in. - **[Members and roles](/docs/manage/organisations/members-and-roles.md):** Send invitations, and choose what each member can do. - **[Billing](/docs/manage/organisations/billing.md):** Payment details, invoices, usage, and spend. - **[Support](/docs/manage/organisations/support.md):** Community help, support plans, and tickets. - **[AWS Marketplace](/docs/manage/organisations/aws-marketplace.md):** Subscribe and pay through AWS. - **[Referrals](/docs/manage/organisations/referrals.md):** The referral link and the rewards it earns. - **[FAQs](/docs/manage/organisations/faqs.md):** General, security, pricing, legal, and troubleshooting questions. ## Related pages - **[Instances](/docs/manage/instances.md):** Deploy and operate databases inside an organisation. - **[surrealctl organisations](/docs/manage/surrealctl/organisations.md):** Organisations, teams, invitations, and tokens from the command line. --- Source: https://surrealdb.com/docs/manage/organisations/aws-marketplace # AWS Marketplace Subscribe through AWS Marketplace so charges appear on your AWS bill, and link the subscription to an organisation. Subscribe through AWS Marketplace to pay for managed instances on your existing AWS bill. You subscribe on the Marketplace listing, and AWS charges you for the usage. SurrealDB holds no card for the organisation. This route suits a procurement process that already runs through AWS: - Consolidated invoicing across vendors. - Spend drawn down against an existing AWS commitment. - A purchasing process that approves Marketplace listings only. Everything else about the instances is the same. ## Subscribe 1. Open the **SurrealDB Cloud** listing in [AWS Marketplace](https://aws.amazon.com/marketplace). 2. Select **Subscribe**. 3. Confirm the contract and the pricing terms. 4. Complete the AWS account and payment linkage as prompted. 5. Follow the set-up steps shown after you subscribe. The last step signs you in to your SurrealDB account, where you authorise the link between the purchase and your organisation. After the link is in place, [deploy instances](/docs/manage/instances/create.md) as you would otherwise. Plans, instance types, regions, and capabilities are the same. Only the invoicing route differs. ## Where your invoices are Invoices and payment information for a Marketplace subscription are in the **AWS Billing and Cost Management** console. They do not appear in SurrealDB Studio, so the [Billing](/docs/manage/organisations/billing.md) page shows no invoices for an organisation billed through the Marketplace. Usage figures stay in Studio under **Usage**. Read them to reconcile what the AWS charge covers. See [Usage and spend](/docs/manage/organisations/billing.md#usage-and-spend). ## Change or cancel a subscription Changes and cancellations go through AWS. Follow the AWS guidance for Marketplace subscriptions. > [!WARNING] > Cancelling the Marketplace subscription removes the billing route for the organisation. Before you cancel, put the organisation on a valid plan, with its own payment method if one is needed. Running instances are otherwise left without a billing route. ## Related pages - **[Billing](/docs/manage/organisations/billing.md):** The direct billing route, and the usage view both routes share. - **[Support](/docs/manage/organisations/support.md):** Support plans, which are added to the organisation whichever billing route it uses. --- Source: https://surrealdb.com/docs/manage/organisations/billing # Billing Organisation-level billing details, payment methods, discount codes, invoices, and the usage that produces them. Set the billing and payment details of your organisation, and check what its instances cost. Billing applies to the organisation, not to each instance. Compute, storage, and network use across every instance in the organisation roll up into one monthly invoice. Charges follow what you provision and what you use, rather than a flat licence fee. SurrealDB invoices you monthly for the usage of the previous month. The cycle runs from the first day of a month to the last, and the invoice arrives by email. Prices on the website exclude VAT and sales tax. Applicable taxes appear on the invoice. Current rates are on the [pricing page](https://surrealdb.com/pricing). > [!IMPORTANT] > Only an organisation **Owner** can change billing and payment details. An Owner or an **Admin** can view invoices. See [Members and roles](/docs/manage/organisations/members-and-roles.md). ## Billing and payment details Open the organisation in [SurrealDB Studio](https://studio.surrealdb.com) and go to **Billing**. ![The Billing page for the Acme Corp organisation in SurrealDB Studio, showing a Billing details card with the name Alex Doe and email alex@example.com, a Payment details card listing a Mastercard ending in 4242, each with an Edit button, and a SurrealDB Agent Memory plan section below with options to upgrade, cancel, or view pricing.](~/assets/img/surrealdb/manage/organisation-billing.webp) **Billing details** holds the name and email address that appear on invoices. **Payment details** holds the payment method. Select **Edit** on either card to change it. Use a billing address that your finance team recognises, and an email address that reaches more than one person. An invoice sent to a person who has left the company can go unnoticed until an instance is suspended. If your organisation has a VAT or tax identification number, enter it in the billing details. The number then appears on every invoice. ## Discount codes Apply a discount code in the **Discount codes** section of the same page. A code applies to later invoices for the organisation, and not to invoices already issued. ## Other subscriptions The **Billing** page also holds subscriptions for other SurrealDB products in the organisation, such as [SurrealDB Agent Memory](/docs/agent-memory.md) context packages. Each subscription has its own plan card, with actions to change it or to cancel it. All subscriptions use the payment method of the organisation, and appear on the same invoice. ## Invoices **Billing** lists each invoice with its date, its status, and its amount, and links to the document itself. The organisation keeps its invoices after you delete the instances that generated them. An audit or an expense query can still be answered later. ## Usage and spend **Usage** shows what produced the charges before the invoice arrives, for the current month or the previous month. ![The Usage page for the Acme Corp organisation in SurrealDB Studio, with a Previous month and Current month toggle, cards showing total spend, total compute hours, total storage, and instance count for the selected period, and a Usage breakdown section listing individual ledger entries.](~/assets/img/surrealdb/manage/organisation-usage.webp) The four cards give the totals: spend, compute hours, storage, and instance count. **Usage breakdown** lists the ledger entries behind those totals, and attributes each charge to a period and a resource. Read the breakdown when a total is higher than you expect. Check usage after you resize an instance. A larger instance type is billed on what it provisions. The effect on spend appears at once, rather than at the end of the month. See [Scaling](/docs/manage/instances/scaling.md). `surrealctl org spend` and `surrealctl org usage` print the same figures for a script or an internal cost report. See [surrealctl organisations](/docs/manage/surrealctl/organisations.md). ## Control what you spend - **[Pause](/docs/manage/instances/configure.md#pause-an-instance) non-production instances when they are idle.** A paused instance keeps its data and stops accruing usage charges. - **Right-size an instance rather than over-provision it.** Storage can only increase, so a generous first allocation is a permanent cost. - **Delete instances you have finished with.** Take an [export](/docs/manage/instances/import-and-export.md) first if the data still matters. ## AWS Marketplace subscriptions If you subscribed through AWS Marketplace, billing runs through your AWS account. Invoices and payment details are in the **AWS Billing and Cost Management** console, and do not appear in Studio. See [AWS Marketplace](/docs/manage/organisations/aws-marketplace.md). ## Billing questions Email [support@surrealdb.com](mailto:support@surrealdb.com) with the details of your query. Write from the address associated with your account, so that the request matches the organisation. The account menu at the top right of SurrealDB Studio shows which address that is. For technical questions rather than billing questions, see [Support](/docs/manage/organisations/support.md). --- Source: https://surrealdb.com/docs/manage/organisations/faqs # FAQs Common questions about managed instances: getting started, limits, security, pricing, legal terms, and troubleshooting. Short answers to the questions asked most often about managed instances, with links to the page that covers each one in full. ## General ### What do I get with a managed instance? A SurrealDB deployment that SurrealDB operates. SurrealDB handles provisioning, high availability, patching, backups, and version upgrades. See [Instances](/docs/manage/instances.md) for what that covers, and [Self-hosted](/docs/manage/self-hosted.md) if you would rather run the server yourself. ### How do I get started? [Create an account](/docs/manage/organisations/sign-in.md) at [studio.surrealdb.com](https://studio.surrealdb.com), [create an organisation](/docs/manage/organisations.md), and [deploy an instance](/docs/manage/instances/create.md). The free instance type needs no payment details. ### Which cloud platforms can I deploy to? AWS and Microsoft Azure, in the regions listed in [Create an instance](/docs/manage/instances/create.md). You choose the provider and the region together when you deploy, and neither can be changed afterwards. ### Which programming languages can I use? Any language with a SurrealDB SDK: - [JavaScript and TypeScript](/docs/reference/javascript.md) - [Python](/docs/reference/python.md) - [Go](/docs/reference/golang.md) - [Java](/docs/reference/java.md) - [Rust](/docs/reference/rust.md) - [PHP](/docs/reference/php.md) - [.NET](/docs/reference/dotnet.md) Any other language can use the [HTTP API](/docs/manage/instances/connect/via-http.md). ### Can I move an existing SurrealDB application onto a managed instance? Yes. Export from your current deployment, then import into the instance. See [Import and export](/docs/manage/instances/import-and-export.md), and [Migrating to SurrealDB](/docs/build/migrating.md) for a move from another database. A dedicated migration tool for existing self-hosted deployments is in development. Until it ships, use `surreal export` and `surreal import`. ### Why am I getting an authentication error when connecting through an SDK? The instance most likely has no user or access method defined yet. Studio authenticates with your own session, so it works before any credentials exist. An SDK needs credentials. See [Connect via SDK](/docs/manage/instances/connect/via-sdk.md#create-credentials). ### Are my instances backed up? Yes. Automated snapshots run on a schedule, with tiered daily, weekly, and monthly retention. The snapshot frequency and the retention of each tier depend on the instance type, and Scale instances allow a configurable frequency. See [Backups and recovery](/docs/manage/instances/backups.md). ### How often can I increase the disk size of an instance? Once every six hours. Storage can increase but never decrease. See [Configure an instance](/docs/manage/instances/configure.md#compute-and-storage). ### Can I transfer an instance to another organisation? No. Deploy an instance in the target organisation, then move the data with [`surreal export` and `surreal import`](/docs/manage/instances/import-and-export.md). ### Can I delete an organisation? Not at present. Delete the instances inside it, and it stops accruing charges. See [Organisations](/docs/manage/organisations.md). ### What are the current limits? - **Server flags and environment variables** cannot be set directly. [Instance capabilities](/docs/manage/instances/configure.md#capabilities) control runtime behaviour instead, in Studio or with `surrealctl instance capabilities`. - **[GraphQL](/docs/learn/querying/graphql/overview.md)** is not yet enabled on managed instances. - **Free instances** become read-only above the 1 GB storage allowance. Existing data stays available, and new writes are refused until you move to a paid instance type. - **Request sizes** are capped per endpoint. See [request size limits](/docs/manage/instances/connect/via-http.md#request-size-limits). ### Where are the documentation and tutorials? - **[SurrealDB documentation](/docs):** Guides and reference, including [SurrealDB Studio](/docs/explore/studio.md). - **[SurrealDB University](/learn):** A self-paced course with hands-on labs. - **[SurrealDB Book](/learn/book):** An in-depth guide to the database. - **[YouTube channel](https://www.youtube.com/@surrealdb):** Release walkthroughs and tutorials. ## Security ### How secure are managed instances? Instances run with encryption at rest and in transit, network isolation, access controls, and monitoring, as set out in the [Security Addendum](/legal/security-addendum). Multi-factor authentication is available through the identity provider you sign in with. SurrealDB holds ISO 27001, SOC 2 (Type 2), and Cyber Essentials Plus certifications. Further industry-specific programmes, such as HIPAA and PCI DSS, are planned. The [Trust Centre](https://trust.surrealdb.com/) holds the current detail. Instances also inherit the [security features](/docs/learn/security/authentication/summary.md#product) of SurrealDB itself, and the [security practices](/docs/learn/security/authentication/summary.md#process) the database is developed under. To assess the service for a workload with specific requirements, use the [contact form](/contact). ### How do I report a security issue? Email [security@surrealdb.com](mailto:security@surrealdb.com). For a vulnerability in SurrealDB itself, submit a report through [GitHub Security Advisories](https://github.com/surrealdb/surrealdb/security/advisories) rather than opening a public issue. SurrealDB asks you to: - Disclose the details privately to SurrealDB first. - Include enough information to reproduce the issue. - Keep the details from the public and from third parties until SurrealDB has addressed them. - Limit any exploitation of the vulnerability to what verifies that it exists. - Get permission before you run automated security tools against SurrealDB infrastructure. In return, SurrealDB: - Acknowledges your report within three business days. - Verifies the issue, and reports progress to you. - Treats your report and any data you share as confidential. - Takes no legal action against you for a report made under this policy. - Credits you in any resulting advisory, unless you prefer otherwise. ### How do I configure user authentication? Connect to the instance, then [define users](/docs/reference/query-language/statements/define/user.md) or [access methods](/docs/reference/query-language/statements/define/access.md) at root, namespace, or database level. Those credentials work across every interface: the [HTTP API](/docs/reference/rest-api.md), the [CLI](/docs/reference/cli/surrealdb-cli/overview.md), the SDKs, and [SurrealDB Studio](https://studio.surrealdb.com). See [Connect via SDK](/docs/manage/instances/connect/via-sdk.md#create-credentials). ### Can I use my own encryption keys? Not at present. SurrealDB manages the keys that encrypt data at rest. The architecture supports customer-managed encryption keys, so they may be offered in future. Use the [contact form](/contact) if you need them. ### Can I control who reaches my instances over the network? Instances are reachable over the public internet by default, protected by the authentication and authorisation defined inside them. Two additional controls exist: - **[AWS PrivateLink](/docs/manage/instances/private-connectivity.md)** gives an instance a private network path from your AWS VPC. In `private` mode the instance gets no public hostname at all. It is available in a subset of regions, and the SurrealDB team onboards you to it. - **[Instance capabilities](/docs/manage/instances/configure.md#capabilities)** restrict what the engine does: which RPC methods and HTTP endpoints it serves, whether guest access is permitted, and which [outbound destinations](/docs/manage/instances/network-access.md) a query can reach. ## Pricing ### How much does it cost? You pay for the compute you consume and the storage you provision. Current rates are on the [pricing page](/pricing). [Usage](/docs/manage/organisations/billing.md#usage-and-spend) shows what an organisation has accrued so far this month. ### Is there a free tier? Yes. The free instance type carries no charge, and provides limited resources for development and evaluation. It has a 1 GB storage allowance, cannot expand its storage, and pauses automatically after seven days without use. ### I am on the free tier. Can I add another instance? Studio prompts you to add [billing information](/docs/manage/organisations/billing.md) to the organisation first. ### How am I billed? Monthly, for the usage of the previous month. The cycle runs from the first day of a month to the last, and the invoice arrives by email. See [Billing](/docs/manage/organisations/billing.md). ### Are invoices itemised, and is tax identified? Yes. Enter your VAT or tax identification number in the billing details of the organisation, and it appears on every invoice. ### Are there additional fees? No. All costs are on the [pricing page](/pricing). Prices quoted on the website exclude VAT and sales tax, and applicable taxes appear on the invoice. ### Who do I contact about billing? Email [support@surrealdb.com](mailto:support@surrealdb.com) from the address associated with your account, and describe the query. ## Legal ### What terms govern my use of the service? The Master Services Agreement governs your use of the managed service. It is on the [legal page](/legal), together with the compliance and privacy documentation. ### Do you serve US-embargoed countries or Russia? SurrealDB Inc. complies with U.S. regulations on embargoed countries and regions. It currently prohibits use of its products and services in Cuba, Iran, North Korea, and Syria. The prohibition also covers the Russian-controlled regions of Crimea, the Luhansk People's Republic (LNR), and the Donetsk People's Republic (DNR). SurrealDB uses geoblocking to prevent logins from restricted regions. A regional block can restrict an entire country, which is **beyond the direct control of SurrealDB**. SurrealDB Ltd is also subject to Article 5n(2b) of Council Regulation 833/2014. That article prohibits the supply of software for the management of enterprises, or for design and manufacture. The prohibition covers the Government of Russia and entities established in Russia. SurrealDB reviews these obligations and updates them as regulations change. ## Troubleshooting ### I cannot connect to my instance Work through these checks in order: 1. Confirm the instance is **running** and not [paused](/docs/manage/instances/configure.md#pause-an-instance). A paused instance is unreachable, and a free instance pauses itself after seven days. 2. Check the **endpoint, namespace, and database** against the Connect menu of the instance in Studio. 3. Check your **credentials**, and confirm the user or access method has permission on that namespace and database. 4. On [PrivateLink](/docs/manage/instances/private-connectivity.md), confirm you connect from inside the VPC. A private hostname does not resolve outside it. 5. Check whether the failing operation depends on a denied [capability](/docs/manage/instances/configure.md#capabilities), such as an outbound HTTP call. 6. Check [metrics and logs](/docs/manage/instances/monitoring.md) for connection saturation or errors around the time the failures started. If the connection still fails, raise a [support ticket](/docs/manage/organisations/support.md) with the instance id. ### How do I report a bug or request a feature? Raise a [support ticket](/docs/manage/organisations/support.md), or ask in the [Discord](https://discord.gg/surrealdb) community. Include steps to reproduce the problem, the instance id, what you expected, and what happened instead. --- Source: https://surrealdb.com/docs/manage/organisations/members-and-roles # Members and roles Invite people to an organisation, track pending invitations, and choose between the Owner, Admin, and Member roles. Invite people to your organisation, and give each one the role that matches the work they do. The **Team** section of the organisation in [SurrealDB Studio](https://studio.surrealdb.com) holds membership. Each member holds exactly one role, and that role decides what the member can see and change across every instance in the organisation. ![The Team page for the Acme Corp organisation in SurrealDB Studio, showing Members and Pending tabs with one pending invitation, a member search box, an Invite member button, and a single row for Alex Doe at alex@example.com labelled OWNER and YOU.](~/assets/img/surrealdb/manage/organisation-members.webp) The **Members** tab lists the people who accepted an invitation. The **Pending** tab lists invitations that have been sent but not yet accepted. ## Invite a member 1. Open **Team**. 2. Select **Invite member**. 3. Enter the email address of the person you invite. 4. Choose the role that person holds. 5. Send the invitation. The invitation appears under **Pending** until the person accepts it. An invitee without an account creates one first, then joins the organisation on the role you chose. See [Accounts and sign-in](/docs/manage/organisations/sign-in.md). Invite the address of a named person rather than a shared mailbox, above all for the Admin and Owner roles. A shared mailbox hides which person acted, and you cannot withdraw it from one reader who leaves. `surrealctl team invite` sends an invitation. `surrealctl team list` reports current membership, which helps when you review access across several organisations. See [surrealctl organisations](/docs/manage/surrealctl/organisations.md). ## What each role can do The person who creates an organisation is its **Owner**. An Owner invites members as **Admin** or **Member**. | Action | Owner | Admin | Member | | --- | --- | --- | --- | | View instances and team | Yes | Yes | Yes | | Deploy, pause, and delete instances | Yes | Yes | No | | Change instance configuration and [capabilities](/docs/manage/instances/configure.md#capabilities) | Yes | Yes | No | | Invite and remove members | Yes | Yes | Own membership only | | Change another member's role | Yes | No | No | | View [invoices](/docs/manage/organisations/billing.md) | Yes | Yes | No | | Change billing and payment details | Yes | No | No | An Admin operates the infrastructure of the organisation, but does not control its payment details or its access list. A Member reads what the organisation holds, and can end its own membership. > [!NOTE] > Each role is a fixed set of permissions. Custom roles and per-user granular permissions are planned. ## Least privilege Grant the lowest role that lets a person do their work, then review the member list at regular intervals. Three actions need particular attention, because each one causes loss or exposure that is hard to reverse: - **Delete an instance:** destroys the data in it. See [Configure an instance](/docs/manage/instances/configure.md#delete-an-instance). - **Change [network access](/docs/manage/instances/network-access.md):** decides which external destinations a query can reach. - **Restore a [backup](/docs/manage/instances/backups.md):** creates a new instance that holds a copy of production data. Every Owner and every Admin can run all three. To let someone operate instances without these permissions, use a separate organisation rather than a lower role. Remove access when a person leaves the project. A member who has left keeps the Admin role until you remove the membership. ## Related pages - **[Organisations](/docs/manage/organisations.md):** The organisation view and its settings. - **[Accounts and sign-in](/docs/manage/organisations/sign-in.md):** How an invitee creates an account. - **[surrealctl organisations](/docs/manage/surrealctl/organisations.md):** Teams, invitations, and tokens from the command line. --- Source: https://surrealdb.com/docs/manage/organisations/referrals # Referrals Share a referral link to earn credits and rewards when someone signs up through it. Share your referral link to earn rewards when someone signs up through it. Every account gets its own referral link. Rewards range from account credit to items from the SurrealDB store. ## How it works 1. Copy your referral link from the [referrals page](https://studio.surrealdb.com/referrals). 2. Share the link with the people you recommend SurrealDB to. When a person signs up through your link, the reward is credited to your account. The Referral Program page in SurrealDB Studio, showing a card with your personal referral link, a copy button and a Share button, a Progress section counting the users you have referred and how many more unlock the next reward, an Unlockable rewards row of cards for free credits, store items and a Discord badge at rising referral counts, and a How does this work section explaining that you share your link, a friend signs up, and you get rewarded. The same page tracks the sign-ups attributed to your link and the rewards they earn. Credit applies to future invoices for your organisation. See [Billing](/docs/manage/organisations/billing.md). --- Source: https://surrealdb.com/docs/manage/organisations/security # Two-step verification Add an authenticator app to your SurrealDB account, and keep a recovery code for when it is unavailable. Two-step verification asks for a code from a second device after your password, so that a leaked password is not enough to reach your account on its own. It is set per account rather than per organisation: turning it on protects your own sign-in, and every organisation you belong to. Security options live in the account portal at [account.surrealdb.com/security](https://account.surrealdb.com/security), under **Security settings**. Studio does not manage them. > [!NOTE] > If you sign in with Google or GitHub, two-factor authentication configured with that provider already applies to your SurrealDB account, because the provider verifies you before returning you to SurrealDB. Adding an authenticator app here covers accounts that sign in with an email address and password. See [Accounts and sign-in](/docs/manage/organisations/sign-in.md). ## Add an authenticator app An authenticator app generates a new six-digit code every thirty seconds, with no network connection needed. Any TOTP app works, including 1Password, Bitwarden, Google Authenticator and Authy. 1. Open [account.surrealdb.com/security](https://account.surrealdb.com/security). 2. Under **Set up a new security option**, choose **Authenticator app**. 3. Scan the QR code with your authenticator app. Where a camera is not available, use the **Manual setup code** shown beside it and enter the key by hand. 4. Enter the code your app displays to confirm the pairing. The method appears under **Two-step verification** once it is confirmed. From then on, signing in asks for a code after your password. One authenticator app can be registered per account. To move to a different app or device, remove the existing method first and add the new one. ## Save your recovery code Setting up your first method produces a single recovery code. It signs you in if your authenticator app is ever unavailable, so store it somewhere you can reach without that device: a password manager, or somewhere safe offline. The code is shown once. Copy it before selecting **I have safely recorded this code**, because it cannot be displayed again afterwards. > [!WARNING] > Losing both your authenticator app and your recovery code locks you out of the account. Neither can be recovered by SurrealDB. Save the recovery code somewhere separate from the device running the app, so that losing one device does not cost you both. ## Remove a method Each registered method can be removed from the same page. Removing your only method returns the account to password-only sign-in, so add a replacement first if you intend to keep two-step verification on. ## Next steps 1. [Invite colleagues](/docs/manage/organisations/members-and-roles.md) so each person signs in with their own account and their own second factor. 2. [Create a token](/docs/manage/surrealctl/authentication.md) for automation rather than sharing an interactive account. --- Source: https://surrealdb.com/docs/manage/organisations/sign-in # Accounts and sign-in Create an account with Google, GitHub, or an email address, and sign in to manage organisations and instances. One account gets you into [SurrealDB Studio](https://studio.surrealdb.com), where you manage [organisations](/docs/manage/organisations.md) and [instances](/docs/manage/instances.md). An account needs no payment details. The free instance type carries no charge, so you can deploy before you add a payment method. To create an account or to sign in, open [studio.surrealdb.com](https://studio.surrealdb.com) and continue to the authentication page. ## Choose a sign-in method Three methods are available. Use the same method every time. If you sign up with Google and later sign in by email on the same address, you create a second account. The first account is not recovered. | Method | What happens | Suited to | | --- | --- | --- | | **Google** | You authorise SurrealDB on Google's own sign-in page | Teams already on Google Workspace | | **GitHub** | You authorise SurrealDB on GitHub's own sign-in page | Developers already signed in to GitHub | | **Email and password** | You set a password, then confirm the address from a verification link | Anyone who prefers not to link a third-party account | > [!IMPORTANT] > Google and GitHub sign-in uses OAuth2. The provider's own page collects your password, so SurrealDB never receives it. SurrealDB requests your email address and name only. After you authorise the connection, the provider returns a token that confirms your identity. Two-factor authentication configured with Google or GitHub also applies to your SurrealDB account, because the provider verifies you before returning you to SurrealDB. For an account that signs in with an email address and password, add an authenticator app instead. See [Two-step verification](/docs/manage/organisations/security.md) for more. ## Create an account With Google or GitHub: 1. Select **Sign up with Google** or **Sign up with GitHub**. 2. Choose your account in the provider's window. 3. Allow the requested permissions. Studio signs you in as soon as the provider returns. With email: 1. Select **Sign up with Email**. 2. Enter your email address. 3. Set a password. 4. Open the verification link sent to your inbox to confirm the address. ## Sign in 1. Open [studio.surrealdb.com](https://studio.surrealdb.com). 2. Select the method you registered with: **Continue with Google**, **Continue with GitHub**, or **Sign in with Email**. 3. Complete any two-factor step the provider asks for. To reset a forgotten email password, select **Forgot password?** on the authentication page. Follow the instructions sent to your address. An account created through Google or GitHub has no SurrealDB password to reset. Recover the provider account instead. ## Accounts for a team Each person needs their own account. Invite colleagues to the organisation instead of sharing credentials. Separate accounts let each member hold a different role, and let you remove someone without changing a password the rest of the team uses. See [Members and roles](/docs/manage/organisations/members-and-roles.md). For automation, use a token from [`surrealctl`](/docs/manage/surrealctl/authentication.md) rather than an interactive account. ## Next steps 1. [Create an organisation](/docs/manage/organisations.md) to hold your instances and billing. 2. [Deploy an instance](/docs/manage/instances/create.md). 3. [Connect to it](/docs/manage/instances/connect.md) from SurrealDB Studio, the CLI, an SDK, or over HTTP. --- Source: https://surrealdb.com/docs/manage/organisations/support # Support Community help, paid support plans, and raising a ticket. Get help from the community, or from a support plan with response-time commitments. Community help is free, and open to everyone. A support plan is what lets you raise a ticket. ## Community support The [SurrealDB Discord](https://discord.gg/surrealdb) answers most questions. `#surrealdb-cloud` covers managed instances, and `#help` or `#general` cover everything else. The SurrealDB team reads the server and escalates the issues that need it. Use the community for questions that carry no confidential detail. Raise a ticket for anything that involves your data, your credentials, or your billing. ## Support plans A support plan applies to the whole organisation. Open **Support** in the organisation sidebar. Without a paid plan, the section shows **View plans**, which opens the [pricing page](/pricing). To add a plan, contact the support team. The Support section of an organisation in SurrealDB Studio, showing a Support Plan panel with a Community card described as help from community members on Discord and GitHub and a View plans button, and a Support History panel stating that a support plan is required for expedited support from the SurrealDB team, with a second View plans button. With a plan in place, the same section shows the plan and the tickets for the organisation. The Support section of an organisation in SurrealDB Studio with a Standard support plan card described as professional support for development teams with business hours coverage and email support, a View plans button, and a Support History panel with a New ticket button. Response times depend on the plan. Check what your plan commits to before an incident needs it. ## Raise a ticket 1. Open **Support** in the organisation sidebar. 2. Select **New ticket**. 3. Enter a **Subject** that names the problem. 4. Describe the problem under **What is your reason for contacting us?** 5. Choose the **Organisation** the ticket belongs to. 6. Choose the **Severity level**. 7. Select **Submit**. The Create new ticket dialog in SurrealDB Studio, with a note that replies are sent to the ticket and to your email address, a Subject field reading I have an issue with my functions, a reason field describing an unresponsive instance, an Organisation selector, a Severity level selector set to partial loss of functionality affecting some users or features, a note that attachments are included by replying to the ticket, and a Submit button. In the description, state what you expected, what happened, and when the problem started. Include the [instance id](/docs/manage/instances/configure.md#the-instance-id) and the [organisation id](/docs/manage/organisations.md#organisation-settings), which you can copy from Studio. Both save a round trip. Studio opens the ticket after you submit it. Replies appear in the ticket and also go to your email address. To attach a log, a query, or a screenshot, add it to a reply. A submitted support ticket in SurrealDB Studio, showing the description and severity level on the left, a panel on the right with the state Submitted, the type Standard and the time of the last update, an updates timeline recording the state change, a reply box with an attachment button, and an All tickets button at the top right. **All tickets** at the top right of the **Support** section lists every open and closed ticket for the organisation. ## Account and billing questions Account and billing questions go to [support@surrealdb.com](mailto:support@surrealdb.com), or through the **Support** section, and need no paid plan. Write from the address associated with your account. See [Billing](/docs/manage/organisations/billing.md). ## Report a security issue Send a security report to [security@surrealdb.com](mailto:security@surrealdb.com) rather than to Discord or a ticket. For a vulnerability in SurrealDB itself, submit a report through [GitHub Security Advisories](https://github.com/surrealdb/surrealdb/security/advisories) rather than a public issue. The disclosure policy is in the [FAQs](/docs/manage/organisations/faqs.md#how-do-i-report-a-security-issue). --- Source: https://surrealdb.com/docs/manage/schema-migration # SurrealKit schema migration SurrealKit is the official schema migration CLI for SurrealDB. Define your schema in .surql files and keep databases in sync across every environment. > [!NOTE] > All of the example commands in this tutorial assume a database running at `http://localhost:8000`, a root user named `root` with the password `secret`, a namespace `main` and a database `main`. > As host `http://localhost:8000`, namespace `main` and database `main` are default values, only the necessary `--user root` and `--pass secret` will be shown alongside each command. > To test these commands without `--user root` and `--pass secret`, authentication can be disabled by passing in the `--unauthenticated` flag when [starting the SurrealDB server](/docs/reference/cli/surrealdb-cli/commands/start.md). SurrealKit is the official schema management and migration CLI for SurrealDB. You define your database schema as plain `.surql` files, commit them alongside your application code, and SurrealKit keeps every environment in sync with those definitions. It has two modes for getting schema into a database: - **Sync:** immediately pushes your schema files to the connected database. Best for local development and ephemeral environments where fast iteration matters and losing data is acceptable. - **Rollouts:** generates a reviewed, phased migration manifest and applies changes in non-destructive then destructive passes, with rollback support. Best for shared, staging, and production databases. Most teams use Sync day-to-day and switch to Rollouts when promoting changes to shared environments. SurrealKit also provides: - **Templates:** scaffold a new project from a template with selectable features (`surrealkit init`). - **Seeding:** apply `.surql` seed data on demand. - **Type generation:** introspect a database to emit JSON and TypeScript types for your application. - **Testing:** a declarative framework for validating schema, permissions, and API endpoints. ## Installation **cargo binstall** (recommended, no compilation required once [cargo binstall is installed](https://github.com/cargo-bins/cargo-binstall#installation)): ```bash cargo binstall surrealkit ``` **Cargo from source:** ```bash cargo install surrealkit ``` **Docker:** ```bash docker pull ghcr.io/surrealdb/surrealkit:latest ``` Prebuilt binaries for Linux (x86_64 / aarch64), macOS (x86_64 / aarch64), and Windows (x86_64) are available on the [GitHub releases page](https://github.com/surrealdb/surrealkit/releases). ## Initialise a project ```bash surrealkit init ``` This scaffolds a project from a template, letting you pick which optional features to include. It always writes the base layout: ```text database/ ├── schema/ # .surql schema definition files ├── rollouts/ # rollout manifests (generated) ├── snapshots/ # schema and catalog snapshots ├── seed/ # optional seed data ├── tests/ # test suites and config └── setup.surql # runs before sync surrealkit.toml # project configuration ``` See [Project templates](/docs/manage/schema-migration/templates.md) for the feature checklist, non-interactive flags, and custom templates. ## Connection configuration SurrealKit resolves connection details in the following order (first match wins): 1. CLI arguments (`--host`, `--ns`, `--db`, `--user`, `--pass`, `--auth-level`) 2. `SURREALDB_*` environment variables 3. `.env` file in the working directory 4. Fallback `DATABASE_*` environment variables | Environment variable | CLI equivalent | Purpose | |---|---|---| | `SURREALDB_HOST` | `--host` | Database endpoint URL | | `SURREALDB_NAMESPACE` | `--ns` | Namespace | | `SURREALDB_NAME` | `--db` | Database name | | `SURREALDB_USER` | `--user` | Username | | `SURREALDB_PASSWORD` | `--pass` | Password | | `SURREALDB_AUTH_LEVEL` | `--auth-level` | `root`, `namespace`/`ns`, or `database`/`db` | The project root (containing `schema/`, `rollouts/`, `snapshots/`, `seed/`, and `tests/`) defaults to `./database`. Override it with the global `--folder` flag or the `SURREALDB_FOLDER` environment variable. Example connecting via CLI flags: ```bash surrealkit --user root --pass secret sync ``` ## Next steps - [New databases](/docs/manage/schema-migration/getting-started/new-databases.md): start a fresh project with SurrealKit from the beginning - [Existing databases](/docs/manage/schema-migration/getting-started/existing-databases.md): adopt SurrealKit in a project that already has a database - [Sync vs Rollouts](/docs/manage/schema-migration/getting-started/sync-vs-rollouts.md): choose the right mode for each environment - [Project templates](/docs/manage/schema-migration/templates.md): scaffold a project with selectable features - [Type generation](/docs/manage/schema-migration/typegen.md): generate JSON and TypeScript types from your schema --- Source: https://surrealdb.com/docs/manage/schema-migration/embed-schema-macro # embed_schema! macro The embed_schema! macro bakes your .surql schema files into the Rust binary at compile time, so schema is always in sync with the application that ships it. The `embed_schema!` macro reads your `database/schema/` directory at compile time and generates a Rust module containing the SQL for every `.surql` file it finds. Because the schema is compiled into the binary, there are no external files to deploy and the schema version is always tied to the application version. Add SurrealKit to your dependencies. The `embed_schema!` macro is re-exported from the main crate, so a single dependency is enough: ```toml [dependencies] surrealkit = "0.7" ``` ## Basic usage Call the macro at the crate root (typically `main.rs` or `lib.rs`): ```rust surrealkit::embed_schema!(); ``` This generates an `embedded_schema` module. Call `sync` on it after connecting to apply any outstanding schema changes: ```rust #[tokio::main] async fn main() -> anyhow::Result<()> { let db = surrealkit::connect( &surrealkit::DbCfg::from_env(None, &Default::default())? ).await?; embedded_schema::sync(&db).await?; // application startup continues here Ok(()) } ``` `sync` behaves identically to `surrealkit sync` from the CLI: it applies new or changed definitions and prunes any that have been removed, using the `__entity` metadata table to track state. ## What the macro generates Given a `database/schema/` directory with these files: ```text database/schema/ ├── users.surql └── orders.surql ``` The macro generates roughly: ```rust pub mod embedded_schema { pub static SCHEMA: &[surrealkit::EmbeddedSchemaFile] = &[ surrealkit::EmbeddedSchemaFile { path: "database/schema/users.surql", sql: "DEFINE TABLE user SCHEMAFULL; ...", }, surrealkit::EmbeddedSchemaFile { path: "database/schema/orders.surql", sql: "DEFINE TABLE order SCHEMAFULL; ...", }, ]; pub async fn sync( db: &surrealkit::Surreal, ) -> surrealkit::anyhow::Result<()> { surrealkit::Sync::embedded(SCHEMA).run(db).await } } ``` The generated `SCHEMA` static is public, so when you need to customise sync behaviour you can pass it to the [`Sync`](/docs/manage/schema-migration/library.md) builder directly instead of calling `embedded_schema::sync`: ```rust use surrealkit::Sync; Sync::embedded(embedded_schema::SCHEMA) .prune(false) .run(&db) .await?; ``` ## Compile-time rebuild behaviour Cargo re-runs the macro whenever a `.surql` file in `database/schema/` changes, because the macro registers each file with `include_str!`. This means schema changes always produce a fresh build, and there is no risk of shipping stale SQL. ## When to use the macro vs runtime loading |-| `embed_schema!` | Runtime `Sync` | |---|---|---| | Schema location | Compiled into binary | Built from files at runtime | | Deployment | No schema files needed | Schema directory must be present | | Dev iteration | Rebuild required on change | Files can be swapped without rebuild | | Best for | Production builds, Docker images | Development, mounted volumes | For most production deployments `embed_schema!` is the right choice. For local development or environments that mount schema as a volume, building an `EmbeddedSchemaFile` slice at runtime and passing it to [`Sync::embedded`](/docs/manage/schema-migration/library.md) is more convenient. --- Source: https://surrealdb.com/docs/manage/schema-migration/getting-started/existing-databases # Existing databases Adopt SurrealKit in a project that already has a SurrealDB database by capturing a baseline snapshot and moving to the Rollouts workflow. If your project already has a running SurrealDB database, you can adopt SurrealKit without disrupting it. A baseline snapshot records the current schema state so SurrealKit can compute diffs from there. ## 1. Install and initialise Install SurrealKit and initialise the project directory if you have not already done so: ```bash cargo binstall surrealkit surrealkit init ``` ## 2. Mirror your existing schema into files Before taking a baseline, write your existing schema definitions into `.surql` files under `database/schema/`. These files should reflect what is currently in the database, as SurrealKit treats them as the target state going forward. You can use `INFO FOR DB` and `INFO FOR TABLE` in SurrealQL to inspect what is currently defined: ```surql INFO FOR DB; INFO FOR TABLE my_table; ``` The following code can be used to return all the define statements in a database as an array of strings. ```surql LET $db = INFO FOR DB; $db.tables.values() + $db.users.values() + $db.tables.keys().map(|$t| { LET $i = INFO FOR TABLE $t; $i.fields.?.values() + $i.indexes.?.values() }).flatten().filter(|$v| !!$v); ``` Copy the returned `DEFINE` statements into appropriately named files in `database/schema/`. ## 3. Take a baseline snapshot Once your schema files mirror the live database, run: ```bash surrealkit rollout baseline --user root --pass secret ``` This connects to the database, captures its current schema state, and writes two snapshot files: ```text database/snapshots/schema_snapshot.json database/snapshots/catalog_snapshot.json ``` These snapshots tell SurrealKit what the database looked like before any SurrealKit-managed rollout. Future `rollout plan` commands diff against this baseline to produce migration manifests. > [!NOTE] > The baseline command does not modify the database. It only reads the current state and writes the snapshot files locally. ## 4. Plan and apply future changes With a baseline in place, use the standard Rollouts workflow for all subsequent schema changes: ```sh # Edit files in database/schema/, then: surrealkit rollout plan --name describe_your_change surrealkit rollout start # Deploy your application changes surrealkit rollout complete ``` See the [Rollouts](/docs/manage/schema-migration/rollouts.md) page for the full reference. ## Using Sync on existing databases You can also use `surrealkit sync` against an existing database, but **be aware that Sync will remove definitions from the database that are not present in your schema files (pruning)**. If your schema files are not yet complete, this could delete definitions you did not intend to remove. To prevent accidental pruning on a shared database, Sync requires an explicit flag: ```bash surrealkit sync --allow-shared-prune ``` For production and shared databases, use the Rollouts workflow to keep full control over what changes are applied and when. ## Next steps - [Rollouts](/docs/manage/schema-migration/rollouts.md): full reference for the phased migration workflow - [Sync vs Rollouts](/docs/manage/schema-migration/getting-started/sync-vs-rollouts.md): a side-by-side comparison --- Source: https://surrealdb.com/docs/manage/schema-migration/getting-started/new-databases # New databases Start a new SurrealDB project with SurrealKit from the beginning, writing schema files and syncing to a local database. If you are starting a new project, SurrealKit can manage your schema from the very first definition. This guide walks through initialising a project, writing your first schema file, and pushing it to a local SurrealDB instance. ## 1. Initialise the project In the root of your repository, run: ```bash surrealkit init ``` This creates a `database/` directory with the following layout: ```text database/ ├── schema/ ├── seed/ ├── tests/ └── rollouts/ ``` ## 2. Write a schema file Create a `.surql` file inside `database/schema/`. Each file can contain one or more `DEFINE` statements. ```surql -- database/schema/users.surql DEFINE TABLE user SCHEMAFULL; DEFINE FIELD email ON user TYPE string ASSERT string::is_email($value); DEFINE FIELD created_at ON user TYPE datetime DEFAULT time::now(); DEFINE INDEX unique_email ON user FIELDS email UNIQUE; ``` Schema files can be organised by table, by domain, or kept in a single file. SurrealKit applies everything in `database/schema/` together. ## 3. Start a local SurrealDB instance ```bash surreal start --user root --pass secret ``` ## 4. Sync your schema ```bash surrealkit --user root --pass secret sync ``` SurrealKit reads every file in `database/schema/`, applies the `DEFINE` statements to the database, and records a content hash for each file in an internal `__entity` metadata table. Future syncs only re-apply files that have changed. ## 5. Enable watch mode during development Pass `--watch` to keep SurrealKit running and automatically re-sync whenever a schema file changes: ```bash surrealkit --user root --pass secret sync --watch ``` ## Vite integration If your project uses Vite, the `vite-plugin-surrealkit` package runs sync automatically when the dev server starts: ```bash npm install --save-dev vite-plugin-surrealkit ``` ```ts // vite.config.ts import { defineConfig } from 'vite'; import { surrealkitPlugin } from 'vite-plugin-surrealkit'; export default defineConfig({ plugins: [ surrealkitPlugin(), ], }); ``` The plugin watches `database/schema/**/*.surql` and re-syncs on any change, with debouncing to avoid overlapping runs. ## Storing connection details Rather than passing flags on every command, store your local connection details in a `.env` file at the project root: ```bash SURREALDB_HOST=http://localhost:8000 SURREALDB_NAMESPACE=dev SURREALDB_NAME=myapp SURREALDB_USER=root SURREALDB_PASSWORD=secret ``` With those set, `surrealkit sync` picks them up automatically. ## Next steps - [Sync](/docs/manage/schema-migration/sync.md): full reference for all sync options - [Sync vs Rollouts](/docs/manage/schema-migration/getting-started/sync-vs-rollouts.md): when to move from Sync to the Rollouts workflow --- Source: https://surrealdb.com/docs/manage/schema-migration/getting-started/sync-vs-rollouts # Sync vs Rollouts Understand when to use SurrealKit's Sync mode for development and Rollouts mode for production, and how teams typically combine both. SurrealKit has two modes for applying schema changes. Which one you use depends on the environment. ## At a glance |-| Sync | Rollouts | |---|---|---| | **Philosophy** | Desired state: your files are truth | Phased migration via reviewed manifests | | **Speed** | Immediate | Planned and staged | | **Destructive changes** | Automatic (pruning) | Explicit, in a separate phase | | **Rollback** | Restore previous files and re-sync | `rollout rollback` command | | **Concurrency safety** | Not guaranteed | Blocked by in-progress rollout | | **Best for** | Local dev, ephemeral, disposable DBs | Shared, staging, production | ## When to use Sync Sync is the right choice when: - You are working on a local or personal development database - The database is ephemeral or disposable (a Docker container, a CI job, a preview environment) - Losing and recreating data is acceptable - You want fast iteration without planning each change explicitly Sync gives you an instant feedback loop: edit a `.surql` file, save it, and the database reflects the change within seconds (especially with `--watch` mode). ## When to use Rollouts Rollouts are the right choice when: - You are modifying a database shared with other developers or services - You are promoting changes to staging or production - The migration involves both non-destructive additions and destructive removals that must happen in separate phases, with your application deployed in between - You need a record of what was changed and when, with the ability to roll back The phased approach (expand first, then contract after the application is updated) means schema changes and application code can be deployed independently, with no downtime or data loss. ## How teams typically combine both A common pattern is: 1. **Local development**: every developer runs `surrealkit sync --watch` against their own local SurrealDB instance. Schema changes are iterated freely. 2. **Pull request / CI**: a `surrealkit test` step validates the schema and permissions against an ephemeral database. Sync is used here too, since the CI database is disposable. 3. **Staging / production**: when changes are ready to promote, a developer runs `surrealkit rollout plan` to generate a migration manifest, commits it, and the deployment pipeline runs `rollout start` and `rollout complete` at the right points around the application release. ## Prune behaviour on shared databases Sync's automatic pruning (removing definitions no longer in your files) is safe on a personal database but dangerous on a shared one. If a colleague has added a definition not yet in your local files, Sync will delete it. SurrealKit protects against this: running `surrealkit sync` against a database that already contains SurrealKit metadata (the `__entity` table) will require you to pass `--allow-shared-prune` explicitly before it will prune anything. This flag is a deliberate speed bump, not a routine option. ## Next steps - [Sync](/docs/manage/schema-migration/sync.md): full Sync reference - [Rollouts](/docs/manage/schema-migration/rollouts.md): full Rollouts reference --- Source: https://surrealdb.com/docs/manage/schema-migration/library # Using SurrealKit as a library Embed SurrealKit directly in a Rust application to connect to SurrealDB, sync schema at startup, run rollouts, and seed data without the CLI. SurrealKit is published as a Rust crate, so you can drive connections, schema sync, rollouts, and seeding from application code rather than the CLI. This suits applications that apply schema inside their own process at startup, for example with an embedded SurrealDB backend (RocksDB, SpeeDB) or when running SurrealDB in the same binary during tests. ## Adding the dependency ```toml [dependencies] surrealkit = "0.7" ``` ## Sync vs rollout SurrealKit gives you two ways to get schema into a database. Pick based on whether the database is disposable or shared. |-| Sync | Rollout | |---|---|---| | Mental model | Declarative *desired state*: "make the database match this schema" | Staged, reviewable *migration* with an explicit undo | | Applies | All changed files, idempotently | Ordered steps across `start` / `complete` / `rollback` phases | | Removes objects | Automatically (prune) | Only in the `complete` phase, via explicit steps | | Reversible | No | Yes (`rollback`) | | Use when | Dev, test, CI, single-owner or embedded databases | Shared and production databases needing expand → contract and a rollback path | The two work together: use **sync** for everyday schema, and use a **rollout** when a change needs to land safely while old and new code run side by side. ## Connecting `DbCfg` reads connection details from the same environment variables as the CLI (`SURREALDB_HOST`, `SURREALDB_NAMESPACE`, and so on), with optional overrides. `connect` builds the `surrealdb::Surreal` client and authenticates: ```rust use surrealkit::{DbCfg, DbOverrides, connect}; let cfg = DbCfg::from_env(None, &DbOverrides::default())?; let db = connect(&cfg).await?; ``` `DbOverrides` lets you override specific fields programmatically while leaving the rest to the environment: ```rust let cfg = DbCfg::from_env(None, &DbOverrides { host: Some("http://localhost:8000".to_string()), ..Default::default() })?; ``` ### In-process SurrealDB For an embedded SurrealDB engine (`mem://`, `rocksdb://`, `speedb://`), construct a `Surreal` client directly and pass it to any library function: ```rust use surrealdb::engine::any::connect; use surrealdb::opt::Config; use surrealdb::opt::capabilities::Capabilities; let db = connect(("mem://", Config::new().capabilities(Capabilities::all()))).await?; db.use_ns("main").use_db("main").await?; ``` ## Syncing schema Use the [`Sync`] builder to apply a slice of schema files to the database. By default it prunes objects that are no longer present and stops on the first error (`prune = true`, `fail_fast = true`): ```rust use surrealkit::{EmbeddedSchemaFile, Sync, Surreal}; use surrealkit::engine::any::Any; static SCHEMA: &[EmbeddedSchemaFile] = &[EmbeddedSchemaFile { path: "database/schema/person.surql", sql: "DEFINE TABLE person SCHEMALESS;", }]; async fn run(db: &Surreal) -> anyhow::Result<()> { // Defaults: prune = true, fail_fast = true. Sync::embedded(SCHEMA).run(db).await?; Ok(()) } ``` The builder methods customise behaviour before calling `run`: ```rust Sync::embedded(SCHEMA) .prune(false) // don't remove objects missing from SCHEMA .allow_all_statements(true) // permit non-DEFINE statements (INSERT/UPDATE/…) .allow_shared_prune(true) // permit pruning on a shared database .dry_run(true) // report what would change without applying .run(db) .await?; ``` `Sync` runs setup internally and reads nothing from the filesystem. To embed your `.surql` files at compile time instead of hand-writing the slice, use the [`embed_schema!` macro](/docs/manage/schema-migration/embed-schema-macro.md). ### `EmbeddedSchemaFile`: `path` vs `sql` The two fields serve different purposes: - **`path` is a stable tracking key**, not a path that must exist on disk. SurrealKit stores it in its metadata tables to identify the file, detect content changes, and prune files that disappear. Keep it stable across releases: renaming it makes SurrealKit treat the old key as deleted and the new one as added. - **`sql` is the content** that gets applied. Changing `sql` while holding `path` constant is exactly what triggers a re-apply on the next sync. ## Rollouts Rollouts are defined entirely in code, with no TOML or `.surql` files on disk required. Build a spec with [`RolloutSpec::builder`] and drive it with the [`Rollout`] facade. ### Status lifecycle ```text planned → running_start → ready_to_complete → running_complete → completed │ └── running_rollback → rolled_back ``` `completed` and `rolled_back` are terminal. `failed` and the `running_*` states are stuck states from an interrupted run; recover them with [`Rollout::abandon`] (or the CLI [`repair`](/docs/manage/schema-migration/rollouts.md) command). Only one rollout may be in a non-terminal state at a time. ### Lifecycle example ```rust use surrealkit::{ Rollout, RolloutSpec, RolloutStep, RolloutPhase, RolloutCompatibility, EmbeddedSchemaFile, EntityKey, EntityKind, Surreal, }; use surrealkit::engine::any::Any; // The desired schema once the rollout completes (used to compute the managed // catalog). Pass `&[]` if your steps fully describe the entity changes. static TARGET: &[EmbeddedSchemaFile] = &[EmbeddedSchemaFile { path: "database/schema/account.surql", sql: "DEFINE TABLE account SCHEMAFULL;", }]; async fn run(db: &Surreal) -> anyhow::Result<()> { let spec = RolloutSpec::builder("20260604__add_account") .name("Add account table") .compatibility(RolloutCompatibility::Phased) // Expand: add the new table (non-destructive). .step(RolloutStep::apply_schema( "create_account", RolloutPhase::Start, "DEFINE TABLE account SCHEMAFULL;", )) // Backfill during complete. run_sql must be safe to re-run. .step(RolloutStep::run_sql( "backfill", RolloutPhase::Complete, "UPDATE account SET active = true WHERE active = NONE;", )) // Undo the expand phase on rollback. .step(RolloutStep::remove_entities( "undo", RolloutPhase::Rollback, vec![EntityKey { kind: EntityKind::Table, scope: None, name: "account".into() }], )) .build(); let rollout = Rollout::new(spec, TARGET); rollout.start(db).await?; // expand (blocks if another rollout is active) // ... deploy new code, drain traffic ... rollout.complete(db).await?; // contract, or call rollout.rollback(db).await? Ok(()) } ``` ### Step actions Each [`RolloutStep`] carries exactly one action, built with a constructor, so invalid combinations cannot be represented: | Constructor | What it does | |---|---| | `RolloutStep::apply_schema(id, phase, sql)` | Apply inline DDL (`OVERWRITE` is injected; safe to retry) | | `RolloutStep::run_sql(id, phase, sql)` | Run data-mutation SQL (must be safe to re-run) | | `RolloutStep::assert_sql(id, phase, sql, expect)` | Assert a query's output equals `expect` | | `RolloutStep::remove_entities(id, phase, entities)` | `REMOVE … IF EXISTS` the given objects | Entities are identified with `EntityKey { kind: EntityKind, scope, name }`, where `EntityKind` is an enum (`Table`, `Field`, `Index`, `Module`, and so on) rather than a string. ### Recovering a stuck rollout If a process dies mid-rollout, the rollout is left in a `running_*` or `failed` state and blocks new rollouts. Inspect the recorded state, then recover it: ```rust use surrealkit::{Rollout, RolloutSpec, Surreal}; use surrealkit::engine::any::Any; async fn run(db: &Surreal, spec: RolloutSpec) -> anyhow::Result<()> { // Inspect the recorded state. let rollout = Rollout::new(spec, &[]); if let Some(report) = rollout.status(db).await? { println!("{:?}: {:?}", report.status, report.last_error); } // Last resort: force a wedged rollout to a terminal state so a new one can // start. This does NOT revert schema changes already applied. Reconcile // those with a fresh sync or a follow-up rollout. Rollout::abandon(db, "20260604__add_account").await?; Ok(()) } ``` ## Seeding [`seed`] runs the `.surql` files in a project's `seed/` directory (lexicographic order), applying template variables: ```rust use surrealkit::{seed, TemplateVars, Surreal}; use surrealkit::engine::any::Any; async fn run(db: &Surreal) -> anyhow::Result<()> { seed(db, "database", &TemplateVars::default()).await?; Ok(()) } ``` The second argument is the project folder (the directory containing `seed/`), matching the CLI's `--folder` / `SURREALDB_FOLDER`. ## Template variables `${VAR}` placeholders in schema, seed, or rollout SQL are substituted from a [`TemplateVars`] map before execution. Lookups are case-insensitive, and an undefined variable is an error naming the missing key and file. Pass them via `Sync::vars(...)`, `Rollout::vars(...)`, or the `seed` argument: ```rust use surrealkit::{Sync, TemplateVars}; let mut vars = TemplateVars::default(); vars.insert("schema_prefix", "acme"); Sync::embedded(SCHEMA).vars(vars).run(db).await?; ``` See [Template variables](/docs/manage/schema-migration/template-variables.md) for the full resolution rules. ## Metadata tables SurrealKit maintains two internal tables in your namespace and database, created automatically: | Table | Purpose | |---|---| | `__entity` | Tracks every schema object SurrealKit manages (content hash, tracking key) | | `__rollout` | Tracks rollout execution state (see the status lifecycle above) | ## Next steps - [Library usage example](/docs/manage/schema-migration/library/example.md): full worked sync and rollout programs - [`embed_schema!` macro](/docs/manage/schema-migration/embed-schema-macro.md): bake schema into the binary at compile time - [Type generation](/docs/manage/schema-migration/typegen.md): generate JSON and TypeScript types from the live schema --- Source: https://surrealdb.com/docs/manage/schema-migration/library/example # Library usage example Worked examples of using SurrealKit's library API for both sync and rollouts in a Rust application. Worked examples of using SurrealKit's library API directly in Rust, covering both sync (for development and ephemeral databases) and rollouts (for shared and production databases). For the macro-based approach to sync, see the [`embed_schema!` macro](/docs/manage/schema-migration/embed-schema-macro.md). ## Shared setup Both examples below assume the following `Cargo.toml` and connection setup. ```toml [package] name = "my-app" version = "0.1.0" edition = "2021" [dependencies] surrealkit = "0.7" surrealdb = "2" tokio = { version = "1", features = ["full"] } anyhow = "1" ``` ```rust use surrealkit::{DbCfg, DbOverrides, connect}; let cfg = DbCfg::from_env(None, &DbOverrides::default())?; let db = connect(&cfg).await?; ``` `DbCfg::from_env` reads `SURREALDB_HOST`, `SURREALDB_NAMESPACE`, `SURREALDB_NAME`, `SURREALDB_USER`, and `SURREALDB_PASSWORD` from the environment or a `.env` file. ## Sync Use sync against local or ephemeral databases where fast iteration matters and pruning is safe. Schema files are embedded at compile time with `include_str!`, then applied with the [`Sync`](/docs/manage/schema-migration/library.md) builder. ```rust use anyhow::Result; use surrealkit::{DbCfg, DbOverrides, EmbeddedSchemaFile, Sync, connect, seed, TemplateVars}; static SCHEMA: &[EmbeddedSchemaFile] = &[ EmbeddedSchemaFile { path: "database/schema/users.surql", sql: include_str!("../database/schema/users.surql"), }, EmbeddedSchemaFile { path: "database/schema/posts.surql", sql: include_str!("../database/schema/posts.surql"), }, ]; #[tokio::main] async fn main() -> Result<()> { let cfg = DbCfg::from_env(None, &DbOverrides::default())?; let db = connect(&cfg).await?; // Defaults: prune = true, fail_fast = true. Sync::embedded(SCHEMA).run(&db).await?; // Optionally load seed data after syncing. seed(&db, "database", &TemplateVars::default()).await?; Ok(()) } ``` Cargo re-compiles whenever an `include_str!`-referenced file changes, so the binary always reflects the latest schema. `Sync::embedded(...).run(...)` applies new or changed definitions and removes any that have been deleted from the schema files. Seed files run on every startup, so use `UPSERT` or guard inserts if idempotency matters. ## Rollouts Use rollouts for shared and production databases. Build a [`RolloutSpec`](/docs/manage/schema-migration/library.md) with the builder, wrap it in the `Rollout` facade, then call `start` and `complete` at the appropriate points around your application deployment. ```rust use anyhow::Result; use surrealkit::{ DbCfg, DbOverrides, connect, Rollout, RolloutSpec, RolloutStep, RolloutPhase, RolloutCompatibility, EntityKey, EntityKind, }; #[tokio::main] async fn main() -> Result<()> { let cfg = DbCfg::from_env(None, &DbOverrides::default())?; let db = connect(&cfg).await?; let spec = RolloutSpec::builder("20260604__add_account") .name("Add account table") .compatibility(RolloutCompatibility::Phased) .step(RolloutStep::apply_schema( "create_account", RolloutPhase::Start, "DEFINE TABLE account SCHEMAFULL;", )) .step(RolloutStep::remove_entities( "rollback_account", RolloutPhase::Rollback, vec![EntityKey { kind: EntityKind::Table, scope: None, name: "account".into() }], )) .build(); // `&[]` here means the steps fully describe the entity changes. Pass the // desired post-rollout schema files to have SurrealKit compute the catalog. let rollout = Rollout::new(spec, &[]); rollout.start(&db).await?; Ok(()) } ``` After `start` succeeds, deploy your application. Once the new version is stable, call `complete` to apply the contract phase: ```rust rollout.complete(&db).await?; ``` If something goes wrong before completing, roll back the start phase: ```rust rollout.rollback(&db).await?; ``` If a process is killed mid-rollout, the `__rollout` row can be left in a `running_*` state. Inspect it with `rollout.status(&db).await?` and recover it with the CLI [`rollout repair`](/docs/manage/schema-migration/rollouts.md) command or `Rollout::abandon(&db, "20260604__add_account").await?`. --- Source: https://surrealdb.com/docs/manage/schema-migration/rollouts # Rollouts Rollouts provide a controlled, phased migration workflow for shared and production SurrealDB databases, with review, staged execution, and rollback support. Rollouts are SurrealKit's migration system for shared and production databases. Rather than immediately pushing all changes (as Sync does), Rollouts generate a reviewed manifest and apply it in two phases: an expand phase that adds new definitions without breaking existing consumers, and a contract phase that removes old ones after your application has been updated. Deploying schema and application changes separately means no downtime. ## The phased migration model A complete rollout has three stages: 1. **Start:** applies non-destructive changes: adding tables, fields, indexes, and access methods that new application code will use. The old application code continues to work alongside the new definitions. 2. **App cutover:** you deploy the new version of your application. Both old and new application code remain compatible with the database during this window. 3. **Complete:** applies destructive changes: removing legacy tables, fields, or indexes that are no longer needed. Only run this after the new application version is stable. ## Command workflow ### For an existing database: baseline first If you are adopting SurrealKit on a database that already exists, capture a baseline before planning any rollout: ```bash surrealkit rollout baseline --user root --pass secret ``` This reads the current database schema and writes snapshot files to `database/snapshots/`. Future plans diff against this baseline. You only need to run this once. ### Plan a rollout After editing your schema files, plan a rollout: ```bash surrealkit rollout plan --name add_customer_indexes ``` SurrealKit computes the diff between the current snapshot and your schema files, then generates a timestamped manifest in `database/rollouts/`: ```text database/rollouts/20260302153045__add_customer_indexes.toml ``` Review this file before proceeding. It lists each step, the SQL it will execute, and which phase (start or complete) it belongs to. Commit it to your repository alongside your schema changes. ### Start the rollout ```bash surrealkit rollout start 20260302153045__add_customer_indexes --user root \ --pass secret ``` This applies all start-phase steps. SurrealKit records resumable state in the `__rollout` metadata table, so if the process is interrupted it can pick up where it left off. Only one rollout can be in progress at a time; concurrent starts are blocked. ### Deploy your application After `start` completes, deploy the new version of your application. At this point both the old and new schema definitions coexist in the database. ### Complete the rollout Once the new application version is healthy: ```bash surrealkit rollout complete 20260302153045__add_customer_indexes --user root --pass secret ``` This applies all complete-phase steps (removing legacy definitions) and marks the rollout as done. ## Rollback If something goes wrong after `start` but before `complete`, you can undo the start-phase changes: ```bash surrealkit rollout rollback 20260302153045__add_customer_indexes --user root --pass secret ``` Rollback reverses what `start` applied. It is not available after `complete` has run, so verify your application is working correctly before calling `complete`. ## Other commands ### status Inspect the state of rollouts stored in the database: ```bash surrealkit rollout status --user root --pass secret ``` ### lint Validate a manifest file without connecting to the database or applying anything: ```bash surrealkit rollout lint 20260302153045__add_customer_indexes --user root \ --pass secret ``` This is useful in CI to catch manifest errors before they reach a shared environment. ### repair If `rollout complete` (or `rollback`) is killed mid-flight, the `__rollout` row can be left in an intermediate state (`running_complete`, `running_rollback`, or `running_start`) even though the schema changes have already been applied. Re-running `complete` or `rollback` will not always heal the metadata, because the underlying SQL steps are already done, and the stuck rollout blocks any new one from starting. Use `repair` to finish the metadata transition without re-running any SQL: ```bash surrealkit rollout repair 20260302153045__add_customer_indexes --user root --pass secret ``` The behaviour depends on the stuck state: | Stuck state | Repair result | |---|---| | `running_complete` | Flips to `completed` and restores the target entities | | `running_rollback` | Flips to `rolled_back` and restores the source entities | | `running_start` | Flips to `failed`; re-run `start` (idempotent) or `rollback` | Repair never re-executes per-step SQL. It only reconciles the `__rollout` and `__entity` tables so subsequent `sync` and `plan` runs see a clean state. ## Rollout state machine Each rollout transitions through the following states, recorded in the `__rollout` table: ```text planned → running_start → ready_to_complete → running_complete → completed │ └── running_rollback → rolled_back ``` | State | Meaning | |---|---| | `planned` | Manifest exists; neither start nor complete has run | | `running_start` | Start phase is in progress or was interrupted | | `ready_to_complete` | Start phase completed successfully | | `running_complete` | Complete phase is in progress or was interrupted | | `completed` | Both phases completed (terminal) | | `running_rollback` | Rollback is in progress or was interrupted | | `rolled_back` | Start phase was reversed (terminal) | | `failed` | A phase failed; recover with `rollback` or `repair` | `completed` and `rolled_back` are terminal. The `running_*` states and `failed` are stuck states left by an interrupted run; recover them with [`repair`](#repair). Only one rollout may be in a non-terminal state at a time. ## Artifacts | Path | Description | |---|---| | `database/rollouts/*.toml` | Generated rollout manifests; commit these | | `database/snapshots/schema_snapshot.json` | Schema state snapshot used for diffing | | `database/snapshots/catalog_snapshot.json` | Catalog snapshot | ## Next steps - [Template Variables](/docs/manage/schema-migration/template-variables.md): parameterise schema files for different environments - [Testing](/docs/manage/schema-migration/testing.md): validate schema and permissions before promoting a rollout --- Source: https://surrealdb.com/docs/manage/schema-migration/sync # Sync Sync pushes your .surql schema files to SurrealDB immediately, keeping the database in desired state. Use it for local development and ephemeral environments. Sync applies all `.surql` files in `database/schema/` to the connected SurrealDB database in a single pass, bringing the database into alignment with your files. It tracks each file by content hash so that unchanged files are skipped on subsequent runs. ## Basic usage ```bash surrealkit sync --user root --pass secret ``` SurrealKit connects to the database, reads your schema files, applies any definitions that have changed, and removes definitions for files that have been deleted (pruning). ## Watch mode During active development, pass `--watch` to keep SurrealKit running and automatically re-sync on any file change: ```bash surrealkit sync --watch --user root --pass secret ``` SurrealKit debounces rapid consecutive saves, so editing multiple files in quick succession triggers a single sync rather than many overlapping ones. ## How it works When Sync runs: 1. All `.surql` files in `database/schema/` are read and their content hashes computed. 2. Hashes are compared against the `__entity` metadata table stored in the database. 3. New or changed files have their SQL statements applied. 4. Files that existed in a previous sync but are no longer present are pruned: their definitions are removed from the database. 5. The `__entity` table is updated to reflect the new state. Any definition that exists in the database but not in a file will be removed on the next sync. Pruning emits `REMOVE … IF EXISTS`, so a tracking entry that points at an object which is already gone no longer errors: drift between SurrealKit's metadata and the database self-heals on the next sync. ## Tracked schema objects Sync manages the full range of SurrealDB schema definitions, tracking each one in the `__entity` table so it can be updated and pruned. This includes `TABLE`, `FIELD`, `INDEX`, `ANALYZER`, `PARAM`, `FUNCTION`, `ACCESS`, and `USER` definitions, as well as `BUCKET`, `SEQUENCE`, `CONFIG`, `MODEL`, and `MODULE`. ## Pruning on shared databases Pruning is safe when you are the sole owner of the database. On a shared database (one already containing SurrealKit metadata from another developer or environment), automatic pruning could remove definitions that others have added. SurrealKit detects this situation and refuses to prune unless you pass an explicit flag: ```bash surrealkit sync --allow-shared-prune --user root --pass secret ``` Do not use this flag routinely on shared databases. Consider switching to [Rollouts](/docs/manage/schema-migration/rollouts.md) for any database that more than one person or service writes to. ## Running non-DEFINE statements By default, schema files may only contain `DEFINE` statements. To allow other statements such as `INSERT`, `UPDATE`, or `CREATE` in your schema files, pass `--allow-all-statements`: ```bash surrealkit sync --allow-all-statements --user root --pass secret ``` This disables catalog entity tracking for the run: SurrealKit can no longer reason about individual objects, so only file-level content hashes are tracked. Pruning of individual definitions is unavailable while this flag is in effect. Prefer keeping data changes in [seed files](/docs/manage/schema-migration/getting-started/new-databases.md) or [rollout steps](/docs/manage/schema-migration/rollouts.md) rather than schema files. ## Metadata tables SurrealKit creates two internal tables in the target database: | Table | Purpose | |---|---| | `__entity` | Stores the content hash and file key for every schema definition SurrealKit manages. Used to detect changes and drive pruning. | | `__rollout` | Stores rollout state. Created when you first use Rollouts; may be present even if you use Sync only. | These tables are managed by SurrealKit and should not be modified directly. ## Vite plugin If you are using Vite, the `vite-plugin-surrealkit` package integrates Sync with the dev server: ```bash npm install --save-dev vite-plugin-surrealkit ``` ```ts // vite.config.ts import { defineConfig } from 'vite'; import { surrealkitPlugin } from 'vite-plugin-surrealkit'; export default defineConfig({ plugins: [ surrealkitPlugin({ syncArgs: [], }), ], }); ``` On `vite dev`, the plugin: 1. Runs `surrealkit sync` once at startup. 2. Watches `database/schema/**/*.surql`. 3. Re-runs sync on any file change, debouncing concurrent runs. ## Reference | Flag | Default | Description | |---|---|---| | `--watch` | off | Re-sync automatically when schema files change | | `--allow-shared-prune` | off | Permit pruning on a database that already has SurrealKit metadata | | `--allow-all-statements` | off | Permit non-`DEFINE` statements (e.g. `INSERT`, `UPDATE`, `CREATE`) in schema files; disables catalog entity tracking | | `--dry-run` | off | Print what would be applied without modifying the database | | `--fail-fast` | off | Stop on the first error rather than continuing | The schema root defaults to `./database`. Override it with the global `--folder` flag or the `SURREALDB_FOLDER` environment variable. --- Source: https://surrealdb.com/docs/manage/schema-migration/template-variables # Template variables Template variables let you parameterise .surql schema files with environment-specific values, substituted at runtime by SurrealKit. Template variables are placeholder tokens in your `.surql` schema files that SurrealKit substitutes with actual values at runtime. A single set of schema files can then cover all environments (development, staging, production) with names, prefixes, or other values varying per environment. ## Syntax Use `${VAR_NAME}` anywhere in a `.surql` file: ```surql -- database/schema/roles.surql DEFINE ROLE ${talent_role} PERMISSIONS FULL; -- database/schema/tables.surql DEFINE TABLE ${schema_prefix}_orders SCHEMAFULL; DEFINE FIELD owner ON ${schema_prefix}_orders TYPE record; ``` Variable names are case-insensitive: `${FOO}`, `${foo}`, and `${Foo}` all match the key `FOO`. If a variable is referenced in a file but has no value configured, SurrealKit exits with an error and a clear message indicating which variable is missing. ## Resolution order SurrealKit resolves variable values in the following order (first match wins): 1. **CLI flag:** `--var KEY=VALUE` (repeatable) 2. **Environment variable:** `SURREALKIT_VAR_` (case-insensitive key matching) 3. **`surrealkit.toml` `[variables]` section** ### CLI flags ```bash surrealkit sync --var schema_prefix=acme --var talent_role=talent_rw surrealkit rollout start my_rollout --var schema_prefix=acme ``` ### Environment variables **Bash** ```bash export SURREALKIT_VAR_SCHEMA_PREFIX=acme export SURREALKIT_VAR_TALENT_ROLE=talent_rw surrealkit sync --user root --pass secret ``` **PowerShell** ```powershell $env:SURREALKIT_VAR_SCHEMA_PREFIX = "acme" $env:SURREALKIT_VAR_TALENT_ROLE = "talent_rw" surrealkit sync --user root --pass secret ``` ### `surrealkit.toml` ```toml [variables] schema_prefix = "myapp" talent_role = "talent_rw" environment = "development" ``` Place `surrealkit.toml` in the root of your project. It is read automatically. The same file also holds the [`[typegen]` section](/docs/manage/schema-migration/typegen.md) when TypeScript generation is enabled. ## Escaping To emit a literal `${...}` string in generated SQL (for example, inside a SurrealQL string value), double the dollar sign: ```surql SET note = 'pass $${MY_VAR} literally'; ``` SurrealKit outputs this as `pass ${MY_VAR} literally` without substitution. ## Which commands apply substitution Template Variables are substituted when applying schema to a database. They are not substituted in commands that only read or plan: | Command | Substitution applied | |---|---| | `sync` | Yes | | `seed` | Yes | | `rollout start` | Yes | | `rollout complete` | Yes | | `rollout rollback` | Yes | | `rollout plan` | No | | `rollout baseline` | No | | `rollout status` | No | | `rollout lint` | No | ## Known limitations **Hash-based re-sync:** SurrealKit tracks files by content hash. Changing a variable value without editing the file will not trigger a re-sync. Touch the file to force re-application: ```bash touch database/schema/tables.surql surrealkit sync --user root --pass secret ``` **Watch mode:** Variables in `surrealkit.toml` are resolved once at startup. If you edit `surrealkit.toml` while `--watch` is running, restart SurrealKit for the new values to take effect. Variables passed via `--var` or environment variables are re-read on each sync cycle. **Catalog snapshots:** Entity names containing `${VAR}` appear literally (without substitution) in `catalog_snapshot.json`. This is cosmetic and does not affect functionality. **String literals:** Substitution is purely textual. `${VAR}` inside a SurrealQL string is replaced with the variable's value, which may or may not be the intended behaviour. Test carefully when embedding variables inside string literals. --- Source: https://surrealdb.com/docs/manage/schema-migration/templates # Project templates surrealkit init scaffolds a project from a template with selectable features. Use the bundled template, pick only the features you need, or supply your own. `surrealkit init` scaffolds a new project from a template and lets you choose which optional features to include. It always writes the base project layout first, then copies in the schema, seed, and test files for the features you select. ```bash surrealkit init ``` In a terminal this shows a checklist of the template's features. Pick the ones you want and SurrealKit writes their files into `database/`. ## Base layout Every `init` creates the base project regardless of which features you choose: ```text database/ ├── schema/ # .surql schema definition files ├── rollouts/ # rollout manifests (generated) ├── snapshots/ # schema and catalog snapshots ├── seed/ # optional seed data ├── tests/ # test suites and config └── setup.surql # runs before sync surrealkit.toml # project configuration ``` ## Choosing features without a prompt When there is no terminal (such as CI), or when you pass any of the flags below, `init` runs without prompting: | Flag | Behaviour | |---|---| | `--feature ` | Enable a feature by id. Repeatable, and pulls in what it requires. | | `-y`, `--yes` | Take the template's default features. | | `--minimal` | Scaffold the base project only, with no features. | | `--force` | Overwrite files that already exist. The default is to skip them. | ```bash surrealkit init --feature organizations --feature teams surrealkit init -y surrealkit init --minimal ``` A feature can depend on other features. Selecting one adds what it requires, and `init` prints what it added. ## Using your own template Point `--from` at a local path or a git repository instead of the bundled template, or pick a bundled template by name with `--template`: ```bash surrealkit init --from ./path/to/template surrealkit init --from https://github.com/your-org/your-template.git surrealkit init --from https://github.com/your-org/your-template.git#v1.0.0 surrealkit init --template default ``` Git sources are cloned with `git clone --depth 1`, so `git` must be on your `PATH`. Pin a branch, tag, or commit with `#rev`, and target a subdirectory with `#rev:subdir`. ## Template layout A template is a directory with a `template.toml` manifest plus the files each feature contributes: ```toml schema_version = 1 name = "default" display_name = "My starter" description = "Shown above the feature checklist" [[features]] id = "organizations" name = "Organizations" description = "Shown next to the feature in the checklist" default = false schema = ["schema/organization/organization.surql"] seed = ["seed/organization_permissions.surql"] suites = ["tests/suites/organization.toml"] fixtures = ["tests/fixtures/organization_seed.surql"] [[features]] id = "teams" name = "Teams" requires = ["organizations"] schema = ["schema/team/team.surql"] ``` Each feature lists the files it adds, grouped by where they land: - `schema` files are copied into `database/schema/` - `seed` files into `database/seed/` - `suites` files into `database/tests/suites/` - `fixtures` files into `database/tests/fixtures/` Set `default = true` to pre-check a feature in the prompt and include it with `-y`. Use `requires` to declare dependencies on other features. ## Bundled template The bundled `default` template provides an organisation and access-control model with four opt-in features: - **Organisations:** organisations, roles that bundle permissions, a per-app permission catalogue, employees, and invitations. - **Teams:** teams within an organisation, with per-member roles. - **Organisation units:** a department and region hierarchy with unit-scoped permissions. - **Subsidiaries and delegation:** parent and child organisations with cross-org delegated permissions. Teams, units, and subsidiaries each require the organisations feature. ## Next steps - [New databases](/docs/manage/schema-migration/getting-started/new-databases.md): start a fresh project with SurrealKit - [Sync](/docs/manage/schema-migration/sync.md): push your scaffolded schema to a database - [Testing](/docs/manage/schema-migration/testing.md): run the test suites the template scaffolded --- Source: https://surrealdb.com/docs/manage/schema-migration/testing # Testing SurrealKit includes a built-in testing framework for validating schema correctness, permissions, and API behaviour across multiple actor types. SurrealKit includes a testing framework that lets you write declarative test suites for your SurrealDB schema. Tests run against an isolated ephemeral database per suite, so they are safe to run in any environment without affecting persistent data. ## Running tests ```bash surrealkit test ``` SurrealKit reads all suite files from `database/tests/suites/*.toml`, runs them in parallel (configurable), and exits non-zero if any case fails. ## Project structure ```text database/tests/ ├── config.toml # global defaults └── suites/ ├── security.toml └── api.toml ``` ### Global config `database/tests/config.toml` sets defaults shared across all suites: ```toml [defaults] timeout_ms = 10000 base_url = "http://localhost:8000" [actors.root] kind = "root" ``` ## Test types SurrealKit supports five test types, specified via the `kind` field on each test case. ### `sql_expect` Runs a SurrealQL statement and asserts whether it succeeds or fails: ```toml [[cases]] name = "guest_cannot_create_order" kind = "sql_expect" actor = "guest" sql = "CREATE order CONTENT { total: 10 };" allow = false error_contains = "permission" ``` Optional `assertions` check the returned data: ```toml [[cases]] name = "user_sees_own_profile" kind = "sql_expect" actor = "user_alice" sql = "SELECT * FROM user WHERE id = $auth.id;" allow = true [[cases.assertions]] path = "0.id" equals_auth = "$auth.id" ``` ### `permissions_matrix` Validates that a single actor has the expected create / select / update / delete permissions on a table or record: ```toml [[cases]] name = "reader_cannot_modify_orders" kind = "permissions_matrix" actor = "reader" table = "order" record_id = "order:test" [[cases.rules]] action = "select" allow = true [[cases.rules]] action = "update" allow = false error_contains = "permission" ``` ### `schema_metadata` Asserts structural facts about the schema: that a field exists with a given type, that an index is defined, and so on. ### `schema_behavior` Tests computed fields, functions, and record relations by asserting on the values returned after specific operations. ### `api_request` Tests HTTP API endpoints, useful when your SurrealDB instance exposes a custom API layer: ```toml [[cases]] name = "orders_endpoint_returns_200" kind = "api_request" actor = "root" method = "GET" path = "/api/orders" expected_status = 200 [[cases.body_assertions]] path = "0.id" exists = true ``` ## Actors Each test case runs as a named actor with a specific authentication method. Actors are defined in `config.toml` or at the suite level. | Actor kind | When to use | |---|---| | `root` | Full root-level access | | `database` | Database-level user credentials | | `record` | Record access via signup / signin | | `token` | JWT token from an environment variable | | `headers` | Custom HTTP headers (e.g. tenant ID) | ```toml [actors.user_alice] kind = "record" access = "app_access" [actors.user_alice.signin_params] email = "alice@example.com" password = "secret" [actors.tenant_a] kind = "headers" headers = { "x-tenant-id" = "tenant_a" } ``` ## Filtering | Flag | Description | |---|---| | `--suite ` | Run only suites whose name matches the glob | | `--case ` | Run only cases whose name matches the glob | | `--tag ` | Run only cases tagged with the given tag (repeatable) | | `--fail-fast` | Stop on the first failure | | `--parallel ` | Number of parallel execution threads | ## Debugging | Flag | Description | |---|---| | `--keep-db` | Preserve the ephemeral database after the run for manual inspection | | `--no-sync` | Skip the schema sync phase before running tests | | `--no-seed` | Skip the seeding phase before running tests | | `--json-out ` | Write a machine-readable JSON report to the specified file | ## Next steps - [CI / CD](/docs/manage/schema-migration/testing/ci-cd.md): integrate tests into automated pipelines with GitHub Actions and Docker Compose --- Source: https://surrealdb.com/docs/manage/schema-migration/testing/ci-cd # CI / CD Run SurrealKit tests in CI pipelines using the official GitHub Action, with JSON output for test reporting and Docker Compose for full end-to-end environments. Tests exit non-zero on any failure, produce machine-readable JSON output, and there is an official GitHub Action that handles installation and execution. ## JSON output Pass `--json-out` to write a structured report alongside the human-readable output: ```bash surrealkit test --fail-fast --json-out results.json ``` The JSON file can be consumed by CI reporting tools or used for dashboards and notifications. ## GitHub Actions The [`surrealkit-action`](https://github.com/surrealdb/surrealkit-action) installs the SurrealKit CLI and runs any SurrealKit command. It supports Linux x64, macOS ARM64, and Windows x64. ### Basic test workflow ```yaml name: tests on: [push, pull_request] jobs: surrealkit-test: runs-on: ubuntu-latest services: surrealdb: image: surrealdb/surrealdb:latest ports: - '8000:8000' options: >- --health-cmd "/surreal is-ready" --health-interval 2s --health-timeout 5s --health-retries 10 env: SURREAL_USER: root SURREAL_PASS: secret steps: - uses: actions/checkout@v4 - uses: surrealdb/surrealkit-action@v1 with: command: test host: http://localhost:8000 user: root pass: secret args: --fail-fast --json-out results.json ``` ### Running a rollout in CI You can also use the action to apply a rollout as part of a deployment workflow: ```yaml - uses: surrealdb/surrealkit-action@v1 with: command: rollout start args: ${{ env.ROLLOUT_ID }} host: ${{ secrets.SURREALDB_HOST }} user: ${{ secrets.SURREALDB_USER }} pass: ${{ secrets.SURREALDB_PASSWORD }} ``` ## Docker Compose For end-to-end testing that mirrors a production-like environment, run SurrealKit alongside SurrealDB in Docker Compose: ```yaml services: surrealdb: image: surrealdb/surrealdb:latest command: start --user root --pass secret healthcheck: test: ["CMD", "/surreal", "is-ready"] interval: 1s timeout: 5s retries: 30 surrealkit: image: ghcr.io/surrealdb/surrealkit:latest depends_on: surrealdb: condition: service_healthy volumes: - ./database:/database:ro command: - --host=http://surrealdb:8000 - --ns=main - --db=main - --user=root - --pass=secret - test - --fail-fast - --json-out=/database/results.json ``` Run with: ```bash docker compose run --rm surrealkit ``` The `database/` directory is mounted read-only; the container reads your schema files and test suites but cannot write back to them. The JSON report is written to `database/results.json` on the host. ## Seeding before tests SurrealKit runs a setup phase before executing tests. By default this includes syncing your schema and running any seed files. To skip individual phases: ```bash surrealkit test --no-sync # skip schema sync, use whatever is already in the DB surrealkit test --no-seed # skip seed data ``` In most CI environments you want both phases to run so each test job starts from a clean, consistent state. --- Source: https://surrealdb.com/docs/manage/schema-migration/typegen # Type generation surrealkit typegen introspects a live database and emits a structured JSON schema document, with optional TypeScript types for the SurrealDB JavaScript SDK. `surrealkit typegen` introspects a live database and emits a structured description of its schema. JSON is the primary output; TypeScript interfaces for the [SurrealDB JavaScript SDK](/docs/reference/javascript.md) can be generated alongside it. ```bash surrealkit typegen --user root --pass secret ``` By default this writes a JSON document to `database/types/schema.json`. SurrealKit's internal bookkeeping tables (`__entity`, `__rollout`) are excluded from the output. ## Command flags | Flag | Default | Description | |---|---|---| | `--out ` | `{folder}/types/schema.json` | Write the JSON to a specific path | | `--stdout` | off | Print the JSON to stdout instead of writing a file | | `--compact` | off | Emit compact, single-line JSON instead of pretty-printed | ```bash # Pretty-printed JSON to the default location surrealkit typegen --user root --pass secret # Pipe compact JSON into another tool surrealkit typegen --stdout --compact --user root --pass secret ``` The JSON document includes the namespace, database, generation timestamp, and a typed description of every table, field, and function in the schema. ## TypeScript output TypeScript generation is opt-in through the `[typegen]` section of `surrealkit.toml`. Set `typescript` to the directory where the generated `index.ts` should be written: ```toml [typegen] typescript = "./src/types" format = "biome check --write" ``` | Key | Description | |---|---| | `typescript` | Directory for the generated `index.ts`. Setting it enables TypeScript output for `typegen` and `sync --watch`. | | `format` | Optional formatter command run on the generated file after writing (for example `biome check --write`, `prettier --write`, or `eslint --fix`). | With `typescript` configured, `surrealkit typegen` writes both the JSON document and an `index.ts`: ```ts // Generated by SurrealKit - do not edit. // Run `surrealkit typegen` to regenerate. import type { RecordId } from 'surrealdb'; export interface User { id: RecordId<'user'>; name: string; email: string; } ``` The emitter targets the SurrealDB JavaScript SDK (v2): one interface per table, every record carrying a typed `id: RecordId<'table'>`, with schema field types mapped to the SDK's wrapper types. ### Formatter When `format` is set, SurrealKit appends the generated file path to the command and runs it, so the output matches your project's house style. The command inherits the working directory so the formatter discovers your project's own config. Formatter failures (a missing binary or a non-zero exit) are reported as warnings and never fail `typegen` or `sync --watch`. ## Regenerating on sync When `[typegen] typescript` is configured, `surrealkit sync --watch` regenerates the TypeScript types whenever the schema changes, so your types stay current with the schema during local development: ```bash surrealkit sync --watch --user root --pass secret ``` Regeneration is gated on actual schema changes (or a missing output file), so idle watch ticks do not re-introspect the database. ## Library API Type generation is also available from the [`surrealkit::typegen`](/docs/manage/schema-migration/library.md) module when embedding SurrealKit in Rust: ```rust use surrealkit::typegen::{generate, render_typescript, write_typescript}; use surrealkit::Surreal; use surrealkit::engine::any::Any; async fn run(db: &Surreal) -> anyhow::Result<()> { // Introspect the database into a structured document (no filesystem IO). let doc = generate(db).await?; // Render TypeScript as a string … let ts = render_typescript(&doc)?; println!("{ts}"); // … or write it to `/index.ts`. write_typescript(&doc, std::path::Path::new("src/types"))?; Ok(()) } ``` `generate` returns a `SchemaTypes` document that the JSON and TypeScript emitters share, so a single introspection can produce both formats. `write_typescript_formatted` writes the file and runs an optional formatter command in the same call. ## Next steps - [Sync](/docs/manage/schema-migration/sync.md): keep the database aligned with your schema files - [Using SurrealKit as a library](/docs/manage/schema-migration/library.md): drive SurrealKit from Rust --- Source: https://surrealdb.com/docs/manage/self-hosted # Self-hosted SurrealDB deployment models on your own infrastructure. Containers, configuration, backups, monitoring, and upgrades. Running SurrealDB yourself is quick to start and gives you full control over storage, configuration, and placement. Everything needed to start a server and to import and export data is in the [command-line tool](/docs/reference/cli/surrealdb-cli/overview.md), packaged and distributed as a single executable that can be downloaded, installed, or run from within Docker. For initial installation, see the [installation guide](/docs/running/installation.md). If you would rather not operate the infrastructure, see [Instances](/docs/manage/instances.md) for the managed option. ## Deployment guides | Guide | Typical use | | --- | --- | | [Deployment models](/docs/manage/self-hosted/deployment-models.md) | Choosing between single-node, multi-node, embedded, and managed | | [Docker](/docs/manage/self-hosted/docker.md) | Quickest path; RocksDB with a volume mount | | [Kubernetes](/docs/manage/self-hosted/kubernetes.md) | Single SurrealDB pod with RocksDB on a persistent volume | | [Managed Kubernetes](/docs/manage/self-hosted/managed-kubernetes.md) | Paths on Amazon EKS, Google GKE, and Azure AKS | Most self-hosted workloads use **RocksDB** on disk - one SurrealDB process per database file. ## Operating a self-hosted instance - [Configuration](/docs/manage/self-hosted/configuration.md) - server startup options, environment variables, and storage engine selection. - [Backups and recovery](/docs/manage/self-hosted/backups-and-recovery.md) - export and import commands, backup strategies, and disaster recovery. - [Monitoring and observability](/docs/manage/self-hosted/monitoring-and-observability.md) - health endpoints, metrics, and tracing integration. - [Upgrades and patching](/docs/manage/self-hosted/upgrades-and-patching.md) - version upgrades, the `surreal fix` migration tool, and compatibility notes. Once an instance is running, the [Observability](/docs/manage/observability.md) section covers the built-in metrics, audit logs and slow-query logs you can scrape over Prometheus or push over OTLP for production monitoring. ## Enterprise Edition Self-hosted clusters can run [Enterprise Edition](/docs/manage/enterprise.md), which adds distributed live queries, audit logging, FIPS-validated cryptography and support tiers on top of the same server. --- Source: https://surrealdb.com/docs/manage/self-hosted/backups-and-recovery # Backups & recovery Create and restore SurrealQL backups with the CLI, plan backup strategies, and prepare for disaster recovery. Reliable backups are central to operating self-hosted SurrealDB. The CLI provides logical backups as SurrealQL using [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md), which dumps namespaces, databases, schemas, and data in a form you can version, diff, and store outside the database. Schedule exports when traffic is lower if large datasets make exports lengthy, and ensure the export process has sufficient disk space and time to finish without overlapping the next run. To **restore**, use [`surreal import`](/docs/reference/cli/surrealdb-cli/commands/import.md) against a running instance, pointing at the correct endpoint, credentials, and target namespace or database. After import, run sanity checks: record counts, critical queries, and application smoke tests before switching production traffic. If you import into a shared cluster, coordinate with application owners so schema or data changes do not race with live writers. **Backup strategies** usually combine scheduled exports with storage-engine or volume **snapshots** where available. Exports are portable and engine-agnostic; snapshots capture disk state quickly but depend on filesystem layout and binary compatibility. Use both when compliance, RPO, or RTO targets demand redundancy across failure modes. **Point-in-time recovery** is not implicit in a single export: each file reflects one moment. Narrow your recovery window by exporting more often, maintaining replicas, or using journaled storage upstream of SurrealDB. Document which artefact (export time, snapshot ID, replica lag) corresponds to a given recovery objective. For **disaster recovery**, keep copies off-site and logically separated from production credentials, encrypt dumps at rest, and test full restores onto clean hardware or namespaces regularly. Maintain runbooks with exact commands, verification queries, and escalation paths so restores succeed when primary systems are unavailable. Validate backup **integrity** periodically: restore to a non-production instance and run application-level checks. Corruption or partial writes are easier to catch in rehearsal than during an incident. Align retention with policy: keep enough history for audits and incident investigation, but prune or tier old exports so storage costs stay predictable. Where legal holds apply, flag those backups and exclude them from automated deletion. --- Source: https://surrealdb.com/docs/manage/self-hosted/configuration # Configuration Configure a self-hosted SurrealDB server: CLI options, environment variables, storage, networking, auth, and TLS. Most server behaviour is fixed at process start. The [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) command is the primary entry point: pass flags for bind addresses, storage paths, authentication, TLS material, logging, and feature toggles. Review the command reference whenever you upgrade, as defaults and supported flags can change between releases. Keep a short changelog of server flags per environment so rollbacks and comparisons stay straightforward. Equivalent settings are often available as [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md), which suits containers, systemd units, and secret managers. Prefer one coherent source of truth (file, env, or flags) so configuration does not diverge silently across hosts. **Storage engines** are selected by the path on [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md). For most self-hosted servers, use **RocksDB** (`rocksdb://…`) for on-disk production data. The default **`memory`** backend (SurrealMX) suits development and tests as well as solutions that make use of its Redis-like persistent storage, while the optional **SurrealKV** (`surrealkv://…`, beta) is an alternative single-node disk engine with a smaller tuning surface. Capacity planning, backups, and versioning options differ by engine - see the [Deployment models](/docs/manage/self-hosted/deployment-models.md) and [Run a single-node, on-disk server](/docs/running/file-backed.md) pages for more. **Bind address and port** determine where clients and operators reach the server. Restrict admin interfaces to trusted networks, use a load balancer or service mesh for TLS termination if that matches your platform, and document ports for HTTP, WebSocket, and any separate metrics endpoints you expose. Firewall rules should default to deny: allow only the subnets and ports your applications and operators require. Set **authentication** and **root user** credentials explicitly in production. Root credentials grant broad access: rotate them on a schedule, store them in a vault, and avoid reusing development passwords. **TLS**: supply certificates and keys via `surreal start` or env vars so wire traffic is encrypted. Renew certificates before expiry, pin trusted CAs where appropriate, and consider a reverse proxy for advanced policies while keeping SurrealDB’s own TLS configuration minimal and auditable. After changes, confirm the effective configuration by inspecting startup logs and probing only the intended listeners. In regulated environments, record who approved credential and TLS updates and when they took effect. --- Source: https://surrealdb.com/docs/manage/self-hosted/deployment-models # Deployment models Deployment models for SurrealDB - managed instances, single-node RocksDB, multi-node clusters, and embedded runtimes - and how to choose between them. SurrealDB separates the query engine (compute) from the underlying storage layer. The same SurrealQL, APIs, and client SDKs work across embedded devices, single-node servers, distributed clusters, and [managed instances](/docs/manage/instances.md) - so you can change how the database runs without rewriting application code. This page explains the available deployment options, storage engines, and how to choose the right architecture for your workload. For how the compute and storage layers interact, see [Architecture](/docs/learn/data-models/architecture.md). ## Deployment models summary | Deployment model | Storage engine(s) | Scaling | High availability | Versioning (`VERSION`) | Best for | Managed option | | --- | --- | --- | --- | --- | --- | --- | | Managed | Start (single-node) to Scale (multi-node cluster) | Vertical and horizontal (by plan) | Fully managed HA on Scale | Where enabled | Production without operating infrastructure | Yes | | Single node | RocksDB (recommended for server workloads); SurrealKV (beta) | Vertical | Filesystem backups | Where enabled (see engine docs) | Development and single-node production | Self-hosted (Community Edition) | | Multi-node | Distributed transactional storage | Horizontal; distributed storage | High (replication and consensus) | Where enabled on the storage tier | Large-scale production workloads | [Scale](https://surrealdb.com/pricing/scale) plan; self-hosted Enterprise | | Embedded | SurrealMX (memory), SurrealKV (beta), RocksDB, IndexedDB (browser) | Application-bound | Application-bound | Where enabled | Offline, edge, browser, and low-latency local apps | No | ## Architecture overview SurrealDB consists of two major layers: **Query layer** - Parses and executes SurrealQL - Authenticates connections and sessions - Enforces table- and field-level permissions on reads and writes - Plans index-backed queries, maintains index entries during writes, and coordinates transactions **Storage layer** - Handles persistence and durability - Determines scalability, temporal versioning, replication, and fault tolerance Because these layers are separated in the architecture, applications can move between deployment models without changing application code or queries. Embedded and browser deployments still use both layers, but they run in-process rather than as separate services. ## Managed instances [Managed instances](/docs/manage/instances.md) provide a fully managed deployment platform built on scalable, fault-tolerant infrastructure. They remove the operational complexity of running clusters while providing production-ready deployments. Plans range from **Start** (single-node, vertically scalable instances) to **Scale** (multi-node clusters on distributed storage, minimum three compute units). Start is enough for many workloads; Scale is aimed at business-critical production where a single-node outage would stop the application and where operating HA yourself (Kubernetes, replication, patching, and backups) would otherwise consume platform team time. See [Architecture](/docs/manage/instances/architecture.md) and [Pricing](https://surrealdb.com/pricing). ### Features - Managed SurrealDB infrastructure - Vertical and horizontal scaling without rebuilding the cluster - High availability - Managed backups - Secure connectivity - Multi-node distributed architecture on the Scale plan - Production monitoring and operations ### Best for - Managed infrastructure - Rapid production deployment - Scaling applications - SaaS platforms - Enterprise workloads ### Benefits compared to self-hosting | Feature | Self-hosted | Managed | | --- | --- | --- | | Infrastructure management | Required | Managed | | Backups | Manual | Managed | | Scaling | Manual | Managed resize, no cluster rebuild | | HA setup | Manual | Built-in | | Upgrades | Manual | Managed | | Cluster operations | Manual | Managed | ## Single-node (RocksDB) Single-node deployments run SurrealDB as a standalone server process using [RocksDB](https://rocksdb.org/). This is the simplest and most widely used production architecture on disk. RocksDB is a high-performance LSM-tree key-value store optimised for high write throughput, SSD storage, and predictable persistence. **Best for** - Small to medium production workloads - Internal tooling - Development environments - Applications without horizontal scaling or built-in cluster fault tolerance ### Limitations - Vertical scaling only - No built-in distributed fault tolerance For setup examples, see [Run a single-node, on-disk server](/docs/running/file-backed.md). ### Common deployment methods **CLI** ```bash surreal start rocksdb://path/to/database ``` **Docker** ```bash docker run --rm \ -p 8000:8000 \ surrealdb/surrealdb:latest \ start --user root --pass secret rocksdb://data/database.db ``` See also [Self-hosted](/docs/manage/self-hosted.md) for Docker, Kubernetes, and platform guides. ### SurrealKV (beta) For single-node and embedded workloads, [SurrealKV](https://github.com/surrealdb/surrealkv) is SurrealDB’s own LSM-backed storage engine, developed in concert with the database rather than as a third-party dependency. That co-development shows up in day-to-day operation: SurrealKV exposes a comparatively small [configuration surface](/docs/reference/cli/surrealdb-cli/commands/start.md#supported-parameters-for-surrealkv) and [environment variable set](/docs/reference/cli/surrealdb-cli/environment-variables.md#surrealkv-environment-variables) next to RocksDB’s extensive tuning knobs, and is aimed at embedded and local-first scenarios. SurrealKV remains **beta**. For conservative production on-disk server deployments today, **prefer RocksDB**. For embedded deployments where smaller resident memory and in-process behaviour are priorities, SurrealKV is the path to evaluate first. It is nonetheless a serious storage path inside the project: features such as temporal reads via the [`VERSION`](/docs/reference/query-language/statements/select.md#the-version-clause) clause were exercised on SurrealKV first and have since been extended to [SurrealMX](/docs/running/in-memory.md) and RocksDB where the engine supports them. To try SurrealKV on a server, see the SurrealKV tab on [Run a single-node, on-disk server](/docs/running/file-backed.md) and the [`surreal start`](/docs/reference/cli/surrealdb-cli/commands/start.md) storage parameters. ## Multi-node clusters For high availability and horizontal scalability, SurrealDB supports distributed deployments backed by a distributed transactional storage layer built for large-scale clusters. ### Architecture In distributed deployments: - Multiple query nodes scale horizontally against shared storage - The storage layer manages replication, consensus, fault tolerance, and distributed transactions - Object-storage backing (rolling out on Scale) places transactional data in commodity object storage while frequently accessed data stays on local disk Diagram of a multi-node cluster: requests to [instance-id].surreal.cloud fan out across three nodes, all backed by centralised storage on AWS S3. This architecture enables zero-downtime scaling, resilient clusters, high-throughput workloads, geographically distributed applications, and (as Scale features roll out) database branching, instant replication and recovery, and lower storage costs at scale. For managed multi-node clusters, use the [Scale](https://surrealdb.com/pricing/scale) plan on a [managed instance](/docs/manage/instances.md). For self-hosted multi-node clusters on Kubernetes, use [SurrealDB Enterprise](https://surrealdb.com/enterprise); the [Managed Kubernetes](/docs/manage/self-hosted/managed-kubernetes.md) page summarises options per cloud. See [Run a multi-node cluster](/docs/running/multi-node.md) for how the model itself works. ### Features **Horizontal scalability** - Scale query and storage nodes independently. **Fault tolerance** - Replication and consensus allow clusters to tolerate node failures. **Distributed ACID transactions** - Strong transactional guarantees across distributed infrastructure. **Large-scale storage** - Designed for very large datasets and high-concurrency production workloads. ### Best for - Enterprise deployments - High-availability applications - Multi-region systems - Large graph workloads - Real-time platforms - AI-native distributed systems ### Operational considerations Distributed deployments add cluster orchestration, node management, monitoring, and replication management. Teams that want distributed scalability without operating that stack should consider [managed instances](/docs/manage/instances.md). ## Embedded deployments Embedded deployments run SurrealDB inside your application process without a separate database server. The query and storage layers share the process (and, in the browser, the same runtime), which removes network latency between your app and the database. This model suits edge computing, mobile and desktop software, offline-first apps, browser PWAs, and AI workflows that need minimal latency. For language-specific setup, see [Embedding SurrealDB](/docs/build/embedding.md) and [Storage engines](/docs/build/embedding/storage-engines.md). ### Supported runtimes SurrealDB supports embedded operation in Rust, Go, JavaScript / TypeScript, WebAssembly, Python, and .NET. Capabilities vary by SDK and storage backend - check the embedding guide for your language. ### Storage options **In-memory (SurrealMX)** - Default in-memory backend since SurrealDB 3.0, with optional snapshots or append-only persistence and support for versioned queries. See [Run a single-node, in-memory server](/docs/running/in-memory.md). **RocksDB** - Persistent on-disk storage with mature tuning for write-heavy, SSD-backed server workloads. See [File-backed storage](/docs/running/file-backed.md). **SurrealKV** - Same beta engine as single-node server deployments; aimed at embedded and local-first in-process workloads where smaller resident memory and simple operational behaviour matter most. **IndexedDB (browser)** - Browser-native persistence with binary serialisation for PWAs and local-first web apps. See [Embedding SurrealDB](/docs/build/embedding.md) and the [Wasm engine](/docs/reference/javascript/engines/wasm.md). ## Choosing the right deployment model **Use a managed instance when** - You want managed infrastructure instead of operating clusters yourself - You need production-ready HA quickly - especially on the **Scale** plan for multi-node fault tolerance - Your team prefers building applications over provisioning servers, Kubernetes, replication, and upgrade runbooks **Use single-node deployments with RocksDB when** - Simplicity matters most - Scaling requirements are moderate - You run small-to-medium production workloads without cluster-level fault tolerance **Use multi-node deployments when** - High availability is required - Workloads need horizontal scaling - Infrastructure spans multiple nodes or availability zones For managed clusters, use the [Scale](https://surrealdb.com/pricing/scale) plan. For self-hosted clusters, see [SurrealDB Enterprise](https://surrealdb.com/enterprise) and [Managed Kubernetes](/docs/manage/self-hosted/managed-kubernetes.md). **Use embedded deployments when** - You run on edge devices or in the browser - Offline operation is required - Minimising latency between app and database is critical ## Conclusion SurrealDB’s architecture lets the same engine and query language run across embedded, single-node, distributed, and managed models. Whether you embed SurrealDB in a browser, run RocksDB on one server, scale horizontally across a multi-node cluster, or use a managed Start or Scale instance, you can match operational and scalability requirements without rewriting queries. ## Next steps - [Managed instances](/docs/manage/instances.md) - Provision a managed SurrealDB instance with built-in monitoring, network access controls and operations tooling. - [Self-hosted](/docs/manage/self-hosted.md) - Run SurrealDB on Docker, Kubernetes (AKS, EKS, GKE) or as a standalone binary with full control over storage and configuration. - [Observability](/docs/manage/observability.md) - Metrics, OTLP and Prometheus access, audit logs and slow-query logs for both Community and Enterprise editions. --- Source: https://surrealdb.com/docs/manage/self-hosted/docker # Docker A tutorial to run SurrealDB from within Docker. Use this tutorial to run SurrealDB from within Docker. ## Running the SurrealDB server using Docker To get started using Docker, you can use the `latest` tag. To view all the available versions and tags, or to use a specific tag visit the [Docker Hub](https://hub.docker.com/r/surrealdb/surrealdb) page. To start a server use the [`start`](/docs/reference/cli/surrealdb-cli/commands/start.md) command. In Docker, SurrealDB listens on port `8000` in all interfaces by default so that the host can connect to the container in the default bridge networking mode. ```bash docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start ``` > [!IMPORTANT] > For local development, use the `latest-dev` image variant (i.e., docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest-dev start). This version includes a shell and package manager, allowing you to install tools and interact with the container's internals. In order to persist data when the Docker instance is restarted or shut down, specify a Docker folder using the Docker `-v` command line argument, and use the on-disk storage engine in SurrealDB using the path prefix chosen as a Docker folder. ```bash mkdir mydata # Create a directory to store the database, owned by the current user docker run --rm --pull always -p 8000:8000 --user $(id -u) -v $(pwd)/mydata:/mydata surrealdb/surrealdb:latest start rocksdb:/mydata/mydatabase.db ``` The default logging level for the database server is `info`. To control the logging verbosity, specify the `--log` argument. The following command starts the database with `debug` level logging, resulting in more logs being output to the terminal. If extra verbosity is not needed, specify a lower level or simply remove the flag, which will default to the `info` level. ```bash mkdir mydata # Create a directory to store the database, owned by the current user docker run --rm --pull always -p 8000:8000 --user $(id -u) -v $(pwd)/mydata:/mydata surrealdb/surrealdb:latest start --log debug rocksdb:/mydata/mydatabase.db ``` ### Configuring authentication Authentication is enabled by default on SurrealDB, while the `--unauthenticated` flag can be used to opt out. To set up access as an authenticated user, configure your initial root-level user by setting the `--user` and `--pass` command-line arguments. The following command starts the database with a top-level user named `root` with a password set to `secret`. The root user will be persisted in storage, which means you don't have to include these two arguments the next time you start SurrealDB. ```bash docker run --rm --pull always -p 80:8000 -v /mydata:/mydata surrealdb/surrealdb:latest start --user root --pass secret rocksdb:mydatabase.db ``` In order to change the default port that SurrealDB uses for web connections and from database clients you can use the Docker `-p` command line argument to tunnel the port to the internal SurrealDB port which SurrealDB is served on. The following command starts the database on port `80`. ```bash docker run --rm --pull always -p 80:8000 -v /mydata:/mydata surrealdb/surrealdb:latest start --user root --pass secret rocksdb:/mydata/mydatabase.db ``` After running the above command, you should see the SurrealDB server start up successfully. ```bash docker run --rm --pull always -p 80:8000 -v /local-dir:/container-dir surrealdb/surrealdb:latest start --user root --pass secret rocksdb:/container-dir/mydatabase.db 2025-08-30T15:06:34.788739Z INFO surreal::dbs: ✅🔒 Authentication is enabled 🔒✅ 2025-08-30T15:06:34.788821Z INFO surrealdb::kvs::ds: Starting kvs store in rocksdb:/container-dir/mydatabase.db 2025-08-30T15:06:34.788859Z INFO surrealdb::kvs::ds: Started kvs store in rocksdb:/container-dir/mydatabase.db 2025-08-30T15:06:34.789222Z INFO surrealdb::kvs::ds: Initial credentials were provided and no existing root-level users were found: create the initial user 'root'. 2025-08-30T15:06:35.205123Z INFO surrealdb::node: Started node agent 2025-08-30T15:06:35.205827Z INFO surrealdb::net: Started web server on 0.0.0.0:8080 ``` For details on the `start` command, and all of the available configuration options and arguments, view the [`start command documentation`](/docs/reference/cli/surrealdb-cli/commands/start.md). ## Using the command-line tools within Docker The Docker container contains both the server, and the command line tools for importing, exporting, and querying a remote SurrealDB server. ```bash docker run --rm --pull always surrealdb/surrealdb:latest help ``` The result should look similar to the output below, confirming that the SurrealDB command-line tool was installed successfully. ```text .d8888b. 888 8888888b. 888888b. d88P Y88b 888 888 'Y88b 888 '88b Y88b. 888 888 888 888 .88P 'Y888b. 888 888 888d888 888d888 .d88b. 8888b. 888 888 888 8888888K. 'Y88b. 888 888 888P' 888P' d8P Y8b '88b 888 888 888 888 'Y88b '888 888 888 888 888 88888888 .d888888 888 888 888 888 888 Y88b d88P Y88b 888 888 888 Y8b. 888 888 888 888 .d88P 888 d88P 'Y8888P' 'Y88888 888 888 'Y8888 'Y888888 888 8888888P' 8888888P' SurrealDB command-line interface and server To get started using SurrealDB, and for guides on connecting to and building applications on top of SurrealDB, check out the SurrealDB documentation (https://surrealdb.com/docs). If you have questions or ideas, join the SurrealDB community (https://discord.gg/surrealdb). If you find a bug, submit an issue on GitHub (https://github.com/surrealdb/surrealdb/issues). We would love it if you could star the repository (https://github.com/surrealdb/surrealdb). ---------- USAGE: surreal [SUBCOMMAND] OPTIONS: -h, --help Print help information SUBCOMMANDS: start Start the database server import Import a SQL script into an existing database export Export an existing database into a SQL script version Output the command-line tool version information sql Start an SQL REPL in your terminal with pipe support help Print this message or the help of the given subcommand(s) ``` For details on the different commands available, visit the [CLI tool documentation](/docs/reference/cli/surrealdb-cli/overview.md). --- Source: https://surrealdb.com/docs/manage/self-hosted/kubernetes # Kubernetes Deploy SurrealDB to Kubernetes with RocksDB on a persistent volume. This guide deploys SurrealDB to a local [KIND](https://kind.sigs.k8s.io/) cluster (Kubernetes in Docker) with **RocksDB** on a persistent volume. That is a **single-node** topology: one SurrealDB pod owns the database file. See [Deployment models](/docs/manage/self-hosted/deployment-models.md#single-node-rocksdb). For **multi-node HA**, use the managed [Scale plan](https://surrealdb.com/pricing/scale) or [SurrealDB Enterprise](https://surrealdb.com/enterprise) on managed Kubernetes - see [Managed Kubernetes](/docs/manage/self-hosted/managed-kubernetes.md). ## Requirements - [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) - [`helm`](https://helm.sh/docs/intro/install/) - [KIND](https://kind.sigs.k8s.io/) and [Docker](https://www.docker.com/) - [Surreal CLI](/docs/reference/cli/surrealdb-cli/overview.md) ## Create a KIND cluster ```bash kind create cluster -n surreal-demo kubectl config current-context # kind-surreal-demo kubectl get ns ``` ## Deploy SurrealDB Use the [SurrealDB Helm chart](https://github.com/surrealdb/helm-charts) with a **ReadWriteOnce** persistent volume. Keep **`replicaCount: 1`** - multiple pods must not share one RocksDB file. ### 1. Add the Helm repository ```bash helm repo add surrealdb https://helm.surrealdb.com helm repo update ``` ### 2. Install with persistence The chart mounts storage at `/home/nonroot` so the non-root container user can write to the volume: ```bash cat <<'EOF' | helm install surrealdb-rocksdb surrealdb/surrealdb -f - strategy: type: Recreate replicaCount: 1 persistence: enabled: true mountPath: /home/nonroot size: 10Gi surrealdb: path: rocksdb:///home/nonroot/data.db unauthenticated: true EOF ``` ### 3. Create initial credentials Port-forward the service, define a root user, then re-enable authentication: ```bash kubectl port-forward svc/surrealdb-rocksdb 8000:8000 ``` In another shell: ```bash surreal sql -e http://localhost:8000 > DEFINE USER root ON ROOT PASSWORD 'StrongSecretPassword!' ROLES OWNER; ``` Upgrade the release without `unauthenticated`: ```bash helm upgrade surrealdb-rocksdb surrealdb/surrealdb -f - <<'EOF' strategy: type: Recreate replicaCount: 1 persistence: enabled: true mountPath: /home/nonroot size: 10Gi surrealdb: path: rocksdb:///home/nonroot/data.db EOF ``` ### 4. Verify persistence ```bash surreal sql -u root -p 'StrongSecretPassword!' -e http://localhost:8000 > USE NS ns DB db; ns/db> CREATE record SET id = record:one; ns/db> SELECT * FROM record; ``` Delete the SurrealDB pod and confirm data survives on the PVC: ```bash kubectl get pod kubectl delete pod kubectl port-forward svc/surrealdb-rocksdb 8000:8000 surreal sql -u root -p 'StrongSecretPassword!' -e http://localhost:8000 > USE NS ns DB db; ns/db> SELECT * FROM record; ``` > [!NOTE] > On a full Kubernetes cluster, set `ingress.enabled=true` when installing the chart to expose SurrealDB through a load balancer instead of port-forwarding. ## Next steps - [Docker](/docs/manage/self-hosted/docker.md) - single-node RocksDB without Kubernetes - [Run a single-node, on-disk server](/docs/running/file-backed.md) - CLI startup options for RocksDB and SurrealKV - [Deployment models](/docs/manage/self-hosted/deployment-models.md) - managed, single-node, and highly available options --- Source: https://surrealdb.com/docs/manage/self-hosted/managed-kubernetes # Managed Kubernetes Options for running SurrealDB on Amazon EKS, Google GKE, and Azure AKS - managed Scale, self-hosted Enterprise clusters, or single-node RocksDB. SurrealDB runs on Amazon EKS, Google GKE and Azure AKS. This page covers the three deployment models available on them - managed Scale, a self-hosted Enterprise cluster, and single-node RocksDB - and which one fits. > [!IMPORTANT] > Production **multi-node** SurrealDB uses shared distributed storage with replication and consensus. For managed HA, use the [Scale plan](https://surrealdb.com/pricing/scale). Self-hosted multi-node clusters on Kubernetes are available with [SurrealDB Enterprise](https://surrealdb.com/enterprise). > [!NOTE] > For a **single-node RocksDB** deployment on Kubernetes - including a cluster you create with EKS, GKE, or AKS - start with [Deploy on Kubernetes](/docs/manage/self-hosted/kubernetes.md). For how the multi-node model works in general, see [Run a multi-node cluster](/docs/running/multi-node.md). ## The managed control planes Each of the three major clouds offers a managed Kubernetes control plane. The control plane is the only part they manage for you; how SurrealDB storage is provided is still your choice - single-node RocksDB on a persistent volume, or a multi-node cluster on distributed storage (Scale or Enterprise). | Provider | Service | | --- | --- | | AWS | [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/) | | Google Cloud | [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine), including Autopilot | | Microsoft Azure | [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/products/kubernetes-service) | ## Choose a path | Goal | Path | | --- | --- | | Managed multi-node HA (recommended for most teams) | A managed [Scale](/docs/manage/instances.md) cluster, with the storage layer operated for you | | Self-hosted multi-node HA on EKS, GKE, or AKS | [SurrealDB Enterprise](https://surrealdb.com/enterprise) with distributed storage (operator and runbooks shipped with Enterprise) | | Single SurrealDB pod, RocksDB on a volume | [Deploy on Kubernetes](/docs/manage/self-hosted/kubernetes.md) on a cluster you manage | ## Next steps - [Deployment models](/docs/manage/self-hosted/deployment-models.md) - single-node vs multi-node vs managed - [Instances](/docs/manage/instances.md) - the managed option - [Scale](https://surrealdb.com/pricing/scale) - pricing for managed multi-node clusters - [Observability](/docs/manage/observability.md) once an instance is running --- Source: https://surrealdb.com/docs/manage/self-hosted/monitoring-and-observability # Monitoring & observability Health checks, OpenTelemetry metrics and traces, audit logs and slow-query logs, and integration with common observability stacks. SurrealDB exposes a built-in **`/health`** HTTP endpoint suitable for load balancer and orchestrator probes. A successful response indicates the process is accepting requests. Combine `/health` with deeper checks so you detect partial failures - slow queries, disk pressure, replication lag - before probes alone would fire. For everything beyond a liveness probe, the [Observability](/docs/manage/observability.md) section is the canonical reference and operator guide. The pages there cover metrics, telemetry access, audit logs and slow-query logs for both Community and Enterprise editions. - [Observability overview](/docs/manage/observability.md) - Edition matrix (Community vs Enterprise), Prometheus and OTLP quickstart, what's new in 3.1, Tokio console, structured logging. - [Metrics reference](/docs/manage/observability/metrics.md) - Access paths, label catalogue, ~60 metrics grouped by signal family, the public allowlist, and the 3.0 → 3.1 migration table. - [Configuration reference](/docs/manage/observability/configuration.md) - Every telemetry, audit-log and slow-query environment variable, plus recommended configurations for local, production and multi-tenant deployments. - [Audit logging (Enterprise)](/docs/manage/observability/audit-logging.md) - Events captured, record shape, rotation, hash chaining, redaction and pipeline self-metrics. - [Slow-query logging (Enterprise)](/docs/manage/observability/slow-query-logging.md) - Threshold-based capture of long-running queries with the same file-sink, hash-chain and redaction options as the audit pipeline. ## Integrating with common observability stacks The OpenTelemetry exporter speaks OTLP gRPC, so anything that ingests OTLP works. Two common patterns: - **Prometheus pull.** Scrape the built-in `/metrics` endpoint. Anonymous scrapers see only the [public allowlist](/docs/manage/observability/metrics.md#public-metrics-allowlist); pass root credentials to unlock the full surface. Build Grafana dashboards on the resulting time series and pair with Alertmanager for the [alert hints](/docs/manage/observability/metrics.md#alert-hints) recommended for production. - **OTLP push.** Point SurrealDB at an OpenTelemetry collector and route from there into Prometheus remote-write, Tempo / Jaeger for traces, and Loki or a SIEM for logs (including audit and slow-query records when their OTel export is opted in). Label streams by environment (`production`, `staging`, `dev`) at the collector so dashboards do not mix traffic accidentally. Either path uses the same `surrealdb.*` instrument namespace and the same `service.edition` resource attribute, so dashboards travel cleanly across deployments. ## Key signals to watch The full alert-hints starter set lives on the [metrics page](/docs/manage/observability/metrics.md#alert-hints), but the must-watch signals for any production deployment are: - **Error rate** - `rate(surrealdb_statement_total{outcome="error"}[5m])` rising sharply against baseline. - **Transaction conflicts** - `rate(surrealdb_transaction_conflicts_total[5m])`, which doubles as the retry-pressure signal. - **Latency tails** - histogram percentiles on `surrealdb_statement_duration_seconds`, `surrealdb_query_duration_seconds`, `surrealdb_http_request_duration_seconds`. - **Active sessions and live queries** - `surrealdb_session_active`, `surrealdb_live_query_active` for capacity planning. - **Audit pipeline health (Enterprise)** - `surrealdb_audit_dropped`, `surrealdb_audit_append_errors`, `surrealdb_audit_queue_depth` to catch lost or delayed audit records. Define SLOs where appropriate - latency and availability targets - and use burn-rate alerts on the resulting error budgets rather than relying on one-off thresholds alone. --- Source: https://surrealdb.com/docs/manage/self-hosted/upgrades-and-patching # Upgrades & patching Upgrade SurrealDB safely: binary replacement, surreal fix for major versions, migrations, and cluster rolling upgrades. **Routine upgrades** typically follow: quiesce or drain clients if your SLA allows, **stop** the server gracefully, **replace** the `surreal` binary with the new release, then **start** with unchanged data paths and reviewed configuration. Read the release notes for breaking changes, new defaults, or removed flags before you cut over. Re-run integration tests against the new version before promoting the change across your organisation. Across **major versions**, on-disk formats may change. When documentation requires it, run [`surreal fix`](/docs/reference/cli/surrealdb-cli/commands/fix.md) to migrate data between layouts, and follow [Migrating from older SurrealDB versions](/docs/build/migrating/from-old-surrealdb-versions/overview.md) plus any linked guides (for example between specific major lines). **Rolling upgrades** in **clustered** setups usually upgrade one node at a time: verify cluster health, upgrade a member, wait for replication or quorum to stabilise, then continue. Never skip staging validation for production-like data volumes. If the cluster spans regions, plan maintenance windows that respect dependency order between tiers. **Before any upgrade**, take a fresh backup - [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) for a logical copy and/or a storage snapshot - so you can revert if migration or client incompatibility surfaces after deploy. Patch **security** releases promptly: subscribe to SurrealDB advisories, test the patch build in staging, then roll out using the same stop - replace - start or rolling pattern your architecture supports. Document the upgraded version in your asset inventory for compliance reviews. ## Automatic data migrations _(since v3.3.0)_ From SurrealDB 3.3.0, a datastore records the version that last opened it and applies any pending data migrations on startup, before it serves the first query. Migrations are recorded in a cluster-wide ledger, so each one runs once no matter how many nodes start against the same datastore, and an interrupted run resumes rather than restarts. A migration that fails aborts startup. A datastore created on 3.3.0 or later records the full migration set as applied when it is created, so it never runs a historical migration. Two consequences for planning an upgrade: - **A downgrade is refused if the datastore has run a migration the older build does not ship.** The server reports the migration by name and stops. Reverting past a migration therefore needs [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) and a reimport into a datastore created by the older version. - **Do not run `DEFINE SEQUENCE` during a rolling upgrade to 3.3.0.** Sequence definitions move to a new key layout. Upgraded nodes still read the old layout, so existing sequences keep working throughout the rollout, but a sequence created on an upgraded node is not visible to a node still on the previous release. Wait until every node has been upgraded. > [!NOTE] > The first migration fixes a key layout in which `DEFINE SEQUENCE` definitions overlapped tables whose names begin with `sq`. On 3.0 to 3.2, a database containing such a table fails `INFO FOR DB`, fails `REMOVE DATABASE` - leaving the database undroppable - and fails an export that includes sequences. No sequence needs to exist for this to happen. Upgrading to 3.3.0 repairs it. --- Source: https://surrealdb.com/docs/manage/surrealctl # surrealctl What the surrealctl control-plane CLI does. When to reach for it instead of SurrealDB Studio or the surreal binary, plus a sixty-second quickstart. `surrealctl` is the command-line interface for the SurrealDB Cloud control plane. Use it to sign in, create and list instances, scale them, pause and resume them, read logs and metrics, manage organisation members, and mint tokens - from a terminal, a shell script, or a CI job. This page covers what the tool is for and gets you through a first session. It is written for operators and developers who already have a SurrealDB Cloud account. For exhaustive flag detail, see the [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md). ## Control plane and data plane SurrealDB ships two command-line tools, and they divide along a clear line. | Tool | Plane | Answers | | --- | --- | --- | | `surrealctl` | Control plane | Which instances exist, what they run, what they cost, who may reach them | | [`surreal`](/docs/reference/cli/surrealdb-cli/overview.md) | Data plane | Queries, imports, exports, and running a server yourself | The two are siblings, not competitors. `surrealctl` never speaks SurrealQL. When you run `surrealctl instance sql`, it resolves the instance, mints a database token, and hands off to the `surreal` binary - the same binary you would run by hand, with the endpoint and credentials already filled in. The same applies to `surrealctl instance import` and `surrealctl instance export`. That is the whole boundary. Three commands cross it; nothing else needs `surreal`. ## When to use which tool - **[SurrealDB Studio](https://studio.surrealdb.com)** - sign-up, browsing data, billing pages, and anything you do once. The browser is the fastest route for one-off work. - **`surrealctl`** - anything you want to repeat, script, schedule, or review in a pull request. Provisioning, scaling, pausing an environment overnight, pulling logs into a report, granting a colleague a role. - **`surreal`** - schema, queries, and data. Also a local server for development. `surrealctl` and Studio talk to the same API, so a change made in one shows up in the other. ## Sixty-second quickstart Install the binary, then sign in and deploy something. ```bash title="First session" surrealctl auth login surrealctl whoami surrealctl org list surrealctl org use acme surrealctl instance create api --type shared-1 --region aws-euw1 surrealctl instance sql api -- --ns app --db main surrealctl instance pause api surrealctl instance delete api --force ``` `auth login` opens a browser and stores the session for later commands. `org use` remembers an organisation so you do not have to name it every time. `instance create` waits until the instance is ready before it returns, so the `sql` command on the next line connects to a running database. For installation instructions, see [Install](/docs/manage/surrealctl/install.md). ## Output that scripts can read Every command that returns a document takes `--json`, and every command sends its data payload to stdout and everything else - progress, prompts, warnings, errors - to stderr. ```bash title="Machine-readable output" surrealctl instance list --json | jq -r '.[] | select(.state == "ready") | .name' ``` So `| jq` always works, `--json > out.json` can still prompt you, and a failed command leaves stdout empty rather than half a document. Exit codes distinguish outcomes a pipeline needs to tell apart: `3` for an expired credential, `5` for a name that does not exist, `10` for a wait that gave up on an operation that is still running. See [Scripting](/docs/manage/surrealctl/scripting.md) for the full contract. ## Command groups | Group | What it manages | | --- | --- | | `auth` | Sign in, sign out, and inspect credentials | | `org` | Organisations, roles, usage, and spend | | `instance` | Instances: lifecycle, endpoints, logs, metrics, capabilities, backups | | `team` | Organisation members | | `invite` | Organisation invitations | | `token` | Personal access tokens | | `catalog` | Regions, instance types, and SurrealDB versions the platform offers | | `spectron` | SurrealDB Agent Memory contexts, keys, and principals | | `config` | Configuration stored on this machine | | `context` | Profiles, and which one is in force | | `api` | Call the API directly | | `open` | Open a dashboard page in a browser | | `status` | Check that everything is configured and reachable | Nouns are singular and verbs are predictable: `list`, `get`, `create`, `update`, `delete`. The plural forms are aliases, so `surrealctl instances ls` works. Only two commands break the pattern, both because the house verb would mislead: `team remove` ends a membership rather than deleting a person, and `team invite` sends an invitation. ## Deliberate omissions Three things `surrealctl` will not do, so you do not go looking for them. - **It does not run queries.** `instance sql` hands off to `surreal`. A wrapper around SurrealQL would be wrong the first time the language gained a feature. - **There is no sticky current instance.** Organisations persist with `org use`; instances are named on each command or picked interactively. A remembered instance means `instance delete` in a forgotten terminal tab deletes production. - **It never accepts legal terms on your behalf.** `surrealctl open terms` prints the links and opens them; accepting is something a person does. ## Topics - [Install](/docs/manage/surrealctl/install.md) - get the binary, sign in, and confirm the setup. - [Authentication](/docs/manage/surrealctl/authentication.md) - login sessions, personal access tokens, and where credentials live. - [Instances](/docs/manage/surrealctl/instances.md) - create, connect, scale, pause, back up, and delete. - [Organisations](/docs/manage/surrealctl/organisations.md) - members, roles, invitations, tokens, usage, and spend. - [Scripting](/docs/manage/surrealctl/scripting.md) - `--json`, exit codes, and unattended runs. - [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md) - every command, flag, and default. --- Source: https://surrealdb.com/docs/manage/surrealctl/authentication # Authenticating surrealctl Choose between a login session and a personal access token, store credentials safely, and authenticate an unattended run. `surrealctl` accepts two kinds of credential. This page explains what each one is good for, how to create and inspect them, and how to hand one to a CI job. It is for anyone setting up their own machine, and for engineers wiring `surrealctl` into a pipeline. For the mechanism underneath - the sign-in flows, what travels on the wire, and how the credential store behaves - see the [authentication reference](/docs/reference/cli/surrealctl/authentication.md). ## Two kinds of credential The two credentials differ in what they can **write**, not in what they can reach. | Credential | How you get it | Use it for | Cloud writes | | --- | --- | --- | --- | | Login session | `surrealctl auth login` | Everyday work at a terminal | Yes | | Personal access token | `surrealctl token create`, or SurrealDB Studio | Scripts, CI jobs, shared runners | No | A login session is the credential that can do everything. A personal access token reads the control plane and cannot change it, which is what makes it safe to leave in a CI secret store. ## Login sessions `surrealctl auth login` signs in through your identity provider and stores a refresh token plus one access token per audience. See [Install](/docs/manage/surrealctl/install.md#sign-in-for-the-first-time) for the three sign-in flows and when each is used. Four commands inspect and maintain the session. ```bash surrealctl auth status # which credential is in use, read from disk surrealctl auth status --verify # the same, plus one request to check it surrealctl auth refresh # renew now, rather than at the next request surrealctl auth logout # remove this profile's stored credentials ``` `auth status` and `auth scopes` touch no network, so they work on a plane, in a container with no egress, and in a CI job that is about to fail for another reason. `whoami` is the command that asks the API who you are. `auth status` also warns when this machine's clock is more than five minutes out of step with the API, because a wrong clock produces authentication failures that look like a rejected credential. `auth logout` revokes the refresh token at the identity provider on a best-effort basis. A failed revocation prints a warning and still removes the credential from this machine. Logging out when nothing is signed in succeeds and still prints a document, so a teardown script does not fail on its second run. > [!NOTE] > `auth scopes` returns an empty list for a login session. Identity scopes are not the authorisation model for a session - your role in the organisation is. Run `surrealctl org permissions` to see what you may do. ## Personal access tokens A personal access token is a long-lived string beginning `sdbp_`. Create one with a login session in place, then hand it to whatever needs it. ```bash surrealctl token scopes surrealctl token create "ci-nightly-report" --scope read:cloud --expires-in 90 | tail -1 ``` ", "required": true, "description": "What this token is for. Shown in every listing. 1 to 120 characters." }, { "name": "--scope", "value": "", "description": "A scope to grant. Repeat the flag to grant several, up to 32." }, { "name": "--expires-in", "value": "", "default": "30", "description": "Days until the token expires, from 1 to 365, or `never`." }, { "name": "--reveal", "description": "Print the secret even when stdout is a terminal." } ]} /> Four things about `token create` are worth knowing before you run it. - **The secret is returned once.** It is the last line on stdout, so `| tail -1` is the whole capture recipe. Nothing can recover it later. - **It refuses to print a secret to a terminal.** Pipe the output, or pass `--reveal` if you meant to read it. The check happens before the token is minted, so a forgotten flag never costs you a credential nobody can use. - **It is the one command with no `--json` form.** The secret can neither go inside the document nor share stdout with it, so `--json` is a usage error here. - **A token with no `--scope` is permitted nothing.** Grant what the job needs and no more. List and revoke tokens by id or label. ```bash surrealctl token list surrealctl token delete ci-nightly-report ``` > [!WARNING] > Revocation is not instant. A revoked token can continue to work for several minutes while the platform's exchange cache expires. Rotate a leaked token and then confirm with `surrealctl token list`. ## What a personal access token cannot do **It cannot manage personal access tokens.** All four `token` verbs - `list`, `create`, `delete`, and `scopes` - refuse a personal access token. The refusal is local, costs no requests, and exits `4`. A leaked token must not be able to mint its own replacements, nor revoke the one you would use to clean up after it. ```text Managing personal access tokens needs an interactive login session, so nothing was sent. Sign in with: surrealctl auth login ``` **It cannot write to the control plane.** The API refuses a personal access token on every mutating route, whatever scopes the token carries. `instance create`, `instance delete`, `org update`, and `team invite` all need a login session. Reads work normally. **It cannot exceed its scopes.** A missing scope produces a 403 naming the scope, and exits `4`. Everything else works: `org list`, `instance list`, `instance logs`, `auth status`, `auth scopes`, and `auth logout`. ## Choosing between them - **At a terminal, sign in.** A login session covers every command and renews itself. - **In CI, use a token** for anything that only reads: nightly usage reports, drift checks, dashboards, alerting on instance state. - **For unattended writes, use a login session** on a dedicated machine account, with the credential file mounted into the job. Scope the account's role in the organisation to what the pipeline needs. ## Supplying a token Two flags, each with an environment variable, and none of them writes the token to disk. ", "env": "SURREALCTL_TOKEN", "description": "Personal access token to authenticate with. Conflicts with `--token-file`." }, { "name": "--token-file", "value": "", "env": "SURREALCTL_TOKEN_FILE", "description": "Read the personal access token from a file, or `-` for stdin." } ]} /> `SURREALCTL_TOKEN` is the usual choice in CI, because the environment is where a secret store puts things. ```bash export SURREALCTL_TOKEN="$(cat /run/secrets/surrealctl)" surrealctl instance list --json ``` `--token` has no short form on purpose: `-t` reads as `--type` on the instance commands, and a credential in `argv` is visible through `ps` and lands in shell history. A credential supplied this way lasts for the one invocation. The only command that stores a token is `auth login --with-token`, which reads it from stdin and remembers it for the profile. ```bash surrealctl auth login --with-token --label laptop < ~/token.txt ``` ## Scopes Five scopes exist: `read:cloud`, `write:cloud-instances`, `write:cloud-organization`, `write:cloud-billing`, and `write:cloud-spectron`. Two commands report on them, and they answer different questions. - `surrealctl token scopes` - the catalogue of scopes a token can be granted. This one is the menu. - `surrealctl auth scopes` - what the credential in your hand carries. This one is the receipt. ## Profiles A profile bundles a credential, an API base URL, and configuration under one name. Use profiles to keep a work account and a personal account apart, or production and staging. ```bash surrealctl context use staging # switch the active profile surrealctl --profile staging auth login # sign that profile in surrealctl context list # every profile, and which is active surrealctl context show # what this invocation resolved, and why ``` Each profile stores its own credential, so switching profiles switches identity. `surrealctl org use` works *within* a profile and remembers an organisation; neither command is expressible as the other. ## Where credentials are stored Credentials live in `credentials.json`, mode `0600`, beside `config.toml` in the configuration directory described in [Install](/docs/manage/surrealctl/install.md#where-surrealctl-keeps-its-files). `config.toml` holds no secrets and is safe to commit; `credentials.json` is not. Two behaviours are worth knowing at a terminal. `surrealctl` refuses to read a `credentials.json` that group or other can read, and tells you the `chmod` to run. And concurrent invocations serialise on an advisory lock, so if a run reports that another `surrealctl` is updating credentials, retry in a moment. The [authentication reference](/docs/reference/cli/surrealctl/authentication.md#where-credentials-are-stored) covers the rest: the resolution order for the directory, why the store is a file rather than the OS keyring, how the lock and the atomic rename work, and how clock skew is learned. ## Troubleshooting | Symptom | Cause | Resolution | | --- | --- | --- | | Exit `3` on any command | No credential, or one that expired and could not be renewed | `surrealctl auth login`, or `surrealctl auth refresh` | | Exit `4` on a `token` command | A personal access token is in use | Sign in with a login session | | Exit `4` on a write | Insufficient scope, insufficient role, or a token where a session is needed | Check `surrealctl org permissions`; use a login session for writes | | `auth status` warns about clock skew | This machine's clock is more than five minutes out | Fix time synchronisation on the host | | Exit `30` | An interactive sign-in did not complete | Run `surrealctl auth login` again; add `--flow device` when there is no browser | | Exit `11` | The credential store could not be read or written | Check the permissions and free space on the configuration directory | | A refusal to read `credentials.json` | The file is readable by group or other | `chmod 600` on the path in the message | To see which requests are being made and which configuration layer won each value, add `--debug`. Credentials are logged as a digest, never in full. ## Next steps - [Instances](/docs/manage/surrealctl/instances.md) - everyday instance work. - [Scripting](/docs/manage/surrealctl/scripting.md) - exit codes, `--json`, and unattended runs. - [Members and roles](/docs/manage/organisations/members-and-roles.md) - what a role permits in an organisation. --- Source: https://surrealdb.com/docs/manage/surrealctl/install # Install Install surrealctl on macOS, Linux, or Windows, verify the download, sign in for the first time, and keep the binary up to date. This page installs the `surrealctl` binary, signs you in to SurrealDB Cloud, and confirms that both worked. Pick one installation method. Everything after it applies whichever you chose. `surrealctl` is a single executable with no runtime dependencies. It is published for macOS, Linux, and Windows, on both `arm64` and `amd64`. > [!NOTE] > The current release is a beta. Until the first stable release, the install script and `surrealctl upgrade` both follow the beta channel, and the version you get carries a `-beta` suffix. ## Install with the install script On macOS and Linux, the install script detects your platform and CPU, downloads the matching asset, checks it against its published SHA-256, and only then puts it in place. ```bash curl -fsSL https://download.surrealdb.com/surrealctl/install.sh | sh ``` The script is served as plain text, so you can open the same URL in a browser and read it before you run it. It installs into the first writable directory of `~/.local/bin` and `/usr/local/bin`, creating `~/.local/bin` if neither exists. **It never uses `sudo`.** If the directory it picks is not on your `PATH`, the script tells you and does not edit your shell profile. ### Options Pass options after `-s --`, which is how `sh` forwards arguments to a script it reads from a pipe. ```bash curl -fsSL https://download.surrealdb.com/surrealctl/install.sh | sh -s -- --to ~/bin ``` | Option | Effect | | --- | --- | | `--version ` | Install this exact version, written `1.0.0` or `v1.0.0` | | `--beta` | Install the newest beta release | | `--alpha` | Install the newest alpha release | | `--to ` | Install into this directory | | `--dry-run` | Print what would be downloaded and installed, then stop | | `-h`, `--help` | Print the options and exit | Two environment variables do the same job as options, which is easier to set in a Dockerfile or a CI step. | Variable | Effect | | --- | --- | | `SURREALCTL_INSTALL_DIR` | Same as `--to` | | `SURREALCTL_DOWNLOAD_ROOT` | Point the script at a different release host, for testing | Use `--dry-run` to see the exact URL and destination before anything is written. ```bash title="Check what the script would do" curl -fsSL https://download.surrealdb.com/surrealctl/install.sh | sh -s -- --dry-run ``` > [!NOTE] > No musl build is published yet. On Alpine and other musl-based distributions the script stops and says so, rather than installing a glibc binary that cannot start. Build from source on those systems. ## Install on Windows Download the executable, then put it somewhere on your `PATH`. ```powershell $version = (Invoke-WebRequest -Uri https://download.surrealdb.com/surrealctl/beta.txt -UseBasicParsing).Content.Trim() Invoke-WebRequest -Uri "https://download.surrealdb.com/surrealctl/$version/surrealctl-$version.windows-amd64.exe" -OutFile surrealctl.exe ``` Read `latest.txt` instead of `beta.txt` once a stable release is published. ## Download a binary Every release is published under `https://download.surrealdb.com/surrealctl/`. Use this route for container images, air-gapped hosts, and anywhere you want the version pinned in source control. Three text files name the current version of each channel. Each contains one line, such as `v1.0.0-beta.1`. | File | Channel | | --- | --- | | `latest.txt` | Stable | | `beta.txt` | Beta | | `alpha.txt` | Alpha | ```bash title="Download and install the current beta" version=$(curl -fsSL https://download.surrealdb.com/surrealctl/beta.txt) curl -fsSL "https://download.surrealdb.com/surrealctl/${version}/surrealctl-${version}.darwin-arm64.tgz" \ | tar -xz -C ~/.local/bin ``` Replace the asset name to match your platform. | Platform | Asset | | --- | --- | | macOS, Apple silicon | `surrealctl-.darwin-arm64.tgz` | | macOS, Intel | `surrealctl-.darwin-amd64.tgz` | | Linux, ARM64 | `surrealctl-.linux-arm64.tgz` | | Linux, x86-64 | `surrealctl-.linux-amd64.tgz` | | Windows, x86-64 | `surrealctl-.windows-amd64.exe` | The archives hold a single file, `surrealctl`, at the root. Windows ships the executable directly, with no archive. ### Verify a download Every asset has a `.txt` file beside it holding its SHA-256, and every version directory has a `SHA256SUMS` covering all of them. The install script and `surrealctl upgrade` both check this automatically; verify by hand when you download the asset yourself. ```bash title="Check one asset" version=$(curl -fsSL https://download.surrealdb.com/surrealctl/beta.txt) asset="surrealctl-${version}.linux-amd64.tgz" base="https://download.surrealdb.com/surrealctl/${version}" curl -fsSLO "${base}/${asset}" curl -fsSL "${base}/${asset}.txt" sha256sum "${asset}" ``` The two digests must match. On macOS, use `shasum -a 256` instead of `sha256sum`. ```bash title="Check every asset you downloaded" curl -fsSLO "${base}/SHA256SUMS" shasum -c --ignore-missing SHA256SUMS ``` ## Build from source Building needs a Rust toolchain. The repository pins the version it wants, so `rustup` fetches the right one on the first build. ```bash git clone https://github.com/surrealdb/surrealctl cd surrealctl cargo install --path . ``` `cargo install` places the binary in `~/.cargo/bin`, so make sure that directory is on your `PATH`. ## Confirm the installation ```bash surrealctl version ``` The command prints the version, the commit it was built from, the build date, and the target triple. If the shell reports that the command was not found, the install directory is not on your `PATH`. `surrealctl --help` lists the command groups. Every group prints its own help, so `surrealctl instance --help` and `surrealctl instance create --help` both work. ## Sign in for the first time ```bash surrealctl auth login ``` The default flow opens a browser, waits for you to approve the sign-in, and stores the result. Three flows are available and `surrealctl` picks the first one that can work here: 1. **Browser loopback** - opens a browser and listens on `127.0.0.1` for the redirect. The default when a browser is available. 2. **Device code** - prints a short code to type on another device. Used automatically when there is no browser, or on request with `--flow device`. 3. **Paste** - prints a URL and takes back the address the browser was redirected to. Used with `--flow paste`, or when neither of the others can work. Over SSH, `surrealctl` skips the browser flow without being asked, because a browser on the far end of an SSH connection opens on the wrong machine. To force a particular flow, name it. A flow you ask for is used or it fails; `surrealctl` does not substitute another one silently. ```bash surrealctl auth login --flow device surrealctl auth login --no-browser ``` If the profile already has a credential, `auth login` reports who is signed in and does nothing. Pass `--force` to sign in again. ### Verify with whoami ```bash surrealctl whoami ``` `whoami` asks the API who you are, so it confirms that the stored credential is accepted and that the network path works. It is the counterpart to `surrealctl auth status`, which reads the local store and touches no network at all. ```bash surrealctl auth status ``` For what the two credential kinds are and how to use a token in CI, see [Authentication](/docs/manage/surrealctl/authentication.md). ## Check the whole setup `surrealctl status` - also spelled `surrealctl doctor` - runs eight checks in dependency order and prints one row for each. No check can abort the command, so you get the whole diagnosis in one run. | Check | What it tells you | | --- | --- | | `credential` | Which credential is in use, and whether it is usable. A failure here is fatal | | `api` | Whether the API base URL answers | | `version` | The minimum client version the API advertises. Reported, never enforced | | `cloud session` | Whether a Cloud session is held. Skipped for a personal access token | | `organization` | Which organisation resolves, and which layer decided it | | `surreal binary` | Which copy the handoff commands will use | | `system message` | Any platform notice currently published | | `release` | Whether a newer `surrealctl` release is available | Because a failure low in the table is often the one above it restated, read from the top: the first failing row is usually the cause and the rest are symptoms. This is the one command that still prints its table when it exits non-zero, because the table is the diagnosis. ## Keep surrealctl up to date `surrealctl` replaces itself. ```bash surrealctl upgrade ``` The command follows the channel this build is on, verifies the download against its published checksum, runs the new binary once to prove it works, and only then replaces the running one. Nothing is overwritten until every one of those steps has passed. | Option | Effect | | --- | --- | | `--check` | Report whether a newer release exists, and install nothing | | `--version ` | Install this exact version instead of the newest | | `--channel ` | Follow `latest`, `beta`, or `alpha` instead of this build's channel | | `--force` | Replace the binary without confirming | | `-o`, `--output ` | Write the new binary here instead of replacing the running one | ```bash title="Ask without installing" surrealctl upgrade --check ``` > [!IMPORTANT] > If the binary was installed by a package manager, `upgrade` refuses and names that tool's own command instead. Replacing a file a package manager owns leaves its records wrong, and the next upgrade through that tool would silently put the old version back. Use `brew upgrade`, `cargo install --force`, or your Nix workflow instead. ### Be told when a release lands `surrealctl status` reports the release check as one of its rows. To be told automatically, at most once a day: ```bash surrealctl config set release_check true ``` That prints a single line on stderr after a command finishes, and only when a terminal is attached - never in a pipe, a CI job, or `--json` output. Set `SURREALCTL_RELEASE_CHECK` to override the setting for one invocation. ## Shell completion `surrealctl completion` writes a completion script to stdout for `bash`, `elvish`, `fish`, `powershell`, or `zsh`. ```bash title="zsh" surrealctl completion zsh > "${fpath[1]}/_surrealctl" ``` ```bash title="bash" surrealctl completion bash > /etc/bash_completion.d/surrealctl ``` ## The `surreal` handoff `instance sql`, `instance import`, and `instance export` hand off to the `surreal` binary. Install it the [usual way](/docs/running/installation.md) and `surrealctl` finds it on your `PATH`. If it is missing when you run one of those three commands, `surrealctl` offers to fetch a copy. ```text `surrealctl instance sql` hands off to the `surreal` binary, and it is not installed. ? Download it from https://download.surrealdb.com now? (y/N) ``` The download lands in `~/.config/surrealctl/bin`, never in `/usr/local/bin`, so it needs no `sudo`. The offer appears only when somebody is there to answer it: an unattended run gets an error naming the install command instead, and `--yes` does not turn the offer into a silent download. `surrealctl` looks for the binary in this order, and the managed copy is last. 1. The `surreal_binary` key in the active profile 2. `SURREALCTL_SURREAL_BINARY` 3. `PATH` 4. `~/.config/surrealctl/bin/surreal` Installing `surreal` properly later therefore takes over with nothing to undo. To point at a specific copy, name it. ```bash surrealctl config set surreal_binary /opt/surrealdb/bin/surreal ``` A path you name explicitly and that does not resolve is an error, not a fallback - silently using a different binary than the one you asked for is how the wrong version gets blamed. ## Where surrealctl keeps its files Four paths, all in one directory under `$XDG_CONFIG_HOME/surrealctl` or `~/.config/surrealctl`. XDG applies on macOS too, because this is a terminal tool that people symlink into a dotfiles repository. | File | Contents | | --- | --- | | `config.toml` | Profiles, the remembered organisation, and other non-secrets. Safe to commit | | `credentials.json` | Tokens, and the values that must be replaced in the same write as them. Mode `0600` | | `credentials.lock` | An empty advisory lock file, so two concurrent runs cannot corrupt a refresh | | `bin/surreal` | The managed `surreal` copy, when one was downloaded | `surrealctl config path` prints the configuration file path. Set `SURREALCTL_CONFIG_DIR` to move the whole directory, or `--config` to point at one configuration file. > [!WARNING] > `surrealctl` refuses to read `credentials.json` if it is readable by group or other, and tells you to run `chmod 600` on it. The file holds a refresh token. ## Next steps - [Authentication](/docs/manage/surrealctl/authentication.md) - login sessions, personal access tokens, and CI credentials. - [Instances](/docs/manage/surrealctl/instances.md) - your first real workflow. - [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md) - every command and flag. --- Source: https://surrealdb.com/docs/manage/surrealctl/instances # Instances Deploy, find, connect to, scale, pause, back up, and delete instances with the surrealctl instance commands. This page walks through everyday instance work from the command line: creating an instance, finding it again, opening a SurrealQL session, resizing it, pausing it, taking backups, and deleting it. It assumes you are signed in and that an organisation resolves. For every flag on every verb, see the [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md). ## Before you start Instances belong to an organisation, and there is no route that lists them across organisations. Pick one for the profile once, and every `instance` command follows it. ```bash surrealctl org use acme ``` `--org` overrides it for a single command, and `SURREALCTL_ORG` overrides it for a shell. See [Organisations](/docs/manage/surrealctl/organisations.md#choosing-an-organisation) for the full order of precedence. Two catalogues tell you what you can deploy. ```bash surrealctl catalog regions surrealctl catalog instance-types surrealctl catalog instance-versions surrealctl org plans ``` `catalog` lists what the platform offers in general. `org plans` is narrower and organisation-specific - it is the one to trust before a `create`. ## Create an instance surrealctl instance create [OPTIONS] --type --region Price it first if the cost matters, then deploy. ```bash surrealctl instance estimate --type shared-1 --region aws-euw1 surrealctl instance create production --type shared-1 --region aws-euw1 ``` ", "required": true, "description": "A name for the instance. 1 to 30 characters." }, { "name": "--type", "value": "", "required": true, "description": "The instance type, by slug. See `surrealctl catalog instance-types`." }, { "name": "--region", "value": "", "required": true, "description": "The region to deploy into, by slug. See `surrealctl catalog regions`." }, { "name": "--version", "value": "", "description": "The SurrealDB version to run. Defaults to the platform's current one." }, { "name": "--compute-units", "value": "", "description": "How many compute units to run, for the types that scale." }, { "name": "--storage", "value": "", "description": "Storage to allocate, in gigabytes." }, { "name": "--access-type", "value": "", "description": "How the instance may be reached: `public`, `private`, or `dual`." }, { "name": "--restore-from", "value": "", "description": "Restore from a snapshot, as `/`." } ]} /> `create` waits until the instance is ready before it returns, so the next line in your script can connect. Pass `--no-wait` to return as soon as the API accepts the request; the output then includes the `surrealctl instance watch` command to pick the state back up. > [!WARNING] > Creating an instance is not idempotent. A retried `instance create` is a second instance and a second bill. `surrealctl` never replays a create automatically for this reason - if a create fails without a clear answer, run `surrealctl instance list` before trying again. ## Naming an instance Every command that takes an instance accepts four spellings: the id, the slug, the name, or `org/name` when you want to be explicit about which organisation. ```bash surrealctl instance get production surrealctl instance get acme/production surrealctl instance get prod-a1b2c3 ``` Omit the name at a terminal and `surrealctl` offers a picker. Omit it in a script and you get a usage error that lists the available instances, so the failure tells you what you should have typed. ## Find instances ```bash surrealctl instance list surrealctl instance list --wide surrealctl instance list --columns name,state,version --sort name surrealctl instance get production ``` `--columns`, `--wide`, `--sort`, `--reverse`, and `--limit` shape the table. The API has no pagination or sort parameter, so all five are applied on your machine after the whole list arrives - and none of them reaches `--json`, where the complete payload is always emitted. A `--limit 1` must not silently truncate a pipeline. ## Connect to an instance ```bash surrealctl instance endpoint production surrealctl instance sql production -- --ns app --db main ``` `instance endpoint` prints the endpoint alone, with no label and no styling, and its output is byte-identical whether you are at a terminal or in a pipe. That makes `$(surrealctl instance endpoint production)` safe to embed in another command. `instance sql` resolves the instance, mints a database token, and becomes `surreal sql`. Everything after `--` is passed to `surreal` verbatim, so the full flag surface of the sibling CLI is available. The token travels in the child process's environment, never in `argv`, where `ps` and shell history can both read it. Imports and exports work the same way, but both need a namespace and a database, because `surreal` requires them. The check happens before a token is minted. ```bash surrealctl instance export production --namespace app --database main -- backup.surql surrealctl instance import staging --namespace app --database main -- seed.surql ``` To authenticate your own client instead, mint a database token. ```bash surrealctl instance token production > token.txt surrealctl instance jwks production ``` `instance token` refuses to print to a terminal, where the token would stay in your scrollback. Pipe it, redirect it, or pass `--reveal` if you meant to read it. `instance jwks` fetches the key set that verifies those tokens; under `--json` it emits the whole key set so it can be piped straight into a verifier. ## Scale an instance ```bash surrealctl instance update production --compute-units 4 surrealctl instance update production --storage 100 surrealctl instance update production --version 3.2.4 ``` Pass at least one of `--type`, `--compute-units`, `--storage`, `--version`, or `--access-type`. With no flags at all, `update` is a usage error rather than a silent no-op. One invocation may be several requests. Each field has its own route, and `surrealctl` applies them in a fixed order - type, compute units, storage, version, access type - stopping at the first failure. If a later change fails, the earlier ones have already been applied. > [!IMPORTANT] > Every change in this list restarts the instance. Plan updates the way you would plan a deployment, and expect a reconnect window. Storage can only be changed once every few hours; `surrealctl` reports the cool-off and when the last change happened rather than sending a request that would be refused. ## Pause and resume ```bash surrealctl instance pause staging surrealctl instance resume staging ``` Pausing stops compute charges and keeps storage and configuration. Pausing an instance that is already paused prints a note, still emits the same document, and exits `0`, so a script that pauses an environment every evening is safe to run twice. ## Watch an operation `instance watch` follows an operation that was started elsewhere - by a colleague, by Studio, or by an earlier `--no-wait` command. ```bash surrealctl instance watch production surrealctl instance watch production --until paused surrealctl instance watch production --json ``` `--until` takes `ready`, `paused`, or `deleted`. Under `--json`, `watch` writes one compact object per line to stdout as the state changes, because the transitions are the answer rather than a progress report. ## Waiting semantics `create`, `update`, `delete`, `pause`, and `resume` wait for the instance to settle before they return. ", "default": "15m", "description": "How long to wait before giving up, such as `90s` or `20m`." } ]} /> Waiting is on in rich, plain, CI, and `--json` output alike. A default that changed with the terminal would mean `CI` changed behaviour rather than presentation, and would break the obvious script: create, then connect. Polling backs off - two seconds for the first thirty, then five, then ten - with jitter so parallel waiters do not convoy. Being rate limited extends the deadline rather than consuming it, up to a couple of minutes of credit, so a busy account does not get spurious timeouts. If the wait gives up, `surrealctl` exits `10` and says so. That is a distinct outcome from a failure: **the operation is still running server-side.** Poll again with `instance get` or `instance watch` rather than rolling back. Interrupting a wait does not stop the operation either. It carries on, and `surrealctl instance watch ` picks the state back up. ## Read state, logs, and metrics ```bash surrealctl instance status production surrealctl instance logs production --level error --limit 100 surrealctl instance logs production --follow surrealctl instance metrics production --metric cpu --from 2026-08-11T10:00:00Z surrealctl instance usage production ``` `instance status` reports the deployment phase and the snapshots the instance can be restored from, as one document. `--level` is a display filter applied on your machine - the API has no level parameter - so `--json` still carries every line the API sent, and a note tells you how many lines were hidden. `--follow` polls every five seconds and, under `--json`, writes one compact object per new line. `--metric` is free-form, because the set of metrics is not published anywhere `surrealctl` can read. `cpu`, `memory`, and `disk` are the usual ones. `--from` and `--to` take RFC 3339 timestamps and are forwarded byte for byte. For the equivalent views in the browser, see [Monitoring](/docs/manage/instances/monitoring.md). ## Backups ```bash surrealctl instance backup list production surrealctl instance backup create production surrealctl instance backup policy get production surrealctl instance backup policy set production --daily 14 --weekly 8 ``` The three retention flags - `--daily`, `--weekly`, `--monthly` - take a number of periods to keep, or the literal `default` to restore your organisation's own value for that tier. `--frequency-hours` sets how often a snapshot is taken, in whole hours up to 24; the plan decides which values are allowed. Only a **reduction** asks for confirmation. Lengthening a retention cannot lose a snapshot, so it is applied without a prompt. Which tiers you may change depends on the organisation's plan. `backup policy get` shows which are editable. `instance backup create` returns as soon as the platform accepts the request. It has no `--wait`, and a new snapshot takes a while to appear in the list, so do not expect to restore from one you took a moment ago. Restoring means creating a new instance from a snapshot, which leaves the original untouched. ```bash snapshot=$(surrealctl instance backup list production --json \ | jq -r 'sort_by(.started_at) | last | .snapshot_id') surrealctl instance create recovery \ --type shared-1 --region aws-euw1 \ --restore-from "production/${snapshot}" ``` See [Backups and recovery](/docs/manage/instances/backups.md) for retention behaviour and what a snapshot contains. ## Capabilities Capabilities decide what SurrealQL running on the instance may do: scripting, guest access, outbound network access, which functions and RPC methods are permitted. ```bash surrealctl instance capabilities get production surrealctl instance capabilities set production --deny-scripting --allow-net api.example.com ``` The endpoint is a full replacement, so `capabilities set` reads the current configuration, applies your flags, prints what would change, and asks before writing. Two consequences matter: - **A list flag replaces that list.** `--allow-net a.example.com,b.example.com` sets the whole list. A later run naming only `a.example.com` removes `b.example.com`. - **Anything you do not name is left exactly as it is**, including capabilities this version of `surrealctl` does not know about. Flags that would change nothing print a note and write nothing. `--force` skips the confirmation. For what each capability means, see [Configure an instance](/docs/manage/instances/configure.md) and [Network access](/docs/manage/instances/network-access.md). ## Delete an instance ```bash surrealctl instance delete staging surrealctl instance delete staging --force ``` The confirmation names the instance, its slug, and its region. Declining changes nothing and exits `0`. `--force` skips the prompt; in an unattended session with neither `--force` nor `--yes`, `delete` exits `2` having sent no request. Deletion removes the instance and everything in it. Take an export or confirm a snapshot first. ## Worked examples ### Stand up a review environment ```bash title="review-env.sh" #!/usr/bin/env bash set -euo pipefail name="review-${1}" surrealctl instance create "$name" --type shared-1 --region aws-euw1 surrealctl instance import "$name" --namespace app --database main -- fixtures/seed.surql surrealctl instance endpoint "$name" ``` `create` returns only once the instance is ready, so the `import` on the next line has something to talk to. The final line prints the endpoint for whatever consumes this script. ### Pause every non-production instance overnight ```bash title="pause-overnight.sh" #!/usr/bin/env bash set -euo pipefail surrealctl instance list --json \ | jq -r '.[] | select(.state == "ready" and .name != "production") | .name' \ | while read -r name; do surrealctl instance pause "$name" --wait-timeout 5m done ``` ### Clone production from its latest snapshot ```bash title="clone-production.sh" #!/usr/bin/env bash set -euo pipefail snapshot=$(surrealctl instance backup list production --json \ | jq -r 'sort_by(.started_at) | last | .snapshot_id') surrealctl instance create scratch \ --type shared-1 --region aws-euw1 \ --restore-from "production/${snapshot}" surrealctl instance endpoint scratch ``` ## Next steps - [Organisations](/docs/manage/surrealctl/organisations.md) - members, roles, invitations, usage, and spend. - [Scripting](/docs/manage/surrealctl/scripting.md) - exit codes and the `--json` contract in full. - [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md) - every instance flag and default. --- Source: https://surrealdb.com/docs/manage/surrealctl/organisations # Organisations Select an organisation, manage members and invitations, check roles, and read usage and spend from the command line. An organisation owns instances, members, and billing. This page covers how `surrealctl` decides which organisation you mean, how to manage the people in it, and how to read what it has used and what it has cost. It is for anyone who administers a team's SurrealDB Cloud account. ## Choosing an organisation Almost every command needs an organisation. `surrealctl` works down a fixed order and stops at the first answer. 1. `--org` on the command line 2. `SURREALCTL_ORG` in the environment 3. `org` in this profile's section of `config.toml` 4. The organisation remembered by `surrealctl org use` 5. Your account's default organisation 6. The only candidate, when you belong to exactly one 7. An interactive picker, at a terminal 8. An error naming the flag ```bash surrealctl org list surrealctl org use acme # remember it for this profile surrealctl --org acme instance list surrealctl org use --clear # forget it again ``` `surrealctl context show` reports which organisation resolved and which layer decided it. Add `--debug` to any command to see the winning layer for every value. In CI, set `SURREALCTL_ORG` to the organisation **id** rather than its name. An id is used as it stands; a name costs a lookup on every command. > [!NOTE] > `SURREALCTL_ORG` outranks the value `org use` remembered. If a remembered organisation seems to be ignored, check whether the variable is exported. `org use` warns you about it at the time. ## Manage organisations ```bash surrealctl org list surrealctl org list --all # include archived organisations surrealctl org get acme surrealctl org create acme-labs --use surrealctl org update acme --name "Acme Ltd" surrealctl org archive acme ``` An organisation name is 1 to 30 characters. `--use` on `create` remembers the new organisation immediately, so the commands after it need no `--org`. `org archive` asks for confirmation unless you pass `--force`. `org get` shows the plan, the state, your role, the member count, the instance allowances, and the billing configuration. ## Members Members are addressed by username or user id. ```bash surrealctl team list surrealctl team get ada surrealctl team update ada --role admin surrealctl team remove ada ``` `team remove` ends someone's membership of the organisation. Their account is untouched, which is why the verb is `remove` and not `delete`. `team list` deliberately does not fold in pending invitations. Someone who has been invited is not yet a member; `invite list` is where they appear. ## Invitations Roles are defined per organisation, so read the list before you send an invitation. ```bash surrealctl org roles surrealctl team invite ada@example.com --role member surrealctl invite list surrealctl invite delete ada@example.com ``` ", "required": true, "description": "The email address to invite." }, { "name": "--role", "value": "", "required": true, "description": "The role to grant. Run `surrealctl org roles` for the ones this organisation defines." } ]} /> `surrealctl team invite` and `surrealctl invite create` are the same command under two names - use whichever reads better where you are. If the API rejects the role, the error points you back at `org roles`. `invite delete` withdraws an invitation, by the email address it was sent to or by its code. The confirmation names the address. ## Roles and permissions ```bash surrealctl org roles # the roles this organisation can assign surrealctl org permissions # what you are permitted to do here ``` `org permissions` answers "what may I do in this organisation" for the credential in hand. It is the question `auth scopes` cannot answer for a login session, where the authorisation model is your role rather than a scope list. For what Owner, Admin, and Member each permit, see [Members and roles](/docs/manage/organisations/members-and-roles.md). ## Personal access tokens Tokens belong to your account rather than to an organisation, but they are how a pipeline reads an organisation's data. ```bash surrealctl token scopes surrealctl token create "ci-read-only" --scope read:cloud | tail -1 surrealctl token list surrealctl token delete ci-read-only ``` All four verbs need a login session, so a token cannot mint its own replacement. See [Authentication](/docs/manage/surrealctl/authentication.md#personal-access-tokens) for scopes, expiry, and how to capture the secret. ## Usage and spend ```bash surrealctl org usage # consumption across every instance surrealctl instance usage production # one instance surrealctl org spend # the current billing period surrealctl org spend --period 03-2026 # a specific month surrealctl org plans # the plans available to this organisation ``` `--period` is written month first, as `MM-YYYY`. If you write it the other way round, `surrealctl` says so and shows the correction rather than sending the request. In text output, `org spend` prints a total on stderr. The total is computed before `--limit` is applied, so it is always the whole bill even when the table is trimmed. > [!IMPORTANT] > Under `--json`, money stays in integer minor units - cents, or millicents where the API uses them. Do not treat `amount` as a decimal. Divide before you display. ```bash title="Total spend for a month, in whole currency units" surrealctl org spend --period 03-2026 --json | jq '[.[].amount] | add / 100' ``` ```bash title="Line items as TSV" surrealctl org spend --json \ | jq -r '.[] | [.effective_at, .resource, .description, .amount] | @tsv' ``` For invoices, payment methods, and how the platform bills, see [Billing](/docs/manage/organisations/billing.md). ## Open the dashboard ```bash surrealctl open org acme surrealctl open billing acme surrealctl open instance production surrealctl open terms ``` The URL is always printed; a browser is only opened when somebody is present to look at it, so this is safe in a script that logs its output. `surrealctl` links to the terms and never accepts them on your behalf. ## Worked examples ### Onboard a new engineer ```bash title="onboard.sh" #!/usr/bin/env bash set -euo pipefail email="$1" surrealctl team invite "$email" --role member surrealctl invite list --columns email,role,status ``` ### Report which instances cost the most ```bash title="top-spend.sh" #!/usr/bin/env bash set -euo pipefail surrealctl org spend --json \ | jq -r 'group_by(.instance_id) | map({instance: .[0].instance_id, cents: ([.[].amount] | add)}) | sort_by(-.cents) | .[] | [.instance, .cents] | @tsv' ``` ### Audit who is in every organisation you belong to ```bash title="audit-members.sh" #!/usr/bin/env bash set -euo pipefail surrealctl org list --json | jq -r '.[].id' | while read -r org; do echo "== ${org}" surrealctl --org "$org" team list --json | jq -r '.[] | [.username, .role] | @tsv' done ``` ## Next steps - [Scripting](/docs/manage/surrealctl/scripting.md) - exit codes, `--json`, and unattended runs. - [Instances](/docs/manage/surrealctl/instances.md) - the instance lifecycle from the command line. - [Organisations](/docs/manage/organisations.md) - the same concepts in SurrealDB Studio. --- Source: https://surrealdb.com/docs/manage/surrealctl/scripting # Scripting Drive surrealctl from CI and shell scripts: the JSON contract, exit codes for control flow, unattended runs, and waiting semantics. This page covers running `surrealctl` where nobody is watching: in CI, in cron, and in shell scripts. It explains the output contract, how to branch on exit codes, how to keep a run from blocking on a prompt, and where to put a credential. It is for engineers automating control-plane work. ## The two streams `surrealctl` splits its output by purpose, and the split is absolute. - **stdout** carries the command's data payload. Nothing else, ever. - **stderr** carries progress, prompts, warnings, hints, errors, and debug output. So `| jq` always works, `--json > out.json` can still ask you a question, and a failed command leaves stdout empty rather than half a document. A broken downstream pipe is not an error: `surrealctl instance list | head -3` exits `0` silently. ## Reading successful output `--json` prints the bare wire payload on stdout, with no envelope. There is no `.data[]` to unwrap, because the stream split and the exit code already answer "did it work". ```bash surrealctl instance list --json | jq -r '.[] | select(.state == "ready") | .name' surrealctl instance get production --json | jq -r '.version' surrealctl org list --json | jq -r '.[] | [.id, .name] | @tsv' ``` Every invocation emits exactly one document. Two commands stream instead, and say so in their own help: `instance watch` and `instance logs --follow` emit newline-delimited JSON, one object per line. Two more are not documents at all: `completion` writes a shell script, and `token create` writes a secret, so it has no `--json` form. A field the API adds appears in `--json` without a new release of `surrealctl`, so treat the payload as open: read the keys you need and ignore the rest. `--limit`, `--sort`, `--columns`, `--wide`, and `--reverse` are presentation only. They shape a table and never reach `--json`, so `--limit 1` cannot silently truncate a pipeline. ## Reading failures On failure, `--json` writes one enveloped object to **stderr** and leaves stdout empty. ```json title="Error envelope" { "kind": "conflict", "message": "could not pause `api`: Instance is not in a valid state", "status": 409, "code": null, "request_id": "01J8XYZ7QK9M3P5R7T9V1W3Y5Z", "hint": "The instance is busy with another change.", "command": "surrealctl instance get --wait", "docs": null, "retry_after_secs": null, "exit_code": 6 } ``` Every key is always present, with an explicit `null` rather than being omitted, so a consumer can index without checking first. | Key | Type | Contents | | --- | --- | --- | | `kind` | string | The error class. Never `null` | | `message` | string | The whole failure chain, including which command failed | | `status` | integer or `null` | The HTTP status, when there was one | | `code` | string or `null` | The API's own error code, when it sent a non-empty one | | `request_id` | string or `null` | The request identifier to quote in a support report | | `hint` | string or `null` | What to do about it | | `command` | string or `null` | A command to run next | | `docs` | string or `null` | A documentation link, for the classes that have a useful one | | `retry_after_secs` | integer or `null` | How long to wait, from the response | | `exit_code` | integer | The process exit code, so the document and the shell agree | One further key appears conditionally: `candidates`, an array of suggested names, when a reference failed to resolve and `surrealctl` had near matches to offer. `kind` is a closed vocabulary of eleven values, so it is safe to match on: `auth`, `forbidden`, `not_found`, `conflict`, `invalid`, `rate_limited`, `upstream`, `network`, `not_available`, `wait_timeout`, `unknown`. ```bash if ! surrealctl instance pause api --json > /dev/null 2> error.json; then jq -r '"\(.kind): \(.message)"' < error.json >&2 fi ``` `not_available` is worth calling out: it means a feature is not enabled for this deployment. Nothing is broken. ## Exit codes Error text is not a contract. These numbers are. | Code | Meaning | | --- | --- | | `0` | Success, including an empty list | | `1` | An unclassified failure | | `2` | Bad invocation, or a destructive command that could not confirm | | `3` | Not authenticated, or the credential expired and could not be renewed | | `4` | Authenticated but not allowed: scope, role, or credential kind | | `5` | The named resource does not exist | | `6` | The resource is not in a state that allows this, or a precondition failed | | `7` | The API rejected the request as invalid | | `8` | Rate limited, after the retry budget was spent | | `9` | The API or its upstream is unreachable, or the feature is not available | | `10` | A wait gave up. **The operation is still running** | | `11` | The credential store could not be read or written | | `30` | An interactive sign-in did not complete | | `130` | Interrupted | `0`, `1`, and `2` keep their conventional meanings, so `if ! surrealctl …` and usage errors behave exactly as they do with the `surreal` CLI. Everything above `2` is additive. `10` earns its own code because "still running" is a genuinely different answer from "failed". A pipeline may reasonably poll again rather than roll back. ### Branching on the code Capture the status rather than testing it inside an `if`, so `set -e` does not end the script and `$?` still means what you think it does. ```bash title="create-if-missing.sh" #!/usr/bin/env bash set -euo pipefail status=0 surrealctl instance get api --json > instance.json 2> error.json || status=$? case "$status" in 0) ;; 5) surrealctl instance create api --type shared-1 --region aws-euw1 --json > instance.json ;; 3) echo "credential expired; re-authenticate the runner" >&2; exit 1 ;; 8) exit 75 ;; # tell the scheduler to requeue *) jq -r '.message' < error.json >&2; exit "$status" ;; esac ``` ## Unattended runs ", "env": "SURREALCTL_ORG", "description": "The organisation to operate on, by id or name." }, { "name": "--token", "value": "", "env": "SURREALCTL_TOKEN", "description": "Personal access token to authenticate with." }, { "name": "--timeout", "value": "", "default": "30s", "env": "SURREALCTL_TIMEOUT", "description": "Maximum time to wait for a single API request." }, { "name": "--retries", "value": "", "default": "3", "env": "SURREALCTL_RETRIES", "description": "How many times to retry a failed request." }, { "name": "--debug", "env": "SURREALCTL_DEBUG", "description": "Log every API request and response to stderr." } ]} /> Every one of these has an environment variable, so a pipeline can set them once for the whole job rather than repeating flags on each step. Two rules govern prompts. **A destructive command refuses rather than proceeds when it cannot ask.** With no `--yes` and no command-specific `--force`, `instance delete` in a non-interactive session exits `2` having sent nothing. A command that quietly went ahead because there was no terminal would be the worst possible default. **`--no-input` makes any session behave as an unattended one**, even at a terminal. Use it to test a pipeline locally: if the script works with `--no-input`, it will work in CI. **Bash** ```bash export SURREALCTL_NO_INPUT=1 surrealctl instance delete staging --force ``` **PowerShell** ```powershell $env:SURREALCTL_NO_INPUT = "1" surrealctl instance delete staging --force ``` ## CI detection and output modes `surrealctl` detects a CI environment and switches to plain output on its own: no spinners, no borders, no relative times. `CI=false`, `CI=0`, and an empty `CI` all mean *not* CI, so a system that sets the variable deliberately is not trapped in plain output. What CI changes is structure, interactivity, and how much progress is reported. What it must not change, and does not: colour, `--json` bytes, exit codes, which requests are made, timeouts, poll intervals, `--wait` defaults, and confirmation semantics. `--json` output is byte-identical whatever the terminal, the width, or the colour settings. Plain output does not depend on terminal width either, so it is stable to diff between runs. For parsing, still prefer `--json`. ## The token in the environment Put a personal access token in the environment and leave it out of `argv`, where `ps` and shell history can both read it. See [Authentication](/docs/manage/surrealctl/authentication.md#personal-access-tokens) for how to create one. ```bash export SURREALCTL_TOKEN="$(cat /run/secrets/surrealctl)" export SURREALCTL_ORG=67upifj5dt6p87ch3nh5t3a8 surrealctl instance list --json ``` Set `SURREALCTL_ORG` to the organisation **id**. An id is used as it stands; a name costs a lookup on every command. ```yaml title=".github/workflows/instance-report.yml" name: instance-report on: schedule: - cron: "0 6 * * *" jobs: report: runs-on: ubuntu-latest env: SURREALCTL_TOKEN: ${{ secrets.SURREALCTL_TOKEN }} SURREALCTL_ORG: ${{ vars.SURREALCTL_ORG_ID }} SURREALCTL_NO_INPUT: "1" steps: - run: curl -fsSL https://download.surrealdb.com/surrealctl/install.sh | sh - run: surrealctl status - run: surrealctl instance list --json > instances.json - uses: actions/upload-artifact@v4 with: name: instances path: instances.json ``` `surrealctl status` early in a job is worth the one second it costs: it names which credential is in use, whether the API answers, and which organisation resolved, all before a later step fails for one of those reasons. > [!IMPORTANT] > A personal access token reads the control plane and cannot write to it. A job that creates, updates, or deletes needs a login session on a dedicated machine account. See [What a personal access token cannot do](/docs/manage/surrealctl/authentication.md#what-a-personal-access-token-cannot-do). ## Waiting in a pipeline `instance create`, `update`, `delete`, `pause`, and `resume` wait for the operation to finish, in every output mode including `--json`. You have three choices. | Choice | Behaviour | Use it when | | --- | --- | --- | | Default | Wait up to 15 minutes | The next step needs the instance ready | | `--wait-timeout ` | Wait that long, then exit `10` if it has not settled | The job has its own time budget | | `--no-wait` | Return as soon as the request is accepted | Something else will follow up | Under `--json`, a waiting command streams progress as newline-delimited JSON on **stderr** and puts one final document on stdout. That split is what keeps "exactly one document on stdout" true for a command that reports progress. ```json title="Progress events, on stderr" {"event":"started","resource":"instance","goal":"ready","state":"pending"} {"event":"transition","resource":"instance","from":"pending","to":"ready","elapsed_secs":86} {"event":"finished","resource":"instance","outcome":"succeeded","state":"ready","elapsed_secs":86,"succeeded":true} ``` Six event types appear: `started`, `transition`, `heartbeat`, `poll_failed`, `throttled`, and `finished`. `instance watch --json` inverts the stream and writes these to stdout instead, because for a watch the transitions *are* the answer. Treat exit `10` as "poll again", not "failed". ```bash title="fail-fast-then-follow.sh" #!/usr/bin/env bash set -euo pipefail status=0 surrealctl instance create api \ --type shared-1 --region aws-euw1 \ --wait-timeout 2m --json > instance.json || status=$? if [ "$status" -eq 10 ]; then echo "still provisioning after two minutes; following the state" >&2 surrealctl instance watch api elif [ "$status" -ne 0 ]; then exit "$status" fi ``` Interrupting a wait does not stop the operation. It continues server-side, and `surrealctl instance watch ` picks the state back up. ## Retries, timeouts, and repeated runs `--timeout` bounds a single request and `--retries` bounds how many times one is retried. Retry behaviour depends on what the request would change: a read is retried on a connection failure or a 5xx, and a create is never retried after a 5xx, because the instance may exist and a replay would bill for two. Rate limits honour the server's `Retry-After`. That leaves a small amount for your script to know about re-runs. | Command | Repeating it | | --- | --- | | `instance create` | Creates a second instance. Check with `instance get` first | | `instance backup create` | Takes a second snapshot | | `instance pause`, `instance resume` | Safe. Already-paused prints a note, emits the same document, exits `0` | | `instance delete` | Safe. A missing instance while waiting counts as done | | `auth logout` | Safe. Signed out already still exits `0` with a document | | `instance capabilities set` | Safe. No change means no write | ## Calling the API directly When a route has no verb yet, `surrealctl api` sends the request with the credential, headers, and retry policy already handled. ```bash surrealctl api get /api/cloud/v0/organizations surrealctl api get /api/cloud/v0/organizations --include surrealctl api post /api/cloud/v0/organizations -d @body.json --force ``` The path is a path, not a URL - the host comes from `--api`. Any write method asks for confirmation unless you pass `--force`. `--query key=value` and `--header name:value` are repeatable. Authentication headers cannot be overridden here; change the credential instead. `--include` prints the status and headers to stderr, and `--raw` prints the body exactly as it arrived. ## Configuration on the machine Values a script should not have to repeat can live in the profile instead. ```bash surrealctl config list surrealctl config set json true surrealctl config set org acme surrealctl config path ``` An environment variable outranks the configured value, and `config set` warns you when the matching variable is exported. `config get` prints the value alone, and nothing at all when there is none, so `[ -z "$(surrealctl config get org)" ]` behaves. ## Checklist before shipping a pipeline - Set `SURREALCTL_NO_INPUT=1` so nothing can block on a prompt. - Set `SURREALCTL_ORG` to an organisation id. - Pass `--json` and parse with `jq`. Never grep stderr. - Branch on exit codes, not on messages. - Treat `10` as "poll again", and `8` as "requeue". - Pass `--yes` or a command's `--force` deliberately, on the steps that need it. - Give a read-only job a personal access token, and nothing more. - Run `surrealctl status` first, so a misconfigured job says so in one line. ## Next steps - [Authentication](/docs/manage/surrealctl/authentication.md) - credentials, scopes, and profiles. - [Instances](/docs/manage/surrealctl/instances.md) - the workflows these scripts drive. - [`surrealctl` reference](/docs/reference/cli/surrealctl/overview.md) - every command, flag, default, and environment variable. --- Source: https://surrealdb.com/docs/reference/cli # CLI Tools Reference for surrealctl, surreal and surqlfmt. The control plane, the data plane, and formatting SurrealQL. SurrealDB ships three command-line tools. They are installed separately and cover different jobs, so most workflows use more than one of them. | Tool | Covers | Reference | | --- | --- | --- | | `surrealctl` | The control plane: organisations, instances, members, tokens, and billing. | [surrealctl](/docs/reference/cli/surrealctl/overview.md) | | `surreal` | The data plane: run a server, query it, and move data in and out. | [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) | | `surqlfmt` | Formatting `.surql` files to a consistent style. | [SurrealQL formatter](/docs/reference/cli/formatter/overview.md) | ## surrealctl Every subcommand has its own page under [surrealctl commands](/docs/reference/cli/surrealctl/commands.md). `surrealctl` manages SurrealDB Cloud from the command line: it signs you in, selects an organisation, and creates, scales, pauses, and deletes instances. Every command can emit JSON, so it is the scriptable path for CI pipelines and provisioning scripts. ```bash surrealctl auth login surrealctl instance create api --type shared-1 --region aws-euw1 ``` `surrealctl` never replaces `surreal`. For SurrealQL work against a Cloud instance it resolves the endpoint and credentials, then hands off to the `surreal` binary. See the [surrealctl reference](/docs/reference/cli/surrealctl/overview.md) for installation, authentication, global flags, and every command. ## SurrealDB CLI The `surreal` binary is the data-plane tool. It starts a server, opens an interactive SurrealQL shell, imports and exports data, validates query files, and reports the version in use. ```bash surreal start --user root --pass secret surreal sql --username root --password secret --pretty ``` See the [SurrealDB CLI reference](/docs/reference/cli/surrealdb-cli/overview.md) for a walkthrough, the [command list](/docs/reference/cli/surrealdb-cli/commands.md) for each subcommand, and [environment variables](/docs/reference/cli/surrealdb-cli/environment-variables.md) for the `SURREAL_*` variables that mirror its flags. ## SurrealQL formatter `surqlfmt` reformats `.surql` files to a consistent style. It is distributed as an npm package rather than as part of the `surreal` binary, which makes it easy to pin in a project and run in CI. ```bash npm install -g @surrealdb/surql-fmt surqlfmt --check ./src/**/*.surql ``` See the [SurrealQL formatter reference](/docs/reference/cli/formatter/overview.md) for its options and usage patterns. --- Source: https://surrealdb.com/docs/reference/cli/formatter/overview # SurrealQL formatter The surqlfmt command automatically formats SurrealQL files for consistent style and readability. The `surqlfmt` tool reformats `.surql` files to follow a consistent style. It adjusts whitespace, indentation, and keyword casing so that SurrealQL code is easy to read and review. `surqlfmt` is a third command-line tool alongside [`surreal`](/docs/reference/cli/surrealdb-cli/overview.md) and [`surrealctl`](/docs/reference/cli/surrealctl/overview.md). It is shipped as an npm package rather than as part of the `surreal` binary, it never connects to a database, and it only reads and writes files. ## Installation The tool can be installed through a single [npm install command](https://www.npmjs.com/package/@surrealdb/surql-fmt): ```bash npm install -g @surrealdb/surql-fmt ``` ## Usage surqlfmt [OPTIONS] [FILES]... ", "description": "Number of indent characters per level." }, { "name": "--indent-char", "value": "", "description": "Character used for indentation, for example `tab`." }, { "name": "--max-line-length", "value": "", "description": "Column at which the formatter wraps long lines." } ]} /> The tool prints the reformatted file by default, and reformats a `.surql` file in place when `--write` is passed. ```bash # Format a single file surqlfmt ./query.surql # Check if files are formatted surqlfmt --check ./src/**/*.surql # Format from stdin cat query.surql | surqlfmt --stdin # Configure indentation and line length surqlfmt --indent 4 --indent-char tab --max-line-length 120 *.surql # Write files in-place surqlfmt --write ./src/**/*.surql ``` For the options of the version you have installed, run `surqlfmt --help`. ## When to use the formatter - **Before committing** - run `surqlfmt` on any `.surql` migration or seed files to keep diffs clean. - **In CI** - add a `surqlfmt --check` step to catch inconsistencies early. - **During development** - pipe ad-hoc queries through the formatter for readability. The `surreal` binary also has a built-in [`format`](/docs/reference/cli/surrealdb-cli/commands/format.md) command, which formats one file at a time and takes no style options. `surqlfmt` is the tool for sweeping a tree of `.surql` files or gating a CI job on formatting. For the other command-line tools, see the [CLI tools overview](/docs/reference/cli.md). --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/authentication # Authentication reference How surrealctl signs in, the difference between a login session and a personal access token, what a token cannot do, and where credentials are stored on disk. `surrealctl` holds one credential per profile. There are two kinds, and they differ in what they can *write* rather than in what they can reach. | Credential | Created by | Suited to | | --- | --- | --- | | **Login session** | `surrealctl auth login` | Interactive use. Can do everything your role allows. | | **Personal access token** | `surrealctl token create`, or the dashboard | CI and automation. Read-only on the Cloud surface. | A login session is an OAuth 2.0 authorisation-code flow with PKCE against `https://auth.surrealdb.com/`. It stores a refresh token and one access token per audience, and renews them as they expire. A personal access token is a long-lived `sdbp_…` string that carries a fixed set of scopes and never expires on a schedule. ## Signing in ```bash surrealctl auth login ``` The command tries three flows in order, and stops at the first one that can run. | Flow | Used when | | --- | --- | | Browser loopback | A browser is available and a `127.0.0.1` port can be bound | | Device code | `--flow device`, or no browser is available | | Paste | `--flow paste`, or nothing else could run | Only *mechanical* failures fall through to the next flow - specifically, the loopback listener failing to bind or serve. If you decline the consent screen, or your email address is unverified, the command says so rather than asking again in a different form. An incomplete interactive flow exits `30`. Naming a flow explicitly means it is used or it fails; there is no silent substitution. ```bash title="Sign in on a machine with no browser" surrealctl auth login --flow device ``` The browser flow is skipped automatically when `SSH_CONNECTION` or `SSH_TTY` is set, because a browser at the far end of an SSH connection opens on the wrong machine. Signing in when the profile already holds a credential is a no-op that exits `0`. Pass `--force` to replace it. ```bash title="Sign in to a second profile" surrealctl --profile staging auth login ``` ### How the browser flow is hardened Worth knowing when a corporate network is in the way: - The listener binds the literal `127.0.0.1`, never `localhost`, which resolves through `/etc/hosts` and DNS and can answer on `::1`. - One of four ports is used - `9375`, `9376`, `9377`, `9378` - because the identity provider matches callback URLs exactly. - Only `/callback` is answered; anything else gets a 404. - The `state` parameter is compared in constant time, and the flow has a 180-second deadline. - The device flow gets 600 seconds, because a device code is typed on a phone. ## Personal access tokens A token is supplied per invocation and is never persisted: **Bash** ```bash title="From the environment - the usual choice in CI" export SURREALCTL_TOKEN=sdbp_... surrealctl instance list ``` **PowerShell** ```powershell title="From the environment - the usual choice in CI" $env:SURREALCTL_TOKEN = "sdbp_..." surrealctl instance list ``` ```bash title="From a file, or from stdin" surrealctl --token-file /run/secrets/surrealctl instance list cat /run/secrets/surrealctl | surrealctl --token-file - instance list ``` `--token` and `--token-file` are mutually exclusive: a credential comes from a flag or from a file, never both. `--token` deliberately has no short form, because `-t` reads as `--type` on the instance commands, and because a credential belongs in the environment or a file rather than in `argv`, where `ps` and shell history can both see it. The only way to *store* a token is `auth login --with-token`, which reads it from stdin: ```bash echo "$SURREALCTL_TOKEN" | surrealctl auth login --with-token --label "ci runner" ``` ### What a personal access token cannot do **It cannot manage personal access tokens.** All four verbs - `token list`, `token create`, `token delete` and `token scopes` - are refused locally, before any request is sent, and exit `4`. A leaked token must not be able to mint its own replacements, nor revoke the one an operator would use to clean up after it. ```text Managing personal access tokens needs an interactive login session, so nothing was sent. Sign in with: surrealctl auth login ``` **It cannot make Cloud writes.** The gateway refuses a token on every non-`GET` Cloud route, so the mutating verbs - `instance create`, `org update`, `spectron key rotate` and the rest - answer 403, exit `4`, even when the token carries the matching write scope. Use a login session for those. **It cannot exceed its scopes.** A missing scope is a 403 naming the scope, relayed rather than predicted. Everything else works, including `org list`, `instance list`, `instance endpoint`, `instance token`, `auth status`, `auth scopes` and `auth logout`. ### Scopes The scope vocabulary is shared with the Cloud MCP tools: | Scope | Grants | | --- | --- | | `read:cloud` | Read access across the Cloud surface | | `write:cloud-instances` | Create, update and delete instances | | `write:cloud-organization` | Change organisation settings and membership | | `write:cloud-billing` | Change billing details | | `write:cloud-spectron` | Manage SurrealDB Agent Memory contexts and keys | Two commands answer two different questions. [`token scopes`](/docs/reference/cli/surrealctl/commands/token.md#token-scopes) is the catalogue of what *can* be granted; [`auth scopes`](/docs/reference/cli/surrealctl/commands/auth.md#auth-scopes) is what the credential in hand *carries*. One is the menu, the other is the receipt. `auth scopes` on a login session returns an empty list and points you at [`org permissions`](/docs/reference/cli/surrealctl/commands/org.md#org-permissions), because OAuth identity scopes are not an authorisation model - your role in the organisation is. ## What each `auth` verb does when nothing is signed in | Verb | Behaviour | | --- | --- | | `auth status` | Succeeds, exit `0` | | `auth logout` | Succeeds, exit `0`, and still emits a document | | `auth scopes` | Exits `3` | | `auth refresh` | Exits `3` | A diagnostic that errors because there is no credential cannot help you fix it, and logging out of nothing is a no-op rather than a failure - a teardown script must not fail on its second run. `scopes` and `refresh` each answer a question *about* a credential, and with none there is no answer to give. `auth status` and `auth scopes` touch no network at all, so they work on a plane, in a container with no egress, and in a CI job that is about to fail for a different reason. [`whoami`](/docs/reference/cli/surrealctl/commands/misc.md#whoami) is the command that asks the API who you are. ## Where credentials are stored One directory holds everything: | Path | Holds | | --- | --- | | `config.toml` | Non-secret configuration: profiles, the persisted organisation | | `credentials.json` | Tokens, the Cloud user id, and the observed clock skew | | `credentials.lock` | A zero-byte advisory lock file | | `bin/surreal` | The managed copy of the `surreal` binary, when one was downloaded | The directory is created mode `0700`, and `credentials.json` mode `0600`. The split between the two files is about **atomicity, not sensitivity**: the few non-secret values that must be replaced in the same transaction as a token live beside it in `credentials.json`. `config.toml` is safe to commit or to sync into a dotfiles repository. `credentials.json` is not. ### Paths The directory is resolved in this order: 1. The directory containing the path given to `--config` 2. `SURREALCTL_CONFIG_DIR` 3. `$XDG_CONFIG_HOME/surrealctl`, or `~/.config/surrealctl` when that variable is unset On Windows the platform configuration directory is used instead - typically `%APPDATA%\surrealctl`. XDG applies on macOS too, rather than `~/Library/Application Support`: this is a terminal tool people symlink into dotfiles repositories, and the macOS path contains a space that shells and documentation both handle badly. `gh` and `aws` make the same choice. ```bash title="Print the resolved path" surrealctl config path ``` ### Permissions are enforced, not warned about Reading `credentials.json` with any group or other bit set is a hard error: ```text /Users/ana/.config/surrealctl/credentials.json is mode 644 and holds a refresh token. Fix it with: chmod 600 /Users/ana/.config/surrealctl/credentials.json ``` The file is created with its final mode, so there is no window in which it is readable by anyone else. Windows has no file mode, and the check does not apply there. ### Why a file, and not the OS keyring This reverses the obvious default deliberately. macOS keychain ACLs are bound to the requesting binary's code signature, so a CLI installed by `cargo install` or Homebrew re-prompts *"surrealctl wants to use your confidential information"* after every upgrade - unanswerable inside a CI job. Headless Linux and containers have no Secret Service at all. `gh`, `aws`, `gcloud` and `flyctl` are all file-based for the same reasons. There is **no keyring backend**. If one is ever added, the rule it has to follow is that selecting it and finding it unavailable fails loudly rather than silently downgrading to the file. ### Concurrent invocations Refresh-token rotation makes a race genuinely dangerous: a losing racer presents an already-consumed refresh token, and the identity provider's breach detection may then invalidate the whole token family, signing you out everywhere. So every mutation takes an exclusive advisory lock on `credentials.lock`, re-reads under the lock in case a sibling has already refreshed, performs the network call inside the lock, and writes through a temporary file in the same directory followed by an atomic rename. Readers take no lock at all, because a rename is atomic - a reader sees either the old file or the new one, never a mix. The lock budget is 30 seconds. Past that: ```text another surrealctl is updating credentials (waited 30s for /Users/ana/.config/surrealctl/credentials.lock). Retry in a moment. ``` ### Clock skew Skew is learned from the `Date` header on every response, successful or not - a 401 caused by a bad clock is exactly the case this fixes - and folded into the next write rather than triggering one. `auth status` warns when the observed skew exceeds five minutes. Only an `invalid_grant` from the identity provider ever deletes a stored session. Local time is an input to scheduling, never to invalidation, which is what stops a broken clock from becoming a lockout. ## Renewing and signing out ```bash title="Renew now, rather than waiting for expiry" surrealctl auth refresh ``` `auth refresh` forces both the access token and the Cloud session. `--all` additionally renews the token used for the account-management routes. On a personal access token it reports that a token does not expire on a schedule and cannot be refreshed, then exits `0` without sending anything. ```bash title="Remove the stored credential for this profile" surrealctl auth logout ``` `auth logout` revokes the refresh token at the identity provider on a best-effort basis and removes only this profile's entry. A failed revocation never blocks the local wipe, but does produce a warning: ```text warning: The refresh token could not be revoked at the identity provider. It was removed from this machine, but may still be usable elsewhere. ``` ## What is sent on the wire Two headers carry the credential, and they are attached in one place in the client rather than by individual commands: | Header | Value | | --- | --- | | `Authorization` | `Bearer ` | | `X-Cloud-Token` | The bare Cloud session JWT, with no `Bearer ` prefix - added only for routes that need it | Every request also carries `X-Request-Id` (a client-minted identifier, echoed back and reported in error output), `X-Client-Name`, `X-Client-Version` and a `User-Agent` naming the version and build target. `--debug` logs every request and response to stderr with the credential reduced to a digest - `Authorization: Bearer ` - so a debug transcript can be pasted into an issue. ## Advanced environment variables Three variables exist for support and for non-production tenants. They are absent from `--help` on purpose. | Variable | Purpose | | --- | --- | | `SURREALCTL_CLOUD_TOKEN` | Supply an already-minted Cloud session token, bypassing sign-in. For debugging and support reproduction only: the session lives about an hour, so it is no use as a CI credential. `auth status` reports when it is set. | | `SURREALCTL_AUTH_CLIENT_ID` | Override the OAuth client id, for a non-production tenant | | `SURREALCTL_AUTH_ISSUER` | Override the OAuth issuer, for a non-production tenant | Use a [personal access token](#personal-access-tokens) for automation. It is the only credential designed to be handed to a machine. ## Related pages - [`auth` commands](/docs/reference/cli/surrealctl/commands/auth.md) - every flag on every verb - [`token` commands](/docs/reference/cli/surrealctl/commands/token.md) - creating and revoking tokens - [Global flags](/docs/reference/cli/surrealctl/global-flags.md) - `--token`, `--token-file`, `--profile` - [Overview](/docs/reference/cli/surrealctl/overview.md) - the rest of the reference - [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) - database credentials, which are a separate concern from these --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands # surrealctl commands A map of every surrealctl command group and leaf command, with links to the page documenting each one. `surrealctl` groups its commands by noun. A group on its own prints its own help; the leaf commands below do the work. Every command accepts the [global flags](/docs/reference/cli/surrealctl/global-flags.md), and every command except [`token create`](/docs/reference/cli/surrealctl/commands/token.md#token-create) accepts `--json`. | Group | Leaf commands | | --- | --- | | [`auth`](/docs/reference/cli/surrealctl/commands/auth.md) | `login`, `logout`, `status`, `refresh`, `scopes` | | [`org`](/docs/reference/cli/surrealctl/commands/org.md) | `list`, `get`, `create`, `update`, `archive`, `use`, `roles`, `permissions`, `usage`, `spend`, `plans` | | [`instance`](/docs/reference/cli/surrealctl/commands/instance.md) | `list`, `get`, `create`, `update`, `delete`, `pause`, `resume`, `watch`, `status`, `endpoint`, `token`, `jwks`, `sql`, `import`, `export`, `metrics`, `logs`, `usage`, `estimate`, `capabilities get`, `capabilities set`, `backup list`, `backup create`, `backup policy get`, `backup policy set` | | [`team`](/docs/reference/cli/surrealctl/commands/team.md) | `list`, `get`, `invite`, `update`, `remove` | | [`invite`](/docs/reference/cli/surrealctl/commands/invite.md) | `list`, `create`, `delete` | | [`token`](/docs/reference/cli/surrealctl/commands/token.md) | `list`, `create`, `delete`, `scopes` | | [`catalog`](/docs/reference/cli/surrealctl/commands/catalog.md) | `regions`, `instance-types`, `storage-types`, `instance-versions`, `billing-countries` | | [`spectron`](/docs/reference/cli/surrealctl/commands/spectron.md) | `context`, `key`, `scoped-key`, `access-token`, `principal`, `package`, `scopes`, `verbs`, `providers`, `usage`, `config` | | [`config`](/docs/reference/cli/surrealctl/commands/config.md) | `list`, `get`, `set`, `unset`, `path`, `edit` | | [`context`](/docs/reference/cli/surrealctl/commands/context.md) | `show`, `use`, `list` | | [Other commands](/docs/reference/cli/surrealctl/commands/misc.md) | `whoami`, `api`, `open`, `status`, `version`, `completion` | ## Aliases Plural spellings of the group nouns work, and so do two verb aliases. Nothing else is aliased. | Alias | Means | | --- | --- | | `orgs`, `instances`, `teams`, `invites`, `tokens`, `contexts` | The singular group | | `ls` | `list` | | `rm` | `delete` - and `team remove` | | `doctor` | `status` | ## Reading these pages Each leaf command gets a usage block, a table of its arguments, a table of its own options, at least one runnable example, and a note on how it refuses. Usage blocks are notation, not commands: `` is a value you must supply, `[NAME]` one you may, `[OPTIONS]` any number of flags, and `...` marks something repeatable. Copy from the examples instead. Flags shared across many commands are documented once: - The [global flags](/docs/reference/cli/surrealctl/global-flags.md), accepted everywhere. - The [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags) - `--columns`, `--wide`, `--no-header`, `--sort`, `--reverse` and `--limit` - carried by every list-shaped command. - The [wait flags](/docs/reference/cli/surrealctl/long-running-operations.md#the-wait-flags) - `--wait`, `--no-wait` and `--wait-timeout` - carried by five `instance` commands. For queries, imports, exports and running a server, use the [`surreal` CLI](/docs/reference/cli/surrealdb-cli/overview.md) instead. See the [overview](/docs/reference/cli/surrealctl/overview.md) for where that boundary sits. --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands/auth # auth Reference for surrealctl auth - signing in, signing out, inspecting the stored credential, renewing it, and listing what it is permitted to do. `surrealctl auth` signs in, signs out, and inspects credentials. Four of its five verbs work entirely from the local credential store; only `login`, `refresh` and `auth status --verify` touch the network. surrealctl auth [OPTIONS] | Verb | Purpose | | --- | --- | | [`login`](#auth-login) | Sign in to SurrealDB Cloud | | [`logout`](#auth-logout) | Remove the stored credentials for a profile | | [`status`](#auth-status) | Show which credentials are in use | | [`refresh`](#auth-refresh) | Renew the stored credentials now | | [`scopes`](#auth-scopes) | Show what the current credential is permitted to do | Behaviour when nothing is signed in is deliberate and differs by verb: `status` and `logout` succeed with exit `0`, while `scopes` and `refresh` exit `3`. See [Authentication](/docs/reference/cli/surrealctl/authentication.md#what-each-auth-verb-does-when-nothing-is-signed-in) for why. ## surrealctl auth login {#auth-login} Sign in to SurrealDB Cloud, storing a credential for the active profile. surrealctl auth login [OPTIONS] This command takes no positional arguments. ", "description": "A label to remember this personal access token by. Requires `--with-token`." }, { "name": "--flow", "value": "", "description": "Which sign-in flow to use: `browser`, `device` or `paste`. Conflicts with `--with-token`." }, { "name": "--no-browser", "description": "Do not open a browser; show a code or a URL instead. Conflicts with `--with-token` and `--flow`." }, { "name": "--force", "description": "Sign in again even if this profile already has a credential." } ]} /> | `--flow` value | Behaviour | | --- | --- | | `browser` | Open a browser and listen on `127.0.0.1` for the redirect | | `device` | Show a code to type on another device, following RFC 8628 | | `paste` | Print a URL and take back the address the browser was redirected to | Left to itself the command tries browser, then device, then paste, and stops at the first one that can run. A flow named explicitly is used or it fails; there is no silent substitution. ```bash title="Sign in interactively" surrealctl auth login ``` ```bash title="Sign in over SSH, where a local browser is no use" surrealctl auth login --flow device ``` ```bash title="Store a personal access token for a profile" echo "$CI_TOKEN" | surrealctl --profile ci auth login --with-token --label "build runner" ``` **Refusals and short-circuits.** A profile that already holds a credential is a no-op that exits `0` and names the remedy: ```text Profile `default` is already signed in as ana@acme.example. Replace it with: surrealctl auth login --force ``` An interactive flow that does not complete exits `30`. A session with nobody present and no way to prompt cannot start a flow at all, and says so rather than printing a device code nobody will read. Only `--with-token` stores a credential. `--token`, `--token-file` and `SURREALCTL_TOKEN` are request-scoped and never written to disk. ## surrealctl auth logout {#auth-logout} Remove the stored credentials for a profile. surrealctl auth logout [OPTIONS] This command takes no positional arguments. ```bash title="Sign out of the active profile" surrealctl auth logout ``` ```bash title="Tear down a profile in a script" surrealctl --profile ci auth logout --force --json ``` Logging out revokes the refresh token at the identity provider on a best-effort basis and removes only this profile's entry. A failed revocation warns but never blocks the local wipe. Signing out when nothing is signed in still succeeds and still emits a document, so `auth logout --json` in a teardown script has the same shape on its second run as on its first: ```json title="Output" { "profile": "ci", "removed": false, "revoked": false } ``` Declining the confirmation prints `Nothing was changed.` and exits `0`. ## surrealctl auth status {#auth-status} Show which credentials are in use, without touching the network. surrealctl auth status [OPTIONS] This command takes no positional arguments. ```bash title="What am I signed in as?" surrealctl auth status ``` ```bash title="Confirm the credential still works" surrealctl auth status --verify ``` The document has the same shape signed in and signed out, so a script can read one field without branching first. `status` warns when the machine's clock is more than five minutes out, and when `SURREALCTL_CLOUD_TOKEN` is supplying a session token. For the API's own view of your identity, use [`whoami`](/docs/reference/cli/surrealctl/commands/misc.md#whoami), which spends a request. ## surrealctl auth refresh {#auth-refresh} Renew the stored credentials now, rather than waiting for them to expire. surrealctl auth refresh [OPTIONS] This command takes no positional arguments. ```bash surrealctl auth refresh ``` The command forces both the access token and the Cloud session. `--all` additionally renews the token used for the account-management routes, which is the one the [`token` commands](/docs/reference/cli/surrealctl/commands/token.md) need. **Refusals.** With nothing signed in, this exits `3`. On a personal access token it reports that a token does not expire on a schedule and cannot be refreshed, then exits `0` without sending anything. ## surrealctl auth scopes {#auth-scopes} Show what the current credential is permitted to do. surrealctl auth scopes [OPTIONS] This command takes no positional arguments. > [!NOTE] > This is the one list-shaped command that carries only `--no-header`. It has no `--columns`, `--wide`, `--sort`, `--reverse` or `--limit`. ```bash title="What may this token do?" surrealctl auth scopes ``` ```text title="Output" SCOPE read:cloud write:cloud-instances ``` The single column is `scope`. **Refusals.** With nothing signed in, this exits `3` - a script asking "what may this credential do" and getting an empty list would read it as "nothing is permitted" rather than "nothing is signed in". On a login session the list is empty, and a sentence on stderr points at [`org permissions`](/docs/reference/cli/surrealctl/commands/org.md#org-permissions): identity scopes are not an authorisation model, and your role in the organisation is what decides what you may do. For the catalogue of scopes a token *can* be granted, use [`token scopes`](/docs/reference/cli/surrealctl/commands/token.md#token-scopes). ## Related pages - [Authentication](/docs/reference/cli/surrealctl/authentication.md) - the credential model, login flows and credential storage - [`token` commands](/docs/reference/cli/surrealctl/commands/token.md) - creating and revoking personal access tokens - [`context` commands](/docs/reference/cli/surrealctl/commands/context.md) - switching between profiles - [Overview](/docs/reference/cli/surrealctl/overview.md) - the rest of the reference - [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) - database-level authentication, which these credentials do not cover --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands/catalog # catalog Reference for surrealctl catalog - the platform-wide lists of regions, instance types, storage types, SurrealDB versions and billing countries. `surrealctl catalog` browses the platform catalogues: the regions, instance types, versions and billing countries SurrealDB Cloud offers. These are the slugs [`instance create`](/docs/reference/cli/surrealctl/commands/instance.md#instance-create) and [`instance update`](/docs/reference/cli/surrealctl/commands/instance.md#instance-update) expect. surrealctl catalog [OPTIONS] | Command | Purpose | | --- | --- | | [`regions`](#catalog-regions) | List the regions instances can be deployed in | | [`instance-types`](#catalog-instance-types) | List the instance types available on the platform | | [`storage-types`](#catalog-storage-types) | List the storage instance types available on the platform | | [`instance-versions`](#catalog-instance-versions) | List the SurrealDB versions instances can run | | [`billing-countries`](#catalog-billing-countries) | List the countries billing details can be registered in | These are nouns rather than `list` verbs, because a catalogue has exactly one thing you can do with it. None takes a positional argument, none takes `--org`, and none carries any flag beyond the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags). > [!IMPORTANT] > These lists are **global**. What an organisation may actually deploy is narrower, and [`org plans`](/docs/reference/cli/surrealctl/commands/org.md#org-plans) is the organisation-scoped answer - the one to trust before a create. ## surrealctl catalog regions {#catalog-regions} List the regions instances can be deployed in. surrealctl catalog regions [OPTIONS] This command takes no positional arguments and no options of its own. Column ids are `slug` and `description`, with `flag` under `--wide`. ```bash surrealctl catalog regions ``` ```text title="Output" SLUG DESCRIPTION aws-euw1 Europe (Ireland) aws-use1 US East (N. Virginia) aws-usw2 US West (Oregon) ``` ```bash title="Just the slugs" surrealctl catalog regions --columns slug --no-header ``` The `slug` is what `--region` accepts. ## surrealctl catalog instance-types {#catalog-instance-types} List the instance types available on the platform. surrealctl catalog instance-types [OPTIONS] This command takes no positional arguments and no options of its own. Column ids are `slug`, `category`, `cpu`, `memory`, `units`, `price_hour`, `storage` and `enabled`, with `display_name`, `restricted` and `description` under `--wide`. ```bash surrealctl catalog instance-types ``` ```bash title="Cheapest first" surrealctl catalog instance-types --sort price_hour ``` The `slug` is what `--type` accepts. Prices stay in integer minor units under `--json`. The `enabled` column is left unset here, because whether a type is available to you is a property of your organisation's plan rather than of the platform - check [`org plans`](/docs/reference/cli/surrealctl/commands/org.md#org-plans), or price a specific combination with [`instance estimate`](/docs/reference/cli/surrealctl/commands/instance.md#instance-estimate). ## surrealctl catalog storage-types {#catalog-storage-types} List the storage instance types available on the platform. surrealctl catalog storage-types [OPTIONS] This command takes no positional arguments and no options of its own. The columns are the same as [`instance-types`](#catalog-instance-types). ```bash surrealctl catalog storage-types ``` An empty list is a success, and says why on stderr: ```text No storage instance types are offered. Distributed storage may not be available here. ``` ## surrealctl catalog instance-versions {#catalog-instance-versions} List the SurrealDB versions instances can run. surrealctl catalog instance-versions [OPTIONS] This command takes no positional arguments and no options of its own. The single column id is `version`. ```bash surrealctl catalog instance-versions ``` ```bash title="The version an update would move to" surrealctl catalog instance-versions --limit 1 --no-header ``` The API's own order is newest-first and is left alone. `--sort version` sorts as strings, so `3.10.0` would come before `3.9.0` - prefer the default order when you want the newest release. An empty list should not happen, and says so: ```text The API returned no versions, which should not happen. Try again, or report it. ``` ## surrealctl catalog billing-countries {#catalog-billing-countries} List the countries billing details can be registered in. surrealctl catalog billing-countries [OPTIONS] This command takes no positional arguments and no options of its own. Column ids are `code` and `name`. ```bash surrealctl catalog billing-countries ``` ```bash title="Check one country is supported" surrealctl catalog billing-countries --json | jq -e '.[] | select(.code == "IE")' > /dev/null ``` ## Related pages - [`instance create`](/docs/reference/cli/surrealctl/commands/instance.md#instance-create) - where these slugs are used - [`org plans`](/docs/reference/cli/surrealctl/commands/org.md#org-plans) - the organisation-scoped subset - [`instance estimate`](/docs/reference/cli/surrealctl/commands/instance.md#instance-estimate) - pricing a type and region before committing to it - [Overview](/docs/reference/cli/surrealctl/overview.md) - the rest of the reference - [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) - for working with the data inside an instance --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands/config # config Reference for surrealctl config - listing, reading, setting and clearing configuration values, printing the file path, and opening it in an editor. `surrealctl config` reads and writes the CLI's own configuration file. Nothing here sends a request: the file never leaves the machine. surrealctl config [OPTIONS] | Verb | Purpose | Alias | | --- | --- | --- | | [`list`](#config-list) | List every configuration key and its value | `ls` | | [`get`](#config-get) | Show one configuration value | | | [`set`](#config-set) | Set a configuration value | | | [`unset`](#config-unset) | Clear a configuration value | | | [`path`](#config-path) | Print the path to the configuration file | | | [`edit`](#config-edit) | Open the configuration file in your editor | | Every verb works on **one profile** - whichever `--profile`, `SURREALCTL_PROFILE` and the file's own `active_profile` settle on - and each one says which profile it touched. ## The keys Five keys are recognised. Each is outranked by an environment variable, and by a command-line flag above that; see the [precedence chain](/docs/reference/cli/surrealctl/global-flags.md#the-precedence-chain). | Key | Type | Description | Outranked by | | --- | --- | --- | --- | | `org` | Text | Default organisation for this profile | `SURREALCTL_ORG` | | `api` | URL | Base URL of the SurrealDB API | `SURREALCTL_API` | | `json` | Boolean | Always emit machine-readable JSON | `SURREALCTL_JSON` | | `plain` | Boolean | Always disable tables, spinners and relative times | `SURREALCTL_PLAIN` | | `surreal_binary` | Text | Path to the `surreal` binary for `sql`, `import` and `export` | `SURREALCTL_SURREAL_BINARY` | The persisted context - what [`org use`](/docs/reference/cli/surrealctl/commands/org.md#org-use) records - is **not** reachable from here. There is no `config set context.org`. The two live in separate tables of the same file so that `org use` never overwrites an `org` value you set by hand, and the hand-written one wins. ```toml title="config.toml" active_profile = "work" [profile.work] org = "acme" # written by a person [profile.work.context] org = "67upif0m8sh1cn1p2c8t" # written by `org use` ``` Keys this build has never heard of are preserved on write, so an older binary editing a newer file does not discard anything. ## surrealctl config list {#config-list} List every configuration key and its value. surrealctl config list [OPTIONS] This command takes no positional arguments and no options of its own. Accepts the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags). Column ids are `key`, `value`, `set` and `description`, with `kind` under `--wide`. ```bash surrealctl config list ``` ```text title="Output" KEY VALUE SET DESCRIPTION org acme yes Default organization for this profile api no Base URL of the SurrealDB API json no Always emit machine-readable JSON plain no Always disable tables, spinners and relative times surreal_binary no Path to the `surreal` binary for sql/import/export ``` Every key is listed whether or not it is set, and the `set` column is what distinguishes a configured value from a default. Only a key this profile has actually set carries a value, so an unset key - `api` above, which falls back to the built-in default - has an empty `value` cell. A note on stderr reports the profile and the file path. ## surrealctl config get {#config-get} Show one configuration value. surrealctl config get [OPTIONS] ", "required": true, "description": "The configuration key." } ]} /> This command has no options of its own. The output is the value alone plus a newline, with no label and no styling, so `$(…)` captures exactly the value. An unset key writes **zero bytes**, so `[ -z "$(surrealctl config get org)" ]` and `wc -l` agree with each other. ```bash ORG=$(surrealctl config get org) ``` ```bash title="Branch on whether a key is set" if [ -z "$(surrealctl config get surreal_binary)" ]; then echo "using surreal from PATH" fi ``` **Refusals.** An unrecognised key is a usage error, exit `2`, and lists the keys that exist. ## surrealctl config set {#config-set} Set a configuration value. surrealctl config set [OPTIONS] ", "required": true, "description": "The configuration key." }, { "name": "", "required": true, "description": "The value to store." } ]} /> This command has no options of its own. ```bash title="Always emit JSON from this profile" surrealctl config set json true ``` ```bash title="Point at a specific surreal build" surrealctl config set surreal_binary /opt/surrealdb/3.2.4/surreal ``` ```bash title="Configure a second profile" surrealctl --profile staging config set org contoso ``` **Warnings.** When the corresponding environment variable is exported, the command says so, because the variable silently outranks what you just wrote: ```text warning: SURREALCTL_JSON is set and takes precedence over this. Unset it for the configured value to apply. ``` **Refusals**, exit `2`: an unrecognised key, or a value the key's type rejects - a non-boolean for `json`, or a URL with a path for `api`. ## surrealctl config unset {#config-unset} Clear a configuration value. surrealctl config unset [OPTIONS] ", "required": true, "description": "The configuration key." } ]} /> This command has no options of its own. ```bash surrealctl config unset json ``` Clearing a key restores the CLI's default rather than falling back to another profile. Clearing an already-unset key is a no-op that succeeds. ## surrealctl config path {#config-path} Print the path to the configuration file. surrealctl config path [OPTIONS] This command takes no positional arguments and no options of its own. The output is the path alone plus a newline. ```bash surrealctl config path ``` ```text title="Output" /Users/ana/.config/surrealctl/config.toml ``` ```bash title="Back the file up before editing it" cp "$(surrealctl config path)" "$(surrealctl config path).bak" ``` `credentials.json` lives beside it in the same directory. See [credential storage](/docs/reference/cli/surrealctl/authentication.md#where-credentials-are-stored) for how that directory is chosen. ## surrealctl config edit {#config-edit} Open the configuration file in your editor. surrealctl config edit [OPTIONS] This command takes no positional arguments and no options of its own. ```bash surrealctl config edit ``` The file is opened with `$EDITOR`. Editing it by hand is supported - [`config list`](#config-list) will show whatever you wrote, and unknown keys survive - but a file written by a newer `surrealctl` is refused rather than migrated: ```text was written by a newer surrealctl (file version 2, this build understands 1). Refusing to touch it - upgrade surrealctl, or point --config elsewhere. ``` ## Related pages - [Global flags](/docs/reference/cli/surrealctl/global-flags.md) - the flags and environment variables these keys sit beneath - [`context` commands](/docs/reference/cli/surrealctl/commands/context.md) - switching between profiles - [`org use`](/docs/reference/cli/surrealctl/commands/org.md#org-use) - remembering an organisation within a profile - [Authentication](/docs/reference/cli/surrealctl/authentication.md#where-credentials-are-stored) - the credential file beside this one - [Overview](/docs/reference/cli/surrealctl/overview.md) - the rest of the reference - [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) - for working with the data inside an instance --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands/context # context Reference for surrealctl context - showing what an invocation resolved, switching between profiles, and listing the profiles that are configured. `surrealctl context` inspects and switches profiles. A profile bundles a credential, an API base URL and a set of configuration values, so switching profile changes all three in one move. surrealctl context [OPTIONS] surrealctl contexts [OPTIONS] | Verb | Purpose | Alias | | --- | --- | --- | | [`show`](#context-show) | Show the context this invocation resolved | | | [`use`](#context-use) | Switch to a different profile | | | [`list`](#context-list) | List the configured profiles | `ls` | `context use` switches *profile*. [`org use`](/docs/reference/cli/surrealctl/commands/org.md#org-use) remembers an organisation *within* a profile. Neither is expressible as the other, and you will usually want both: one profile per account or tenant, one remembered organisation inside each. ## surrealctl context show {#context-show} Show the context this invocation resolved. surrealctl context show [OPTIONS] This command takes no positional arguments and no options of its own. ```bash surrealctl context show ``` ```text title="Output" Profile work Organization acme Id 67upif0m8sh1cn1p2c8t Source persisted context (`org use`) API https://api.surrealdb.com Config /Users/ana/.config/surrealctl/config.toml Store /Users/ana/.config/surrealctl/credentials.json ``` The command resolves through the **same chain every other command uses**, including the interactive picker when several organisations are available and nothing has chosen - so what it reports is genuinely what the next command will do, not an approximation of it. A failure to resolve an organisation is reported as *part of the answer* rather than as an error: the profile, the API base and both file paths are still printed, followed by a note and the hint `surrealctl org use `. ```bash title="Which organisation would this command hit?" surrealctl context show --json | jq -r '.organization.name + " via " + .source' ``` ## surrealctl context use {#context-use} Switch to a different profile. surrealctl context use [OPTIONS] ", "required": true, "description": "The profile to make active." } ]} /> This command has no options of its own. ```bash title="Switch" surrealctl context use staging ``` ```bash title="Create a profile and sign into it" surrealctl context use staging surrealctl auth login ``` The profile is created if it does not exist. A **new** profile gets a note that it has no stored credentials, plus the hint naming the login command for it. **Refusals.** An empty or whitespace-only name is a usage error, exit `2`: ```text Ensure the profile name is not empty. ``` **Warnings.** The command warns when `SURREALCTL_PROFILE` is set, because the variable outranks the file it just wrote. For a single command against a different profile, pass `--profile` rather than switching: ```bash surrealctl --profile staging instance list ``` ## surrealctl context list {#context-list} List the configured profiles. surrealctl context list [OPTIONS] This command takes no positional arguments and no options of its own. Accepts the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags). Column ids are `profile`, `active`, `org` and `api`, with `source` under `--wide`. ```bash surrealctl context list ``` ```text title="Output" PROFILE ACTIVE ORGANIZATION API default no contoso https://api.surrealdb.com work yes acme https://api.surrealdb.com ``` ```bash title="See where each organisation came from" surrealctl context list --wide ``` The profile in force is always listed, even when the file has no entry for it - so `--profile scratch context list` shows `scratch`. The order is stable between runs. ## Related pages - [Global flags](/docs/reference/cli/surrealctl/global-flags.md#the-precedence-chain) - the chain `context show` reports on - [`config` commands](/docs/reference/cli/surrealctl/commands/config.md) - the values a profile holds - [`org use`](/docs/reference/cli/surrealctl/commands/org.md#org-use) - remembering an organisation within a profile - [`auth` commands](/docs/reference/cli/surrealctl/commands/auth.md) - signing a profile in - [Overview](/docs/reference/cli/surrealctl/overview.md) - the rest of the reference - [SurrealDB CLI](/docs/reference/cli/surrealdb-cli/overview.md) - for working with the data inside an instance --- Source: https://surrealdb.com/docs/reference/cli/surrealctl/commands/instance # instance Reference for surrealctl instance - creating, scaling, pausing and deleting instances, reading logs, metrics and endpoints, minting tokens, and managing capabilities and backups. `surrealctl instance` manages instances: the databases themselves. It is the largest group in the CLI, and covers provisioning, scaling, observability, credentials, capabilities and backups. surrealctl instance [OPTIONS] surrealctl instances [OPTIONS] | Verb | Purpose | Alias | | --- | --- | --- | | [`list`](#instance-list) | List the instances in an organisation | `ls` | | [`get`](#instance-get) | Show one instance | | | [`create`](#instance-create) | Create an instance | | | [`update`](#instance-update) | Change an instance's type, size, or version | | | [`delete`](#instance-delete) | Delete an instance and everything in it | `rm` | | [`pause`](#instance-pause) | Pause a running instance | | | [`resume`](#instance-resume) | Resume a paused instance | | | [`watch`](#instance-watch) | Watch an operation that was started elsewhere | | | [`status`](#instance-status) | Show an instance's deployment phase and restorable snapshots | | | [`endpoint`](#instance-endpoint) | Print the endpoint a client connects to | | | [`token`](#instance-token) | Mint a database token for an instance | | | [`jwks`](#instance-jwks) | Fetch the key set that verifies an instance's tokens | | | [`sql`](#instance-sql) | Open a SurrealQL session against an instance | | | [`import`](#instance-import) | Import a file into an instance | | | [`export`](#instance-export) | Export an instance to a file | | | [`metrics`](#instance-metrics) | Show an instance's resource metrics | | | [`logs`](#instance-logs) | Show an instance's logs | | | [`usage`](#instance-usage) | Show what an instance has consumed | | | [`estimate`](#instance-estimate) | Estimate what an instance would cost | | | [`capabilities get`](#instance-capabilities-get) | Show an instance's capability configuration | | | [`capabilities set`](#instance-capabilities-set) | Change an instance's capability configuration | | | [`backup list`](#instance-backup-list) | List the snapshots an instance can be restored from | `ls` | | [`backup create`](#instance-backup-create) | Take a backup of an instance now | | | [`backup policy get`](#instance-backup-policy-get) | Show the backup retention policy | | | [`backup policy set`](#instance-backup-policy-set) | Change the backup retention policy | | ## Naming an instance Four spellings are accepted: the id, the slug, the name, or `org/name`. Only a bare id avoids a lookup against the organisation. A slug always contains a hyphen, so it can never be mistaken for an id. ```bash surrealctl instance get production surrealctl instance get production-6xk2 surrealctl instance get acme/production surrealctl instance get 67upif0m8sh1cn1p2c8t ``` Omit the reference entirely and, on a terminal, you get a picker. Elsewhere it is a usage error, exit `2`, listing what was available: ```text No instance was given and this session cannot prompt. Name one, or pass --org to change which organization is searched. Available: production staging ``` There is deliberately no `instance use`. A sticky implicit target is fine for a noun whose verbs are mostly reads, and wrong for one whose verbs include `delete`. ## surrealctl instance list {#instance-list} List the instances in an organisation. surrealctl instance list [OPTIONS] This command takes no positional argument, deliberately: a positional here would read as an *instance*. The organisation comes from `--org` and the [precedence chain](/docs/reference/cli/surrealctl/global-flags.md#the-precedence-chain). Accepts the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags) and nothing else. Column ids are `name`, `state`, `type`, `version`, `region`, `compute_units` and `storage`, with `id`, `slug`, `host`, `access_type` and `organization_id` under `--wide`. ```bash surrealctl instance list ``` ```text title="Output" NAME STATE TYPE VERSION REGION UNITS STORAGE production ready production-2 3.2.4 aws-euw1 4 100 GB staging paused shared-1 3.2.4 aws-euw1 1 10 GB ``` ```bash title="Ready instances only, as names" surrealctl instance list --json | jq -r '.[] | select(.state == "ready") | .name' ``` ## surrealctl instance get {#instance-get} Show one instance. surrealctl instance get [OPTIONS] [INSTANCE] This command has no options of its own. ```bash surrealctl instance get production ``` ```bash title="One field, for a script" surrealctl instance get production --json | jq -r .state ``` ## surrealctl instance create {#instance-create} Create an instance. surrealctl instance create [OPTIONS] --type --region ", "required": true, "description": "A name for the instance. Between 1 and 30 characters." } ]} /> ", "required": true, "description": "The instance type, by slug. See `surrealctl catalog instance-types`." }, { "name": "--region", "value": "", "required": true, "description": "The region to deploy into, by slug. See `surrealctl catalog regions`." }, { "name": "--version", "value": "", "description": "The SurrealDB version to run. Defaults to the platform's current one." }, { "name": "--compute-units", "value": "", "description": "How many compute units to run, for the types that scale. Minimum 1." }, { "name": "--storage", "value": "", "description": "Storage to allocate, in gigabytes. Minimum 1." }, { "name": "--access-type", "value": "", "description": "How the instance may be reached: `public`, `private` or `dual`." }, { "name": "--restore-from", "value": "", "description": "Restore from a snapshot, as `/`." }, { "name": "--wait", "description": "Wait for the instance to become ready. On by default." }, { "name": "--no-wait", "description": "Return as soon as the API accepts the request." }, { "name": "--wait-timeout", "value": "", "default": "15m", "description": "How long to wait before giving up." } ]} /> | `--access-type` value | Meaning | | --- | --- | | `public` | Reachable over the public internet. The API's own default | | `private` | Reachable over PrivateLink only. Needs PrivateLink on the organisation | | `dual` | Reachable both ways | ```bash title="Create and wait for it to be ready" surrealctl instance create api --type shared-1 --region aws-euw1 ``` ```bash title="A production instance with room to grow" surrealctl instance create production \ --type production-2 \ --region aws-euw1 \ --compute-units 4 \ --storage 100 ``` ```bash title="Clone from a snapshot" surrealctl instance create staging \ --type shared-1 \ --region aws-euw1 \ --restore-from production/rsnapshot-20260811-150405 ``` With the wait, the command reports progress and then emits a fresh detail view. With `--no-wait` it emits the accepted request and hints at `surrealctl instance watch `. **Refusals**, all exit `2` and all before any request: - A name outside 1 to 30 characters, with the length it counted. - `--compute-units` below 1, or `--storage` below 1. - A malformed `--restore-from`, which is split on the last `/`: ```text `prod-snap-1` is not a snapshot reference. Use /, for example production/rsnapshot-20250128-150405. Run `surrealctl instance status ` to list the snapshots an instance can be restored from. ``` > [!WARNING] > The create route accepts no idempotency key, so a retried create is a second instance and a second bill. `surrealctl` never retries it after a `502` for exactly that reason. If a create times out, run [`instance list`](#instance-list) before running it again. ## surrealctl instance update {#instance-update} Change an instance's type, size, or version. surrealctl instance update [OPTIONS] [INSTANCE] ", "description": "Move to this instance type, by slug." }, { "name": "--compute-units", "value": "", "description": "Scale to this many compute units. Minimum 1." }, { "name": "--storage", "value": "", "description": "Grow storage to this many gigabytes. Minimum 1." }, { "name": "--version", "value": "", "description": "Upgrade to this SurrealDB version." }, { "name": "--access-type", "value": "", "description": "Change how the instance may be reached: `public`, `private` or `dual`." }, { "name": "--wait", "description": "Wait for the instance to become ready again. On by default." }, { "name": "--no-wait", "description": "Return as soon as the API accepts the last change." }, { "name": "--wait-timeout", "value": "", "default": "15m", "description": "How long to wait before giving up." } ]} /> There is no single update route on the API - each field has its own - so one invocation is several requests. They are applied in a fixed order and stop at the first failure: 1. `--type` 2. `--compute-units` 3. `--storage` 4. `--version` 5. `--access-type` > [!IMPORTANT] > Every one of these restarts the instance. A single `instance update` that changes three fields restarts it once per field, in the order above. ```bash title="Scale up" surrealctl instance update production --compute-units 8 ``` ```bash title="Upgrade the SurrealDB version" surrealctl instance update production --version 3.2.4 ``` ```bash title="Change type and grow storage in one go" surrealctl instance update production --type production-4 --storage 250 ``` **Refusals**, all exit `2`: - No flags at all: ```text Nothing to update. Pass at least one of --type, --compute-units, --storage, --version, or --access-type. ``` - `--compute-units` or `--storage` below 1. - A storage cool-off still in force, naming the interval and, when known, when storage was last changed. - `--compute-units` outside the current type's range, when `--type` is not also being changed: ```text `shared-1` accepts 1-2 compute units; 8 is outside that. ``` ## surrealctl instance delete {#instance-delete} Delete an instance and everything in it. surrealctl instance delete [OPTIONS] [INSTANCE] ", "default": "15m", "description": "How long to wait before giving up." } ]} /> ```bash title="Delete, with a confirmation" surrealctl instance delete staging ``` ```bash title="Delete from a teardown script" surrealctl instance delete staging --force --json ``` The confirmation names the instance, its slug and its region, so the wrong terminal tab is caught before the request. Declining prints `Nothing was deleted.` and exits `0` with no document. In a non-interactive session without `--force` or `--yes`, it exits `2` having sent nothing. The delete route answers with no body, so the emitted document is synthesised: ```json title="Output" { "id": "67upif0m8sh1cn1p2c8t", "name": "staging", "slug": "staging-4jd1", "state": "deleted" } ``` While waiting, a `404` is the success condition - there is nothing left to fetch. ## surrealctl instance pause {#instance-pause} Pause a running instance. surrealctl instance pause [OPTIONS] [INSTANCE] ", "default": "15m", "description": "How long to wait before giving up." } ]} /> ```bash surrealctl instance pause staging ``` Pausing something already paused prints a note and still emits a document, exit `0`. The shape of the answer does not change with remote state, so a script does not need to know which case it hit. ## surrealctl instance resume {#instance-resume} Resume a paused instance. surrealctl instance resume [OPTIONS] [INSTANCE] ", "default": "15m", "description": "How long to wait before giving up." } ]} /> ```bash surrealctl instance resume staging ``` Resuming something already ready behaves like pausing something already paused: a note, a document, exit `0`. ## surrealctl instance watch {#instance-watch} Watch an operation that was started elsewhere. surrealctl instance watch [OPTIONS] [INSTANCE] ", "default": "ready", "description": "The state to wait for: `ready`, `paused` or `deleted`." }, { "name": "--wait-timeout", "value": "", "default": "15m", "description": "How long to watch before giving up." } ]} /> There is no `--wait` or `--no-wait` here, because a watch is nothing but a wait. | `--until` value | Finishes when | | --- | --- | | `ready` | The instance is running and healthy | | `paused` | The instance is deliberately stopped | | `deleted` | The instance is gone - a `404` while polling is what finishes this one | ```bash title="Pick up a create started in an earlier CI job" surrealctl instance watch api ``` ```bash title="Wait for a pause to complete" surrealctl instance watch staging --until paused ``` > [!NOTE] > This is the one command where `--json` writes to stdout continuously. The transitions *are* the answer, so the newline-delimited stream goes to stdout and nothing else is emitted. In text mode the run ends with a fresh detail view, except under `--until deleted`, where there is nothing left to fetch. ```bash title="Follow the transitions as they happen" surrealctl instance watch api --json | jq -r '.event + " " + (.state // "")' ``` ## surrealctl instance status {#instance-status} Show an instance's deployment phase and restorable snapshots. surrealctl instance status [OPTIONS] [INSTANCE] Also accepts the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags), which arrange the snapshot table. Column ids are `snapshot_id`, `started_at`, `tiers` and `on_demand`, with `valid_versions` under `--wide`. ```bash surrealctl instance status production ``` The answer is one composite document: a phase, then the snapshots the instance can be restored from. In plain mode the phase is written as `phase` before the table, so `grep ^phase` works the same way it does on [`instance get`](#instance-get). An instance with no restorable snapshots gets a note on stderr rather than an empty table with no explanation. Snapshot ids from here are what [`instance create --restore-from`](#instance-create) expects. ## surrealctl instance endpoint {#instance-endpoint} Print the endpoint a client connects to. surrealctl instance endpoint [OPTIONS] [INSTANCE] This command has no options of its own. The output is the bare endpoint plus a newline, byte-identical in rich and plain output, so `$(…)` captures exactly the value with no label and no styling. ```bash ENDPOINT=$(surrealctl instance endpoint production) surreal sql --endpoint "$ENDPOINT" --namespace app --database main ``` ```bash title="Under --json, the hosts come too" surrealctl instance endpoint production --json ``` ```json title="Output" { "endpoint": "wss://production-6xk2.aws-euw1.surreal.cloud", "host": "production-6xk2.aws-euw1.surreal.cloud", "private_host": null, "access_type": "public" } ``` A private-only instance produces a warning on stderr rather than a refusal - the endpoint is still correct, it is simply not reachable from where you are. ## surrealctl instance token {#instance-token} Mint a database token for an instance. surrealctl instance token [OPTIONS] [INSTANCE] This is a *database* token - the credential a SurrealDB client authenticates with. It is not a [personal access token](/docs/reference/cli/surrealctl/commands/token.md), which authenticates against the control plane. ```bash title="Copy it to the clipboard" surrealctl instance token production | pbcopy ``` ```bash title="Use it with the surreal CLI" SURREAL_TOKEN=$(surrealctl instance token production) \ surreal sql --endpoint "$(surrealctl instance endpoint production)" --namespace app --database main ``` **Refusals.** Printing a credential into a terminal's scrollback is refused, exit `2`, and the check happens *before* the token is minted - so a forgotten `--reveal` never costs a credential nobody can recover: ```text Refusing to print a database token to a terminal, where it would stay in your scrollback. Pipe it: surrealctl instance token production | pbcopy Or pass --reveal if you meant to see it. ``` Under `--json` the answer is a one-field document carrying the instance id and the token. ## surrealctl instance jwks {#instance-jwks} Fetch the key set that verifies an instance's tokens. surrealctl instance jwks [OPTIONS] [INSTANCE] Also accepts the [list presentation flags](/docs/reference/cli/surrealctl/output-and-exit-codes.md#list-presentation-flags). Column ids are the JWK member names - `kid`, `kty`, `alg` and `use`, with `crv` under `--wide` - so a member the key type does not carry renders as missing rather than empty. ```bash title="Inspect the keys" surrealctl instance jwks production ``` ```bash title="Pipe the whole key set into a verifier" surrealctl instance jwks production --json > jwks.json ``` Under `--json` the whole key set is emitted, `keys` wrapper included, so it can be handed straight to a JWT library. ## surrealctl instance sql {#instance-sql} Open a SurrealQL session against an instance. surrealctl instance sql [OPTIONS] [INSTANCE] [-- ...] ", "description": "The namespace to open." }, { "name": "--database", "value": "", "description": "The database to open." } ]} /> This command does not speak SurrealQL. It resolves the instance, mints a database token, and becomes [`surreal sql`](/docs/reference/cli/surrealdb-cli/commands/sql.md). Everything after `--` is forwarded to that command verbatim, because the sibling's flag surface is large and moves independently. ```bash title="Open a shell" surrealctl instance sql production --namespace app --database main ``` ```bash title="Pass flags through to surreal sql" surrealctl instance sql production -- --ns app --db main --pretty ``` ```bash title="Run a query from a pipeline" echo "SELECT count() FROM person GROUP ALL;" \ | surrealctl instance sql production --namespace app --database main ``` `--namespace` and `--database` are optional here, because `surreal sql` accepts a session without them. See [the `surreal` handoff](/docs/manage/surrealctl/install.md#the-surreal-handoff) for how the binary is located and what is passed in the child's environment. **Warnings**, neither of which stops the handoff: a private-only instance, and an instance that is not in a ready phase. ## surrealctl instance import {#instance-import} Import a file into an instance. surrealctl instance import [OPTIONS] [INSTANCE] [-- ...] ", "required": true, "description": "The namespace to import into." }, { "name": "--database", "value": "", "required": true, "description": "The database to import into." } ]} /> ```bash surrealctl instance import production --namespace app --database main -- ./seed.surql ``` **Refusals.** `surreal import` requires both a namespace and a database, so this command does too. The check fires before a token is minted: ```text `surrealctl instance import` needs both --namespace and --database; `surreal import` requires them. ``` See [`surreal import`](/docs/reference/cli/surrealdb-cli/commands/import.md) for the flags available after the `--`. ## surrealctl instance export {#instance-export} Export an instance to a file. surrealctl instance export [OPTIONS] [INSTANCE] [-- ...] ", "required": true, "description": "The namespace to export from." }, { "name": "--database", "value": "", "required": true, "description": "The database to export from." } ]} /> ```bash title="Export to a file" surrealctl instance export production --namespace app --database main -- ./backup.surql ``` ```bash title="Export to stdout and compress" surrealctl instance export production --namespace app --database main -- - | gzip > backup.surql.gz ``` Like `import`, this refuses a missing namespace or database before minting anything. See [`surreal export`](/docs/reference/cli/surrealdb-cli/commands/export.md) for the flags available after the `--`. For a managed snapshot rather than a SurrealQL dump, use [`instance backup create`](#instance-backup-create). ## surrealctl instance metrics {#instance-metrics} Show an instance's resource metrics. surrealctl instance metrics [OPTIONS] --metric [INSTANCE] ", "required": true, "description": "Which metric to read, such as `cpu`, `memory` or `disk`." }, { "name": "--from", "value": "