Embedding SurrealDB

SurrealDB is designed to be run in many different ways and environments. Due to the separation of the storage and compute layers, SurrealDB can be run in embedded mode from within a number of different language environments. In Python, SurrealDB can be run as an in-memory database, or it can persist data using a file-based storage engine.

In-Memory Database

The in-memory database is perfect for embedded applications, development, testing, caching, or temporary data. It provides the fastest performance as all data is stored in RAM, but data is lost when the connection closes.

Installation

The embedded database functionality is included in the standard surrealdb package:

pip install surrealdb

Note

This SDK works seamlessly with SurrealDB versions v2.0.0 to v2.3.6, ensuring compatibility with the latest features. The embedded database functionality is included in pre-built wheels on PyPI.

Pre-built wheels are available for:

• Linux (x86_64, aarch64)
• macOS (x86_64, ARM64)
• Windows (x64)

Note

For Python 3.9, 3.10, 3.11, 3.12, and 3.13. For source builds, you’ll need a Rust toolchain and maturin.

Using the in-memory database

The simplest way to use an in-memory database instance of SurrealDB is to create an instance with the mem:// or memory endpoint.

Async example:

import asyncio
from surrealdb import AsyncSurreal

async def main():
    """Run basic async embedded database operations."""
    # Create an in-memory database (can use 'mem://' or 'memory')
    async with AsyncSurreal("mem://") as db:
        # Use a namespace and database
        await db.use("test", "test")
        
        # Note: Embedded databases don't require authentication
        
        # Create a person
        person = await db.create(
            "person",
            {"name": "John Doe", "age": 30, "email": "john@example.com"}
        )
        print(f"Created person: {person}")
        
        # Query all people
        people = await db.select("person")
        print(f"All people: {people}")
        
        # Update the person
        updated = await db.update(
            "person",
            {"name": "John Doe", "age": 31, "email": "john@example.com"}
        )
        print(f"Updated person: {updated}")
        
        # Run a SurrealQL query
        result = await db.query(
            """
            SELECT * FROM person WHERE age > $min_age
            """,
            {"min_age": 25}
        )
        print(f"Query result: {result}")
        
        # Delete all people
        await db.delete("person")
        print("Deleted all people")

if __name__ == "__main__":
    asyncio.run(main())

Sync example:

The embedded database also supports the blocking API:

from surrealdb import Surreal

# In-memory (can use 'mem://' or 'memory')
with Surreal("memory") as db:
    db.use("test", "test")
    
    person = db.create("person", {"name": "Jane"})
    print(person)

File-Based Database

The file-based database is a more advanced storage engine that can be used to persist data to disk. It uses the SurrealKV storage engine for good performance while maintaining persistence across connections.

Installation

The same surrealdb package includes file-based database support:

pip install surrealdb

Using the file-based database

The file-based database can be created using the file:// (rocksdb:// ) or surrealkv:// endpoint with a path to the database file.

Async example:

import asyncio
from surrealdb import AsyncSurreal

async def main():
    """Demonstrates file-based persistent storage."""
    # Create a file-based database (can use "file://mydb" or "surrealkv://mydb")
    async with AsyncSurreal("file://mydb") as db:
        await db.use("test", "test")
        
        # Data persists across connections
        company = await db.create("company", {"name": "TechStart"})
        print(company)
    
    # Reconnect to the same database
    async with AsyncSurreal("file://mydb") as db:
        await db.use("test", "test")
        
        # Data persists!
        companies = await db.select("company")
        print(companies)

if __name__ == "__main__":
    asyncio.run(main())

Sync example:

from surrealdb import Surreal

# File-based database
with Surreal("file://mydb") as db:
    db.use("test", "test")
    
    company = db.create("company", {"name": "TechStart"})
    print(company)

Performance Considerations

In-Memory (memory or mem://)

• Fastest: All data in RAM
• Non-persistent: Data lost when connection closes
• Best for: Tests, caches, temporary data

File-Based (file:// (rocksdb:// ) or surrealkv://)

• Persistent: Data saved to disk
• Good performance: SurrealKV storage engine
• Best for: Local apps, development, single-node deployments

When to Use Embedded vs Remote

Use Embedded (memory, mem://, file:// (rocksdb:// ), or surrealkv://) when:

• Building desktop applications with local data storage
• Running tests (in-memory is very fast)
• Local development without server setup
• Embedded systems or edge computing devices without network access
• Single-application data storage

Use Remote (ws:// or http://):

• Multiple applications need to share data
• Building distributed systems or microservices
• Requiring scalability across multiple instances
• Cloud deployments with scalable cloud infrastructure
• Need for horizontal scaling or centralized data management

Examples

For more examples, see the examples/embedded/ directory in the repository:

The repository includes three example scripts to help you get started:

• basic_async.py - Demonstrates async operations with an in-memory database using mem:// (or memory), including CRUD operations, SurrealQL queries, and context managers for automatic cleanup. Run with: python examples/embedded/basic_async.py

• basic_sync.py - Shows the same operations using the synchronous Surreal API without await, making it perfect for scripts and simple applications. Run with: python examples/embedded/basic_sync.py

• persistence.py - Demonstrates file-based persistent storage using file:// (rocksdb://) or surrealkv:// URLs, showing how data persists across multiple connections and can be updated over time. Run with: python examples/embedded/persistence.py

Connection URLs

In-Memory Database

# Non-persistent, fastest performance  
db = AsyncSurreal("mem://")
# Or
db = AsyncSurreal("memory")

File-Based Database

# Persistent storage using file path
db = AsyncSurreal("file://path/to/database")

# Or using surrealkv:// protocol
db = AsyncSurreal("surrealkv://path/to/database")

Remote Database (for comparison)

# WebSocket connection to remote server
db = AsyncSurreal("ws://localhost:8000")

# HTTP connection to remote server
db = AsyncSurreal("http://localhost:8000")

Key Differences from Remote Connections

Authentication Behavior

Embedded databases don’t require authentication, though you can still call signin() if needed:

# Embedded - auth optional (can be skipped)
async with AsyncSurreal("mem://") as db:
    await db.use("test", "test")
    # Ready to use!

# Or with signin (works but not required)
async with AsyncSurreal("file://mydb") as db:
    await db.signin({"username": "root", "password": "root"})
    await db.use("test", "test")

# Remote - requires authentication
async with AsyncSurreal("ws://localhost:8000") as db:
    await db.signin({"user": "root", "pass": "root"})
    await db.use("test", "test")

Automatic Connection Management

With embedded databases, the connection is established immediately when creating the client. Use context managers for automatic cleanup:

# Good practice - automatic cleanup
async with AsyncSurreal("mem://") as db:
    await db.use("test", "test")
    # Do work here
    
# Connection automatically closed when exiting context

Single Process Access

Embedded databases are accessed by a single process. For multi-process or distributed access, use a remote SurrealDB server.

Best Practices

Use Context Managers

Always use context managers (with or async with) to ensure proper cleanup:

async with AsyncSurreal("mem://") as db:
    # Your code here
    pass
# Connection automatically closed

Choose the Right Storage Type

• Tests: Use mem:// for speed and isolation
• Development: Use file:// (rocksdb:// ) for persistence
• Production (single-node): Use file:// (rocksdb:// ) with backups
• Production (distributed): Use remote SurrealDB server

Testing with Temporary Databases

Use temporary directories for isolated test databases:

import tempfile
import asyncio
from surrealdb import AsyncSurreal

async def test_example():
    with tempfile.TemporaryDirectory() as tmpdir:
        async with AsyncSurreal(f"file://{tmpdir}/test.db") as db:
            await db.use("test", "test")
            # Run tests
            # Database automatically deleted when tmpdir is cleaned up

Troubleshooting

Import Errors

If you encounter import errors, ensure you have the correct version installed:

pip install --upgrade surrealdb

Connection Issues

For file-based databases, ensure the directory exists and you have write permissions:

import os
db_path = "mydata/database"
os.makedirs("mydata", exist_ok=True)
db = AsyncSurreal(f"file://{db_path}")

Performance Issues

If experiencing slow performance with file-based storage:

• Consider using in-memory mode for non-persistent data
• Ensure disk I/O is not a bottleneck
• Use appropriate indexes for your queries

Additional Resources

• Python SDK GitHub Repository
• SurrealQL Query Language
• Python SDK Documentation