Nov 21st, 2024
We are thrilled to announce the release of SurrealDB 2.1.0! This update includes major enhancements to graph querying and a prominent new role for our custom-built storage backend SurrealKV. Explore our release page for a detailed overview of all the updates, including enhancements in security, performance and stability, as well as a comprehensive list of bug fixes.
SurrealDB 2.1.0 marks an important release, expanding our suite of graph querying capabilities, now introducing the ability to recursively walk graph relations or record links with an easy to use syntax. With the ability to retrieve a flat result or tree-like data structures, allowing you to collect details at each level of depth, this feature unlocks numerous new use-cases previously not possible.
-- Select all people who connect to a person, 3 levels deep SELECT *, @{3}->knows->person AS connections_3rds FROM person; -- Select page:main with its id, name and nested pages, at least 3 and at most 5 pages deep -- Here, the recursive pattern will repeat at the `@` symbol, creating an arbitrary depth tree-like data structure page:main.{3..5}.{ id, name, contains: <-contained_in<-page.@ }; -- Recursive idioms work with any idiom parts, not limited to graphs -- Here, we recursively fetch friends and then collect their names person:tobie.{1..5}(.friend).name
In 2.1.0
we have added a cross-transaction caching layer to the document processing layer, ensuring that all DEFINE FIELD
, DEFINE INDEX
, DEFINE EVENT
, DEFINE TABLE ... AS
, and LIVE SELECT
statement definitions are loaded initially, and then cached. Subsequent requests or transactions can then use the cached statement definitions, removing the need to perform 5 additional range scans.
The caching works by specifying a cache version (an uuid value) for each definition type on the DEFINE TABLE
statement definition. As this table definition is always fetched for each table (for which a document is being processed) within each transaction, we can always detect when the cached statement definitions are in need of being refreshed. When we detect an invalid cache entry, we refetch the relevant definitions from the storage engine. Importantly, each statement definition type (fields, events, indexes, foreign tables, live queries) are cached separately, so modifying any one of those statement types will not invalidate the other types. This change greatly reduces the number of requests on the key-value store, and allows for cross-transaction caching in embedded, single-node, and distributed environments.
The only downside to this addition is that any concurrent additions or modifications of fields, events, indexes, foreign tables, or live queries, will now need to update the table definition, leading to potential concurrent editing conflicts if executed at exactly the same time. This is probably not an issue for DEFINE
statements, but might be more visible when there are many concurrent LIVE SELECT
or KILL
statements on a table.
The cache is bounded, only storing 1000 entries (although this is configurable with the SURREAL_DATASTORE_CACHE_SIZE
environment variable). Least recently used cache entries will be replaced with more recent entries.
So for instance if you run the following query, we will use 5 cache entries (1 for the fields, 1 for the indexes, 1 for the events, 1 for the foreign tables, and 1 for the live queries):
SELECT * FROM person:tobie;
If you then run a different query, then we will use an additional 5 cache entries (1 for the fields, 1 for the indexes, 1 for the events, 1 for the foreign tables, and 1 for the live queries):
SELECT * FROM product:tshirt;
So effectively using the default caching parameter, we can cache the definitions for 200 tables. This is for a single SurrealDB node, and the items which are cached could be different on another SurrealDB node in a cluster, depending on the queries which are executed against each node.
With the new release, imports are now streamed. Previously, when running an import, the entire import had to be parsed before it could be run. This introduced limitations on the size of imports as they had to fit within the available system memory. With the introduction of streaming imports, the import query is now parsed and run statement by statement. This makes it possible to import datasets without issue, even if they exceed a computer’s available memory
SurrealKV is now used for our in-memory instances. Our previous storage solution did not allow concurrent transactions. SurrealKV allows concurrent transactions and is therefore significantly faster. Additionally, we have optimised for the database load times when restarting surrealkv, further improving efficiency.
In 2.1
the behaviour of the UPSERT
statement from versions 2.0.0
-> 2.0.4
has also been improved to correctly handle existing index entries, WHERE
conditions, and when generating new records without specifically setting the RecordID. Let’s take a look at some examples of the improvements.
When using a UNIQUE
index combined with the UPSERT
statement, SurrealDB will now fetch the record from the index if it already exists. If a specific RecordID has been specified for the statement, then an error will be returned in the event that the index is already populated with a matching unique value for a different record.
-- This will define a unique index on the table DEFINE INDEX OVERWRITE testing ON person FIELDS one, two, three UNIQUE; -- This will create a record, and populate the unique index with this record id UPSERT person SET one = "something", two = "something", three = "something"; -- This will update the record, returning the same record id created in the statement above UPSERT person SET one = "something", two = "something", three = "something"; -- This will throw an error, because the unique index already has a record with a different record id UPSERT person:test SET one = "something", two = "something", three = "something";
Another example regarding the WHERE
clause is that in 2.1
when no RecordID has been specifically set, the iterator will first fetch records from storage. If records already exist which match the WHERE
clause, then these records will be updated. Otherwise, a new record will be created if no records are found.
-- This will return the created record as no matching records exist UPSERT person SET name = 'Jaime' WHERE name = 'Jaime'; -- This will return the updated record, because a matching record exists, and the WHERE clause matches UPSERT person SET name = 'Tobie' WHERE name = 'Jaime'; -- This will return a newly created record, because the WHERE clause does not match UPSERT person SET name = 'Tobie' WHERE name = 'Jaime';
For more information on the UPSERT
statement works with records, please refer to the UPSERT
documentation
This release also introduces some new functions, and improvements to existing ones.
The type::field()
and type::fields()
functions can now be used in more projections, like WHERE
clauses in SELECT
statements.
LET $field = "foo"; SELECT type::field($field) FROM person WHERE type::field($field) == "bar";
We introduced two new array methods too: array::fold()
and array::reduce()
. These methods allow you to walk over array values and collect them into something else, like an object, string or number as an example.
"I am a forwards string" .split('') .fold("", |$one, $two| $two + $one); -- “gnirts sdrawrof a ma I”
Lastly, we introduced new string distance and similarity functions, allowing you to find distance and similarity between two strings.
// No distance (0) RETURN string::distance::hamming("", ""); RETURN string::distance::hamming("hamming", "hamming"); // Distance of 3 RETURN string::distance::hamming("hamming", "hammers"); // Distance of 2 RETURN string::distance::hamming("hamming", "h香mmüng");; // Distance of 14 RETURN string::distance::hamming("Friedrich Nietzs", "Jean-Paul Sartre");
Some additional note-worthy changes include:
DEFAULT
clauseTo see the full list of enhancement please check out our release notes, and documentation. Thank you for your continued support and enthusiasm for SurrealDB. We are excited to see how you integrate these new capabilities into your applications! Install the new release and let us know your feedback.