Similarity search

Vector search finds records by meaning rather than by the words they contain. This page runs similarity queries over an example dataset, covering the vector:: distance and similarity helpers and how to filter KNN results with predicates.

Vector search in SurrealDB

Vector search in SurrealDB can be used in place of full-text search, or together with it.

For example, still using the same liquids table, you can store the chemical composition of the liquid samples in a vector format.

-- Insert a sample & content field into a liquids table
INSERT INTO liquidsVector [
    {
        sample:'Sea water', 
        content: 'The sea water contains some amount of lead', 
        embedding: [0.1, 0.2, 0.3, 0.4] },
    {
        sample:'Tap water', 
        content:
          'The team lead by Dr. Rose found out that the tap water in was potable',
        embedding:[1.0, 0.1, 0.4, 0.3]
    },
    {
        sample:'Sewage water', 
        content: 'High amounts of a were found in Sewage water', 
        embedding : [0.4, 0.3, 0.2, 0.1]
    }
];

Notice that we have added an embedding field to the table. This field will store the vector embeddings of the content field so we can perform vector searches on it.

In the example above you can see that the results are more accurate. The search pulled up only the results in which the word "lead" was used to mean the material, while the final liquidsVector record had the lowest score. This is the advantage of using vector search over full-text search.

Another use case for vector search is in the field of facial recognition. For example, if you wanted to search for an actor or actress who looked like you from an extensive dataset of movie artists, you would first use an LLM model to convert the artist's images and details into vector embeddings and then use SurrealQL to find the artist with the most resemblance to your face vector embeddings. The more characteristics you decide to include in your vector embeddings, the higher the dimensionality of your vector will be, potentially improving the accuracy of the matches but also increasing the complexity of the vector search.

Computation on vectors: "vector::" package of functions

SurrealDB provides vector functions for most of the major numerical computations done on vectors. They include functions for element-wise addition, division and even normalisation.

They also include similarity and distance functions, which help in understanding how similar or dissimilar two vectors are. Usually, the vector with the smallest distance or the largest cosine similarity value (closest to 1) is deemed the most similar to the item you are trying to search for.

The choice of distance or similarity function depends on the nature of your data and the specific requirements of your application.

In the liquids examples, we assumed that the embeddings represented the harmfulness of lead (as a substance). We used the vector::similarity::cosine function because cosine similarity is typically preferred when absolute distances are less important, but proportions and direction matter more.

Filtering through vector search

The vector::distance::knn() function from SurrealDB returns the distance computed between vectors by the KNN operator. This operator can be used to avoid recomputation of the distance in every select query.

Consider a scenario where you’re searching for actors who look like you but they should have won an Oscar. You set a flag, which is true for actors who’ve won the golden trophy.

Let’s create a dataset of actors and define an approximate vector index on the embeddings field. This walkthrough uses HNSW; from SurrealDB 3.1 you can instead use DISKANN when your vectors no longer fit comfortably in memory. See the page on vector indexes for trade-offs and supported TYPE / DIST combinations.

-- Create a dataset of actors with embeddings and flags
CREATE actor:1 SET
    name = 'Actor 1',
    embedding = [0.1, 0.2, 0.3, 0.4],
    flag = true;
CREATE actor:2 SET
    name = 'Actor 2',
    embedding = [0.2, 0.1, 0.4, 0.3],
    flag = false;
CREATE actor:3 SET
    name = 'Actor 3',
    embedding = [0.4, 0.3, 0.2, 0.1],
    flag = true;
CREATE actor:4 SET
    name = 'Actor 4',
    embedding = [0.3, 0.4, 0.1, 0.2],
    flag = true;

-- Define an embedding to represent a face
LET $person_embedding = [0.15, 0.25, 0.35, 0.45];

-- Define an HNSW index on the actor table
DEFINE INDEX hnsw_pts ON actor FIELDS embedding HNSW DIMENSION 4;

-- Select actors who look like you and have won an Oscar
SELECT id, flag, vector::distance::knn() AS distance FROM actor
  WHERE flag = true AND embedding <|2,40|> $person_embedding ORDER BY distance;

[
	[
		{
			distance: 0.09999999999999998f,
			flag: true,
			id: actor:1
		},
		{
			distance: 0.412310562561766f,
			flag: true,
			id: actor:4
		}
	]
];

actor:1 and actor:4 have the closest resemblance with your query vector among those who have also won an Oscar.

How the filter is applied

Because embedding is indexed, the flag = true condition is not applied after the nearest neighbours have been gathered. Instead SurrealDB applies it during the approximate (HNSW or DISKANN) search, so candidates that fail the condition are discarded as the graph is traversed and never occupy one of the K result slots. This keeps the result both correct and efficient:

• •

  Correct, because you still receive up to K records that match the condition.

• •

  Efficient, since the search does not spend its slots on records that would only be filtered out afterwards.

With the four actors above, the two nearest records are actor:1 and actor:2, and actor:2 has flag = false. Filtering afterwards would leave one result, while filtering during the search skips actor:2 and fills the second slot with actor:4:

SurrealDB applies the condition in one of two ways:

• •

  Checked on each candidate. The search reads the record of each candidate it visits and tests the condition. The example above runs this way, because flag has no index of its own.

• •

  Pre-filtered through an index. When another index gives exactly the records that the condition matches, SurrealDB reads that index first into an allow-list, and the search admits only records on the list without reading them. See pre-filtering with an index.

Confirming the pushed-down filter with EXPLAIN

Available since: v3.1.5

The EXPLAIN clause can be used in the same query to see the query plan. The pushed-down condition appears as a predicate attribute on the KnnScan operator:

EXPLAIN SELECT id, flag, vector::distance::knn() AS distance FROM actor
  WHERE flag = true AND embedding <|2,40|> $person_embedding ORDER BY distance;

'SelectProject [ctx: Db] [projections: id, flag, distance]
    Sort [ctx: Db] [sort_keys: vector::distance::knn() ASC]
        Filter [ctx: Db] [predicate: flag = true]
            KnnScan [ctx: Db] [index: hnsw_pts, k: 2, ef: 40, dimension: 4, predicate: flag = true]'

The predicate: flag = true on the KnnScan line is the condition being evaluated inside the index search. If the attribute is missing and no bitmap operator appears beneath KnnScan, the condition is not applied during the search (for example, when the vector field is not indexed). A DISKANN index produces an identical KnnScan line.

Pre-filtering with an index

Available since: v3.3.0

A condition is pre-filtered when an index identifies exactly the records it matches. That is the case for:

• •

  An equality or range condition, or an OR of them, on a field whose standard or unique index has never stored an array. An array value has one index entry per element, so an index that holds one could list records the condition does not match. An index records this from SurrealDB 4.0, so an index built by an earlier release qualifies only after REBUILD INDEX. In SurrealDB 3.3, the field needs a declared type that cannot hold an array, such as TYPE bool or TYPE string.

• •

  A full-text @@ condition on a field with a full-text index.

A NOT condition, or any other condition, is still checked on each candidate. When a WHERE clause mixes the two kinds, the qualifying conditions form the allow-list and the rest are checked on each candidate. A query with a VERSION clause is not pre-filtered. From SurrealDB 4.0, a WITH INDEX clause limits which indexes build the allow-list, and the vector index still runs the search, so the clause does not change which records are returned:

• •

  A WITH INDEX clause that names the index on the filtered field, such as idx_flag below, pre-filters through that index.

• •

  A WITH INDEX clause that names only the vector index, such as hnsw_pts, checks every condition on each candidate.

Before SurrealDB 4.0, any WITH clause turned pre-filtering off.

Defining an index for flag in the example above is enough for its condition to qualify, because none of its values is an array:

DEFINE INDEX idx_flag ON actor FIELDS flag;

EXPLAIN ANALYZE SELECT id, flag, vector::distance::knn() AS distance FROM actor
  WHERE flag = true AND embedding <|2,40|> $person_embedding ORDER BY distance;

'SelectProject [ctx: Db] [projections: id, flag, distance] {rows: 2, batches: 1, elapsed: 472.29µs}
    Sort [ctx: Db] [sort_keys: vector::distance::knn() ASC] {rows: 2, batches: 1, elapsed: 472.29µs}
        Filter [ctx: Db] [predicate: flag = true] {rows: 2, batches: 1, elapsed: 472.29µs}
            KnnScan [ctx: Db] [index: hnsw_pts, k: 2, ef: 40, dimension: 4, prefilter_tier: exact] {rows: 2, batches: 1, elapsed: 472.29µs}
                BitmapIndexScan [ctx: Db] [index: idx_flag, access: = true] {rows: 3, batches: 0, elapsed: 0ns}

Total rows: 2'

The condition now appears as a BitmapIndexScan beneath KnnScan, which found the three actors with flag = true. KnnScan carries no predicate, because no condition is left to check during the search.

The prefilter_tier attribute, reported under EXPLAIN ANALYZE, shows how the search used the allow-list. The choice depends on the number of records on it:

The thresholds are set with SURREAL_KNN_PREFILTER_EXACT_THRESHOLD, SURREAL_KNN_PREFILTER_EF_BOOST_THRESHOLD, SURREAL_KNN_PREFILTER_EF_BOOST and SURREAL_KNN_PREFILTER_EF_MAX, and SURREAL_KNN_PREFILTER_ENABLED=false turns pre-filtering off. See environment variables. The allow-list is built with the same bitmaps that combine indexes in other queries.

Ordering by distance

Available since: v4.0.0

A search for the nearest records can also be written as an ordering by a distance function with a LIMIT. When the function matches the distance that an HNSW or DISKANN index was defined with, the index answers the query, and a WHERE condition is pre-filtered in the same way as with the <|K, EF|> operator. The index in the example above uses the default EUCLIDEAN distance:

SELECT id, vector::distance::euclidean(embedding, $person_embedding) AS distance FROM actor
  WHERE flag = true
  ORDER BY vector::distance::euclidean(embedding, $person_embedding)
  LIMIT 2;

[
	{
		distance: 0.09999999999999998f,
		id: actor:1
	},
	{
		distance: 0.412310562561766f,
		id: actor:4
	}
]

EXPLAIN shows a KnnScan with k: 2, the LIMIT, beneath the sort:

EXPLAIN SELECT id FROM actor
  WHERE flag = true
  ORDER BY vector::distance::euclidean(embedding, $person_embedding)
  LIMIT 2;

'SelectProject [ctx: Db] [projections: id]
    Limit [ctx: Db] [limit: 2]
        Sort [ctx: Db] [sort_keys: vector::distance::euclidean(embedding, $person_embedding) ASC, keep: 2]
            Filter [ctx: Db] [predicate: flag = true]
                KnnScan [ctx: Db] [index: hnsw_pts, k: 2, ef: 150, dimension: 4]
                    BitmapIndexScan [ctx: Db] [index: idx_flag, access: = true]'

The index answers the ordering only when all of the following are true. Otherwise the query reads and sorts every matching record, and returns the exact result.

• •

  The ordering is ascending, has a single key, and the statement has a LIMIT without START.

• •

  The function is vector::distance::cosine, euclidean, manhattan, chebyshev, hamming or minkowski, and matches the DIST of the index. For an index defined with DIST COSINE, use vector::distance::cosine.

• •

  Every record of the table has a vector in the indexed field, so the index holds an entry for each of them. A record whose field is NONE or NULL stops the index from answering the ordering.

• •

  The index was defined or rebuilt on SurrealDB 4.0. An older index answers the ordering after REBUILD INDEX.

The search runs with the search width that <|K|> uses, which is the larger of K and the EFC of an HNSW index or the L_BUILD of a DISKANN index. Like the <|K, EF|> operator, it is approximate. To get the exact nearest records, add WITH NOINDEX, which sorts every record by the distance function.

Filtering after the search

A pre-filtered search returns up to K records that match the condition. To find the K nearest records first and then keep only those that match, put the search in a subquery and the condition in the outer WHERE clause. The condition is not moved into the subquery, so the result can hold fewer than K records:

SELECT id, flag FROM (
  SELECT id, flag FROM actor WHERE embedding <|2,40|> $person_embedding
) WHERE flag = true;

[
	{
		flag: true,
		id: actor:1
	}
]

The two nearest records are actor:1 and actor:2, and only actor:1 has flag = true.

For HNSW and DISKANN configuration and the KNN cheat sheet, see Vector indexes.