DEFINE ANALYZER statement

In the context of a database, an analyzer plays a crucial role in text processing and searching. It is defined by its name, a set of tokenizers, and a collection of filters.

The output of an analyzer can be experimented with by using the search::analyze() function.

Requirements

• You must be authenticated as a root, namespace, or database user before you can use the DEFINE ANALYZER statement.
• You must select your namespace and database before you can use the DEFINE ANALYZER statement.

Statement syntax

SurrealQL Syntax
DEFINE ANALYZER [ OVERWRITE | IF NOT EXISTS ] @name [ TOKENIZERS @tokenizers ] [ FILTERS @filters ] [ COMMENT @string ]

Tokenizers

Tokenizers are responsible for breaking down a given text into individual tokens based on a set of instructions. There are a couple of tokenizers that can be used while defining an analyzer as seen below:

blank

The blank tokenizer breaks down a text into tokens by creating a new token each time it encounters a space, tab, or newline character. It’s a straightforward way to split text into words or chunks based on whitespace.

DEFINE ANALYZER example_blank TOKENIZERS blank;
search::analyze("example_blank", "hello world");

Output
[
	'hello',
	'world'
]

camel

The camel tokenizer is used for identifying and creating tokens when the next character in the text is uppercase. This is particularly useful for processing camelCase or PascalCase text, common in programming, to split them into meaningful words.

DEFINE ANALYZER example_camel TOKENIZERS camel;
search::analyze("example_camel", "helloWorld");

Output
[
	'hello',
	'World'
]

class

The class tokenizer segments text into tokens by detecting changes (digit, letter, punctuation, blank) in the Unicode class of characters. It creates a new token when the character class changes, distinguishing between digits, letters, punctuation, and blanks. This allows for flexible tokenization based on character types.

DEFINE ANALYZER example_class TOKENIZERS class;
search::analyze("example_class", "123abc!XYZ");

Output
[
	'123',
	'abc',
	'!',
	'XYZ'
]

punct

The punct tokenizer generates tokens by breaking the text whenever a punctuation character is encountered. It’s suitable for tokenizing sentences or breaking text into smaller units based on punctuation marks.

DEFINE ANALYZER example_punct TOKENIZERS punct;
search::analyze("example_punct", "Hello, World!");

Output
[
	'Hello',
	',',
	'World',
	'!'
]

Filters

Filters take on the task of transforming these tokens for further processing and analysis.

ascii

The ascii filter is responsible for processing tokens by replacing or removing diacritical marks (accents and special characters) from the text. It helps standardize text by converting accented characters to their basic ASCII equivalents, making it more suitable for various text analysis tasks.

DEFINE ANALYZER example_ascii TOKENIZERS class FILTERS ascii;
search::analyze("example_ascii", "résumé café");

Output
[
	'resume',
	'cafe'
]

lowercase

The lowercase filter converts tokens to lowercase, ensuring that text is consistently in lowercase format. This is often used to make text case-insensitive for search and analysis purposes.

DEFINE ANALYZER example_lowercase TOKENIZERS class FILTERS lowercase;
search::analyze("example_lowercase", "Hello World");

Output
[
	'hello',
	'world'
]

uppercase

The uppercase filter converts tokens to uppercase, ensuring text consistency in uppercase format. It can be useful when case-insensitivity is required for specific analysis or search operations.

For example, if you had the text “Hello World”, the uppercase filter would create two tokens, [“HELLO”, “WORLD”]. Below is an example of how to use the uppercase filter:

DEFINE ANALYZER example_uppercase TOKENIZERS class FILTERS uppercase;
search::analyze("example_uppercase", "Hello World");

Output
[
	'HELLO',
	'WORLD'
]

edgengram(min,max)

The edgengram filter is used to create tokens that represent prefixes of terms. It generates a sequence of tokens that gradually build up a term, which can be useful for autocomplete or searching based on partial words. It accepts two parameters min and max which define the minimum and maximum amount of characters in the prefix.

For example, if you had the text “apple banana”, the edgengram filter would create six tokens, [“a”, “ap”, “app”, “b”, “ba”, “ban”]. Below is an example of how to use the edgengram filter:

DEFINE ANALYZER example_edgengram TOKENIZERS class FILTERS edgengram(1,3);
search::analyze("example_edgengram", "apple banana");

[
	'a',
	'ap',
	'app',
	'b',
	'ba',
	'ban'
]

ngram(min,max)

The ngram filter is used to create a sequence of ‘n’ tokens from a given sample of text or speech. These items can be syllables, letters, words or base pairs according to the application. It accepts two parameters min and max which indicates that you want to create n-grams starting from min to size of max.

DEFINE ANALYZER example_ngram TOKENIZERS class FILTERS ngram(1,3);
search::analyze("example_ngram", "apple banana");

Output
[
	'a',
	'ap',
	'app',
	'p',
	'pp',
	'ppl',
	'p',
	'pl',
	'ple',
	'l',
	'le',
	'e',
	'b',
	'ba',
	'ban',
	'a',
	'an',
	'ana',
	'n',
	'na',
	'nan',
	'a',
	'an',
	'ana',
	'n',
	'na',
	'a'
]

snowball(language)

The snowball filter applies Snowball stemming to tokens, reducing them to their root form and converts the case to lowercase. The following supported languages can be passed as a parameter in snowball: Arabic, Danish, Dutch, English, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, Turkish.

DEFINE ANALYZER english_snowball TOKENIZERS class FILTERS snowball(english);
DEFINE ANALYZER german_snowball TOKENIZERS class FILTERS snowball(german);

RETURN [
    search::analyze("english_snowball", "Looking at some running cats"),
    search::analyze("german_snowball", "Sollen wir was trinken gehen?")
];

Output
[
	[
		'look',
		'at',
		'some',
		'run',
		'cat'
	],
	[
		'soll',
		'wir',
		'was',
		'trink',
		'geh',
		'?'
	]
]

Using IF NOT EXISTS clause

Available since: v1.3.0

The IF NOT EXISTS clause can be used to define an analyzer only if it does not already exist. You should use the IF NOT EXISTS clause when defining an analyzer in SurrealDB if you want to ensure that the analyzer is only created if it does not already exist. If the analyzer already exists, the DEFINE USER statement will return an error.

It’s particularly useful when you want to safely attempt to define a analyzer without manually checking its existence first.

On the other hand, you should not use the IF NOT EXISTS clause when you want to ensure that the analyzer definition is updated regardless of whether it already exists. In such cases, you might prefer using the OVERWRITE clause, which allows you to define a analyzer and overwrite an existing one if it already exists, ensuring that the latest version of the analyzer definition is always in use

-- Create an ANALYZER if it does not already exist
DEFINE ANALYZER IF NOT EXISTS example TOKENIZERS blank;

Using OVERWRITE clause

Available since: v2.0.0

The OVERWRITE clause can be used to create an analyzer and overwrite an existing one if it already exists. You should use the OVERWRITE clause when you want to modify an existing analyzer definition. If the analyzer already exists, the DEFINE ANALYZER statement will overwrite the existing analyzer definition with the new one.

-- Create an ANALYZER and overwrite if it already exists
DEFINE ANALYZER OVERWRITE example TOKENIZERS blank;

More examples

Examples on application of analyzers to indexes can be found in the documenation on DEFINE INDEX statement

This example creates an analyzer that tokenizes text based on the class of characters and then applies the lowercase filter to the tokens.

-- Creates a simple analyzer removing diacritics marks
DEFINE ANALYZER ascii TOKENIZERS class FILTERS lowercase,ascii;

This example creates an analyzer specifically designed for processing English texts.

-- Creates an analyzer suitable for English text
DEFINE ANALYZER english TOKENIZERS class FILTERS snowball(english);

This example creates an analyzer specifically designed for auto-completion tasks.

-- Creates an analyzer suitable for auto-completion.
DEFINE ANALYZER autocomplete FILTERS lowercase,edgengram(2,10);

This example creates an analyzer specifically designed for source code analysis.

-- Creates an analyzer suitable for source code analysis.
DEFINE ANALYZER code TOKENIZERS class,camel FILTERS lowercase,ascii;