Interface: TokenizeOptions¶

Options for tokenizing a full-text search query without a table index.

Properties¶

optional asciiFolding: boolean;

Whether to fold ASCII characters.

optional baseTokenizer: BaseTokenizer;

The tokenizer to use. The default is "simple".

optional customStopWords: string[];

Custom stop words that replace the built-in list for language.

This option only affects tokenization when removeStopWords is true.

undefined keeps the built-in language list. An empty array explicitly replaces it with no stop words.

optional language: string;

Language for stemming and stop words.

optional lowercase: boolean;

Whether to lowercase tokens.

optional maxTokenLength: number;

Maximum token length; tokens longer than this are ignored.

optional ngramMaxLength: number;

N-gram maximum length.

optional ngramMinLength: number;

N-gram minimum length.

optional prefixOnly: boolean;

Whether to only emit token prefixes for the n-gram tokenizer.

optional removeStopWords: boolean;

Whether to remove stop words.

optional stem: boolean;

Whether to stem tokens.