What options are available for the 'tokenizer' parameter and what is the default?

1 vues

Réponse

The 'tokenizer' parameter specifies which tokenizer to use. The default is 'STANDARD', which applies a language-specific tokenizer. The alternative is 'BASIC', which separates words by white spaces and punctuation, and is available for Chinese, Japanese, and Korean to enhance rule matching.
Did this answer help you?
compileCategory

textRuleDevelop

See technical action
Other questions about compileCategory
What is the primary function of the compileCategory action? What are the required parameters for the compileCategory action? What does the 'casOut' parameter represent? Can I use an existing concept model (LI binary) when compiling a category model? What is the purpose of the 'language' parameter?
Copyright © WeAreCAS 2023-2025