What options are available for the 'tokenizer' parameter and what is the default?

4 vues

Réponse

The 'tokenizer' parameter specifies which tokenizer to use. The default is 'STANDARD', which applies a language-specific tokenizer. The alternative is 'BASIC', which separates words by white spaces and punctuation, and is available for Chinese, Japanese, and Korean to enhance rule matching.
Did this answer help you?
compileCategory

textRuleDevelop

See technical action
Other questions about compileCategory
What are the required parameters for the compileCategory action? Can I use an existing concept model (LI binary) when compiling a category model? What is the primary function of the compileCategory action? What does the 'casOut' parameter represent? What is the purpose of the 'language' parameter?
Copyright © WeAreCAS 2014-2026
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. WeAreCAS is an independent community site and is not affiliated with SAS Institute Inc.