What is the 'tokenizer' parameter and when should I use the 'BASIC' option?

4 vues

Réponse

The 'tokenizer' parameter specifies the tokenizer to use. The default is 'STANDARD', which applies a language-specific tokenizer. The 'BASIC' option uses a tokenizer that separates words by white spaces, punctuation, and CJKT characters. The 'BASIC' tokenizer is only available for Chinese, Japanese, and Korean and can enhance rule matching for specific texts in these languages.
Did this answer help you?
compileConcept

textRuleDevelop

See technical action
Other questions about compileConcept
What is the purpose of the 'casOut' parameter in this action? How can I include predefined entities in my concept model compilation? What languages are supported by the compileConcept action? What is the primary function of the compileConcept action? How can I extend a predefined sentiment model?
Copyright © WeAreCAS 2014-2026
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. WeAreCAS is an independent community site and is not affiliated with SAS Institute Inc.