Package opennlp.tools.tokenize.lang
package opennlp.tools.tokenize.lang
-
ClassesClassDescriptionProvides the language dependent tokenizer resources: the
TokenizerCharacterPolicythat decides which tokens the alphanumeric optimization accepts without further splitting, and theTokenContextGenerator.