Uses of Package
opennlp.tools.tokenize
Packages that use opennlp.tools.tokenize
-
Classes in opennlp.tools.tokenize used by opennlp.tools.tokenizeClassDescriptionThe
BPEModelstores learned BPE merge operations and can be serialized and deserialized for reuse.Represents a pair of adjacent symbols in BPE.ABaseToolFactoryfor BPE tokenization that manages the BPE merge rules artifact and its serialization within aBPEModel.A basicTokenizerimplementation which performs tokenization using character classes.An immutable policy for testing token eligibility against explicit character sets.The factory that providesTokenizerdefault implementation and resources.TheTokenizerModelis the model used by a learnableTokenizer. -
Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.langClassDescriptionAn immutable policy for testing token eligibility against explicit character sets.