Class TokenizerCharacterPolicy
- All Implemented Interfaces:
Predicate<CharSequence>
A token consists of one or more letters or digits. Each letter or digit may be followed by zero or more marks. A mark cannot start a token. The supplied sets express a caller's tokenization policy; this class does not infer Unicode character categories.
This predicate tests a complete candidate without finding token boundaries or changing its text.
- Since:
- 3.0.0
-
Method Summary
Modifier and TypeMethodDescriptionstatic TokenizerCharacterPolicyascii()Returns a policy for ASCII letters and digits, with no marks.booleanopennlp.tools.util.normalizer.CodePointSetReturns the immutable digit set.opennlp.tools.util.normalizer.CodePointSetReturns the immutable letter set.opennlp.tools.util.normalizer.CodePointSetgetMarks()Returns the immutable mark set.inthashCode()static TokenizerCharacterPolicyof(opennlp.tools.util.normalizer.CodePointSet letters, opennlp.tools.util.normalizer.CodePointSet digits, opennlp.tools.util.normalizer.CodePointSet marks) Creates a policy from explicit, pairwise-disjoint code point sets.static TokenizerCharacterPolicyParses the text form written bytoString(): the three fieldsletters=,;digits=and;marks=in that order, each followed by zero or more comma separated hex code points (0041) or inclusive ranges (0041-005A).booleantest(CharSequence input) toString()
-
Method Details
-
of
public static TokenizerCharacterPolicy of(opennlp.tools.util.normalizer.CodePointSet letters, opennlp.tools.util.normalizer.CodePointSet digits, opennlp.tools.util.normalizer.CodePointSet marks) Creates a policy from explicit, pairwise-disjoint code point sets. Whitespace separates tokens and so cannot be part of one; every other non-surrogate code point is left to the caller.- Parameters:
letters- Code points treated as letters.digits- Code points treated as digits.marks- Code points permitted after a letter, a digit, or another mark.- Returns:
- The immutable policy.
- Throws:
IllegalArgumentException- Thrown if a set isnull,lettersanddigitsare both empty, a code point occurs in more than one set, or a set contains a surrogate code point orUnicodeWhitespacemember.
-
ascii
Returns a policy for ASCII letters and digits, with no marks.- Returns:
- a policy for ASCII letters and digits, with no marks
-
parse
Parses the text form written bytoString(): the three fieldsletters=,;digits=and;marks=in that order, each followed by zero or more comma separated hex code points (0041) or inclusive ranges (0041-005A). Hex digits may use either case; no prefix or whitespace is accepted.- Parameters:
text- The text form to read.- Returns:
- The policy the text describes.
- Throws:
IllegalArgumentException- Thrown iftextisnullor not of that form, or if the sets it describes are rejected byof(CodePointSet, CodePointSet, CodePointSet).
-
test
Returns
trueif the entire input is a non-empty token accepted by this policy. Malformed UTF-16 input (an unpaired or reversed surrogate) returnsfalse.- Specified by:
testin interfacePredicate<CharSequence>- Throws:
IllegalArgumentException- Thrown ifinputisnull.
-
getLetters
public opennlp.tools.util.normalizer.CodePointSet getLetters()Returns the immutable letter set.- Returns:
- the immutable letter set
-
getDigits
public opennlp.tools.util.normalizer.CodePointSet getDigits()Returns the immutable digit set.- Returns:
- the immutable digit set
-
getMarks
public opennlp.tools.util.normalizer.CodePointSet getMarks()Returns the immutable mark set.- Returns:
- the immutable mark set
-
equals
-
hashCode
public int hashCode()Consistent with
equals(Object): computed from the letter, digit and mark sets. -
toString
Writes the text form read by
parse(String), for exampleletters=0041-005A,0061-007A;digits=0030-0039;marks=forascii(): runs of adjacent code points become ranges, and each code point is written with at least four upper-case hex digits.
-