Class TokenizerCharacterPolicy

java.lang.Object
opennlp.tools.tokenize.TokenizerCharacterPolicy
All Implemented Interfaces:
Predicate<CharSequence>

public final class TokenizerCharacterPolicy extends Object implements Predicate<CharSequence>
An immutable policy for testing token eligibility against explicit character sets.

A token consists of one or more letters or digits. Each letter or digit may be followed by zero or more marks. A mark cannot start a token. The supplied sets express a caller's tokenization policy; this class does not infer Unicode character categories.

This predicate tests a complete candidate without finding token boundaries or changing its text.

Since:
3.0.0
  • Method Summary

    Modifier and Type
    Method
    Description
    Returns a policy for ASCII letters and digits, with no marks.
    boolean
    opennlp.tools.util.normalizer.CodePointSet
    Returns the immutable digit set.
    opennlp.tools.util.normalizer.CodePointSet
    Returns the immutable letter set.
    opennlp.tools.util.normalizer.CodePointSet
    Returns the immutable mark set.
    int
    of(opennlp.tools.util.normalizer.CodePointSet letters, opennlp.tools.util.normalizer.CodePointSet digits, opennlp.tools.util.normalizer.CodePointSet marks)
    Creates a policy from explicit, pairwise-disjoint code point sets.
    parse(String text)
    Parses the text form written by toString(): the three fields letters=, ;digits= and ;marks= in that order, each followed by zero or more comma separated hex code points (0041) or inclusive ranges (0041-005A).
    boolean

    Methods inherited from class Object

    getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface Predicate

    and, negate, or
  • Method Details

    • of

      public static TokenizerCharacterPolicy of(opennlp.tools.util.normalizer.CodePointSet letters, opennlp.tools.util.normalizer.CodePointSet digits, opennlp.tools.util.normalizer.CodePointSet marks)
      Creates a policy from explicit, pairwise-disjoint code point sets. Whitespace separates tokens and so cannot be part of one; every other non-surrogate code point is left to the caller.
      Parameters:
      letters - Code points treated as letters.
      digits - Code points treated as digits.
      marks - Code points permitted after a letter, a digit, or another mark.
      Returns:
      The immutable policy.
      Throws:
      IllegalArgumentException - Thrown if a set is null, letters and digits are both empty, a code point occurs in more than one set, or a set contains a surrogate code point or UnicodeWhitespace member.
    • ascii

      public static TokenizerCharacterPolicy ascii()
      Returns a policy for ASCII letters and digits, with no marks.
      Returns:
      a policy for ASCII letters and digits, with no marks
    • parse

      public static TokenizerCharacterPolicy parse(String text)
      Parses the text form written by toString(): the three fields letters=, ;digits= and ;marks= in that order, each followed by zero or more comma separated hex code points (0041) or inclusive ranges (0041-005A). Hex digits may use either case; no prefix or whitespace is accepted.
      Parameters:
      text - The text form to read.
      Returns:
      The policy the text describes.
      Throws:
      IllegalArgumentException - Thrown if text is null or not of that form, or if the sets it describes are rejected by of(CodePointSet, CodePointSet, CodePointSet).
    • test

      public boolean test(CharSequence input)

      Returns true if the entire input is a non-empty token accepted by this policy. Malformed UTF-16 input (an unpaired or reversed surrogate) returns false.

      Specified by:
      test in interface Predicate<CharSequence>
      Throws:
      IllegalArgumentException - Thrown if input is null.
    • getLetters

      public opennlp.tools.util.normalizer.CodePointSet getLetters()
      Returns the immutable letter set.
      Returns:
      the immutable letter set
    • getDigits

      public opennlp.tools.util.normalizer.CodePointSet getDigits()
      Returns the immutable digit set.
      Returns:
      the immutable digit set
    • getMarks

      public opennlp.tools.util.normalizer.CodePointSet getMarks()
      Returns the immutable mark set.
      Returns:
      the immutable mark set
    • equals

      public boolean equals(Object o)

      Two policies are equal if their letter, digit and mark sets are equal.

      Overrides:
      equals in class Object
    • hashCode

      public int hashCode()

      Consistent with equals(Object): computed from the letter, digit and mark sets.

      Overrides:
      hashCode in class Object
    • toString

      public String toString()

      Writes the text form read by parse(String), for example letters=0041-005A,0061-007A;digits=0030-0039;marks= for ascii(): runs of adjacent code points become ranges, and each code point is written with at least four upper-case hex digits.

      Overrides:
      toString in class Object