Class StringUtil

java.lang.Object
opennlp.tools.util.StringUtil

public class StringUtil extends Object
  • Field Details

    • ASCII_DIGIT_STRINGS

      public static final List<String> ASCII_DIGIT_STRINGS
      The immutable list of the ten ASCII digit strings "0" to "9", indexed by digit value.
  • Constructor Details

    • StringUtil

      public StringUtil()
  • Method Details

    • isWhitespace

      public static boolean isWhitespace(char charCode)
      Determines if the specified Character is a whitespace under the active WhitespaceMode: the Unicode White_Space property by default, or, when the "opennlp.whitespace.mode" system property selects WhitespaceMode.LEGACY, the union of Character.isWhitespace(int) and the Unicode Zs category (Character.SPACE_SEPARATOR) that OpenNLP 1.x and 2.x used.

      Tokenization, corpus format parsing, and feature generation all resolve whitespace through this method, so they share one WhitespaceMode for the life of the process; use isUnicodeWhitespace(char) directly where the Unicode definition should apply unconditionally, such as user-text normalization with no trained-model dependency.

      Parameters:
      charCode - The character to check.
      Returns:
      true if charCode represents a white space under the active WhitespaceMode, false otherwise.
    • isWhitespace

      public static boolean isWhitespace(int charCode)
      Determines if the specified code point is a whitespace under the active WhitespaceMode; see isWhitespace(char) for details.
      Parameters:
      charCode - An int representation of a character to check.
      Returns:
      true if charCode represents a white space under the active WhitespaceMode, false otherwise.
    • isUnicodeWhitespace

      public static boolean isUnicodeWhitespace(char charCode)
      Determines if the specified Character is a whitespace under the Unicode White_Space property; delegates to UnicodeWhitespace.isWhitespace(int). This is also what isWhitespace(char) resolves to under the default WhitespaceMode.
      Parameters:
      charCode - The character to check.
      Returns:
      true if charCode has the White_Space property, false otherwise.
    • isUnicodeWhitespace

      public static boolean isUnicodeWhitespace(int charCode)
      Determines if the specified code point is a whitespace under the Unicode White_Space property; delegates to UnicodeWhitespace.isWhitespace(int). This is also what isWhitespace(int) resolves to under the default WhitespaceMode.
      Parameters:
      charCode - An int representation of a character to check.
      Returns:
      true if charCode has the White_Space property, false otherwise.
    • splitOnUnicodeWhitespace

      public static String[] splitOnUnicodeWhitespace(CharSequence input)
      Splits input on runs of Unicode White_Space. Leading and trailing runs are ignored, so whitespace-only input yields an empty array.
      Parameters:
      input - The text to split. Must not be null.
      Returns:
      The non-whitespace terms in order.
      Throws:
      IllegalArgumentException - If input is null.
    • splitOnWhitespace

      public static String[] splitOnWhitespace(CharSequence input)
      Splits input on runs of whitespace under the active WhitespaceMode, see isWhitespace(char). Leading and trailing runs are ignored, so whitespace-only input yields an empty array. Use splitOnUnicodeWhitespace(CharSequence) where the Unicode definition should apply regardless of the mode.
      Parameters:
      input - The text to split. Must not be null.
      Returns:
      The non-whitespace fields in order.
      Throws:
      IllegalArgumentException - If input is null.
    • split

      public static String[] split(CharSequence input, char separator)
      Splits input around literal occurrences of separator, with the same result as String.split(String) called with the single-character pattern Character.toString(separator) and a limit of 0: leading and middle empty fields are kept, trailing empty fields are removed, and empty input yields an array containing one empty string. The scan compares characters, so separator is always treated as a literal, including when it is a regular-expression metacharacter such as '.', '$' or '|', and supplementary code points in the input pass through untouched.
      Parameters:
      input - The text to split. Must not be null.
      separator - The literal field separator. Must not be a surrogate.
      Returns:
      The fields in order.
      Throws:
      IllegalArgumentException - If input is null or separator is a surrogate.
    • split

      public static String[] split(CharSequence input, char separator, int limit)
      Splits input around literal occurrences of separator, with the same limit contract as String.split(String, int) applied to the single-character pattern Character.toString(separator): a positive limit applies at most limit - 1 splits and keeps trailing empty fields; 0 applies unlimited splits and removes trailing empty fields, exactly like split(CharSequence, char); a negative limit applies unlimited splits and keeps trailing empty fields. Empty input yields an array containing one empty string at every limit. The scan compares characters, so separator is always treated as a literal, including when it is a regular-expression metacharacter, and supplementary code points in the input pass through untouched.
      Parameters:
      input - The text to split. Must not be null.
      separator - The literal field separator. Must not be a surrogate, so that a supplementary code point in input is never split in half.
      limit - The maximum number of fields, or 0 for the default behavior, or negative for no limit.
      Returns:
      The fields in order.
      Throws:
      IllegalArgumentException - If input is null or separator is a surrogate.
    • startsWithByteOrderMark

      public static boolean startsWithByteOrderMark(CharSequence input)
      Tests whether input starts with a byte order mark, U+FEFF, which a UTF-8 or UTF-16 text file may carry as its first character.
      Parameters:
      input - The text to test. Must not be null.
      Returns:
      true if the first character of input is U+FEFF.
      Throws:
      IllegalArgumentException - If input is null.
    • stripByteOrderMark

      public static String stripByteOrderMark(String input)
      Removes one leading byte order mark, U+FEFF, from input. A U+FEFF anywhere else is left in place, since there it is a zero width no-break space and part of the text.
      Parameters:
      input - The text to strip. Must not be null.
      Returns:
      input without its leading byte order mark, or input itself if it has none.
      Throws:
      IllegalArgumentException - If input is null.
    • splitNonEmpty

      public static String[] splitNonEmpty(CharSequence input, char... separators)
      Splits input around literal occurrences of any of the separators and drops the empty fields that leading, trailing or adjacent separators would produce, so a run of separators acts as one and input made only of separators yields an empty array. The scan compares characters, so every separator is a literal, including a regular-expression metacharacter, and supplementary code points in the input pass through untouched.
      Parameters:
      input - The text to split. Must not be null.
      separators - The literal field separators. Must not be null or empty, and must not contain a surrogate.
      Returns:
      The non-empty fields in order.
      Throws:
      IllegalArgumentException - If input or separators is null, separators is empty, or a separator is a surrogate.
    • containsAsciiUpperCase

      public static boolean containsAsciiUpperCase(CharSequence input)
      Tests whether input contains an ASCII capital letter, A to Z. Capital letters outside ASCII do not count.
      Parameters:
      input - The text to check. Must not be null.
      Returns:
      true if at least one character is an ASCII capital letter.
      Throws:
      IllegalArgumentException - If input is null.
    • containsAsciiDigit

      public static boolean containsAsciiDigit(CharSequence input)
      Tests whether input contains an ASCII digit, 0 to 9. Digits outside ASCII do not count.
      Parameters:
      input - The text to check. Must not be null.
      Returns:
      true if at least one character is an ASCII digit.
      Throws:
      IllegalArgumentException - If input is null.
    • isAsciiLetter

      public static boolean isAsciiLetter(int codePoint)
      Tests for an ASCII letter, a to z or A to Z.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint is an ASCII letter.
    • isAsciiLowerCase

      public static boolean isAsciiLowerCase(int codePoint)
      Tests for an ASCII lowercase letter, a to z.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint is an ASCII lowercase letter.
    • isAsciiDigit

      public static boolean isAsciiDigit(int codePoint)
      Tests for an ASCII digit, 0 to 9.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint is an ASCII digit.
    • isAsciiWhitespace

      public static boolean isAsciiWhitespace(int codePoint)
      Tests for an ASCII whitespace character: space, tab, line feed, vertical tab, form feed or carriage return. This is the set matched by \s in a Pattern compiled without UNICODE_CHARACTER_CLASS.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint is an ASCII whitespace character.
    • endOfAsciiDigits

      public static int endOfAsciiDigits(CharSequence text, int from)
      Finds the end of the run of ASCII digits that starts at from.
      Parameters:
      text - The text to scan. Must not be null.
      from - The offset the run starts at, between 0 and text.length().
      Returns:
      The offset after the last digit of the run, or from if no digit is there.
      Throws:
      IllegalArgumentException - If text is null or from is out of range.
    • isLineTerminator

      public static boolean isLineTerminator(int codePoint)
      Tests for a line terminator: line feed U+000A, carriage return U+000D, next line U+0085, line separator U+2028, or paragraph separator U+2029. This is the set Pattern treats as line terminators outside UNIX_LINES mode. It is narrower than UnicodeWhitespace.lineBreakCodePointSet(), which also holds vertical tab U+000B and form feed U+000C.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint ends a line.
    • isUnpairedSurrogate

      public static boolean isUnpairedSurrogate(int codePoint)
      Checks whether a code point taken from a CharSequence is a lone surrogate, U+D800 to U+DFFF, rather than a character. String.codePoints() and Character.codePointAt(CharSequence, int) return a surrogate only when it is not part of a valid high and low surrogate pair.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if codePoint is in the surrogate range.
    • indexOfLineTerminator

      public static int indexOfLineTerminator(CharSequence text, int from)
      Finds the first line terminator at or after from, as defined by isLineTerminator(int).
      Parameters:
      text - The text to scan. Must not be null.
      from - The offset to start at, between 0 and text.length().
      Returns:
      The offset of the first line terminator, or -1 if there is none.
      Throws:
      IllegalArgumentException - If text is null or from is out of range.
    • trimUnicodeWhitespace

      public static String trimUnicodeWhitespace(CharSequence input)
      Trims leading and trailing runs of Unicode White_Space, the same set splitOnUnicodeWhitespace(CharSequence) breaks terms on.
      Parameters:
      input - The text to trim. Must not be null.
      Returns:
      The trimmed string; may be empty when input is whitespace-only.
      Throws:
      IllegalArgumentException - If input is null.
    • isUnicodeBlank

      public static boolean isUnicodeBlank(CharSequence input)
      true when input is null, empty, or consists only of Unicode White_Space code points.
      Parameters:
      input - The text to test; null is treated as blank.
      Returns:
      true if there is no non-whitespace code point.
    • toLowerCase

      public static String toLowerCase(CharSequence string)
      Converts a CharSequence to lower case, independent of the current Locale via Character.toLowerCase(int) which uses mapping information from the UnicodeData file.
      Parameters:
      string - The CharSequence to transform.
      Returns:
      The lower-cased String.
    • toLowerCaseCharBuffer

      public static CharBuffer toLowerCaseCharBuffer(CharSequence sequence)
    • toUpperCase

      public static String toUpperCase(CharSequence string)
      Converts a CharSequence to upper case, independent of the current Locale via Character.toUpperCase(int) which uses mapping information from the UnicodeData file.
      Parameters:
      string - The CharSequence to transform.
      Returns:
      The upper-cased String
    • isEmpty

      public static boolean isEmpty(CharSequence theString)
      Determines whether a CharSequence is empty.
      Parameters:
      theString - The CharSequence to examine. Must not be null.
      Returns:
      true if CharSequence.length() is 0, otherwise false.
      Throws:
      IllegalArgumentException - Thrown if theString is null.
      Since:
      1.5.1
    • isBlank

      public static boolean isBlank(CharSequence theString)
      Determines whether a CharSequence is blank: empty, or made up entirely of code points that isWhitespace(int) accepts. Unlike String.isBlank(), this follows the toolkit's whitespace definition, which includes the no-break spaces the JDK predicate leaves out, so a value spelled entirely from them cannot pass a blank check as content. Unlike isUnicodeBlank(CharSequence), it resolves through the active WhitespaceMode and does not treat null as blank.
      Parameters:
      theString - The CharSequence to examine. Must not be null.
      Returns:
      true if theString is empty or all whitespace.
      Throws:
      IllegalArgumentException - Thrown if theString is null.
    • levenshteinDistance

      public static int[][] levenshteinDistance(String wordForm, String lemma)
      Computes the Levenshtein distance of two strings in a matrix.

      Based on this pseudo-code which in turn is based on the paper Wagner, Robert A.; Fischer, Michael J. (1974), "The String-to-String Correction Problem", Journal of the ACM 21 (1): 168-173

      Parameters:
      wordForm - The form as input.
      lemma - The target lemma.
      Returns:
      A 2-dimensional Levenshtein distance matrix.
    • computeShortestEditScript

      public static void computeShortestEditScript(String wordForm, String lemma, int[][] distance, StringBuffer permutations)
      Computes the Shortest Edit Script (SES) to convert a word into its lemma. This is based on Chrupala's PhD thesis (2008).
      Parameters:
      wordForm - The token.
      lemma - The target lemma.
      distance - A 2-dimensional Levenshtein distance matrix.
      permutations - The number of permutations.
    • decodeShortestEditScript

      public static String decodeShortestEditScript(String wordForm, String permutations)
      Reads the predicted Shortest Edit Script (SES) by a lemmatizer model and applies the permutations to obtain the lemma from the wordForm.
      Parameters:
      wordForm - The wordForm as input.
      permutations - The permutations predicted by the lemmatizer model.
      Returns:
      The decoded lemma.
    • getShortestEditScript

      public static String getShortestEditScript(String wordForm, String lemma)
      Parameters:
      wordForm - The word as input.
      lemma - The target lemma.
      Returns:
      Retrieves the Shortest Edit Script (SES) required to go from a word to a lemma.