Class HexCodePoints
java.lang.Object
opennlp.tools.util.normalizer.HexCodePoints
Reads the hex code point notation of Unicode data files and similar code point tables: one
code point as hex digits (
1F600), a sequence as hex digits separated by whitespace
(1F468 200D 1F469), and an inclusive range as two code points joined by two dots
(1F3FB..1F3FF). Also removes the # comments these files carry.-
Method Summary
Modifier and TypeMethodDescriptionstatic String[]Splits a sequence into its hex code point tokens.static StringDecodes a sequence of hex code points separated by runs of Unicode whitespace into the characters they name.static StringdecodeSequence(CharSequence hex, char separator) Decodes a sequence of hex code points separated by exactly oneseparatoreach into the characters they name.static intParses hex digits as one code point.static intparseCodePoint(CharSequence hex, int start, int end) Parses the hex digits between two indexes as one code point.static int[]parseRange(CharSequence hex) Parses one code point or an inclusiveXXXX..YYYYrange.static StringstripComment(String line) Removes a trailing comment from a data file line.
-
Method Details
-
stripComment
Removes a trailing comment from a data file line.- Parameters:
line- The raw line. Must not benull.- Returns:
- The line up to but excluding the first
#, orlineitself when it has none. Surrounding whitespace is kept. - Throws:
IllegalArgumentException- Thrown iflineisnull.
-
parseCodePoint
Parses the hex digits between two indexes as one code point.- Parameters:
hex- The text holding the digits. Must not benull.start- The index of the first digit.end- The index after the last digit.- Returns:
- The code point.
- Throws:
IllegalArgumentException- Thrown ifhexisnull, the region lies outsidehex, is empty, holds a character that is not a hex digit, or names a value outside[0, U+10FFFF].
-
parseCodePoint
Parses hex digits as one code point.- Parameters:
hex- The digits. Must not benull.- Returns:
- The code point.
- Throws:
IllegalArgumentException- Thrown ifhexisnullor empty, holds a character that is not a hex digit, or names a value outside[0, U+10FFFF].
-
decodeSequence
Decodes a sequence of hex code points separated by runs of Unicode whitespace into the characters they name. Leading and trailing whitespace is ignored.- Parameters:
hex- The sequence. Must not benull.- Returns:
- The decoded characters, in order.
- Throws:
IllegalArgumentException- Thrown ifhexisnull, holds no code point, or one of its code points is malformed.
-
decodeSequence
Decodes a sequence of hex code points separated by exactly oneseparatoreach into the characters they name. Two adjacent separators or a leading separator leave an empty code point, which is rejected; trailing separators are ignored.- Parameters:
hex- The sequence. Must not benull.separator- The character between two code points. Must not be a surrogate.- Returns:
- The decoded characters, in order.
- Throws:
IllegalArgumentException- Thrown ifhexisnullor empty,separatoris a surrogate, or one of the code points is malformed.
-
codePointTokens
Splits a sequence into its hex code point tokens.- Parameters:
hex- The sequence. Must not benull.- Returns:
- The tokens in order, at least one.
- Throws:
IllegalArgumentException- Thrown ifhexisnullor holds no token.
-
parseRange
Parses one code point or an inclusiveXXXX..YYYYrange.- Parameters:
hex- The code point or range. Must not benull.- Returns:
- The first and the last code point of the range; both the same for one code point.
- Throws:
IllegalArgumentException- Thrown ifhexisnull, a code point is malformed, or the range ends before it starts.
-