Class EmojiCharSequenceNormalizer
- All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer
CharSequenceNormalizer implementation that replaces each run of complete emoji with
one space.
An emoji is a fully-qualified or minimally-qualified sequence of Unicode Emoji 18.0 in the
sense of UTS #51: a single emoji, an emoji
with U+FE0F, a flag, a keycap, a skin tone modifier sequence, a ZWJ sequence or a tag
sequence, as listed in emoji-test.txt. A minimally-qualified ZWJ sequence lacks U+FE0F
after a later element only, such as a woman raising hand written without U+FE0F after
U+2640. Adjacent emoji form one run and become one space; one redundant U+FE0F right after an
emoji is removed with it. Everything else is kept: a symbol with text presentation such as
U+2764 without U+FE0F, an unqualified sequence starting with such a symbol, letters and
ideographs from other planes,
private use characters, unpaired surrogates, and an emoji with a stray joiner, modifier,
selector, flag letter or tag next to it, which is kept as a whole instead of leaving a
fragment behind.
Since 3.0.0 only complete emoji sequences are removed. Under
CompatibilityMode.LEGACY the earlier output is produced for language detector models
trained with it: each maximal run of ASCII hyphens, characters from U+E000 to U+FFFF (private
use characters, CJK compatibility ideographs, Arabic and Hebrew presentation forms, fullwidth
and halfwidth forms, variation selectors), supplementary characters up to U+10FC00 whether
they are emoji or not, and unpaired surrogates from U+D83C on becomes one space, while emoji
in the Basic Multilingual Plane such as U+231A are kept.
- See Also:
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic EmojiCharSequenceNormalizerReturns the shared, stateless instance.normalize(CharSequence text)
-
Constructor Details
-
EmojiCharSequenceNormalizer
public EmojiCharSequenceNormalizer()
-
-
Method Details
-
getInstance
Returns the shared, stateless instance.- Returns:
- the shared, stateless instance
-
normalize
- Specified by:
normalizein interfaceopennlp.tools.util.normalizer.CharSequenceNormalizer
-