Class UrlCharSequenceNormalizer
- All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer
CharSequenceNormalizer implementation that replaces each http or
https URL and each email address with one space.
A URL is a scheme, ://, and a body. The scheme is http or https
in any letter case and starts the text or follows a code point that is neither a letter,
a digit, nor one of _ + - . :, so the https inside git+https://
or blob:https:// is not a URL of its own. The body runs to the end of the text or
to the first Unicode whitespace, control character, unpaired surrogate, typographic
quotation mark, or one of < > "; it therefore includes a port, userinfo, percent
escapes, a bracketed IPv6 host, and non-ASCII host names and paths. Trailing sentence
punctuation (. , ; : ! ?), a trailing apostrophe, and a closing bracket without its
opening bracket inside the body are not part of the URL. A URL with another scheme is kept
as it is, and no email address is matched inside it.
An email address is a maximal run of the local-part set [-+_.0-9A-Za-z] whose
left neighbor is outside that set, an @, and a domain run out of
[-.0-9A-Za-z] that must not start with a dot and must span at least two chars.
Since 3.0.0 URLs are removed as a whole. Under CompatibilityMode.LEGACY the
earlier output is produced for language detector models trained with it: a URL is a
lowercase http:// or https:// followed by a run of
[-_.?&~;+=/#0-9A-Za-z], and email addresses are then matched in the result, also
inside URLs with another scheme.
- See Also:
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic UrlCharSequenceNormalizerReturns the shared, stateless instance.normalize(CharSequence text) URLs and email addresses are found in one pass, so an address inside a URL of any scheme is never matched on its own.
-
Constructor Details
-
UrlCharSequenceNormalizer
public UrlCharSequenceNormalizer()
-
-
Method Details
-
getInstance
Returns the shared, stateless instance.- Returns:
- the shared, stateless instance
-
normalize
URLs and email addresses are found in one pass, so an address inside a URL of any scheme is never matched on its own.- Specified by:
normalizein interfaceopennlp.tools.util.normalizer.CharSequenceNormalizer
-