Class UrlCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.UrlCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer

public class UrlCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.CharSequenceNormalizer
A CharSequenceNormalizer implementation that replaces each http or https URL and each email address with one space.

A URL is a scheme, ://, and a body. The scheme is http or https in any letter case and starts the text or follows a code point that is neither a letter, a digit, nor one of _ + - . :, so the https inside git+https:// or blob:https:// is not a URL of its own. The body runs to the end of the text or to the first Unicode whitespace, control character, unpaired surrogate, typographic quotation mark, or one of < > "; it therefore includes a port, userinfo, percent escapes, a bracketed IPv6 host, and non-ASCII host names and paths. Trailing sentence punctuation (. , ; : ! ?), a trailing apostrophe, and a closing bracket without its opening bracket inside the body are not part of the URL. A URL with another scheme is kept as it is, and no email address is matched inside it.

An email address is a maximal run of the local-part set [-+_.0-9A-Za-z] whose left neighbor is outside that set, an @, and a domain run out of [-.0-9A-Za-z] that must not start with a dot and must span at least two chars.

Since 3.0.0 URLs are removed as a whole. Under CompatibilityMode.LEGACY the earlier output is produced for language detector models trained with it: a URL is a lowercase http:// or https:// followed by a run of [-_.?&~;+=/#0-9A-Za-z], and email addresses are then matched in the result, also inside URLs with another scheme.

See Also:
  • Constructor Details

    • UrlCharSequenceNormalizer

      public UrlCharSequenceNormalizer()
  • Method Details

    • getInstance

      public static UrlCharSequenceNormalizer getInstance()
      Returns the shared, stateless instance.
      Returns:
      the shared, stateless instance
    • normalize

      public CharSequence normalize(CharSequence text)
      URLs and email addresses are found in one pass, so an address inside a URL of any scheme is never matched on its own.
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer