Class Factory

java.lang.Object
opennlp.tools.tokenize.lang.Factory

public class Factory extends Object
Provides the language dependent tokenizer resources: the TokenizerCharacterPolicy that decides which tokens the alphanumeric optimization accepts without further splitting, and the TokenContextGenerator.
  • Field Details

    • DEFAULT_CHARACTER_POLICY

      public static final TokenizerCharacterPolicy DEFAULT_CHARACTER_POLICY
      The policy for languages without a dedicated one: ASCII letters and digits, no marks.
    • DEFAULT_ALPHANUMERIC

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public static final Pattern DEFAULT_ALPHANUMERIC
      Deprecated, for removal: This API element is subject to removal in a future version.
      Use DEFAULT_CHARACTER_POLICY, which accepts the same characters. This constant will be removed in 4.0.
      The alphanumeric pattern of earlier releases for languages without a dedicated one.
  • Constructor Details

    • Factory

      public Factory()
  • Method Details

    • getCharacterPolicy

      public TokenizerCharacterPolicy getCharacterPolicy(String languageCode)
      Gets the character policy for a language.
      Parameters:
      languageCode - An ISO 639-1 code, or an ISO 639-2 code in its terminology or bibliographic form (deu or ger). If null, or unknown, the DEFAULT_CHARACTER_POLICY will be returned.
      Returns:
      The TokenizerCharacterPolicy for the language, or the default policy.
    • getAlphanumeric

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public Pattern getAlphanumeric(String languageCode)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Use getCharacterPolicy(String). This method will be removed in 4.0.
      Gets the alphanumeric pattern that releases before 3.0 used for a language.
      Parameters:
      languageCode - An ISO 639-1 code, or an ISO 639-2 code in its terminology or bibliographic form. If null, or unknown, the DEFAULT_ALPHANUMERIC pattern will be returned.
      Returns:
      The Pattern that accepts the same characters as getCharacterPolicy(String) for the language.
    • policyOfLegacyExpression

      @Internal @Deprecated(since="3.0.0", forRemoval=true) public static TokenizerCharacterPolicy policyOfLegacyExpression(String expression)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Serves only the deprecated Pattern API and will be removed with it in 4.0.
      Returns the policy of the language whose alphanumeric expression of an earlier release is expression, or null if there is none.

      Note: Internal use only, for the deprecated Pattern API.

      Parameters:
      expression - An alphanumeric expression of an earlier release.
      Returns:
      the policy of the language whose alphanumeric expression of an earlier release is expression, or null if there is none
    • legacyExpressionOf

      @Internal @Deprecated(since="3.0.0", forRemoval=true) public static String legacyExpressionOf(TokenizerCharacterPolicy policy)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Serves only the deprecated Pattern API and will be removed with it in 4.0.
      Returns the alphanumeric expression of an earlier release that accepts the same characters as policy, or null if there is none.

      Note: Internal use only, for the deprecated Pattern API.

      Parameters:
      policy - A character policy, or null.
      Returns:
      the alphanumeric expression of an earlier release that accepts the same characters as policy, or null if there is none
    • legacyPatternOf

      @Internal @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public static Pattern legacyPatternOf(TokenizerCharacterPolicy policy)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Serves only the deprecated Pattern API and will be removed with it in 4.0.
      Returns the alphanumeric pattern of an earlier release that accepts the same characters as policy, or null if there is none.

      Note: Internal use only, for the deprecated Pattern API.

      Parameters:
      policy - A character policy, or null.
      Returns:
      the alphanumeric pattern of an earlier release that accepts the same characters as policy, or null if there is none
    • createTokenContextGenerator

      public opennlp.tools.tokenize.TokenContextGenerator createTokenContextGenerator(String languageCode, Set<String> abbreviations)
      Initializes a customized TokenContextGenerator via a set of abbreviations.
      Parameters:
      languageCode - The ISO_639-1 code to be used.
      abbreviations - The abbreviations to be used for new instance.