Class Factory
java.lang.Object
opennlp.tools.tokenize.lang.Factory
Provides the language dependent tokenizer resources: the
TokenizerCharacterPolicy
that decides which tokens the alphanumeric optimization accepts without further splitting,
and the TokenContextGenerator.-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final PatternDeprecated, for removal: This API element is subject to removal in a future version.static final TokenizerCharacterPolicyThe policy for languages without a dedicated one: ASCII letters and digits, no marks. -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionopennlp.tools.tokenize.TokenContextGeneratorcreateTokenContextGenerator(String languageCode, Set<String> abbreviations) Initializes a customizedTokenContextGeneratorvia a set ofabbreviations.getAlphanumeric(String languageCode) Deprecated, for removal: This API element is subject to removal in a future version.getCharacterPolicy(String languageCode) Gets the character policy for a language.static StringDeprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.static PatternDeprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.static TokenizerCharacterPolicypolicyOfLegacyExpression(String expression) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.
-
Field Details
-
DEFAULT_CHARACTER_POLICY
The policy for languages without a dedicated one: ASCII letters and digits, no marks. -
DEFAULT_ALPHANUMERIC
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public static final Pattern DEFAULT_ALPHANUMERICDeprecated, for removal: This API element is subject to removal in a future version.UseDEFAULT_CHARACTER_POLICY, which accepts the same characters. This constant will be removed in 4.0.The alphanumeric pattern of earlier releases for languages without a dedicated one.
-
-
Constructor Details
-
Factory
public Factory()
-
-
Method Details
-
getCharacterPolicy
Gets the character policy for a language.- Parameters:
languageCode- An ISO 639-1 code, or an ISO 639-2 code in its terminology or bibliographic form (deuorger). Ifnull, or unknown, theDEFAULT_CHARACTER_POLICYwill be returned.- Returns:
- The
TokenizerCharacterPolicyfor the language, or the default policy.
-
getAlphanumeric
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public Pattern getAlphanumeric(String languageCode) Deprecated, for removal: This API element is subject to removal in a future version.UsegetCharacterPolicy(String). This method will be removed in 4.0.Gets the alphanumeric pattern that releases before 3.0 used for a language.- Parameters:
languageCode- An ISO 639-1 code, or an ISO 639-2 code in its terminology or bibliographic form. Ifnull, or unknown, theDEFAULT_ALPHANUMERICpattern will be returned.- Returns:
- The
Patternthat accepts the same characters asgetCharacterPolicy(String)for the language.
-
policyOfLegacyExpression
@Internal @Deprecated(since="3.0.0", forRemoval=true) public static TokenizerCharacterPolicy policyOfLegacyExpression(String expression) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.Returns the policy of the language whose alphanumeric expression of an earlier release isexpression, ornullif there is none.Note: Internal use only, for the deprecated
PatternAPI.- Parameters:
expression- An alphanumeric expression of an earlier release.- Returns:
- the policy of the language whose alphanumeric expression of an earlier release is
expression, ornullif there is none
-
legacyExpressionOf
@Internal @Deprecated(since="3.0.0", forRemoval=true) public static String legacyExpressionOf(TokenizerCharacterPolicy policy) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.Returns the alphanumeric expression of an earlier release that accepts the same characters aspolicy, ornullif there is none.Note: Internal use only, for the deprecated
PatternAPI.- Parameters:
policy- A character policy, ornull.- Returns:
- the alphanumeric expression of an earlier release that accepts the same characters
as
policy, ornullif there is none
-
legacyPatternOf
@Internal @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API") public static Pattern legacyPatternOf(TokenizerCharacterPolicy policy) Deprecated, for removal: This API element is subject to removal in a future version.Serves only the deprecatedPatternAPI and will be removed with it in 4.0.Returns the alphanumeric pattern of an earlier release that accepts the same characters aspolicy, ornullif there is none.Note: Internal use only, for the deprecated
PatternAPI.- Parameters:
policy- A character policy, ornull.- Returns:
- the alphanumeric pattern of an earlier release that accepts the same characters as
policy, ornullif there is none
-
createTokenContextGenerator
public opennlp.tools.tokenize.TokenContextGenerator createTokenContextGenerator(String languageCode, Set<String> abbreviations) Initializes a customizedTokenContextGeneratorvia a set ofabbreviations.- Parameters:
languageCode- The ISO_639-1 code to be used.abbreviations- The abbreviations to be used for new instance.
-
DEFAULT_CHARACTER_POLICY, which accepts the same characters.