Class TokenizerAnnotator
java.lang.Object
opennlp.tools.tokenize.TokenizerAnnotator
- All Implemented Interfaces:
opennlp.tools.document.DocumentAnnotator
public final class TokenizerAnnotator
extends Object
implements opennlp.tools.document.DocumentAnnotator
Adapts a
Tokenizer to the document pipeline: provides Layers.TOKENS.
When Layers.SENTENCES is present, each sentence is tokenized separately and
the token spans are shifted back to document coordinates; a present-but-empty sentence
layer therefore yields a present-but-empty token layer. Only when the sentence layer
is absent is the whole text tokenized at once. Either way, every token span refers to
the original document text.
The adapter holds no per-call state; it is as thread-safe as the tokenizer it wraps.
- Since:
- 3.0.0
-
Constructor Summary
ConstructorsConstructorDescriptionTokenizerAnnotator(opennlp.tools.tokenize.Tokenizer tokenizer) Initializes the adapter. -
Method Summary
Modifier and TypeMethodDescriptionopennlp.tools.document.Documentannotate(opennlp.tools.document.Document document) Tokenizes the document and adds theLayers.TOKENSlayer, sentence by sentence when a sentence layer is present and over the whole text otherwise.Set<opennlp.tools.document.LayerKey<?>> provides()toString()Returns the adapter's simple class name, which names it in pipeline validation messages.Methods inherited from interface opennlp.tools.document.DocumentAnnotator
requires
-
Constructor Details
-
TokenizerAnnotator
public TokenizerAnnotator(opennlp.tools.tokenize.Tokenizer tokenizer) Initializes the adapter.- Parameters:
tokenizer- The tokenizer to delegate to. Must not benull.- Throws:
IllegalArgumentException- Thrown iftokenizerisnull.
-
-
Method Details
-
annotate
public opennlp.tools.document.Document annotate(opennlp.tools.document.Document document) Tokenizes the document and adds theLayers.TOKENSlayer, sentence by sentence when a sentence layer is present and over the whole text otherwise.- Specified by:
annotatein interfaceopennlp.tools.document.DocumentAnnotator- Parameters:
document- The document to annotate. Must not benull.- Returns:
- A new
Documentwith theLayers.TOKENSlayer added. Nevernull. - Throws:
IllegalArgumentException- Thrown ifdocumentisnullor already carries theLayers.TOKENSlayer.
-
provides
- Specified by:
providesin interfaceopennlp.tools.document.DocumentAnnotator
-
toString
-