Class TermVectorAnnotator

java.lang.Object
opennlp.tools.termvector.TermVectorAnnotator
All Implemented Interfaces:
opennlp.tools.document.DocumentAnnotator

public final class TermVectorAnnotator extends Object implements opennlp.tools.document.DocumentAnnotator
Rolls the token layer up into a term vector layer for index consumers: one TermVector per distinct term, carrying the term string, its occurrence count, and (in full mode) the occurrence offsets.

Term identity comes from the configured normalizer: without one, the term is the token's covered text; with a CharSequenceNormalizer, each token is normalized on its own; with an OffsetAwareNormalizer, the document is normalized once and each token span is mapped into the normalized text. Occurrence spans always point into the original text. A token whose term is empty is omitted.

The layer is document-scoped: each TermVector is a whole-document statistic, so the annotations carry no span of their own and the occurrence offsets live inside the payload. The layer preserves first-occurrence order: the first token of a term fixes its position in the layer.

The annotator holds no per-call state; it is as thread-safe as the normalizer it was built with.

Since:
3.0.0
  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    How much each TermVector records.
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector>
    The key of the term vector layer this annotator provides: a document-scoped layer of TermVector values, one per distinct term.
  • Constructor Summary

    Constructors
    Constructor
    Description
    Initializes a full mode annotator that groups tokens by their covered text as-is.
    Initializes an annotator that groups tokens by their covered text as-is.
    TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)
    Initializes a full mode annotator that groups tokens by their per-token normalized form.
    TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode)
    Initializes an annotator that groups tokens by their per-token normalized form.
    TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer)
    Initializes a full mode annotator that groups tokens by their normalized form through a whole-document alignment.
    TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode)
    Initializes an annotator that groups tokens by their normalized form through a whole-document alignment.
  • Method Summary

    Modifier and Type
    Method
    Description
    opennlp.tools.document.Document
    annotate(opennlp.tools.document.Document document)
    Aggregates the token layer into the TERM_VECTORS layer.
    Set<opennlp.tools.document.LayerKey<?>>
    Set<opennlp.tools.document.LayerKey<?>>
    Returns the adapter's simple class name, which names it in pipeline validation messages.

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
  • Field Details

    • TERM_VECTORS

      public static final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector> TERM_VECTORS
      The key of the term vector layer this annotator provides: a document-scoped layer of TermVector values, one per distinct term.
  • Constructor Details

    • TermVectorAnnotator

      public TermVectorAnnotator()
      Initializes a full mode annotator that groups tokens by their covered text as-is.
    • TermVectorAnnotator

      public TermVectorAnnotator(TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their covered text as-is.
      Parameters:
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if mode is null.
    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)
      Initializes a full mode annotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so any CharSequenceNormalizer works, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to each token's covered text. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer is null.
    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so any CharSequenceNormalizer works, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to each token's covered text. Must not be null.
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer or mode is null.
    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer)
      Initializes a full mode annotator that groups tokens by their normalized form through a whole-document alignment. Prefer the plain-normalizer constructor as the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer is null.
    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their normalized form through a whole-document alignment. Prefer the plain-normalizer constructor as the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not be null.
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer or mode is null.
  • Method Details

    • annotate

      public opennlp.tools.document.Document annotate(opennlp.tools.document.Document document)
      Aggregates the token layer into the TERM_VECTORS layer. A present-but-empty token layer yields a present-but-empty term vector layer.
      Specified by:
      annotate in interface opennlp.tools.document.DocumentAnnotator
      Parameters:
      document - The document to annotate. Must not be null and must contain the Layers.TOKENS layer.
      Returns:
      A new Document with the TERM_VECTORS layer added. Never null.
      Throws:
      IllegalArgumentException - Thrown if document is null, lacks the Layers.TOKENS layer, or already carries the TERM_VECTORS layer.
    • requires

      public Set<opennlp.tools.document.LayerKey<?>> requires()
      Specified by:
      requires in interface opennlp.tools.document.DocumentAnnotator
    • provides

      public Set<opennlp.tools.document.LayerKey<?>> provides()
      Specified by:
      provides in interface opennlp.tools.document.DocumentAnnotator
    • toString

      public String toString()
      Returns the adapter's simple class name, which names it in pipeline validation messages.
      Overrides:
      toString in class Object
      Returns:
      the adapter's simple class name, which names it in pipeline validation messages