Class TokSpanEventStream

java.lang.Object
opennlp.tools.util.AbstractEventStream<opennlp.tools.tokenize.TokenSample>
opennlp.tools.tokenize.TokSpanEventStream
All Implemented Interfaces:
AutoCloseable, opennlp.tools.util.ObjectStream<opennlp.tools.ml.model.Event>

public class TokSpanEventStream extends AbstractEventStream<opennlp.tools.tokenize.TokenSample>
This class reads the samples via an Iterator and converts the samples into events which can be used by the maxent library for training.
  • Constructor Details

    • TokSpanEventStream

      public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenizerCharacterPolicy characterPolicy, opennlp.tools.tokenize.TokenContextGenerator cg)
      Initializes a new event stream based on the data stream using a TokenContextGenerator.
      Parameters:
      tokenSamples - The data stream for this event stream.
      skipAlphaNumerics - Whether alphanumerics are skipped, or not.
      characterPolicy - The TokenizerCharacterPolicy that decides which candidate tokens count as alphanumeric. Must not be null.
      cg - A TokenContextGenerator which should be used for the event stream d.
      Throws:
      IllegalArgumentException - Thrown if characterPolicy is null.
    • TokSpanEventStream

      @Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, opennlp.tools.tokenize.TokenContextGenerator cg)
      Deprecated, for removal: This API element is subject to removal in a future version.
      Initializes a new event stream that decides which candidate tokens count as alphanumeric with a regular expression.
      Parameters:
      tokenSamples - The data stream for this event stream.
      skipAlphaNumerics - Whether alphanumerics are skipped, or not.
      alphaNumeric - A custom alphanumeric Pattern, or null for Factory.DEFAULT_CHARACTER_POLICY.
      cg - A TokenContextGenerator which should be used for the event stream d.
    • TokSpanEventStream

      public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, opennlp.tools.tokenize.TokenContextGenerator cg)
      Initializes a new event stream based on the data stream using a TokenContextGenerator and Factory.DEFAULT_CHARACTER_POLICY.
      Parameters:
      tokenSamples - The data stream for this event stream.
      skipAlphaNumerics - Whether alphanumerics are skipped, or not.
      cg - A TokenContextGenerator which should be used for the event stream d.
    • TokSpanEventStream

      public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics)
      Initializes a new event stream based on the data stream using a TokenContextGenerator that relies on a DefaultTokenContextGenerator.
      Parameters:
      tokenSamples - The data stream for this event stream.
      skipAlphaNumerics - Whether alphanumerics are skipped, or not.