Class TokSpanEventStream
java.lang.Object
opennlp.tools.util.AbstractEventStream<opennlp.tools.tokenize.TokenSample>
opennlp.tools.tokenize.TokSpanEventStream
- All Implemented Interfaces:
AutoCloseable, opennlp.tools.util.ObjectStream<opennlp.tools.ml.model.Event>
This class reads the
samples via an Iterator
and converts the samples into events which
can be used by the maxent library for training.-
Constructor Summary
ConstructorsConstructorDescriptionTokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics) Initializes a new event stream based on the data stream using aTokenContextGeneratorthat relies on aDefaultTokenContextGenerator.TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, opennlp.tools.tokenize.TokenContextGenerator cg) Deprecated, for removal: This API element is subject to removal in a future version.TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, opennlp.tools.tokenize.TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGeneratorandFactory.DEFAULT_CHARACTER_POLICY.TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenizerCharacterPolicy characterPolicy, opennlp.tools.tokenize.TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGenerator. -
Method Summary
Methods inherited from class AbstractEventStream
close, read, reset
-
Constructor Details
-
TokSpanEventStream
public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenizerCharacterPolicy characterPolicy, opennlp.tools.tokenize.TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGenerator.- Parameters:
tokenSamples- Thedata streamfor this event stream.skipAlphaNumerics- Whether alphanumerics are skipped, or not.characterPolicy- TheTokenizerCharacterPolicythat decides which candidate tokens count as alphanumeric. Must not benull.cg- ATokenContextGeneratorwhich should be used for the event streamd.- Throws:
IllegalArgumentException- Thrown ifcharacterPolicyisnull.
-
TokSpanEventStream
@Deprecated(since="3.0.0", forRemoval=true) @SuppressForbidden("user-supplied alphanumeric pattern, kept for the deprecated API") public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, opennlp.tools.tokenize.TokenContextGenerator cg) Deprecated, for removal: This API element is subject to removal in a future version.UseTokSpanEventStream(ObjectStream, boolean, TokenizerCharacterPolicy, TokenContextGenerator). This constructor will be removed in 4.0.Initializes a new event stream that decides which candidate tokens count as alphanumeric with a regular expression.- Parameters:
tokenSamples- Thedata streamfor this event stream.skipAlphaNumerics- Whether alphanumerics are skipped, or not.alphaNumeric- A custom alphanumericPattern, ornullforFactory.DEFAULT_CHARACTER_POLICY.cg- ATokenContextGeneratorwhich should be used for the event streamd.
-
TokSpanEventStream
public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics, opennlp.tools.tokenize.TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGeneratorandFactory.DEFAULT_CHARACTER_POLICY.- Parameters:
tokenSamples- Thedata streamfor this event stream.skipAlphaNumerics- Whether alphanumerics are skipped, or not.cg- ATokenContextGeneratorwhich should be used for the event streamd.
-
TokSpanEventStream
public TokSpanEventStream(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> tokenSamples, boolean skipAlphaNumerics) Initializes a new event stream based on the data stream using aTokenContextGeneratorthat relies on aDefaultTokenContextGenerator.- Parameters:
tokenSamples- Thedata streamfor this event stream.skipAlphaNumerics- Whether alphanumerics are skipped, or not.
-
TokSpanEventStream(ObjectStream, boolean, TokenizerCharacterPolicy, TokenContextGenerator).