Class TokenSampleStream
java.lang.Object
opennlp.tools.tokenize.lang.en.TokenSampleStream
- All Implemented Interfaces:
Iterator<opennlp.tools.tokenize.TokenSample>
public class TokenSampleStream
extends Object
implements Iterator<opennlp.tools.tokenize.TokenSample>
Class which produces an Iterator<TokenSample> from a file of whitespace delimited
tokens. This class uses a number of English-specific heuristics to un-separate tokens which
are typically found together in text.
Tokens are separated by runs of Unicode White_Space, see
StringUtil.splitOnUnicodeWhitespace(CharSequence), independent of the whitespace
mode; leading and trailing whitespace adds no token. A line without a token resets the
quote state, and a token holding a letter or digit of any script is a word rather than
punctuation.
-
Constructor Details
-
TokenSampleStream
- Throws:
IOException
-
-
Method Details
-
hasNext
-
next
-
remove
-