Class WordpieceEncoder
- All Implemented Interfaces:
SubwordTokenizer
SubwordTokenizer implementing the BERT tokenization stages: basic tokenization
(control removal, whitespace normalization, CJK isolation, optional lower casing with accent
stripping, punctuation isolation) followed by greedy longest-match wordpiece segmentation.
Each result includes a vocabulary id and range in the original text. The range
refers to the input before normalization. encode(CharSequence) always starts with the
classification entry, with an empty range at offset 0, and ends with the separator entry,
with an empty range at the end of the text.
Segmentation is greedy longest-match-first with the ## continuation marker, as in
the BERT reference implementation.
Ids follow the line-number convention of BERT vocab.txt files. List constructors use
the list index, while the map constructor uses the supplied ids. The classification, separator,
and unknown tokens must all be present in the
vocabulary, because each emitted piece must have an id. Vocabulary entries starting with
## are continuation pieces and can match only after the first piece of a word.
Lower casing applies the Unicode simple case mapping of Character.toLowerCase(int),
one code point to one code point, plus the Final_Sigma context, so a word-final Greek
capital sigma becomes U+03C2 as in the reference implementation. Other
SpecialCasing
mappings are not applied; for example U+0130 becomes i without a combining dot.
A word exceeding the configured maximum number of normalized Unicode code points becomes
the unknown piece. The default is 100, the value used by the Hugging Face transformers
BERT tokenizer; the original google-research/bert code uses 200. Both count code
points, and a constructor parameter selects another limit.
- Since:
- 3.0.0
- See Also:
-
Constructor Summary
ConstructorsConstructorDescriptionWordpieceEncoder(List<String> vocabulary) Instantiates an encoder for an uncased BERT model with the BERT special tokens.WordpieceEncoder(List<String> vocabulary, boolean lowerCase) Instantiates an encoder with the BERT special tokens.WordpieceEncoder(List<String> vocabulary, boolean lowerCase, int maxWordCodePoints) Instantiates an encoder with the BERT special tokens and a custom word-length limit.WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken) Instantiates an encoder with custom special tokens.WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints) Instantiates an encoder with custom special tokens and a custom word-length limit.WordpieceEncoder(Map<String, Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken) Instantiates an encoder from an explicit piece-to-id mapping for vocabularies with noncontiguous ids.WordpieceEncoder(Map<String, Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints) Instantiates an encoder from a piece-to-id mapping with a custom word-length limit. -
Method Summary
Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface SubwordTokenizer
encodeToIds, encodeToPieces
-
Constructor Details
-
WordpieceEncoder
Instantiates an encoder for an uncased BERT model with the BERT special tokens.- Parameters:
vocabulary- The ordered vocabulary; the list index becomes the id. Must not benullor containnull, empty, or duplicate entries.- Throws:
IllegalArgumentException- Thrown if the vocabulary isnull, contains anull, empty, or duplicate entry, or is missing a BERT special token.
-
WordpieceEncoder
Instantiates an encoder with the BERT special tokens.- Parameters:
vocabulary- The ordered vocabulary; the list index becomes the id. Must not benullor containnull, empty, or duplicate entries.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.- Throws:
IllegalArgumentException- Thrown if the vocabulary isnull, contains anull, empty, or duplicate entry, or is missing a BERT special token.
-
WordpieceEncoder
Instantiates an encoder with the BERT special tokens and a custom word-length limit.- Parameters:
vocabulary- The ordered vocabulary; the list index becomes the id. Must not benullor containnull, empty, or duplicate entries.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.maxWordCodePoints- The non-negative maximum number of normalized Unicode code points in one word.- Throws:
IllegalArgumentException- Thrown if an argument is invalid or a BERT special token is missing.
-
WordpieceEncoder
public WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken) Instantiates an encoder with custom special tokens.- Parameters:
vocabulary- The ordered vocabulary; the list index becomes the id. Must not benullor containnull, empty, or duplicate entries.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.classificationToken- The CLS token; must not benullor empty and must be in the vocabulary.separatorToken- The SEP token; must not benullor empty and must be in the vocabulary.unknownToken- The UNK token; must not benullor empty and must be in the vocabulary.- Throws:
IllegalArgumentException- Thrown if any argument isnull, the vocabulary contains anull, empty, or duplicate entry, or a special token is empty or missing.
-
WordpieceEncoder
public WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints) Instantiates an encoder with custom special tokens and a custom word-length limit.- Parameters:
vocabulary- The ordered vocabulary; the list index becomes the id. Must not benullor containnull, empty, or duplicate entries.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.classificationToken- The CLS token; must not benullor empty and must be in the vocabulary.separatorToken- The SEP token; must not benullor empty and must be in the vocabulary.unknownToken- The UNK token; must not benullor empty and must be in the vocabulary.maxWordCodePoints- The non-negative maximum number of normalized Unicode code points in one word.- Throws:
IllegalArgumentException- Thrown if an argument is invalid or a special token is missing.
-
WordpieceEncoder
public WordpieceEncoder(Map<String, Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken) Instantiates an encoder from an explicit piece-to-id mapping for vocabularies with noncontiguous ids.- Parameters:
vocabularyIds- The piece-to-id mapping. Must not benullor containnullor empty keys,nullvalues, or negative ids.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.classificationToken- The CLS token; must not benullor empty and must be in the vocabulary.separatorToken- The SEP token; must not benullor empty and must be in the vocabulary.unknownToken- The UNK token; must not benullor empty and must be in the vocabulary.- Throws:
IllegalArgumentException- Thrown if any argument isnull, the mapping contains anullor empty key,nullvalue, or negative id, or a special token is empty or missing.
-
WordpieceEncoder
public WordpieceEncoder(Map<String, Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints) Instantiates an encoder from a piece-to-id mapping with a custom word-length limit.- Parameters:
vocabularyIds- The piece-to-id mapping. Must not benullor containnullor empty keys,nullvalues, or negative ids.lowerCase-truefor uncased models (lower casing and accent stripping),falsefor cased models.classificationToken- The CLS token; must not benullor empty and must be in the vocabulary.separatorToken- The SEP token; must not benullor empty and must be in the vocabulary.unknownToken- The UNK token; must not benullor empty and must be in the vocabulary.maxWordCodePoints- The non-negative maximum number of normalized Unicode code points in one word.- Throws:
IllegalArgumentException- Thrown if an argument is invalid or a special token is missing.
-
-
Method Details
-
encode
Encodes text into subword pieces.- Specified by:
encodein interfaceSubwordTokenizer- Parameters:
text- The text to encode; must not benull.- Returns:
- The pieces in model order; may be empty.
-