speechbrain.tokenizers

Package defining the SentencePiece tokenizer

speechbrain.tokenizers.SentencePiece

Library for Byte-pair-encoding (BPE) tokenization. Authors * Abdelwahab Heba 2020 * Loren Lugosch 2020.

speechbrain.tokenizers.discrete_SSL_tokenizer

Tokenizer for semantic tokens.