speechbrain.tokenizers.discrete_SSL_tokenizer module

Tokenizer for semantic tokens.

Author

  • Pooneh Mousavi 2024

Summary

Classes:

DiscreteSSLTokenizer

This class is tokenizer for DiscreteSSL models that apply post-processing on the semnatic tokens extracted from DiscreteSSL model.

Reference

class speechbrain.tokenizers.discrete_SSL_tokenizer.DiscreteSSLTokenizer(num_clusters)[source]

Bases: object

This class is tokenizer for DiscreteSSL models that apply post-processing on the semnatic tokens extracted from DiscreteSSL model. It makes the token ids of each layer to be unique by adding the token IDs of each layer by layer_num*sunmber_of _cluster. It applies deduplication for each layer independently if the field is set to true for the layer and padded all items with zero. It applies subwording for each layer independently if the sentence piece tokenizer is set to for the layer and padded all items with zero. If subwording is not applied, all token IDs are incremented by one to avoid conflict between pad_id(0) and cluster with centroid zero.

Parameters:

num_clusters (List[int]) – determine the number of clusters of the kmeans models. It could be varying for each layer.

Example

>>> import torch
>>> inputs = torch.randint(0,1000,(3, 6, 2))
>>> ssl_layer_num = [7,23]
>>> deduplicate =[False, True]
>>> bpe_tokenizers=[None, None]
>>> num_clusters = [1000,2000]
>>> tokenizer = DiscreteSSLTokenizer(num_clusters=num_clusters)
>>> tokens= tokenizer.encode(inputs,SSL_layers=ssl_layer_num, deduplicates=deduplicate, bpe_tokenizers=bpe_tokenizers)
>>> print(tokens.shape)
torch.Size([3, 6, 2])
textify(tokens)[source]

Convert token ID to char to be used for training sentencepiece tokenizer. :param tokens: A (Batch x Seq ) tensor of audio tokens :type tokens: torch.Tensor :param Returns: :param ———: :param processed_tokens: A (Batch x Seq) list of corresponding char for each token ID. :type processed_tokens: list

encode(input, SSL_layers=[7], deduplicates=[False], bpe_tokenizers=[None])[source]

Takes an input tokenized wavform and return its corresponding processed tokens.

Parameters:

  • tokens (torch.Tensor) – A (Batch x Seq x num_SSL_layers) tensor of audio tokens

  • SSL_layers (List[int] (default: [7]):) – determine which layers of SSL should be used to extract information.

  • deduplicates (List[boolean] (default: [False]):) – determine to apply deduplication(remove duplicate subsequent tokens) on the tokens extracted for the corresponding layer.

  • bpe_tokenizers (List[int] (default: [None]):) – determine to apply subwording on the tokens extracted for the corresponding layer if the sentencePiece tokenizer is trained for that layer.

  • Returns

  • ---------

  • processed_tokens (torch.Tensor) – A (Batch x Seq x num_SSL_layers) tensor of audio tokens after applying deduplication and subwording if necessary.