Nj ddlZddlZddlZddlmZddlmZmZddlmZeje Z GddZ dZ dZd e fd ZGd d eZy) N)nn)BaseConditioner TokenizedText)download_if_necessaryc4eZdZdZdededdfdZdedefdZy) SentencePieceTokenizeraThis tokenizer should be used for natural language descriptions. For example: ["he didn't, know he's going home.", 'shorter sentence'] => [[78, 62, 31, 4, 78, 25, 19, 34], [59, 77, PAD, PAD, PAD, PAD, PAD, PAD]] Args: n_bins (int): should be equal to the number of elements in the sentencepiece tokenizer. tokenizer_path (str): path to the sentencepiece tokenizer model. nbinstokenizer_pathreturnNctjd|t|}tjt ||_||j jk(s&Jd|j jd|dy)Nz'Loading sentencepiece tokenizer from %sz'sentencepiece tokenizer has vocab size=z but nbins=z was specified)loggerinfor sentencepieceSentencePieceProcessorstrsp vocab_size)selfr r s m/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/pocket_tts/conditioners/text.py__init__zSentencePieceTokenizer.__init__sw =~N.~>66s>7JK**,, 5dgg6H6H6J5K;W\V]]k l ,textcttj|jj |t dddfS)N)out_type)rtorchtensorrencodeint)rrs r__call__zSentencePieceTokenizer.__call__"s1U\\$''...*LMdTUgVWWr) __name__ __module__ __qualname____doc__rrrrrrrrr s7  c 3 4 XSX]Xrrizehf://kyutai/pocket-tts-without-voice-cloning/tokenizer.model@d4fdd22ae8c8e1cb3634e150ebeff1dab2d16df3r c*tttS)zReturn a SentencePieceTokenizer with the default model path and vocab size. Downloads the tokenizer model from HuggingFace on first use. )rDEFAULT_TOKENIZER_N_BINSDEFAULT_TOKENIZER_PATHr$rrget_default_tokenizerr(-s "":