NjxddlmZddlZddlZddlmZej eZ ddZ ddZ d dZ y) ) annotationsN)TokenizerModelc<t}d}d}d}|j}|r=|Dcgc]}|j|s|} }|j| }t | }|j } t|}g} g} |D]=}| j |} t | dkrtjd|d|dz }Bt | dkDrq| Dcgc]}d|d }}dj|}tjd|d|d||jvr|j|}| j|| d}||vrtjd|d |dz }|r1|j|r tjd|d |dz }|j|| j|@d |_|j}|j!| d }|j#| ddd }t | }t%|||||Scc}wcc}w)auClean a vocabulary by removing duplicates and tokens that were already in the vocabulary. This function removes duplicate tokens and tokens that are already in the model's vocabulary. It also removes the tokenizer's post-processor, which we do not use anyway. :param model: The tokenizer model to clean. :param vocabulary_to_add: The vocabulary to add to the model. Any tokens in this vocabulary that are split according to the pretokenizer are added as multiword tokens. :param token_remove_regex: A regex pattern to remove tokens from the vocabulary. :return: The cleaned tokenizer model. rzToken 'z ' was empty after preprocessing.',z7' was split into multiple tokens after preprocessing: [z#], adding it as a multi-word token.z ' was already in the vocabulary.z!' was removed due to regex match.NT)preprocess_tokensF) is_special single_word normalized)setsorted_vocabularymatchremove_tokens_from_vocabularylen preprocessor preprocessloggerwarningjoin vocabularyremove_token_from_vocabularyappendaddpost_processorprune_added_tokensadd_tokens_to_vocabularyadd_addedtokens_report_statistics)modelvocabulary_to_addtoken_remove_regex seen_tokens n_duplicaten_emptyn_regex_removedinternal_tokenstokentokens_to_remover tokens_to_addadded_tokens_to_add preprocessedsubword tokens_as_str split_into n_multiwords n/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/model2vec/tokenizer/tokenizer.pyclean_and_create_vocabularyr2 sX %KKGO!&!8!8O/>beBTBZBZ[`BaEb334DE./%%Lo&K!M%'"#..u5 | q NNWUG+KL M qLG  | q ;GH<q ^? h//JKL hwi~67c:|jxsg|jxsg}}|j}|j}g}|D]W}|j |}||j g|||.|j |j |jY|S)zConvert a list of Token objects to their corresponding token ID sequences. :param tokens: List of Token objects to convert :param model: The tokenizermodel of the tokenizer. :return: List of token IDs corresponding to the input tokens )bos_idseos_idsr to_tokenizergetrencodeids) tokensr prefixsuffixr tokenizer token_idsr(token_ids r1turn_tokens_into_idsrC_s]](b%--*=2FF!!J""$I!#I>>%(     9v9x9&9 :   Y--e488 9  r5)r rr! list[str]r"zre.Pattern[str] | Nonereturnr) r0intr$rFr&rFr%rFrENone)r=rDr rrEzlist[list[int]]) __future__rloggingre skeletokenr getLogger__name__rr2rrCr5r1rOs\" %   8 $E E E/E EP 8r5