^Nj0 XUddlZddlmZddlmZddlmZddlmZm Z m Z ddl Z ddl Z ddlmZddlmZmZmZmZddlmZmZdd lmZmZdd lmZdd lmZmZgd Z ed dddded e Dcgc]}|d c}dd gZ!e"ee#d<GddeZ$GddeZ%ycc}w)N) defaultdict)get_all_start_methods)Path)AnyIterableType)SnowballStemmer)define_cache_dir iter_batchget_all_punctuationremove_non_alphanumeric)ParallelWorkerPoolWorker)SparseEmbeddingSparseTextEmbeddingBase)SimpleTokenizer)SparseModelDescription ModelSource)arabicdanishdutchenglishfinnishfrenchgermangreek hungarianitalian norwegian portugueseromanianrussianspanishswedishtamilturkishz Qdrant/bm25z6BM25 as sparse embeddings meant to be used with Qdrantz apache-2.0g{Gz?)hf.txtTz mock.file) model vocab_size descriptionlicense size_in_GBsourcesadditional_files requires_idf model_filesupported_bm25_modelsceZdZdZ d$dededzdedededed ed ed edzd effd Z e de e fdZ e dedede efdZ d%dededeeezdededzded edzdeefdZ d&deeezdededzd edeef dZde ede efdZde ede efdZdeeezd edefdZde edeeeffdZe dedefdZd eeezd edeefd!Ze ded"fd#ZxZS)'Bm25aImplements traditional BM25 in a form of sparse embeddings. Uses a count of tokens in the document to evaluate the importance of the token. WARNING: This model is expected to be used with `modifier="idf"` in the sparse vector index of Qdrant. BM25 formula: score(q, d) = SUM[ IDF(q_i) * (f(q_i, d) * (k + 1)) / (f(q_i, d) + k * (1 - b + b * (|d| / avg_len))) ], where IDF is the inverse document frequency, computed on Qdrant's side f(q_i, d) is the term frequency of the token q_i in the document d k, b, avg_len are hyperparameters, described below. Args: model_name (str): The name of the model to use. cache_dir (str, optional): The path to the cache directory. Can be set using the `FASTEMBED_CACHE_PATH` env variable. Defaults to `fastembed_cache` in the system's temp directory. k (float, optional): The k parameter in the BM25 formula. Defines the saturation of the term frequency. I.e. defines how fast the moment when additional terms stop to increase the score. Defaults to 1.2. b (float, optional): The b parameter in the BM25 formula. Defines the importance of the document length. Defaults to 0.75. avg_len (float, optional): The average length of the documents in the corpus. Defaults to 256.0. language (str): Specifies the language for the stemmer. disable_stemmer (bool): Disable the stemmer. Raises: ValueError: If the model_name is not in the format / e.g. BAAI/bge-base-en. N model_name cache_dirkbavg_lenlanguagetoken_max_lengthdisable_stemmerspecific_model_pathkwargsc  t |||fi| |tvrt|d||_||_||_||_|j|} tt||_ | |_ |j| |j|j|j|_||_t#t%|_||_|r"t#|_d|_t2|_yt#|j/|j|j|_t1||_t2|_y)Nz language is not supported)local_files_onlyr=)super__init__supported_languages ValueErrorr:r7r8r9_get_model_descriptionstrr r6_specific_model_pathdownload_model_local_files_only _model_dirr;setr punctuationr< stopwordsstemmer_load_stopwordsr r tokenizer) selfr5r6r7r8r9r:r;r<r=r>model_description __class__s f/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/fastembed/sparse/bm25.pyrBz Bm25.__init__[s$ Y9&9 . .z)CDE E$DM  77 C-i89$7!--  NN!33 $ 9 9 . !1245. '*uDNDL )!!5!5doot}}!UVDN*84DL(returnctS)zLists the supported models. Returns: list[SparseModelDescription]: A list of SparseModelDescription objects containing the model information. )r2clss rT_list_supported_modelszBm25._list_supported_modelss %$rU model_dirc||dz }|jsgSt|d5}|jjcdddS#1swYyxYw)Nr(r)existsopenread splitlines)rYr[r:stopwords_pathfs rTrOzBm25._load_stopwordssN"z%66$$&I .# &!668&&(' & &s AA documents batch_sizeparallelr@c #ZKd}t|tr|g}d}t|trt||krd}||r+t ||D]} |j | Ed{y|dk(rt j}dtvrdnd} |||j|j|j|j|j|j||d } t|xsd|j!| } | j"t ||fi| D] } | D]} | y7ǭw) NFTr forkserverspawn) r5r6r7r8r9r:r;r<r@r=) num_workersworker start_method) isinstancerFlistlenr raw_embedos cpu_countrr7r8r9r:r;r<r_get_worker_class ordered_map)rQr5r6rdrerfr@r=is_smallbatchrmparamspoolrecords rT_embed_documentszBm25._embed_documentss- i %" IH i &9~ *  x#Iz:>>%000;1}<<>+7;P;R+R 1, data-parallel encoding will be used, recommended for offline encoding of large datasets. If 0, use all available cores. If None, don't use data-parallel processing, use default onnxruntime threading instead. Returns: List of embeddings, one per document )r5r6rdrerfr@r=N)r{r5rFr6rIrG)rQrdrerfr>s rTembedz Bm25.embedsS,(($..)!!33 $ 9 9)   sA AAAtokensc g}|D]}|j}||jvr"||jvr1t||jkDrJ|j r|j j |n|}|sv|j||SN)lowerrLrMrpr;rN stem_wordappend)rQr~stemmed_tokenstoken lower_token stemmed_tokens rT_stemz Bm25._stems$&E++-K(((dnn,5zD111CG< token_numtextrr~s rT token_countzBm25.token_countsW %eS1uD.t4H^^,,X6F V $IrUc Xi}tt}|D]}||xxdz cc<t|}|D]t}|j|}||}||jdzz||<||xx||jd|j z |j |z|j z zzzzcc<v|S)agCalculate the term frequency part of the BM25 formula. ( f(q_i, d) * (k + 1) ) / ( f(q_i, d) + k * (1 - b + b * (|d| / avg_len)) ) Args: tokens (list[str]): The list of tokens in the document. Returns: dict[int, float]: The token_id to term frequency mapping. rj)rintrpcompute_token_idr7r8r9)rQr~tf_mapcounterrdoc_lentoken_idnum_occurrencess rTrzBm25._term_frequencys$&)4S)9#M M "a ' "$f+$M,,];H%m4O.$&&1*=F8  8 $&&DFF TVVg- <<3!   % rUrc>ttj|Sr)absmmh3hash)rYrs rTrzBm25.compute_token_id6s499U#$$rUqueryc +xKt|tr|g}|D]}t|}jj |}j |}t jttfd|Dt j}t j|}t||yw)zTo emulate BM25 behaviour, we don't need to use weights in the query, and it's enough to just hash the tokens and assign a weight of 1.0 to them. c3@K|]}j|ywr)r).0rrQs rT z#Bm25.query_embed..FsR>%..u5>s)dtype)indicesvaluesN) rnrFr rPrrnparrayrorKint32 ones_liker)rQrr>rr~r token_idsrs` rT query_embedzBm25.query_embed:s eS !GED*40D^^,,T2F!ZZ/NSR>RRShhI\\),F!)FC CsB7B: Bm25WorkerctSr)rrXs rTrtzBm25._get_worker_classLsrU)Ng333333?g?gp@r(FN)NFN)rN)__name__ __module__ __qualname____doc__rFfloatrboolrrB classmethodrorrZrrOrrr{r}rrqrdictrrrrrt __classcell__)rSs@rTr4r4=s@!%! " %*..).):.)  .)  .)  .).).).)!4Z.).)`%t,B'C%%)))S ))#!&*./!/!/!#& /!  /! * /!/!!4Z/! / "/!h#  #&  *    / "  @DI$s)( 9  o  x}!4d3iDe4D<%S%S%%Dx}!4DDQ`HaD$$|"4rUr4c eZdZdededefdZededededdfdZdee e efdee e e e ffdZ ededededefd Zy ) rr5r6r>c 6|j||fi||_yr)init_embeddingr))rQr5r6r>s rTrBzBm25Worker.__init__Rs )T((YI&I rUrVc |d||d|SN)r5r6r)rYr5r6r>s rTstartzBm25Worker.startZsHjIHHHrUitemsc#bK|D]&\}}|jj|}||f(ywr)r)rq)rQridxrw onnx_outputs rTprocesszBm25Worker.process^s5 JC**..u5K{" " s-/c td||d|Sr)r4)r5r6r>s rTrzBm25Worker.init_embeddingesIzYI&IIrUN)rrrrFrrBrrrtuplerrorr staticmethodr4rrrUrTrrQsJJJ JIsIsIcIlII#eCHo.# %T/223 4#J3J3J#J$JJrUr)&rr collectionsrmultiprocessingrpathlibrtypingrrrrnumpyrpy_rust_stemmersr fastembed.common.utilsr r r r fastembed.parallel_processorrr&fastembed.sparse.sparse_embedding_baserr fastembed.sparse.utils.tokenizerr"fastembed.common.model_descriptionrrrCr2ro__annotations__r4r)langs0rTrs #1&& , D=R,L}-4GH4GDTF$-4GH  7t23 Q "QhJJwIs1 B'