^Nj:;ddlZddlmZddlmZddlmZeezZdZ eje dDcgc]}e |jdc}ejZd ej d ej fd ZGd d ZGddZedk(rBej*j-dddZedddddZeD]Zej5eyycc}w)N) NumpyArray) LateInteractionTextEmbeddingBase)&LateInteractionMultimodalEmbeddingBaseA1)dtypeidsreturnct|}tj|dddf|dddf}|jtjj ||d}t |jdS)zCompute full Hamming distance matrix Args: ids: shape (n,) - array of ids, only size of the array matters Return: np.ndarray (n, n) - hamming distance matrix Naxis)lennp bitwise_xorviewuint8reshape POPCOUNT_LUTsum)r nxor_vals bytes_views m/Users/ahmed/devFolder/Ultron/claude-voice/.venv/lib/python3.12/site-packages/fastembed/postprocess/muvera.pyhamming_distance_matrixrsg CA~~c!T'lCaL9Hrxx(00Aq9J  # ' 'Q ' //ceZdZdZdededej jfdZdejdejfdZ y ) SimHashProjectiona  SimHash projection component for MUVERA clustering. This class implements locality-sensitive hashing using random hyperplanes to partition the vector space into 2^k_sim clusters. Each vector is assigned to a cluster based on which side of k_sim random hyperplanes it falls on. Attributes: k_sim (int): Number of SimHash functions (hyperplanes) dim (int): Dimensionality of input vectors simhash_vectors (np.ndarray): Random hyperplane normal vectors of shape (dim, k_sim) k_simdimrandom_generatorcR||_||_|j||f|_y)a@ Initialize SimHash projection with random hyperplanes. Args: k_sim (int): Number of SimHash functions, determines 2^k_sim clusters dim (int): Dimensionality of input vectors random_generator (np.random.Generator): Random number generator for reproducibility sizeN)r!r"normalsimhash_vectors)selfr!r"r#s r__init__zSimHashProjection.__init__.s, /66S%L6Irvectorsr ct||jz}|dkDdtj|jzz}|S)a$ Compute the cluster IDs for a given vector using SimHash. The cluster ID is determined by computing the dot product of the vector with each hyperplane normal vector, taking the sign, and interpreting the resulting binary string as an integer. Args: vectors (np.ndarray): Input vectors of shape (n, dim,) Returns: np.ndarray: Cluster IDs in range [0, 2^k_sim - 1] Raises: AssertionError: If a vector shape doesn't match expected dimensionality r)r(raranger!)r)r+ dot_product cluster_idss rget_cluster_idsz!SimHashProjection.get_cluster_ids<s?$ d** * #Q1 $**0E+EF rN) __name__ __module__ __qualname____doc__intrrandom Generatorr*ndarrayr1rrr r sJ  Jc J Jryy?R?R Jrzzbjjrr ceZdZdZ ddededededef dZe ddededededed df d Zd efd Z e d efd Z d e d e fdZ d e d e fdZ dd e deded e fdZy)Muveraa MUVERA (Multi-Vector Retrieval Architecture) algorithm implementation. This class creates Fixed Dimensional Encodings (FDEs) from variable-length sequences of vectors by using SimHash clustering and random projections. The process involves: 1. Clustering vectors using multiple SimHash projections 2. Computing cluster centers (with different strategies for docs vs queries) 3. Applying random projections for dimensionality reduction 4. Concatenating results from all projections Attributes: k_sim (int): Number of SimHash functions per projection dim (int): Input vector dimensionality dim_proj (int): Output dimensionality after random projection r_reps (int): Number of random projection repetitions random_seed (int): Random seed for consistent random matrix generation simhash_projections (List[SimHashProjection]): SimHash instances for clustering dim_reduction_projections (np.ndarray): Random projection matrices of shape (R_reps, d, d_proj) r"r!dim_projr_reps random_seedch||kDrtd|d|d||_||_||_||_t j j|}t|Dcgc]$}t|j|j|&c}|_ |jddg|||f|_ ycc}w) aQ Initialize MUVERA algorithm with specified parameters. Args: dim (int): Dimensionality of individual input vectors k_sim (int, optional): Number of SimHash functions (creates 2^k_sim clusters). Defaults to 5. dim_proj (int, optional): Dimensionality after random projection (must be <= dim). Defaults to 16. r_reps (int, optional): Number of random projection repetitions for robustness. Defaults to 20. random_seed (int, optional): Seed for random number generator to ensure reproducible results. Defaults to 42. Raises: ValueError: If dim_proj > dim (cannot project to higher dimensionality) z4Cannot project to a higher dimensionality (dim_proj=z > dim=))r!r"r#r-r%N) ValueErrorr!r"r=r>rr7 default_rngranger simhash_projectionschoicedim_reduction_projections)r)r"r!r=r>r? generator_s rr*zMuvera.__init__js2 c>FxjPWX[W\\]^     II))+6 6]$ " DJJDHHy Y"$  *3)9)92q'QTV^H_)9)`& $ s )B/modelr c0||j||||S)a Create a Muvera instance from a multi-vector embedding model. This class method provides a convenient way to initialize a MUVERA that is compatible with a given multi-vector model by automatically extracting the embedding dimensionality from the model. Args: model (MultiVectorModel): A late interaction text or multimodal embedding model that provides multi-vector embeddings. Must have an `embedding_size` attribute specifying the dimensionality of individual vectors. k_sim (int, optional): Number of SimHash functions (creates 2^k_sim clusters). Defaults to 5. dim_proj (int, optional): Dimensionality after random projection (must be <= model's embedding_size). Defaults to 16. r_reps (int, optional): Number of random projection repetitions for robustness. Defaults to 20. random_seed (int, optional): Seed for random number generator to ensure reproducible results. Defaults to 42. Returns: Muvera: A configured MUVERA instance ready to process embeddings from the given model. Raises: ValueError: If dim_proj > model.embedding_size (cannot project to higher dimensionality) Example: >>> from fastembed import LateInteractionTextEmbedding >>> model = LateInteractionTextEmbedding(model_name="colbert-ir/colbertv2.0") >>> muvera = Muvera.from_multivector_model( ... model=model, ... k_sim=6, ... dim_proj=32 ... ) >>> # Now use postprocessor with embeddings from the model >>> embeddings = np.array(list(model.embed(["sample text"]))) >>> fde = muvera.process_document(embeddings[0]) )r"r!r=r>r?)embedding_size)clsrKr!r=r>r?s rfrom_multivector_modelzMuvera.from_multivector_models'`$$#   rcXd|jz}|j|z|jzS)z Get the output dimension of the MUVERA algorithm. Returns: int: Output dimension (r_reps * num_partitions * dim_proj) where b = 2^k_sim r)r!r>r=)r)num_partitionss r_get_output_dimensionzMuvera._get_output_dimensions)DJJ{{^+dmm;;rc"|jS)N)rR)r)s rrMzMuvera.embedding_sizes))++rr+c*|j|ddS)a Encode a document's vectors into a Fixed Dimensional Encoding (FDE). Uses document-specific settings: normalizes cluster centers by vector count and fills empty clusters using Hamming distance-based selection. Args: vectors (NumpyArray): Document vectors of shape (n_tokens, dim) Returns: NumpyArray: Fixed dimensional encodings of shape (r_reps * b * dim_proj,) Tfill_empty_clustersnormalize_by_countprocessr)r+s rprocess_documentzMuvera.process_documents||GRV|WWrc*|j|ddS)a Encode a query's vectors into a Fixed Dimensional Encoding (FDE). Uses query-specific settings: no normalization by count and no empty cluster filling to preserve query vector magnitudes. Args: vectors (NumpyArray]): Query vectors of shape (n_tokens, dim) Returns: NumpyArray: Fixed dimensional encoding of shape (r_reps * b * dim_proj,) FrUrXrZs r process_queryzMuvera.process_querys||GSX|YYrrVrWc :|jd|jk(s!Jd|jd|jg}d|jz}tj|}|r t |nd}t |jD]\}} tj||jf} |D cic]} | g} } d} d}| j|}t|t |D])\}\}}| |xx|z cc<| |j|+|s|rtj||} | dk(}|r"|J| J|}| |xx| |dddfzcc<|r|J|Jtj|dddft|}tj|d}tj ||Dcgc] }|| |dc}j#d |j}|| |<|j$|jkrX|j&|}dtj(|j$z | |zz}|j|j+|j| j+tj,|Scc} wcc}w) a Core encoding method that transforms variable-length vector sequences into FDEs. The encoding process: 1. For each of r_reps random projections: a. Assign vectors to clusters using SimHash b. Compute cluster centers (sum of vectors in each cluster) c. Optionally normalize by cluster size d. Fill empty clusters using Hamming distance if requested e. Apply random projection for dimensionality reduction f. Flatten cluster centers into a vector 2. Concatenate all projection results Args: vectors (np.ndarray): Input vectors of shape (n_vectors, dim) fill_empty_clusters (bool): Whether to fill empty clusters using nearest vectors based on Hamming distance of cluster IDs normalize_by_count (bool): Whether to normalize cluster centers by the number of vectors assigned to each cluster Returns: np.ndarray: Fixed dimensional encoding of shape (r_reps * b * dim_proj) where B = 2^k_sim is the number of clusters Raises: AssertionError: If input vectors don't have expected dimensionality r-zExpected vectors of shape (n, z), got rN) minlengthrrrB)shaper"r!rr.r enumeraterFzerosr1zipappendbincountwhereMAX_HAMMING_DISTANCEargminarrayrr=rHsqrtflatten concatenate)r)r+rVrWoutput_vectorsrQcluster_center_idsprecomputed_hamming_matrixprojection_indexsimhashcluster_centerscluster_center_idcluster_center_id_to_vectorscluster_vector_counts empty_maskvector_cluster_ids cluster_idvec_idxvecnon_empty_maskmasked_hammingnearest_non_empty fill_vectorsdim_reduction_projectionprojected_centerss rrYzMuvera.processsD MM!  ( M +DHH:WW]]O L M (DJJYY~6;N #$6 7TX #*343K3K)L % g hh'ABO?QB?Q*;!2%?Q )B%) !J")!8!8!A .12DiPWFX.Y* NWc +s2+,Z8??H/Z"%8(* 4FR`(a%2a7 !!---,888",/3H3XYZ\`Y`3aa/#!---1===!#tQw')=?Y"%'IIn1$E!!xx+>%! %%&7&?&?&AB  ! !/"9"9"; s 4 J3J N)*)TT)r2r3r4r5r6r* classmethodMultiVectorModelrOrRpropertyrMrr[r]boolrYr:rrr<r<Ts90 )a )a)a )a  )a  )aV 5 5 5  5  5  5  5 5 nrs- 46\\rxxE#J?JqQc*J?rxxP  0 0 011hN.N.b z YY__Rc *F CAr2 &Fu o @sC