$HJjB 8UddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZd dlmZe rddl m!Z!ddl"m#Z#ddl$m%Z%er ddl&Z&ddl'm(Z(ejRe*Z+dedededZ,erddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6ddl-m7Z8ddl-m9Z:ddl-m;Z<ddl-m=Z>ej~jddjZBeBevZCdeDfdZ;deDfd Z=daEeFdzeGd!<d"eFfd#ZHd"eFfd$ZIdSd%ZJn/d&ZCd'Z;d(Z=Gd)d*Z1Gd+d,Z2Gd-d.Z0d/Z5d0Z4d1ZJd2d d3d4d d3d4d d3d5d6d3d7dd3d8d9d:d;ZKeFeDeFeDeDffeGd<<iZLeFeDe dzfeGd=<d>eDdzd"eMfd?ZN dTd>eDd@edzdAeMd"e dzfdBZOeLfdCeDdDeFeDe dzffdEZPdSdUdFZ9 dVdCeDdGeDdzdHeQeDzdzdAeMd"e f dIZ7dJeReezfdKZSd&aTedLZUdMeVdNeReDdOeVd"eVfdPZW dWdQZXgdRZYy)XN)Callable)contextmanager)Path) ModuleType) TYPE_CHECKING)!get_checkpoint_conversion_mapping®ister_checkpoint_conversion_mapping)register_patch_mapping)ENV_VARS_TRUE_VALUESlogging)is_flash_attention_requested)KERNELS_MAX_VERSIONKERNELS_MIN_VERSIONis_kernels_availableis_rocm_platformis_torch_available)flash_attention_forward)PretrainedConfig)PreTrainedModel) KernelConfigzh`kernels` is either not installed or uses an incompatible version. Please install a compatible version (z <= version < z), e.g. `pip install kernels==`) CUDAPropertiesDeviceFuncRepositoryLayerRepositoryLocalLayerRepositoryModeregister_kernel_mappingreplace_kernel_forward_from_hubuse_kernel_mapping) get_kernel) kernelize)use_kernel_forward_from_hub)use_kernel_func_from_hubUSE_HUB_KERNELSYES layer_namecbtr t|StjdtdS)NFkernels hub usage is disabled through the environment USE_HUB_KERNELS=c|SNclss o/Users/ahmed/devFolder/claude-voice/.venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.pyz-use_kernel_forward_from_hub..Zss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)r)s r1r%r%Ss2 7 C C   XYvXwx # "r3 func_namecbtr t|StjdtdS)Nr+c|Sr-r.funcs r1r2z*use_kernel_func_from_hub..csr3)r4!_kernels_use_kernel_func_from_hubr6r7r8)r9s r1r&r&\s2 4Y? ?   XYvXwx % $r3_KERNEL_MAPPING_CACHEreturncdtddditdtddtd d d d idtjtj ztdddtj tj ztdddiidtjtj ztdddtj tj ztdddiidtj tj ztdddiitj tdddtjtddditj tdddtjtddditjtddditjtddditj tdddtjtdddidtj tdddtjtddditj tdddtjtddditjtddditjtdddiddtjtj ztdddiidtjtj ztdd diidtjtj ztdd!diidtjtj ztdd"diidtjtj ztdd#diidtjtj ztdd$diid% }tjtd&d'd(itd&d'd(tjtd)d'd(id*dtj tj ztdd+d(iid,}||z}|S)-Ncudaz!kernels-community/deformable-detrMultiScaleDeformableAttentionr)repo_idr)versiony)min_capabilitymax_capability)type propertieszAtlas-Inference/gdnQwen3_5GatedDeltaNet(ef12347fc77d6ddf1cb72c0bd0af1c7d6cc69172T)rDr)revisiontrust_remote_codezkernels-community/liger-kernelsLigerSwiGLUMLPrLigerTiledSwiGLUMLP LigerGEGLUMLPLigerTiledGEGLUMLP LigerLinear LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)rBrocmxpumpsnpuzkernels-community/megablocksMegaBlocksMoeMLPCPUMegaBlocksMoeMLP)rBrVrWcpuzkernels-community/activationFastGELU QuickGELUNewGELUSiluGeluGeluTanh) rCrK SwiGLUMLPGeGLUMLPLinearrUrZr]r^r_SiLUGeLUrbzkernels-community/rotaryapply_rotary_transformers)rDr9rEzkernels-community/aiter-rope)rWrBrVLigerForCausalLMLoss)rotary_pos_embForCausalLMLoss)rrrr INFERENCE TORCH_COMPILETRAININGr)_KERNEL_MAPPING_FUNCTION_KERNEL_MAPPINGs r1_build_kernel_mappingrqks?>."-SQTU#15G&*  %NNT%7%77 A#3 !: MMD$6$66 A#8 !9  NNT%7%77 A#2 !: MMD$6$66 A#7 !9  MMD$6$66 A#0 !9MM? A#1 !$ NNO A#1 !% MM? A#1 !$ NNO A#1 !% NNO ;#, !%NNO ?#, !%MM? A#1 !$ NNO A#1 !% Q4nMM? >#5 !$ NNO >#5 !% MM? >#5 !$ NNO >#5 !% NNO >#5 !%NNO >#8 !%A'!RNNT%7%77 >#- !:NNT%7%77 >#. !:NNT%7%77 >#, !:NNT%7%77 >6[\:NNT%7%77 >6[\:NNT%7%77 >:_`:QOi jNNN :Fakl% '6B]ghNNN >Jeop% MMD$6$66 AMcmn9 !$  0*,DDr3c.t tatS)zPReturn the default transformers kernel mapping, building it lazily on first use.)r?rqr.r3r1get_kernel_mapping_transformersrsZs ! ($9$; !$$r3c2| t}t|yr-)rsr )mappings r1$register_kernel_mapping_transformersrvas ?57G(r3Fc d}|S)Nc|Sr-r.r/s r1 decoratorz.use_kernel_forward_from_hub..decoratorlsJr3r.argskwargsrys r1r%r%ks r3c d}|S)Nc|Sr-r.r<s r1ryz+use_kernel_func_from_hub..decoratorrsKr3r.rzs r1r&r&qs r3ceZdZdZy)rctd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeErrorselfr{r|s r1__init__zLayerRepository.__init__xsop pr3N__name__ __module__ __qualname__rr.r3r1rrws qr3rceZdZdZy)rctd)NzSLocalLayerRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzLocalLayerRepository.__init__|stu ur3Nrr.r3r1rr{s vr3rceZdZdZy)rctd)NzMFuncRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzFuncRepository.__init__sno or3Nrr.r3r1rrs pr3rctd)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rr{r|s r1r!r!s l  r3ctd)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rrs r1r r ssttr3ctd)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rrs r1rvrvs q  r3zkernels-community/causal-conv1d)rDrEzkernels-community/mamba-ssmz!kernels-community/finegrained-fp8zkernels-community/deep-gemmzkernels-community/sonic-moez ep-support)rDrM)z causal-conv1dz mamba-ssmzfalcon_mamba-ssmzfinegrained-fp8z deep-gemmz sonic-moe_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationc>|duxrtjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rs r1 is_kernelrs- 4' c II@BU V^b br3attention_wrapperallow_all_kernelscddlm}ddlm}d|vr|j ddn|}t |syt sttd|vr%|j d\}}|j}n|}d}|j}|jd\}}} |j}| r| jnd} d} | t|d } | rd nd} t|| | | } d}t| dr |t }|}n)t| drddlm}||n|}d}n| t'| |}|j)||j)|||| S#t$rt$r} td |d| dd} ~ wwxYw)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continuous batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|rN:@)"requested_attention_implementationrErrMrErz-An error occurred while trying to load from 'z': .flash_attention_2flash_attn_varlen_funcsparse_atten_func)msa_attention_forwardsdpa) masking_utilsrmodeling_utilsrsplitrr ImportError_MISSING_KERNELS_MESSAGEstrip partitionrr# ValueError Exceptionhasattrr msa_attentionrgetattrregister)rrrrractual_attn_namerD kernel_name_revrEis_fa4kernelemask_implementationkernel_functionrs r1load_and_register_attn_kernelrs =8.attach_hidden_kernels,s{(92>DDFHD"4 5 5 788!"bii0$=dVDBB &&tR0Gr3cZt|diD]}t||st||y)Nr)rrdelattr)rrs r1detach_hidden_kernelsz(kernelize..detach_hidden_kernels7s,F$5r:Dvt$% ;r3NrBrV)rI)inherit_mapping)devicemodeT)rrrapplytrainingrrlrnrrIrr kernel_configuse_local_kernelr"kernel_mapping_kernels_kernelize _use_kernels)modelrrr device_typerrs r1r$r$'s  !233 1&+ )*%*^^t~~$,\`ll'' & %5%7 K[)    *"'"5"5"F"FFO#E$7$7$F$FXgh"5dCih uV$ ?! )*ih  )*s$B8D$D(D$D!D$$D7rMrEclddlm}tsttd||d}t |||||S)Nr) __version__ transformers) frameworkrErD)rMrE user_agentrN)rrrrget_kernel_hub)rrMrErrrs r1r#r#Rs?   !233-+R]^J hJbs r3 module_namesc8ttrgfd}|S)aO This decorator attaches the target function within the module as a plain attribute (not as a submodule). Keep in mind that this registration is only meant for `kernelize` to recognize its target modules (i.e. function exchanged for a weightless `nn.Module` with the same forward) to then exchange to the kernel variation (in-place) if the conditions are met. We cache each of these function-based registrations: After proper registration and exchange it is removed from the module's `_modules` dict as it does not really act as `nn.Module` but a base function. c<|jfd}||_|S)Nc|g|i||jjdi}D]B}t|ddxst|ddxs t|dd}|td||||<Dy)Nrrkernel_layer_namer9z)Could not infer kernel function name for )__dict__ setdefaultrr)rr{r|hidden_kernelsrrr orig_inits r1new_initz8use_kernelized_func..decorator..new_initss d ,T ,V ,"]]556GLN"B D16r#6=6r;5 <$'PQSPV%WXX(*t$#r3)r)r0rrrs @r1ryz&use_kernelized_func..decoratorpsLL  *"   r3)rr)rrys` r1use_kernelized_funcrcs#,)$~ . r3c#,K dadday#dawxYww)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)rr.r3r1allow_all_hub_kernelsrs""  "Es  parent_cls child_names kernel_clsc|jfd}td|j|fd|i}d|j|_|S)a6 Create a new class that inherits from `parent_cls` and fuses the child modules specified in `child_names with the provided `kernel_cls`. The first child in `child_names` will be replaced with the `kernel_cls`, and the rest will be replaced with `nn.Identity()` to keep the same interface. c|g|i|Dcgc]}t||}}|}t|d|ddD]!}t||tj#ycc}w)Nrr)rsetattrrIdentity) rr{r|rchildrenkernel_instancerr original_inits r1 patched_initz9make_parent_class_for_kernel_fusion..patched_initsrd,T,V,4?@KDGD$'K@$h/k!no6OD D$ .$AsA&Fusedr)rrIrr)rrrr patched_clsr s `` @r1#make_parent_class_for_kernel_fusionrsW''M/z2234zmjR^E_`K!&z'>'>&?@K r3c  #t|drt|jdstd|jd|jj}i}i}d}|j j D]n\}}t|ttfrd|i}t|tr*t|jdk7rtd|td|d|tt|j}t|trY|\} } | jd d} | jd d} | jd d xst } | d | ind | i} | d | iz} | | f}n|} dt d } | | f}| j#d\}}}|r|std| d||j$r;|j'dj)dd}t+t-|||}nt/d+||d| }|j1}|td|t2j4j|j6}|rt9||jddnd}|:d|j:vr,t=j>|j@d}||_ t|tr0||||<A|j|_!|||<|||j<nt|tr|&td|jd|jd|j|_!|Dcgc]}|d }}|Dcgc]}|jEddd}}ttG|dk7rtd||djIdd}|Dcgc]}|jEddd}}|&tKjLd 5||}dddd }|jOD]u\}#tQjR||stU#fd!|Dstd"|d#|d$|jd%}tW#} tY| |||| j<w|std&|d'|jd(t[|d%)t|d*r5t]|}!t_|j`}"|!|!|"z}"tc||"d%)|||j<q||_ycc}wcc}wcc}w#1swY)xYw),N config_class model_typezModel z# has no config_class or model_type.rzLExpected exactly one kernel repo regardless of device/mode specificity, got zInvalid hub repo z for layer rMrErNF)rErNrzInvalid kernel repo string /) repo_path package_namer))rDr)z*Could not load kernel class from hub_repo=Layoutforwardcyr-r.rs r1 _noop_forwardz?register_kernel_replacements_and_fusions.._noop_forwardsr3z Fused kernel z* requires a companion layout class named 'zLayout' in the same module.rrzGAll patterns for a fused kernel must share the same parent module, got *z\w+metac36K|]}t|ywr-)r).0childrs r1 z;register_kernel_replacements_and_fusions..4sK{e7651{szModule z* does not have the expected child modules z required for the fused kernel TzNo module matched pattern z for fused kernel z3. Provide the full dotted path from the model root.) overwriteconversion_mappingr.)2rrrrrrrrstrtuplerlenvaluesnextiterrrrrrstriprrrrloadsysmodulesrrr functoolswrapsrrrsplitsetrtorchr named_modulesr fullmatchallrIrr r listr$r )$r0configrr patch_mapping new_mapping meta_modelr)hub_reporepo_strmetadatarMrErN final_reporDrlayer_name_in_reporrepor kernel_mod layout_clsritem glob_patternspparent_patternsparent_patternr matched_anyr module_clsexisting transformsrs$ @r1(register_kernel_replacements_and_fusionsrMs 3 'ws7G7G/V6#,,/RSTT!!,,J%'MKJ - < < B B D H he -h'H h %8??$%* bckblm0 K ~VW WX__./0 h &!) Hh||J5Hll9d3G ( -@% H ]L] /6/B 7+U]H^H ,.?@ @H"H-JH#$;LMH"H-J)1););C)@&&0:8,kR\Q_`a a  ) )">>#.44S9"=L'w-)-D #-D YY[  I(VW W[[__Z%:%:; R\WZJ,?,?+@)GNbf  !iz7J7J&J __Z// 0 1 "/J  j# &!*4 J',6+>+>J ((2M* %0:K ++ , E *! #J$7$7#:;(1122MO ,6+>+>J (1;<T!WM<environrupperr8r4r%r?r__annotations__rqrsrvrrboolrrrintr7rrrrIrrM__all__r.r3r1rdsN $% j4185602   H %N+>*??]^q]rrsu    V$&JJNN3De$L$R$R$T!48LL##%C%*.4$;-m4m^%T%)   qqvvpp u "CqQ:qI$AaP#FSTU:qI: U 2T#tCH~-.8:S*t"33493:$diKK19DK\`K$K\Pf1 #1 S*t:K5K0L1 h(+Z $# Dj3Y    "$d8nx&?$P " " c 4W/ W/ W/"W/ W/t r3