GJj.ddlmZddlmZddlmZGddeZGddeZGdd eZ Gd d eZ Gd d eZ y))TupleN)ModulecteZdZdZ d dededeffd ZdZde jde jfd Z xZ S) InstanceNormaApplies instance normalization [1] on the inputs. Computes .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta, where :math:`\gamma` and :math:`\beta` are learned per feature dimension parameters initialized at 1 and 0 respectively. Both are of size :attr:`dims`, if :attr:`affine` is ``True``. Args: dims (int): The number of features of the input. eps (float): A value added to the denominator for numerical stability. Default: ``1e-5``. affine (bool): Default: ``False``. Shape: - Input: :math:`(..., C)` where :math:`C` is equal to :attr:`dims`. - Output: Same shape as the input. Examples: >>> import mlx.core as mx >>> import mlx.nn as nn >>> x = mx.random.normal((8, 4, 4, 16)) >>> inorm = nn.InstanceNorm(dims=16) >>> output = inorm(x) References: [1]: https://arxiv.org/abs/1607.08022 dimsepsaffinect||r6tj|f|_tj |f|_||_||_yN) super__init__mxonesweightzerosbiasrr)selfrrr __class__s e/Users/ahmed/devFolder/claude-voice/.venv/lib/python3.12/site-packages/mlx/nn/layers/normalization.pyr zInstanceNorm.__init__*sE  ''4'*DK$)DI cB|jd|jdd|vSN, eps= , affine=rrrrs r _extra_reprzInstanceNorm._extra_repr7'))F488*Ih$6F5GHHrxreturnc:ttd|jdz }tj||d}tj ||d}||z tj ||jzz}d|vr|j|z|jzS|S)NTaxiskeepdimsr) tuplerangendimrmeanvarrsqrtrrrrrreduction_axesr)r*s r__call__zInstanceNorm.__call__:suQ 34wwq~=ffQ^d; X#.1 108D0@ a$))+GaGr)h㈵>F) __name__ __module__ __qualname____doc__intfloatboolr rrarrayr. __classcell__rs@rrr sVF    IH"((HrxxHrrc DeZdZdZ d dedededeffd ZdZdZ xZ S) LayerNormaApplies layer normalization [1] on the inputs. Computes .. math:: y = \frac{x - E[x]}{\sqrt{Var[x]} + \epsilon} \gamma + \beta, where :math:`\gamma` and :math:`\beta` are learned per feature dimension parameters initialized at 1 and 0 respectively. [1]: https://arxiv.org/abs/1607.06450 Args: dims (int): The feature dimension of the input to normalize over eps (float): A small additive constant for numerical stability affine (bool): If True learn an affine transform to apply after the normalization bias (bool): If True include a translation to the affine transformation. If set to False the transformation is not really affine just scaling. rrr rct||r8tj|f|_|rtj |f|_||_||_yr ) r r rrrrrrr)rrrr rrs rr zLayerNorm.__init__]sH  ''4'*DKHHdW-  rcB|jd|jdd|vSrrrs rrzLayerNorm._extra_reprhrrcd|vr |jnd}d|vr |jnd}tjj ||||j S)Nrr)rrrfast layer_normr)rrrrs rr.zLayerNorm.__call__ksD (D 0d"dNtyyww!!!VT488<488<|j}|j^}}}|j|d|}tj|dd}tj |dd}||z tj ||jzz}|j|g||}|S)NrPr"Tr#)rKrErSrr)r*r+r)rrrKrUrVrmeansr*s r _group_normzGroupNorm._group_norms__ WWd IIeR ,D1ffQQ. Y"((3>2 2 AIIe )d )D )rc|jr |jn |j}||}d|vr|j|z|jzS|SrG)rLrXr[rr)rr group_norms rr.zGroupNorm.__call__sT&&  / /!!  qM08D0@ a$))+GaGr)r/TF) r0r1r2r3r4r5r6r rrXr[r.r8r9s@rrJrJs^B#( 555 5  5 ! 5"  HrrJc eZdZdZ ddededededef fd ZfdZd Z d e jd e e je jffd Z d e jd e jfd ZxZS) BatchNormaWApplies Batch Normalization over a 2D or 3D input. Computes .. math:: y = \frac{x - E[x]}{\sqrt{Var[x]} + \epsilon} \gamma + \beta, where :math:`\gamma` and :math:`\beta` are learned per feature dimension parameters initialized at 1 and 0 respectively. The input shape is specified as ``NC`` or ``NLC``, where ``N`` is the batch, ``C`` is the number of features or channels, and ``L`` is the sequence length. The output has the same shape as the input. For four-dimensional arrays, the shape is ``NHWC``, where ``H`` and ``W`` are the height and width respectively. For more information on Batch Normalization, see the original paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `_. Args: num_features (int): The feature dimension to normalize over. eps (float, optional): A small additive constant for numerical stability. Default: ``1e-5``. momentum (float, optional): The momentum for updating the running mean and variance. Default: ``0.1``. affine (bool, optional): If ``True``, apply a learned affine transformation after the normalization. Default: ``True``. track_running_stats (bool, optional): If ``True``, track the running mean and variance. Default: ``True``. Examples: >>> import mlx.core as mx >>> import mlx.nn as nn >>> x = mx.random.normal((5, 4)) >>> bn = nn.BatchNorm(num_features=4, affine=True) >>> output = bn(x) num_featuresrmomentumr track_running_statsczt|||_||_||_||_|r6t j|f|_t j|f|_ |j rLt j|f|_ t j|f|_ |jddgdyy)N running_mean running_varFkeysrecurse)r r r`rrarbrrrrrrdrefreeze)rr`rrar rbrs rr zBatchNorm.__init__s (  #6 '' rrr cttd|jdz }tj||}tj ||}||fS)z Calculate the mean and variance of the input tensor across the batch and spatial dimensions. Args: x (array): Input tensor. Returns: tuple: Tuple containing mean and variance. rr")r$)r&r'r(rr)r*r,s r _calc_statszBatchNorm._calc_stats>sGuQ 34wwq~.ffQ^,Syrc0|jdks|jdkDrtd|j|j|\}}|jrS|jrG|j }d|z |j z||zz|_d|z |jz||zz|_n$|jr|j }|j}||z tj||jzz}d|vr|j|z|jzS|S)z Forward pass of BatchNorm. Args: x (array): Input tensor. Returns: array: Normalized output tensor. rQzr{sZ%9H69Hx)=)=X=f=B[H[H|{H{Hr