pyc • Lines: 110�
���i � �� � d dl Z ej j d � j � G d� d� Z G d� d� Z G d� d� Z
G d � d
� Z y# e$ rZ ede� �� Y dZ[�>dZ[ww xY w) � NzGPU not found or error: c � � e Zd Zd� Zd� Zd� Zy)�Linearc � � || _ || _ t j j ||� t j
d|z � z | _ y )Ng @)� in_features� out_features�np�random�randn�sqrt�weight)�selfr r s �G/home/palash/git/iron_learn/python_scripts/transformer/simple_layers.py�__init__zLinear.__init__ s< � �&��� (����i�i�o�o�k�<�@�2�7�7�3�Q\�K\�C]�]�� � c �P � || _ t j || j � S �N)�inputsr �dotr �r
�xs r �forwardzLinear.forward s � ��� ��v�v�a����%�%r c �� � t j || j j � }t j | j j |� }| xj ||z z c_ |S r )r r r �Tr )r
� grad_output�
learning_rate�
grad_input� grad_weights r �backwardzLinear.backward sK � ��V�V�K���� � �7�
��f�f�T�[�[�]�]�K�8� � � � �}�{�2�2� ��r N��__name__�
__module__� __qualname__r r r � r r r r s � �^�
&�r r c � � e Zd Zd� Zd� Zd� Zy)�ReLUc � � d | _ y r ��input�r
s r r z
ReLU.__init__ s � ���
r c �< � || _ t j d|� S �Nr )r( r �maximumr s r r z ReLU.forward s � ���
��z�z�!�Q� � r c �* � d|| j dk < |S r+ r' )r
r s r r z
ReLU.backward s � �'(� �D�J�J�!�O�$��r Nr r# r r r% r% s � �� �r r% c � � e Zd Zd� Zd� Zd� Zy)�Softmaxc � � d | _ y r )�outputr) s r r zSoftmax.__init__% s � ��� r c � � t j |t j |dd�� z
� }|t j |dd�� z | _ | j S �N�����T)�axis�keepdims)r �exp�max�sumr1 )r
r �exp_xs r r zSoftmax.forward( sE � ����q�2�6�6�!�"�t�<�<�=���b�f�f�U��d� C�C�� ��{�{�r c �p � | j |t j || j z dd�� z
z }|S r3 )r1 r r9 )r
r r s r r zSoftmax.backward- s3 � ��[�[�K�"�&�&��t�{�{�9R�Y[�fj�2k�$k�l�
��r Nr r# r r r/ r/ $ s � ���
r r/ c � � e Zd Zd� Zd� Zd� Zy)�CrossEntropyLossc � � d | _ d | _ y r )� predictions�targetsr) s r r zCrossEntropyLoss.__init__2 s � � ����� r c �@ � || _ || _ | j j d }| j t j |� | j f }t j
|dd� }t j |dz � }t j |� |z }|S )Nr gV瞯�<g�������?)r? r@ �shaper �arange�clip�logr9 )r
r? r@ �m�p�log_likelihood�losss r r zCrossEntropyLoss.forward6 s� � �&��� �� � �L�L� � �q� !�� � � �R�Y�Y�q�\�4�<�<� 7� 8�� �G�G�A�u�i� (�� �&�&��U��+�+���v�v�n�%��)��� r c �� � | j j d }| j j � }|t j
|� | j fxx dz cc< ||z }|S )Nr � )r@ rB r? �copyr rC )r
rF �grads r r zCrossEntropyLoss.backward@ sY � � �L�L� � �q� !��� � �$�$�&�� �R�Y�Y�q�\�4�<�<�
'�(�A�-�(� �� ��� r Nr r# r r r= r= 1 s � � ��r r= ) �cupyr �cuda�Device�use� Exception�e�printr r% r/ r= r# r r �<module>rU st �� �*��G�G�N�N�1������ �
�
� � �� ��Y � *� � $�Q�C�
(�)�)��*�s �)A �A2� A-�-A2