pyc • Lines: 514�
�L�i�9 � �r � d dl Zd dlmZmZmZ ej j d � j � e d� G d� d� Z G d� d� Z
G d � d
� Z G d� d� Z G d
� d� Z G d� d� Z G d� d� Z G d� d� Z G d� d� Zy# e
$ rZ e de� �� Y dZ[�pdZ[ww xY w)� N)�Linear�ReLU�Softmaxz(GPU found and will be used for training.zGPU not found or error: c � � e Zd Zd� Zd� Zd� Zy)� Embeddingc � � || _ || _ t j j ||� t j
d|z � z | _ y )N� @)�
vocab_size�d_model�np�random�randn�sqrt�weight)�selfr
r s �E/home/palash/git/iron_learn/python_scripts/transformer/transformer.py�__init__zEmbedding.__init__ s9 � �$�������i�i�o�o�j�'�:�R�W�W�S�:�EU�=V�V��� c �Z � |j dk( sJ d� �|| _ | j | S )N� �+Input should be a 1D array of token indices)�ndim�inputsr �r �xs r �forwardzEmbedding.forward s- � ��v�v��{�I�I�I�{�����{�{�1�~�r c �v � t j j | j | j | |z � y �N)r �add�atr r �r �grad_output�
learning_rates r �backwardzEmbedding.backward s( � �
��� � �$�+�+�t�{�{�]�N�[�,H�I�r N��__name__�
__module__�__qualname__r r r$ � r r r r
s � �W�
�
r r c � � e Zd Zd� Zd� Zd� Zy)�PositionalEncodingc � � || _ || _ t j j | j | j � dz | _ y )Ng{�G�z�?)�embedding_dimension�sequence_lengthr r
r r )r r- r. s r r zPositionalEncoding.__init__ s: � �#6�� �.����i�i�o�o�d�&:�&:�D�<T�<T�U�X\�\��r c � � |j dk( r|j d | j k( sJ d� �|| _ || j d |j d �d d �f z S )N� r z?Input should be of shape (sequence_length, embedding_dimension)r )r �shaper- r r r s r r zPositionalEncoding.forward sc � ��v�v��{�q�w�w�q�z�T�-E�-E�E� I� HI� I�E�����4�;�;�{�����
�{�A�~�.�.�.r c �Z � |j d }| j d |xxx ||z z ccc |S �Nr )r1 r )r r"