pyc • Lines: 115�
��i:
� �� � d dl Z d dlZd dlZd dlmZ d dlmZm Z ej j d � j � e
d� G d� d� Z G d� d � Zy# e$ rZ e
de� �� Y dZ[�*dZ[ww xY w)
� N)� Transformer)�Softmax�CrossEntropyLossz(GPU found and will be used for training.zGPU not found or error: c � � e Zd Zd� Zd� Zd� Zy)�CorpusTokenizerc �� � |j � }|D �cg c]# }|j � s�|j � ��% }}g d�| _ t � }|D ]! }|j |j � � �# | j t
t |� � z | _ t | j � D ��ci c] \ }}||��
c}}| _
t | j � D ��ci c] \ }}||��
c}}| _ | j d | _ | j d | _
| j d | _ t | j � | _ y c c}w c c}}w c c}}w )N��<PAD>�<SOS>�<EOS>r
r r )�
splitlines�strip�special_tokens�set�update�split�sorted�list�vocab� enumerate�
char_to_id�
id_to_char�pad_id�sos_id�eos_id�len�
vocab_size)�self� corpus_text�lines�line�words�i�ws �D/home/palash/git/iron_learn/python_scripts/transformer/tokenizers.py�__init__zCorpusTokenizer.__init__ s � ��&�&�(��*/�@�$�4�:�:�<���� �@��@� :��� ���� '�D� �L�L���� � &� '� �(�(�6�$�u�+�+>�>��
�,5�d�j�j�,A�B�D�A�q�1�a�4�B���,5�d�j�j�,A�B�D�A�q�1�a�4�B����o�o�g�.�� ��o�o�g�.�� ��o�o�g�.�� � �d�j�j�/����' A�� C��Bs �E�E�6
E�$
E c �l � |j � }| j g|D �cg c] }|| j v s�| j | ��" c}z | j gz }t |� |k r || j
g|t |� z
z z
}n|d | }t
j |t j �� S c c}w �N)�dtype) r r r r r r �np�array�int32)r �text�seq_lenr" r$ �tokenss r% �encode_sentencez CorpusTokenizer.encode_sentence' s� � ��
�
� �� �+�+� �e�!\��q�D�O�O�G[�$�/�/�!�"4�!\�\�`d�`k�`k�_l�l�� �v�;�� � �t�{�{�m�w��V��'<�=� =�F��H�W�%�F��x�x��b�h�h�/�/�� "]s
�B1�B1c �� � t |d� r|j � }dj |D �cg c]) }t |� dkD s�| j t |� ��+ c}� S c c}w )N�get� � ��hasattrr2 �join�intr )r �idsr# s r% �decodezCorpusTokenizer.decode6 sJ � � �3�� �c�g�g�i���x�x�#�L�Q��Q��!��� � ��Q��0�L�M�M��L� �A!� A!N��__name__�
__module__� __qualname__r& r0 r: � � r% r r s � �*�0
0� NrA r c � � e Zd Zd� Zd� Zd� Zy)�
CharTokenizerc �� � g d�| _ d| _ d| _ d| _ t t t
|� � � }t | j � D ��ci c] \ }}||��
c}}| _ t |� D ]* \ }}|t | j � z | j |< �, | j j � D ��ci c] \ }}||��
c}}| _ t | j � | _ y c c}}w c c}}w )Nr r � r4 )
r r r r r r r r r r �itemsr r )r r- � unique_charsr# �tok�ch�k�vs r% r& zCharTokenizer.__init__< s� � � 9����� ��� ��� � �d�3�t�9�o�.� � 1:�$�:M�:M�0N�O�f�a��3��6�O��� �|�,� ?�E�A�r�"#�c�$�*=�*=�&>�">�D�O�O�B� � ?� -1�O�O�,A�,A�,C�D�D�A�q�1�a�4�D��� �d�o�o�.���� P�� Es �
C0�?
C6c � � |D �cg c] }|| j v s�| j | ��" }}t |� |k r || j g|t |� z
z z
}n|d | }t j |t j
�� S c c}w r( )r r r r* r+ r, )r r- r. rI �indicess r% r0 z CharTokenizer.encode_sentenceN sz � �15�O�2��t���9N�4�?�?�2�&�O��O� �w�<�'� !� �� � �}��#�g�,�(>�?� ?�G� �h�w�'�G��x�x��r�x�x�0�0��
Ps
�B�Bc �� � t |d� r|j � }dj |D �cg c]) }t |� dkD s�| j t |� ��+ c}� S c c}w )Nr2 � r4 r5 )r rM r# s r% r: zCharTokenizer.decodeX sM � � �7�E� "� �k�k�m�G� �w�w��O�A�C��F�Q�J�����A��/�O�P�P��Or; Nr< r@ rA r% rC rC ; s � �/�$1�QrA rC )�time�cupyr* �numpy�onp� transformerr �
simple_layersr r �cuda�Device�use�print� Exception�er rC r@ rA r% �<module>r\ sz �� � � � #� 3�*��G�G�N�N�1����� �
4�5�+N� +N�Z"Q� "Q��a � *� � $�Q�C�
(�)�)��*�s �1A" �"A<�' A7�7A<