📄 llm.cpython-312.pyc
/home/palash/git/local-ai/server/features/__pycache__/llm.cpython-312.pyc
Language: pyc • Lines: 782
�

�]�j�Q����dZddlZddlZddlZddlZddlmZddd�Zdd�Zdd�Z	dd	�Z
dd
�Zdd�Zd�Z
dd
edefd�Zd�Zd�Zd�Zd�Zd�Zd�Zdd�Zdd�Zdd�Zdd�Zd�Zdd�Zd�Zy) a�llama-server lifecycle and the streaming LLM round-trip worker.

Two llama-server processes run concurrently:

* the **GPU** server on ``LLAMA_BASE`` (8081) serving interactive chat UI
  users, and
* the **CPU** server on ``LLAMA_BASE_CPU`` (8079) serving automated self-chat
  agents.

Every function below takes a ``mode`` (``"gpu"`` or ``"cpu"``) so loads,
unloads and completions always hit the right server without ever stopping the
other one.
�N)�Mzgpu_slot0.kvzcpu_slot0.kv��gpu�cpuc�<�tj|td�S)zGCheckpoint filename (relative to ``LLAMA_SLOT_SAVE_DIR``) for ``mode``.r)�_SLOT_CHECKPOINT_FILES�get��modes �server/features/llm.py�slot_checkpoint_filer
s��!�%�%�d�,B�5�,I�J�J�c�p�tjjtjt|��S)z5Absolute path of the KV-checkpoint file for ``mode``.)�os�path�joinr�LLAMA_SLOT_SAVE_DIRr
r
s r�slot_checkpoint_pathr#s#��
�7�7�<�<��-�-�/C�D�/I�J�Jrc�t�tj5dtj|<ddd�y#1swYyxYw)ucFlag the lane's slot KV as changed by an outgoing completion.

    Called right before any request is sent to ``mode``'s chat-completions
    endpoint — processing a prompt always mutates the server's slot KV. The
    flag decides whether :func:`save_slot_checkpoint` snapshots again on the
    next unload or the on-disk snapshot is already up to date.TN)r�
_data_lock�_slot_kv_dirtyr
s r�mark_slot_kv_dirtyr(s0��
���&�!%������&�&�&�s�.�7c
���tj5|dk(rtjntj}tjj|d�}tjj|�}ddd�dk7rysryt|�}	tjtj|��d�|tj|�d�d�	�}|jd
k(r�|j�jdd�}tj5|tj|�tj�|d
�tj|<dtj|<ddd�td|�d|�d|���ytd|�d|j�d|j dd
���y#1swY��>xYw#1swY�WxYw#t"$r}td|�d|���Yd}~yd}~wwxYw)uSnapshot the llama-server's current KV cache to disk.

    Calls ``POST /slots/0?action=save`` so the KV of everything processed so
    far survives an imminent model unload. Skipped when the model is not
    loaded or no completion has run since the last save/restore (the on-disk
    snapshot is already current). Failures (slot busy, media tokens in the
    slot, feature unavailable) never block the caller — they just mean the
    next reload re-prefills from scratch, exactly like before this
    optimization existed.
    rFN�chat_loadedTz/slots/0?action=save��filename�model���json�timeout���n_tokensr)�filer�tsr#�[llama] z slot KV checkpointed (z tokens) -> z slot KV save failed (�): z slot KV save error: )rr�_cpu_model_status�model_statusrr	�_slot_checkpointsr
�requests�post�server_base�server_model_id�status_coder �time�print�text�	Exception)r�ms�dirty�cpr�rr#�es        r�save_slot_checkpointr93s���
���+�$(�E�M�Q�
 �
 �q�~�~��� � �$�$�T�5�1��
�
 �
 �
$�
$�T�
*��+�
�]����R��#�D�)�H�9��M�M��}�}�T�"�#�#7�8�'��1B�1B�4�1H�I��
��
�=�=�C���v�v�x�|�|�J��2�H����
/�$��.�.�t�4��)�)�+� (�	-��#�#�D�)�*/�� � ��&�
/�
��4�&� 7��z��h�Z�X�
��
��t�f�2�1�=�=�/��Q�V�V�D�S�\�N�S�	
�
�K+�+��(
/�
/���9�
����3�A�3�7�8�8���9�sD�A%F2�BG�AF?�)G�+G�2F<�?G�G�	G/�G*�*G/c
� �tj5ttjj	|�xsi�}ddd�sy|j	d�xst|�}tjjtjjtj|��s:tj5tjj|d�ddd�y|j	d�tj|�k7rqtd|�d|j	d��dtj|��d��tj5tjj|d�ddd�y	tjtj |��d	�|tj|�d
�d��}|j"d
k(ro|j%�j	d|j	dd��}tj5dtj&|<ddd�td|�d|�d��ytd|�d|j"�d|j(dd
���tj5tjj|d�ddd�y#1swY��pxYw#1swYyxYw#1swYyxYw#1swY��xYw#t*$r}td|�d|���Yd}~��d}~wwxYw#1swYyxYw)aQRestore the previously saved KV cache into slot 0 of ``mode``'s server.

    Called right after a model load. The restored prefix is only a *cache*:
    the next completion still verifies its prompt against the restored tokens
    and evaluates whatever is new, so a stale snapshot costs time but can
    never produce wrong output.
    NFr$rz[llama] Dropping stale z KV checkpoint (saved for 'z', now 'z')z/slots/0?action=restorerrrr"r#rr&z# slot KV restored from checkpoint (z tokens)Tz slot KV restore failed (r'z slot KV restore error: )rr�dictr*r	r
rr�existsrr�popr.r1r+r,r-r/r rr2r3)rr6rr7r#r8s      r�restore_slot_checkpointr>fs���
���7�
�!�%�%�)�)�$�/�5�2�
6��7�
���v�v�f�~�;�!5�d�!;�H�
�7�7�>�>�"�'�'�,�,�q�'<�'<�h�G�H�
�\�\�	0�
���#�#�D�$�/�	0��	�v�v�g��!�+�+�D�1�1�	�%�d�V�,��6�6�'�?�+�8�A�4E�4E�d�4K�3L�B�
P�	
��\�\�	0�
���#�#�D�$�/�	0��<��M�M��}�}�T�"�#�#:�;�&��1B�1B�4�1H�I��	
��
�=�=�C���v�v�x�|�|�J����z�1�0E�F�H����
/�).�� � ��&�
/�
��4�&� C�H�:�X�V�
��
��t�f�5�a�m�m�_�C����t�PS��~�V�	
�

���,�	������d�+�,��Y7�7��	0���	0���
/�
/���<�
����6�q�c�:�;�;��<��,��sf�-J,�!J9�!K�BK�K�,K�+K�!L�,J6�9K�K�K�K�	L�&K<�<L�L
c��y�N�)�args�kwargss  r�_consult_workerrD�s��r�promptrc��ddlm}m}ddl}|dk(r!|�r|jd�|�r�|j||d�}y)z@
    Executes a prompt against the expert/agent model pool.
    r)�
_llm_pools�_human_priority_activeNrg�?)�server.features.staterGrHr0�sleepr	)rErrCrGrHr0�pools       r�consult_expert_modelrL�sF��I���u�}�$�&��D�J�J�s�O�%�&��>�>�$�
�5� 1�2�Drc��tj5tjj|�}|s
	ddd�y|jdd�}|jd�}t	|jd��}ddd�rytj
rytjvrdvr|S|tjvrtjSdS#1swY�YxYw)u�Return the llama-server mode a task must run on.

    Tasks posted by agent users (self-chat: editor, moderator, ...) run on the
    server selected by ``SELF_CHAT_MODE`` (``"cpu"`` or ``"gpu"``); tasks from
    interactive users always use the GPU server. A per-task ``mode`` override
    (set at /api/chat admission, e.g. by ``self-chat.py --gpu``) wins over the
    global flag for agent tasks. An interactive user who explicitly opted into
    the CPU lane for a research task (task marked ``cpu``) always runs on the
    CPU server. When ``FORCE_GPU_LANE`` is set (test-time), every non-flagged
    task — agent or not — runs on the GPU lane.
    Nr�_user�rrr)rr�tasksr	�bool�FORCE_GPU_LANE�_agent_users�SELF_CHAT_MODE)�task_id�t�userr�cpu_flaggeds     r�	task_moderY�s���
���)�
�G�G�K�K�� ����)�)��u�u�W�b�!���u�u�V�}���1�5�5��<�(��
)��������q�~�~��$�.�"8���#�q�~�~�5�1���@�5�@�)�)�s�#C�=C�Cc�L�|dk(rtjStjS)zGBase URL of the llama-server for ``mode`` (defaults to the GPU server).r)r�LLAMA_BASE_CPU�
LLAMA_BASEr
s rr-r-�s��#�u�}�1���>�!�,�,�>rc�L�|dk(rtjStjS)z6Chat-completions URL of the llama-server for ``mode``.r)r�
LLAMA_URL_CPU�	LLAMA_URLr
s r�
server_urlr`�s��"�e�m�1�?�?�<����<rc�p�|dk(r"tjxstjStjS)z9Model filename the llama-server for ``mode`` should load.r)r�MODEL_ID_CPU�MODEL_IDr
s rr.r.�s&���u�}��~�~�+����+��:�:�rc��tj5|dk(rtjntjcddd�S#1swYyxYw)zkModel status of the llama-server for ``mode`` ("unloaded", "loading",
    "chat_loaded", "unloading", ...).rN)rrr(r)r
s r�
server_statusre�s9��
���H�&*�e�m�q�"�"����H�H�H�s�%A�A	c�L�|dk(rtjStjS)z0Idle timestamp of the llama-server for ``mode``.r)r�_cpu_last_llm_use�
_last_llm_user
s r�server_last_useri�s��"&�%�-�1���D�Q�_�_�Drc��t|�S)z8Backwards-compatible model filename lookup for ``mode``.)r.r
s r�active_model_idrk�s
���4� � rc��|�tj}	tj|�d�d��}|jdk(S#t
$rYywxYw)z~True when the llama-server at ``base`` answers /health.

    Defaults to the GPU server so existing callers keep working.
    z/health�)r!r"F)rr\r+r	r/r3)�baser7s  r�is_llama_alivero�sO��
�|��|�|����L�L�D�6��)�1�5���}�}��#�#������s�(=�	A	�A	c	��tj5tj5|dk(rtjntjdk(r	ddd�ddd�y	ddd�td|�d��tj|�tj5|dk(rdt_ndt_ddd�	tjtj|��d�d	tj|�id
��}|jdk(rNtd
|�d��tj5|dk(rdt_ndt_ddd�	ddd�ytd|j�d|jdd���tjtj|��}tj5|dk(r|rdndt_n|rdndt_ddd�	ddd�y#1swY���xYw#1swY��OxYw#1swY��xYw#t$r}td|���Yd}~��d}~wwxYw#1swY�]xYw#1swYyxYw)z4Unload the model from the llama-server for ``mode``.r�unloadedNTz[llama] Requesting z model unload from VRAM/RAM...�	unloadingz/models/unloadr�rr"r&z model unloadedz[llama] Unload response: � z[llama] Unload error: rF)r�_model_transition_lockrr(r)r1r9r+r,r-r.r/r2r3ro)rr7r8�alives    r�unload_llama_modelrws��	
�	!�	!�+�
�\�\�	�'+�u�}��#�#�!�.�.�Z�W��	�+�+�W�	�	�#�D�6�)G�H�I�	
���t�$�
�\�\�	-��u�}�&1��#�!,���		-�	0��
�
��=�=��&�'�~�6��q�0�0��6�7���A�
�}�}��#�����o�6�7��\�\�4��u�}�.8��+�)3���	4�
�?+�+�@
�-�a�m�m�_�A�a�f�f�T�c�l�^�L�M�
� � ����t�!4�5��
�\�\�	H��u�}�7<�m�*��#�27��Z���		H�
�W+�+�	�	��	-�	-��4�4���	0��*�1�#�.�/�/��	0��
	H�	H��M+�+�s��I
�*G7�I
�<I
�H�7I
�A1H�1H�H� (H�8I
�%I�%	I
�7H	�<I
�H	�	I
�H	�H�	H>�&H9�4I
�9H>�>I
�I
	�I
�
Ic���tj5|dk(rtjntjdk(}|dk(rdt_ndt_ddd�tj|�}tj
|�}t
d|�d|�d��	tj|�d�d	|id
��}|jdvr�td
�D]�}tj|�r�t
d|�d|dz�d��tj5|dk(r)dt_tj�t_
n(dt_tj�t_ddd�rtj|�ytj d���n(t
d|j�d|j"dd���tj|�r�tj5|dk(r)dt_tj�t_
n(dt_tj�t_ddd�rtj|�ytj5|dk(rdt_ndt_ddd�y#1swY��DxYw#1swY��BxYw#t$$r}t
d|���Yd}~��	d}~wwxYw#1swY��xYw#1swYyxYw)aTLoad the model on the llama-server for ``mode`` and wait for it to be
    ready, tracking the per-server model status and idle timestamp.

    When the load follows an unload (image generation, idle release), the KV
    checkpoint saved by :func:`save_slot_checkpoint` is restored so the next
    completion only has to evaluate new tokens.rrq�loadingNz([llama] Sending load request for model 'z' to z...z/models/loadrrr)r"��rsr&z model ready (attempt ��)rT�z[llama] Load failed (r'r"z[llama] Load exception: F)rrr(r)r.r-r1r+r,r/�rangeror0rgrhr>rJr2r3)r�was_unloaded�model_idrnr7�ir8s       r�load_llama_modelr�2se��
���	'�$(�5�=�A���a�n�n�
����5�=�"+�A��&�A�N�	'�� � ��&�H��=�=���D�	�4�X�J�e�D�6��
M�N�.��M�M��f�L�!��(�(;�S�
��
�=�=�J�&��2�Y�
���#�#�D�)��H�T�F�*@��1���Q�G�H����:��5�=�2?�A�/�26�)�)�+�A�/�-:�A�N�.2�i�i�k�A�O�
:�$��1�1�$�7���
�
�1�
�
�"
�)�!�-�-���A�F�F�4�C�L�>�J�K�
	�����
�\�\�	.��u�}�&3��#�&*�i�i�k��#�!.���"&�)�)�+���
	.��
�%�%�d�+��	
���(��5�=�",�A��'�A�N�	(�
�s	'�	'��*:�:���.�
�(���,�-�-��.��
	.�	.��(�
�s^�AJ�A3J2�AJ%�% J2�AJ2�+AK�2K#�J"�%J/	�*J2�2	K�;K�K�K �#K,c���t|�}tt|�dz
dd�D]�}||}|jd�dk7r�|jd�}t	|t
�s�?	t
j|�}|jd�dur|jd�nd	}|s�~tj|�}|s��i|�dd
d|�d�d
�dd|id�gi�||<|S|S#ttf$rY��wxYw)aGAttach the bytes of the most recent ``read_image`` result to its tool
    message so the model can actually see the image this round.

    The stored tool result stays a tiny JSON blob (url only); the image bytes
    are embedded only in this round's payload. A copy is returned so the
    stored session is never mutated.
    r{������role�tool�content�okT�	image_urlNr2z[Image loaded from �])�typer2�url)r�r�)�listr~�lenr	�
isinstance�strr �loads�	TypeError�
ValueErrorr�_image_to_data_url)�messages�outr��mr��datar��data_urls        r�_inject_read_imager�us���x�.�C�
�3�s�8�a�<��R�
(�����F���5�5��=�F�"���%�%�	�"���'�3�'��	��:�:�g�&�D�(,�x�x��~��'=�d�h�h�{�#�4�����'�'��,����
��
���+>�s�e�1�)E�F�$�E�8�3D�E��
��A��	��J�3�2�J��!�:�&�	��	�s�%C�C,�+C,c�b�	tj|�tjkDrtj|d�tj|||�}t|�}|D�cgc]*}t
|t�s�|jd�dk(s�)|��,}}|rtd|�dt|��d��tj5tjj|i�jdd�}tjj|i�jd	d
�}	ddd�tjvxs	}
tj|�||
rgntj|
rdndtj d
�}d|d<tj"|�t%j&tj(|�|dd��}|j*dk7rE|j,r|j,ddnd|j*��}
t/d|j*�d|
����d|_d}d}i}tj5tjj|i�jdd�}ddd�|j3d��D�]�}|r|j5d�s�|dd}|j7�dk(r�n�	t9j:|�}|jdg�}|s�^|djdi�}|jd �}|rI||z
}tj5|tjvr|ztj|d<ddd�|jd!�}|r||z
}|jd"�}|s��|D]�}|jd#d�}||vr`|jd$i�}||jd%d�|jd&d$�|jd'd�|jd(d�d)�d*�||<�y||}|jd%�r|d%|d%<|jd$�}|s��|jd'�r|d'|d$d'<|jd(�s��|d$d(xx|d(z
cc<�����td|�d+t|��d,t|��d-t|����d.||zd/�}|rt?|jA��|d"<dd0|igi}d|vrtjBd1||||�2�ytjBd3|d4||�5�ycc}w#1swY���xYw#1swY���xYw#t8j<$rY���wxYw#1swY���xYw#tD$rU} tG| �}!d6|!jI�vsd7|!jI�vrd8}!tjBd3||!||�5�Yd} ~ yd} ~ wwxYw)9Nu1Context is full — compressing older messages...r�r�z[llm_round] Round z
 includes z$ tool message(s) with search resultsrNrO�no_toolsF�none�auto)rr��tools�tool_choice�
max_tokensT�streamiX)r r�r!r"i�zHTTP zLLM server returned z: zutf-8�	reasoning)�decode_unicodezdata: �z[DONE]�choicesr�delta�reasoning_contentr��
tool_calls�index�function�idr��name�	arguments)r�r�)r�r�r�r�z done: reasoning_buf=z chars, content_buf=z chars, tool_calls=�	assistant)r�r�r��message�llm_ok)�body�round�sid�llm_errzUnexpected response)�errorr�r��image�visionzmThe current model does not support image input. Please use a vision-capable model or send text-only messages.)%r�estimate_tokens�AUTO_COMPACT_THRESHOLD�
set_status�prepare_context_for_llmr�r�r;r	r1r�rrP�TOOL_FREE_AGENTSr.�TOOLS�MAX_INPUT_TOKENSrr+r,r`r/r2�RuntimeError�encoding�
iter_lines�
startswith�stripr r��JSONDecodeErrorr��values�_event_postr3r��lower)"rUr��	round_num�msgsrr�r��	tool_msgs�	task_user�
task_no_tools�	tool_free�payloadr7�err_body�
reasoning_buf�content_buf�tool_calls_map�prev_reasoning�line�data_str�chunkr�r��rc�c�tc_list�tc�idx�fn�existing�msgr�r8�err_texts"                                  r�_llm_workerr��s���hT����T�"�Q�%=�%=�=�
�L�L��"U�V��,�,�S�$��=��%�h�/�� (�\�1�J�q�$�,?�A�E�E�&�M�U[�D[�Q�\�	�\���&�y�k��C�	�N�;K�Ko�p�q�
�\�\�	L������G�R�0�4�4�W�b�A�I��G�G�K�K���4�8�8��U�K�M�	L���!3�!3�3�D�}�	��&�&�t�,� �$�R�!�'�'�%.�6�F��,�,�
��!����	���T�"��M�M�!�,�,�t�,�7�4�QT�U���=�=�C��'(�v�v�q�v�v�d�s�|�U�1�=�=�/�3J�H��!5�a�m�m�_�B�x�j�Q�R�R���
��
�����
�\�\�	K��W�W�[�[��"�5�9�9�+�r�J�N�	K��L�L��L�5�/	U�D��t���x�8���A�B�x�H��~�~��8�+��
��
�
�8�,���i�i�	�2�.�G����A�J�N�N�7�B�/�E����.�/�B����#�
��\�\�W��!�'�'�)�8F��8V�����(��5�W��	�	�)�$�A���q� ���i�i��-�G��!�U�B��&�&��!�,�C��.�0��V�V�J��3��%(�"$�&�&��r�"2�$&�F�F�6�:�$>�(*���v�r�(:�-/�V�V�K��-D�)�	/��s�+�$2�#�#6���6�6�$�<�-/��X�H�T�N��V�V�J�/���!�v�v�f�~�?A�&�z��� 4�V� <�!�v�v�k�2� (�� 4�[� A�R��_� T� A�-U�3/	U�`	�"�9�+�-B�3�}�CU�BV�Vj�kn�oz�k{�j|�}P�QT�Uc�Qd�Pe�f�	g��"�!/�-�!?�
��
� $�^�%:�%:�%<� =�C����Y��,�-�.�����
�M�M�(�G�$�i�S�Q�
�M�M���+���
��q]�	L�	L��.	K�	K���'�'�
��
��W�W��d�T��q�6���h�n�n�&�&�(�h�n�n�6F�*F�G�H�	�
�
�i���	�s�S�S��	T�s��A"U�$T
�:T
�T
�/U�A!T�#DU�71T�(A
U�6T)�AU�,U�	3U�=B,U�*-U�BU�/U�
U�T�U�T&�!U�)U�<U�?U�U�U
	�U�	V.�AV)�)V.c�V�tj|�}tj|�tj5|dk(rtjntj
}ddd�dk7rtj|�tj5tjj|�}|s
	ddd�yd|d<||d<ttjj|g��}ddd�td|�d|�d|�d	t��d
�	�tj||dk(rdnd
|�d��tjj|tjd�}|jtj |||||�y#1swY��3xYw#1swY��xYw)Nrr�llm_waiting�_state�_roundzlm_round] Starting round z
 for task z on z
 server with z
 raw messagesrzThinking...zThinking (round z)...)rrY�ensure_llama_serverrr(r)r�rPr	r��sessionsr1r�r�rG�submitr�)rUr�r�rr4rVr�rKs        r�_start_llm_roundr�sd���;�;�w��D����$��	
���F�$(�E�M�Q�
 �
 �q�~�~��F�	�]��	���4� �	
���1�
�G�G�K�K�� ����1�1�$��(����(����
�
���s�B�/�0��
1�
�'�	�{�*�W�I�T�$��}�]`�ai�]j�\k�kx�
y�z��L�L��)�q�.��8H���SW�6X��
�<�<���D�!�,�,�u�"5�6�D��K�K��
�
�w��Y��$�G�!F�F��1�1�s�&F�#F�?3F�F�F()r)rr@)�__doc__r rr0r+rIrrr
rrr9r>rDr�rLrYr-r`r.rerirkrorwr�r�r�r�rArr�<module>r�s�����	���#�"0��G��K�
K�
&�0�f4�n	�

3��
3�C�
3�(A�8?�
=�
�H�E�
!�
�-�`@�F"�JiT�XHr