📄 monitoring.cpython-312.pyc
/home/palash/git/local-ai/server/features/__pycache__/monitoring.cpython-312.pyc
Language: pyc • Lines: 623
�

�j^B����dZddlZddlZddlZddlmZddlZddlmZddd�Zd�Z	d	�Z
d
�Zd#d�Zd�Z
d$d
�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zddd�Zd�Zd�Zd�Zd�Zd �Zda da!d!�Z"d"�Z#y)%a�Health monitoring, server lifecycle and the background maintenance loops.

Two llama-server processes run concurrently on separate ports:

* the **GPU** server on ``LLAMA_BASE`` (8081) for interactive chat UI users, and
* the **CPU** server on ``LLAMA_BASE_CPU`` (8079) for automated self-chat
  agents.

Each is started, killed, health-checked and idle-unloaded independently so an
agent run never disturbs interactive users (and vice versa).
�N)�datetime)�M�8081�8079��gpu�cpuc���tj5tjtjtjtj
tjd�cddd�S#1swYyxYw)N)�model�predicted_per_second�
overheated�gpu_temp�ram_evacuating)r�
_data_lock�model_status�	_last_tps�_overheated�	_gpu_temp�_ram_evacuating���7/home/palash/git/local-ai/server/features/monitoring.py�model_status_snapshotrsH��	
���
��^�^�$%�K�K��-�-�����/�/�
�
�
�
�s�AA)�)A2c��	tjgd�ddd��}t|jj	��S#t
$rYywxYw)N)z
nvidia-smiz--query-gpu=temperature.gpuz--format=csv,noheaderT���capture_output�text�timeout)�
subprocess�run�int�stdout�strip�	Exception)�rs r�get_gpu_tempr'%sK��	��N�N�R����	
���1�8�8�>�>�#�$�$������s�=A�	A�Ac��	tjddgddd��}|jj�j	d�}|dj	�}t|d�}t|d�}||z
|zd	zS#t$rYywxYw)
N�freez-mTrr�
���d)r r!r#r$�splitr"r%)r&�lines�parts�total�	availables     r�
get_ram_usager32s�����N�N�F�D�>�$�T�ST�U������� �&�&�t�,���a���� ���E�!�H�
����a��M�	��	�!�U�*�S�0�0������s�A=B�	B�Bc�H�|�Htjgd�d��tjd�tjgd�d��yt|}d|��}tjdd	|gd��tjd�tjdd
d	|gd��y)z�Kill llama-server process(es).

    ``mode`` is ``"gpu"`` (port 8081), ``"cpu"`` (port 8079) or ``None`` to
    kill both servers at once (emergency RAM evacuation, full restart).
    N)�pkill�-f�llama-serverT�rr+)r5�-9r6r7zllama-server.*--port r5r6r9)r r!�time�sleep�_LLAMA_PORTS)�mode�port�patterns   r�kill_llama_serverr@>s����|����6�t�L��
�
�1�
����<�T�R�����D�%�d�V�,�G��N�N�G�T�7�+�D�A��J�J�q�M��N�N�G�T�4��1�$�Grc�6�tjgd�d��y)N)r5r6zmain.py.*lowvramTr8)r r!rrr�kill_comfyuirBPs���N�N�6�t�Lrc���|dk(rtjntj}tjjd�}t
tjj||�d��d�}tjtjg|z||d��tj�dz}tj�|krjtjd�	tj|�d	�d
��}|jdk(rt!d
|�d|���y	tj�|kr�jt!d
|�d��tj$|�y#t"$rY�HwxYw)zGLaunch a llama-server with the given argument list and wait for health.r	�~/local-ai-filesz-llama-server.log�aT)r#�stderr�start_new_session�x�z/health��r��z[restart] llama-server (z
) healthy on u.) did not respond within 2 minutes — killingF)r�LLAMA_BASE_CPU�
LLAMA_BASE�os�path�
expanduser�open�joinr �Popen�LLAMA_SERVER_PATHr:r;�requests�get�status_code�printr%r@)�argsr=�base�log_dir�llm_log�deadliner&s       r�_start_llama_processr_Ts4��#�u�}�1���!�,�,�D��g�g� � �!3�4�G��2�7�7�<�<��D�6�1B�)C�D�c�J�G����	
�	�	���$����	��y�y�{�S� �H�
�)�)�+��
 ��
�
�1�
�	�����v�W�-�q�9�A��}�}��#��0���m�D�6�J�K��$�	�)�)�+��
 �
�$�T�F�*X�
Y�Z��������	�	��	�s�!:E�	E&�%E&c�Z�td|�d��tj|�tjd�tj
5|dk(rdt_ndt_ddd�|dk(rtjntj}t||�y#1swY�;xYw)z�Restart the llama-server for ``mode`` (``"gpu"`` or ``"cpu"``) using its
    own argument set and port, leaving the other server untouched.z![llama] Restarting llama-server (�)r+r	�unloadedNr)rYrr@r:r;r�_cpu_model_statusr�LLAMA_SERVER_ARGS�LLAMA_SERVER_ARGS_CPUr_)r=rZs  r�restart_llama_serverrfns���
�-�d�V�1�
5�6�������J�J�q�M�	
���(��5�=�",�A��'�A�N�	(�
#'�%�-�1���Q�5L�5L�D���t�$�
(�(�s�
B!�!B*c��|dk(rtjntj}tj|�ryt	d|�d��t|�y)zGMake sure the llama-server for ``mode`` is running, starting it if not.r	Nz[llama] u+ llama-server not reachable — starting...)rrMrN�is_llama_aliverYrf)r=r[s  r�ensure_llama_serverri}sD��#�u�}�1���!�,�,�D�������	�H�T�F�E�
F�G���rc���tj5|tjvr
	ddd�y	ddd�tj|�}tj|�y#1swY�4xYw)z�Make sure the llama-server the task's author needs is running.

    Tasks posted by agent users (self-chat: editor, moderator, ...) run on the
    CPU server; tasks from interactive users use the GPU server.
    N)rr�tasks�	task_moderi)�task_idr=s  r�_ensure_llama_server_for_taskrn�sZ��
�����!�'�'�!����!��
�;�;�w��D����$��	��s�A"�"A+c�b�tjrytj5tjr
	ddd�y	ddd�tjd5ttjd�dkDxstjdducddd�S#1swY�\xYw#1swYyxYw)auTrue if the CPU self-chat lane has (or is about to have) work.

    The CPU llama-server is only started when a self-chat agent is registered
    or an agent task is queued/running on the cpu lane. This keeps the machine
    from booting a second llama-server that nothing ever uses. Under the
    test-time ``FORCE_GPU_LANE`` flag the CPU lane is never needed at all.
    FNTr	r)r�FORCE_GPU_LANEr�_agent_users�_queue_locks�len�_task_queues�_current_task_idsrrr�_cpu_lane_neededrv�s���	����	
�����>�>������
����	�X��1�>�>�%�(�)�A�-�W��1D�1D�U�1K�SW�1W�X�X����X�X�s�B�6B%�B"�%B.c	��td�tj�tj�t	j
d�tjjd�}ttjj|d�d�}tjtjjtj�ddtjdtjd	gtj ||d
��tj"5dt_dt_ddd�t)tj*d
�t-�rt)tj.d�ytd�y#1swY�TxYw)NzRestarting serversr+rD�comfyui.logrE�main.py�--output-directory�--input-directory�	--lowvramT��cwdr#rFrGrbrr	z@[llama] Skipping CPU llama-server start (no agent lane activity))rYrr@rBr:r;rOrPrQrRrSr rT�VENV_PYTHON�COMFYUI_OUTPUT�
COMFYUI_INPUT�COMFYUI_DIRrrrcr_rdrvre)r\�	comfy_logs  r�restart_serversr��s��	�
�������N�N���J�J�q�M��g�g� � �!3�4�G��R�W�W�\�\�'�=�9�3�?�I�����G�G�L�L����'�� �
����
�O�O��	
�
�M�M�����
���)�#���(���)���,�,�e�4����Q�4�4�e�<�
�P�Q�)�)�s�F�Fc	���	tjtj�d�d��}|jdkry	t
d�tj�tjd�tjjd�}ttjj|d�d	�}tj tjjtj"�d
dtj$dtj&d
gtj(||d��tj�dz}tj�|krrtjd�	tjtj�d�d��}|jdkrt
d�y	tj�|kr�rt
d�y#t
$rY���wxYw#t
$rY�?wxYw)Nz/promptrJrKi�u'[comfyui] Not reachable — starting...r+z
~/local-airxrEryrzr{r|Tr}rHrIz[comfyui] Healthyz*[comfyui] Did not respond within 2 minutes)rVrWr�COMFYUI_URLrXr%rYrBr:r;rOrPrQrRrSr rTrr�r�r�)r&r\r�r^s    r�ensure_comfyui_runningr��s���
��L�L�A�M�M�?�'�2�A�>���=�=�3����
�
3�4��N�N���J�J�q�M��g�g� � ��.�G��R�W�W�\�\�'�=�9�3�?�I�����G�G�L�L����'�� �
����
�O�O��	
�
�M�M������y�y�{�S� �H�
�)�)�+��
 ��
�
�1�
�	�����
�
��g�6��B�A��}�}�s�"��)�*��#�	�)�)�+��
 �
�
6�7��A�
��
��<�	��	�s$�7G�2AG*�	G'�&G'�*	G6�5G6c�n�	tjd�dD�]}tj|5t	tj
|�dkDxstj|du}ddd�tj5|dk(rtjntj}|dk(rtjntj}ddd�dk(s��tj�z
dkDs��r��td|�d��tj|�����#1swY��xYw#1swY�bxYw)	N�
rrr	�chat_loaded�,z
[idle] No z2 LLM activity for 300s, releasing model weights...)r:r;rrrrsrtrurrcr�_cpu_last_llm_use�
_last_llm_userY�unload_llama_model)r=�queue_active�ms�lus    r�_idle_unload_loopr��s��
��
�
�2��#�	+�D�����%�
f�"�1�>�>�$�#7�8�1�<�e��@S�@S�TX�@Y�ae�@e��
f����
O�,0�E�M�Q�(�(�q�~�~��,0�E�M�Q�(�(�q����
O��]�"��	�	��b�(8�3�(>���
�4�&�(Z�[�\��$�$�T�*�	+��
f�
f��
O�
O�s�7D�?AD+�D(	�+D4	c�^�		tj�j�}tjd|f�}|D]3}td|d�d|d���tjd|df��5	tjd	���#t$r}td|���Yd}~�2d}~wwxYw)
Nz�SELECT * FROM tasks WHERE reminder_at IS NOT NULL AND reminder_at <= ? AND reminded=0 AND status NOT IN ('completed','cancelled')z[reminder] Task '�titlez	'. User: �user_idz&UPDATE tasks SET reminded=1 WHERE id=?�idz[reminder] Error: i��)
r�now�	isoformatr�	_db_fetchrY�_db_runr%r:r;)r��due�task�es    r�_reminder_loopr�s���
�	,��,�,�.�*�*�,�C��+�+�b�eh�dj�k�C��
S���)�$�w�-��	�$�y�/�AR�S�T��	�	�B�T�$�Z�M�R�
S�
	
�
�
�5�����	,��&�q�c�*�+�+��	,�s�A1B�	B,�B'�'B,c��dt_td�dD�]}tj|5tj|}|r�tj
5tjj|�}|r�|jd�dvr�||jdd�|jdd�|jd	�|jd
d�|jd�d�}tj|jd
|�d|d<d|d<d|d<td|�d|�d��ddd�ddd���tj�tj�td�	tjd�tj�}|�5|tjkr"td|d�dtj�d��n�atj �dt_y#1swY��xYw#1swY���xYw)NTz[ram] Emergency RAM evacuationr�status)�done�error�
session_id��_original_message�_original_image�_user�_client_timestamp)rmr��message�image�user�client_timestamprr�u"Server ran out of RAM — requeuedrz[ram] Requeued z task z to front of its lanez%[ram] Killed llama-server and ComfyUIrz
[ram] RAM �.0fu% ≤ z%, restarting serversF)rrrYrrrurrkrWrt�insertr@rBr:r;r3�RAM_RESUME_THRESHOLDr�)r=�tid�t�entry�rams     r�
_evacuate_ramr�
s����A��	�
*�+��X��
�^�^�D�
!�	X��%�%�d�+�C���\�\�X������C�(�A��Q�U�U�8�_�4E�E�'*�*+�%�%��b�*A�'(�u�u�-@�"�'E�%&�U�U�+<�%=�$%�E�E�'�2�$6�01���6I�0J�
!�����t�,�3�3�A�u�=�&-��(��%I��'�
�/3��+�,����v�V�C�5�@U�V�W�X�	X�	X�X�(�����N�N��	�
1�2�
��
�
�1�
��o�o����?�s�a�&<�&<�<��J�s�3�i�v�a�.D�.D�-E�EZ�[�\��������A��5X�X��	X�	X�s$�&G:�CG.�&G:�.G7�3G:�:H	c�v�	tjd�tj�}tj5|t_|�O|tjk\r<tjs|td|�dtj�d��dt_nPtjr@|�|tjkr+td|�dtj�d��dt_ddd�tjr�tjd	5tjd	du}ddd�sstj5tj}ddd�d
k(r!td�tjd	�n$|dk(rtd
�tj�tjs^tj �}|�H|tj"k\r5td|d�dtj"�d��tj$���#1swY��0xYw#1swY��xYw#1swY��xYw)NTr�z[thermal] GPU u°C >= u°C, OVERHEATEDu°C <= u°C, resumedFrr�u1[thermal] Overheated — unloading GPU chat model�image_activeu-[thermal] Overheated — freeing ComfyUI VRAMz[ram] RAM usage r�z% >= �%)r:r;rr'rr�TEMP_THRESHOLD_ONrrY�TEMP_THRESHOLD_OFFrrrurr��free_comfyui_vramrr3�RAM_EVAC_THRESHOLDr�)�temp�busyr�r�s    r�_thermal_monitorr�3s���
��
�
�2���~�~���
�\�\�
	&��A�K���D�A�,?�,?�$?��}�}��(���g�a�6I�6I�5J�/�Z��%)�A�M����D�L�D�A�<P�<P�4P���t�f�G�A�4H�4H�3I��V�W� %��
�
	&�
�=�=�����&�
>��*�*�5�1��=��
>���\�\�(����B�(���&��M�N��(�(��/��>�)��I�J��'�'�)�� � ��/�/�#�C���3�!�*>�*>�#>��(��S�	��q�7K�7K�6L�A�N�O����!�G�
	&�
	&�� 
>�
>��(�(�s$�B-H�H#�H/�H �#H,�/H8c�^�	tjddd��j�}tjd|�d�dd��}|j�D]2}d|vs�d|vs�
|j	�dj	d	�d
cS	y#t
$r}t
d|���Yd}~yd}~wwxYw�
z.Get this machine's stable global IPv6 address.z1ip -6 route show default | awk '{print $5; exit}'T��shellrzip -6 addr show �
 scope global�inet6�	temporaryr+�/rzdns] Failed to get IPv6: N�r �check_outputr$�
splitlinesr.r%rY��iface�output�liner�s    r�_get_current_ipv6r�Z����1��'�'�?�t�RV�
�
�%�'�	��(�(��u�g�]�3�4�d�
���%�%�'�	5�D��$��;�d�#:��z�z�|�A��,�,�S�1�!�4�4�	5�
���1�
�+�A�3�/�0�0���1��*�AB�B�!&B�B�	B,�B'�'B,c��	tjddd��j�}tjd|�d�dd��}|j�D]O}|j�}|j	d�s�%d|vs�*|j�djd	�d
cS	y#t$r}td|���Yd}~yd}~wwxYw)
z8This machine's LAN IPv4 on the default (WiFi) interface.z1ip -4 route show default | awk '{print $5; exit}'Tr�zip -4 addr show r�zinet �	secondaryr+r�rz%[heartbeat] Failed to get WiFi IPv4: N)r r�r$r��
startswithr.r%rYr�s    r�_get_wifi_ipv4r�ks���;��'�'�?�t�RV�
�
�%�'�	��(�(��u�g�]�3�4�d�
���%�%�'�	5�D��:�:�<�D����w�'�K�t�,C��z�z�|�A��,�,�S�1�!�4�4�	5����;�
�5�a�S�9�:�:���;�s*�A6B(�9B(�>&B(�%B(�(	C	�1C�C	g��ip�tsc��tj�}tdr|tdz
dkr	tdSdD]r}	tj|d��}|jj�}|jdk(r/|jd�d	k(rtj||�
�|cS�ttd�tdS#t$rY��wxYw)z@Public WAN IPv4 as seen from the internet, cached for 5 minutes.r�r�r�)zhttps://api.ipify.orgzhttps://ifconfig.me/iprrKrL�.rJr�z+[heartbeat] Could not determine public IPv4)r:�_public_ipv4_cacherVrWrr$rX�count�updater%rY)r��urlr&r�s    r�_get_public_ipv4r��s���
�)�)�+�C��$��C�*<�T�*B�$B�S�$H�!�$�'�'�B���	����S�!�,�A�������B��}�}��#�����
��(:�"�)�)�R�C�)�8��	��
�
�
7�8��d�#�#���	��	�s�A,C�	C
�C
c��t�t�t�d�}tjt
j|d��}|j�|S)zBPOST this machine's addresses to the GCP receiver over the tunnel.)�ipv6�public_ipv4�	wifi_ipv4r)�jsonr)r�r�r�rV�postr�
HEARTBEAT_URL�raise_for_status)�payloadr&s  r�_send_heartbeatr��sF��"�#�'�)�#�%��G�
	�
�
�a�o�o�G�Q�?�A������Nrc�X�ttjxrtj�S)zGTrue when the GoDaddy API credentials are available in the environment.)�boolr�GODADDY_API_KEY�GODADDY_API_SECRETrrr�
_ddns_enabledr��s����!�!�:�a�&:�&:�;�;rc�^�	tjddd��j�}tjd|�d�dd��}|j�D]2}d|vs�d|vs�
|j	�dj	d	�d
cS	y#t
$r}t
d|���Yd}~yd}~wwxYwr�r�r�s    rr�r��r�r�c�X�dtj�dtj��}dtj�dtj��dd�}tj|||dd�g�	�}|jd
k(rtd|���ytd
|j�d|j���y)Nz#https://api.godaddy.com/v1/domains/z/records/AAAA/zsso-key �:zapplication/json)�
AuthorizationzContent-TypeiX)�data�ttl)�headersr�rLz[ddns] GoDaddy AAAA updated to Tz[ddns] GoDaddy update failed (�): F)
r�DDNS_DOMAIN�DDNS_SUBDOMAINr�r�rV�putrXrYr)�new_ipr�r��resps    r�_update_godaddy_aaaar��s���/��
�
��n�Q�M]�M]�L^�
_�C�#�A�$5�$5�#6�a��8L�8L�7M�N�*��G��<�<��W�V�C�4P�3Q�R�D����3��
�/��x�8�9��
�.�t�/?�/?�.@��D�I�I�;�O�P�rc���t�sytjxsd}tj�}|tz
|kry|at�}|sy|tk7rt|�r|ayyy)uOCall on every ConnectionManager tick — self-throttles to DDNS_CHECK_INTERVAL.Nr�)r�r�DDNS_CHECK_INTERVALr:�_last_dns_checkr��_last_known_ipv6r�)�intervalr��
current_ips   r�maybe_update_dnsr�sk���?���$�$�+��H�
�)�)�+�C�
�_��x�'���O�"�$�J����%�%��
�+�)��,�&rc���		t�}td|���t�t
jd��9#t$r)}tdtj�d|���Yd}~�Md}~wwxYw)Nz[+] heartbeat sent: z[-] GCP unreachable (r�r�)r�rYr%rr�rr:r;)r�r�s  r�_connection_managerr�sm��
�	C�%�'�G��(��	�2�3�
	���
�
�2�����	C��)�!�/�/�):�#�a�S�A�B�B��	C�s�;�	A-�A(�(A-)N)r)$�__doc__rOr r:rrV�server.features.staterr<rr'r3r@rBr_rfrirnrvr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�rrrrr�<module>rs���
�
�����#��f�-��	
�
�	�H�$M��4%��
 �X�"R�B%8�P+�&
�#�L$"�N�"�$!��,��$�$	�<�
�"�����*�$r