관리-도구
편집 파일: stat_sender.cpython-37.pyc
B � d� � @ sd d Z dZddlZddlZddlZddlmZ ddlmZ ddl m Z m Z ddl m Z ddlmZ dd lmZmZ dd lmZ dd lmZ dd lmZ d dlmZ d dlm Z ddl m Z ddl m Z ddl!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z( G dd� de�Z)e*dk�r`e&� ej+dej,d� ye)� �-� W n4 e k �r^ Z. ze/e.� e0d ��W ddZ.[.X Y nX dS )zd This module contains SSA classes for sending e-mails with report and sending reports to ClickHouse z ssa.modules� N)� EmailMessage)�partial)�Optional�Any)�get_admin_email)�is_cl_solo_edition)�Session�Response)� HTTPAdapter)�RequestException)�Retry� )�Common)� DecisionMaker� )� stat_server)�SSAError)�Mailer�render_report_table)� read_sys_id� sentry_init� duration_cast� format_datec s� e Zd ZdZ� fdd�Zdedd�dd�Zeee d�d d �Z ee d �d d �Z ee d �dd�Z ee d�dd�Zeeed�dd��Zeed�dd�Zee d�dd�Z� ZS ) �StatisticsSenderz4 Send report to ClickHouse and over e-mails c s� t � �� t�d�| _| j�dt� t� | _dt � d�| _ t � | _ t dtdg�tddd g�dd �}t|d �}t� | _| j�d|� t| jjd d �| j_d S )N� stat_senderz StatisticsSender enabled: %szhttps://z /api/clos-ssa� �POSTi� i� i� )�total�method_whitelist�status_forcelist�backoff_factor)Z max_retries� )�timeout)�super�__init__�logging� getLogger�logger�info� __package__r �sys_idr � ch_endpointr � mail_senderr � frozensetr r �sessionZmountr �request)�selfZ retry_conf�adapter)� __class__� �H/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/stat_sender.pyr$ * s zStatisticsSender.__init__N)�report�returnc C s0 |dkrt � �� }| jr"| �|� | �|� dS )zi Send given report over e-mail If no report given, get it from DecisionMaker API N)r �get_json_report� summary_notification_enabled� email_report�clickhouse_report)r0 r5 r3 r3 r4 �send; s zStatisticsSender.send)� report_viewr6 c C s� | � � }|r�|�d�r�t|�\}}| jj|dt|d �|d�}|jt�|�dd|d � d�d� |j|d d|d � d �d� | j� |� |S d S ) z4 Create and send e-mail with report �domainsZ ssa_report�date)Z recipient�templater> �html�jsonZreport_z.json)�subtype�filenamer@ z.htmlN) �get_mail_recipient�getr r, �_messager �add_attachmentrA �dumpsZ_send)r0 r< Zmail_toZ report_table�mail�msgr3 r3 r4 r9 G s z StatisticsSender.email_report)r6 c C sH y| � � pt� S tk rB } z| j�dt|�� W dd}~X Y nX dS )z� Retrieve a recipient's e-mail: 1. get address from a wmt-api utility 2. if command failed or address is empty, get address of server admin z get_admin_email failed with: %sN)�wmt_api_report_emailr � Exceptionr' �error�str)r0 �er3 r3 r4 rD ` s z#StatisticsSender.get_mail_recipientc C s� t dd�rdnd}y&tjd|� �dgdddd�j�� }W n> tjtttfk rv } z| j � dt |�� d S d }~X Y nX yt � |��d ��d �S t jk r� } z| j � d t |�� W d d }~X Y n" tk r� | j � d |� Y nX d S )zD Retrieve a recipient's e-mail address from WMT API T)�skip_jwt_checkz wmt-api-solozwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-get)�check�text�capture_outputzwmt-api utility failed: %sN�configZ report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)r � subprocess�run�stdout�strip�CalledProcessError�AttributeError�OSError� ValueErrorr' rM rN rA �loadsrE �JSONDecodeError)r0 �_utilZ api_responserO r3 r3 r4 rK l s( z%StatisticsSender.wmt_api_report_emailc C s� |� d�r�| j�d| j� y"| jj| j| �| �|��d�}W nN tk r� } z0| jj d|d| jid� t dt |�� ��|�W dd}~X Y nX | � |�S | j�d � d S dS ) z+ Send report to ClickHouse r= zSending POST request to %s)rA zPOST failed with %s�endpoint)�extraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF) rE r' r( r+ r. Zpost�_ch_pack�clickhouse_formatr rM r rN �_process_response)r0 r< �resprO r3 r3 r4 r: � s $ z"StatisticsSender.clickhouse_report)�valuer6 c C s t | d�S )z< Pack given value into data field of a dict )�data)�dict)rf r3 r3 r4 rb � s zStatisticsSender._ch_pack)�original_reportr6 c C s` t � }xT|�d�D ]F}|�| j|�d�|�d�|�d�|�d�dd� |�d�D �d �� qW |S ) zd Format local report for sending to ClickHouse (required structures differ) r= �name� slow_urls� slow_reqs� total_reqsc S s< g | ]4}|� d �|� d�t|� d��t|� dd��d��qS )rj �reqs_num�average_duration� correlationr )�urlZcount_requests� avg_durationrp )rE r �float)�.0�ur3 r3 r4 � <listcomp>� s z6StatisticsSender.clickhouse_format.<locals>.<listcomp>�urls)� system_id�domainZcount_slow_urlsZcount_slow_requestsZtotal_requests�details)�listrE �appendr* )r0 ri Z ch_reportry r3 r3 r4 rc � s z"StatisticsSender.clickhouse_format)�responser6 c C s� |j s(| jjd|j|jd|jid� dS | j�d|j|j|j� |�� }|d dkrr| jjd|d d |id� dS | j�d � d S ) z� Check received response :param response: a requests.Response object :return: True in case of success, False otherwise z&Unable to connect to server with %s:%sZ resp_text)ra Fz [%s:%s] Response received %s�status�okz Received response with status %sr} z Sent to ClickHouse successfullyT) r r' rM � status_code�reasonrR r( rq rA )r0 r} �resultr3 r3 r4 rd � s z"StatisticsSender._process_response)N)�__name__� __module__� __qualname__�__doc__r$ rh r; r r r9 rN rD rK �boolr: � staticmethodr rb r{ rc r rd � __classcell__r3 r3 )r2 r4 r % s r �__main__zstat_sender_standalone.log)rC �level)1r� r) rA r% rU � email.messager � functoolsr �typingr r �clcommon.cpapir �clcommon.lib.cleditionr Zrequestsr r Zrequests.adaptersr Zrequests.exceptionsr Z$requests.packages.urllib3.util.retryr �commonr �decision_makerr �internal.constantsr �internal.exceptionsr Zinternal.mailerr r �internal.utilsr r r r r r� � basicConfig�INFOr; �exc�print� SystemExitr3 r3 r3 r4 �<module> s< 0