관리-도구
편집 파일: autotracer.cpython-37.pyc
B � dYR � @ s� d Z dZddlZddlZddlmZ ddlmZmZm Z ddl m Z m Z ddl m Z ddlmZmZmZmZmZ dd lmZ ddlZdd lmZ d d lmZ d d l m Z m Z m Z ddl m!Z!m"Z"m#Z# ddl$m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+m,Z,m-Z- yddl.m/Z0 W n e1k �r& dZ0Y nX eG dd� d��Z2eG dd� d��Z3eG dd� d��Z4G dd� de�Z5e6e5d �d d �Z7G d d � d e�Z8e9d!k�r�e-� ej:d"ej;d#� e8� Z<e<� dS )$z' This module contains Autotracer class z ssa.modules� N)� defaultdict)� dataclass�field�asdict)�datetime� timedelta)� fnmatchcase)�Iterator�Optional� NamedTuple�List�Iterable)�urlparse)�setup_database� )�Common)�iter_domains_data�iter_urls_data�get_url_durations� )�status�disabled_users�is_edition_supported)� load_tunables)�autotracing_tunables_schema)�autotracing_stats_file)�is_xray_version_supported�is_kernel_version_supported� sentry_initc @ sJ e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S ) �URLzURL data container�uri� avg_duration� max_duration�total_slow_reqs� total_reqs� belongs_to�belongs_to_userN)�__name__� __module__� __qualname__�__doc__�str�__annotations__�int� r. r. �G/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/autotracer.pyr / s r c @ s� e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ eed< dZ eed< dZ eed < dZ eed < dZeed < dZeed < dZeed < dS )� RulesRejectszRejects counters containerr �non_wp� throttled�density�slowness� max_slowness�disabled�nginx� frequency� server_limit� domain_limit� no_domainN)r' r( r) r* r1 r- r, r2 r3 r4 r5 r6 r7 r8 r9 r: r; r. r. r. r/ r0 ; s r0 c @ sD e Zd ZU dZeed< dZeed< dZeed< e e d�Z e ed< dS ) �Statsz AutoTracer statistics container� rules_versionr �urls_processed� urls_selected)�default_factory�rejectsN) r' r( r) r* r+ r, r> r- r? r r0 rA r. r. r. r/ r< K s r< c @ s" e Zd ZU dZeed< eed< dS )�URLSz" Representation of an URL � domain_name�uri_pathN)r' r( r) r* r+ r, r. r. r. r/ rB T s rB )�url�returnc C s\ t | �}|jrd|j� �nd}|jr2|j� |� �nd}|j�dd�}|�d�d }t||�S )z� Split URL into domain_name and uripath including query string :param url: URL of format protocol://domain/path;parameters?query#fragment :return: namedtuple URL(domain_name, uripath) �?� �/zwww.�:r )r �query�path�netloc�replace�splitrB )rE � fragments�qsr Z_no_www_netlocZ_no_port_netlocr. r. r/ � url_split\ s rR c s� e Zd ZdZd7� fdd� Z� fdd�Zeed�dd ��Zee e e d �d d ��Z e e d �dd�Zed�dd�Zd8e e ed�dd�Ze e d�dd�Zeee d�dd�Zee e d�d d �Zee e d�d d �Ze e ee d �d!d"�Ze e e d#�d$d%�Zee d�d&d'�Z d9e e ee d(�d)d*�Z ee d�d+d,�Z e e d�d-d.�Z dd�d/d0�Z!dd�d1d2�Z"ed�d3d4�Z#ed�d5d6�Z$� Z%S ):� AutoTracerz0 SSA autotracing module implementation. Nc sN t � �� t�d�| _| j�dt� t� | _t� | _ d | _ |rB|nt � | _ d S )N� auto_tracerzAutoTracer enabled: %s) �super�__init__�logging� getLogger�logger�info� __package__�listr � tasks_list�statsr �engine)�selfr_ )� __class__r. r/ rV o s zAutoTracer.__init__c s& t � s| j�d� d S t� s,| j�d� d S t� sB| j�d� d S td krZ| j�d� d S tj}|d k �r| j�d� t� �� | j� d| j � t | j �| _ x|| �� D ]p}| j�d|j� | j jd7 _y||j| jd � W q� tk �r } z| j�d t|�� W d d }~X Y q�X q�W | �� d S ) Nz*AutoTracer skipped: unsupported CL editionz.AutoTracer skipped: unsupported kernel versionz-AutoTracer skipped: unsupported X-Ray versionz*AutoTracer skipped: unable to import X-RayzAutoTracer startedz AutoTracer loaded config: %szStarting auto task for %sr )rE Z tracing_countzFailed to start task: %s)r rY rZ r r �xray_libZstart_autotracingrU rV �debug�configr< r= r^ �urls_scheduledr r? Zrequest_number� Exception�errorr+ �save_iteration_stats)r` Z start_toolrE �e)ra r. r/ �__call__y s4 (zAutoTracer.__call__)rF c C s t dt�S )z$ Load configuration zautotracing.json)r r r. r. r. r/ � load_conf� s zAutoTracer.load_conf)rC rF c C s t dkr dS t �| �S )z; Gets domain info for the specified domain N)rb � domain_info)rC r. r. r/ �gets_domaininfo� s zAutoTracer.gets_domaininfo)� user_namerF c C s" | j r tdkrdS t�|�jS dS )z> Says if nginx is enabled for a specific user NF)Z skip_nginxrb ZNginxUserCache� is_enabled)r` rn r. r. r/ �nginx_is_enabled� s zAutoTracer.nginx_is_enabledc C s, t dd� | jD ��}| j�d|� | j| S )z>Recalculate limit per server taking into account running tasksc S s( g | ] }|d dkr|d dkr|�qS )r �running�userz *autotracing*r. )�.0�taskr. r. r/ � <listcomp>� s z<AutoTracer.per_server_limit_recalculated.<locals>.<listcomp>z&Number of running autotracing tasks %s)�lenr] rY rc Zper_server_limit)r` Z running_countr. r. r/ � per_server_limit_recalculated� s z(AutoTracer.per_server_limit_recalculated)� full_listrF c sD |dkr t �� }| j�d|� t�� t| jd� � � fdd�|D �S )zQ Excludes tasks older than N days from the general list of tasks NzTask list loaded %s)�daysc s8 g | ]0}|� d �dk r |� d �ndt� �� �kr|�qS )Z createtimeNr )�getr- � timestamp)rs rt )� n_days_agor. r/ ru � s z1AutoTracer.excludes_old_tasks.<locals>.<listcomp>)rb r] rY rc r �nowr Zmin_retracing_interval)r` rx r. )r| r/ �excludes_old_tasks� s z AutoTracer.excludes_old_tasks)� current_urlrF c C s| t |�}xn| jD ]d}t |d �}|j|jko6|j|jk}t|j|j�oRt|j|j�}|s\|r| j�d||d � dS qW dS )z� Excludes url from the list if it completely matches the current url or if domain names match and "*" follows the domain name in the list rE z2Skipped: URL %s was traced recently. Matched by %sTF)rR r] rC rD r rY rc )r` r �cZ task_data�tZ direct_matchZwildcard_matchr. r. r/ �exclude_thesame_urls� s z AutoTracer.exclude_thesame_urls)�url_total_reqs�domain_total_reqsrF c C s4 | j r0t�t�||��}| j�d|� || jkS dS )z-Check that URL density passes given thresholdzCalculated density %sT)r3 �npZaminZcorrcoefrY rc Zdensity_threshold)r` r� r� Z url_densityr. r. r/ �pass_by_density� s zAutoTracer.pass_by_density)� url_durationsrF c C s0 d| j }t�||�}| j�d||� || jkS )z� The measure of "slowness" for URL is: at least N% of its requests take more than X seconds. N% -- self.slow_duration_percentage X -- self.slow_duration_threshold �d z.Calculated %sth percentile %s for min duration)Zslow_duration_percentager� � percentilerY rc Zslow_duration_threshold)r` r� Zreversed_percentileZreversed_percentile_valuer. r. r/ �pass_by_slowness_percentile� s z&AutoTracer.pass_by_slowness_percentilec C s* t �|| j�}| j�d| j|� || jkS )a The opposite to pass_by_slowness_percentile method. The measure of "much slowness" for URL is: at least N% of its requests must take less than X seconds. N% -- self.max_slow_duration_percentage X -- self.max_slow_duration_threshold z.Calculated %sth percentile %s for max duration)r� r� Z max_slow_duration_percentagerY rc Zmax_slow_duration_threshold)r` r� Zpercentile_valuer. r. r/ � pass_by_max_slowness_percentile� s z*AutoTracer.pass_by_max_slowness_percentile)�url_throttled_reqsr� rF c C s8 |dkr dS t |�t |� d }| j�d|� || jkS )zY Check that percent of throttled requests per URL passes given threshold NFr� z Calculated throttled percent %s)�sumrY rc Z allowed_throttling_percentage)r` r� r� Zthrottled_percentr. r. r/ �pass_by_allowed_throttling s z%AutoTracer.pass_by_allowed_throttling)� wp_statusrF c C s | j rt|�S dS )zx Check that URLs of a particular domain should be analyzed. For now we skip non-wordpress sites T)Zonly_wp�bool)r` r� r. r. r/ �pass_by_engine s zAutoTracer.pass_by_enginec c s �x t | j�D �]�}d}| �|j�sZ| j�d� |j}| jj j |7 _ | j j |7 _ qt t | j|j ��}�x�t| j|j t|�� ��D �]|\}}|| jkr�q�| j�d|� | j j d7 _ | �|�d�|d �s�| j�d� | jj jd7 _q�| �|d |j��s&| j�d� | jj jd7 _q�|| }| �|��sZ| j�d � | jj jd7 _q�| �|��s�| j�d � | jj jd7 _q�|dk�r�| � |j �}|dk�r�| j�d � | jj j d7 _ q�n|j }t |t t!�"|��t#|d �t#|d �t$|�|j |d �V q�W qW dS )a Select all URLs suitable for auto tracing by very basic rules: - WP site - suitable throttling - suitable density - measure of "slow" URL ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS Nz Skipped by engine: non-wordpresszProcessing URL %sr r� r� z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain infoZ url_slow_reqs)r! r# r$ r" r% r& )%r r_ r� Zis_a_wordpress_domainrY rc Z urls_numberr^ rA r1 r> �dictr rC r r\ �keysZnon_url_fieldsr� rz r2 r� r� r3 r� r4 r� r5 rm r; rr r r- r� Zmeanr� �max) r` Z domain_data� domain_ownerZ skipped_countZdomain_url_durationsrE �dataZ durationsrl r. r. r/ � urls_computationally_filtered sb z(AutoTracer.urls_computationally_filtered)�stats_collectedrF c c s� | � � xv| �� D ]j}|j| jkrF| j�d|j� | jj jd7 _q| � |j�rv| j�d|j� | jj j d7 _ q|V qW dS )z� From selected by computed thresholds URLs take those for which: - autotracing enabled - nginx disabled ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS z'Skipped: autotracing is disabled for %sr z Skipped: nginx is enabled for %sN) �fill_in_disabled_usersr� r& r rY rc r^ rA r6 rp r7 )r` r� rE r. r. r/ r? t s zAutoTracer.urls_selectedc c s� d}t t�}| �� }| j�d|� | �� | _| �� }x�|D ]�}| �|j �r^| j j j d7 _ q:||k r�||j | jk r�|d7 }||j d7 < |V q�| j�d|j | j� | j j jd7 _q:| j�d|j |� | j j jd7 _q:W dS )a= Schedule autotracing by sorted list taking into account the limits: - no same task for 10 days - limit per server - limit per domain ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS Return resulting list of URLs scheduled for auto tracing r zSorted scheduled list %sr z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r- � urls_sortedrY rc r~ r] rw r� r r^ rA r8 r% Zper_domain_limitr: r9 )r` Zgeneral_tasks_counterZtasks_counter_per_domainZ sorted_urlsZper_server_smart_limitrE r. r. r/ re � s. zAutoTracer.urls_scheduledc C s, t t| �� �dd� dd�}t |dd� dd�S )zd Sort URLs by total number of requests first and by average duration second c S s | j S )N)r! )�ur. r. r/ �<lambda>� � z(AutoTracer.urls_sorted.<locals>.<lambda>T)�key�reversec S s | j S )N)r$ )r� r. r. r/ r� � r� )�sortedr\ r? )r` Z first_serier. r. r/ r� � s zAutoTracer.urls_sortedc C s t � | _ dS )z6 Fill internal list of disabled users N)r )r` r. r. r/ r� � s z!AutoTracer.fill_in_disabled_usersc C sp | j dk rly,ttd��}t�t| j �|� W dQ R X W n4 tk rj } z| j�dt |�� W dd}~X Y nX dS )zI Save collected statistics for current iteration to file N�wz*Unable to save iteration stats to file: %s) r^ �openr �json�dumpr �OSErrorrY �warningr+ )r` � stats_fileri r. r. r/ rh � s z AutoTracer.save_iteration_statsc C sv yFt t��4}t�|�}t|d |d |d tf |d ��}W dQ R X W n& ttjtfk rl t| j �}Y nX t |�S )z@ Load statistics for latest iteration from file r= r> r? rA N) r� r r� �loadr< r0 r� �JSONDecodeError�KeyErrorr= r )r` r� �_dataZ stat_datar. r. r/ �load_iteration_stats� s z AutoTracer.load_iteration_statsc C s* | � � }|�tt� d tt� �d�� |S )rH r )r Zdisabled_users_quantity)r� �updater� r rv r )r` Z stats_loadedr. r. r/ � get_stats� s zAutoTracer.get_stats)N)N)N)&r' r( r) r* rV rj � staticmethodr� rk r+ r �objectrm r� rp r- rw r\ r~ r� r� r r� r� r� r� r r r� r? re r r� r� rh r� r� � __classcell__r. r. )ra r/ rS j s4 W, rS �__main__zauto_tracer_standalone.log)�filename�level)=r* r[ r� rW � collectionsr Z dataclassesr r r r r �fnmatchr �typingr r r r r � urllib.parser Znumpyr� Zssa.dbr �commonr Zstorager r r Z autotracingr r r Z configurationr Zconfiguration.schemesr �internal.constantsr �internal.utilsr r r Zxray.shared_libraryZshared_libraryrb � ImportErrorr r0 r< rB r+ rR rS r' � basicConfig�DEBUGr� r. r. r. r/ �<module> sJ