관리-도구
편집 파일: storage.cpython-37.pyc
B � d � @ s� d Z ddlZddlmZ ddlmZmZmZmZ ddl Z ddl m Z m Z m Z m Z mZmZ ddlmZmZ eG dd� d��Zee d �d d �Zd d � Zeeee f d �dd�ZdS )zF Represents storage where ssa data is collected, stored and extracted � N)� dataclass)�List�Iterator�Tuple�Dict)�func�cast�case�literal_column�distinct�text)� session_scope� RequestResultc @ s2 e Zd ZU eed< ee ed< eed< eed< dS )� DomainData� domain_name�domain_total_reqs�is_a_wordpress_domain� urls_numberN)�__name__� __module__� __qualname__�str�__annotations__r �int�bool� r r �D/opt/alt/python37/lib/python3.7/site-packages/ssa/modules/storage.pyr s r )�returnc c s� t | ���}|�tjt�dtj�t�tj�t� tj �t� t tj ����tjt�dtj���tjt�dtj��}tj|dd� d�}xf|D ]^\}}t|�}d}dgd }x,|D ]$\} } } } } | |t| �< t || �}q�W t||| |d�V q�W W dQ R X dS ) z7 Iterates data from database domain-by-domain. z%Hc S s | d S )Nr r )�itemr r r �<lambda>/ � z#iter_domains_data.<locals>.<lambda>)�keyr � )r r r r N)r �queryr �domainr �strftime� created_at�Count�id�max� wordpress�countr �path�group_by�order_by� itertools�groupby�tupler r )�engine�dbZresults_by_hourZresults_by_hour_groupedr �groupZdomain_results_by_hourr Zrequests_number_by_hour�_�hourZ requests_numZ is_wordpressZurlsr r r �iter_domains_data s0 r7 c c s� t | ���x}|�tjt�dtj�t�ttj t j ��� d�t� tj�� d�t�ttjt j ��� d���tj|k��ttj�|�jjddid�j���tjt�dtj���tjt�dtj��}d}d gd d gd d gd }}}x�|D ]~\} } } } } |�r<|| k�r<|t||||d �fV d gd d gd d gd }}}| |t| �< | |t| �< | |t| �< | }q�W | t| |||d �fV W dQ R X dS ) z6 Iterates urls data from database url-by-url. z%H�url_throttled_reqs�url_total_reqs� url_slow_reqs� literal_bindsT)�compile_kwargsNr r" )r, r8 r9 r: )r r# r r, r r% r& ZSumr �hitting_limits� sqlalchemy�Integer�labelr' r( �is_slow_request�filterr$ r �in_� expression�compile�stringr- r. �dictr )r2 r Z all_pathsr3 � urls_dataZ previous_pathr8 r9 r: r, r6 Zurl_throttled_reqZ url_total_reqZ url_slow_reqr r r �iter_urls_dataC sH "" " rI c c sv t | ��d}|�tjtj��tj|k��tj�}t� t |�dd� �}x$|D ] \}}|dd� |D �fV qHW W dQ R X dS )zA Get information about durations of requests url-by-url. c S s | d S )Nr r )r r r r r � r z#get_url_durations.<locals>.<lambda>c S s g | ] \}}|�qS r r )�.0r5 �durationr r r � <listcomp>� s z%get_url_durations.<locals>.<listcomp>N) r r# r r, rK rB r$ r. r/ r0 �list)r2 r r3 rH Zdurations_by_pathr! r4 r r r �get_url_durations{ s rN )�__doc__r/ � dataclassesr �typingr r r r r> r r r r r r �ssa.dbr r r r7 rI r r rN r r r r �<module> s '8