관리-도구
편집 파일: md.cpython-37.pyc
B �bG � @ s< d dl mZ d dlmZmZ ddlmZmZ ddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZ G dd� d�ZG dd � d e�ZG d d � d e�ZG d d � d e�ZG dd� de�Z G dd� de�Z G dd� de�Z G dd� de�Z G dd� de�Z ee! ee! e"d�dd�Z#edd �d#e!e$e"e$d �d d!��Z%d"S )$� )� lru_cache)�List�Optional� )� COMMON_SAFE_ASCII_CHARACTERS� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated�is_ascii�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai� remove_accent� unicode_rangec @ sP e Zd ZdZeed�dd�Zedd�dd�Zdd�d d �Ze e d�d d ��Z dS ) �MessDetectorPluginzy Base abstract class used for mess detection plugins. All detectors MUST extend and implement given methods. )� character�returnc C s t �dS )z@ Determine if given character should be fed in. N)�NotImplementedError)�selfr � r �F/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/md.py�eligible s zMessDetectorPlugin.eligibleNc C s t �dS )z� The main routine to be executed upon character. Insert the logic in witch the text would be considered chaotic. N)r )r r r r r �feed$ s zMessDetectorPlugin.feed)r c C s t �dS )zB Permit to reset the plugin to the initial state. N)r )r r r r �reset+ s zMessDetectorPlugin.resetc C s t �dS )z� Compute the chaos ratio based on what your feed() has seen. Must NOT be lower than 0.; No restriction gt 0. N)r )r r r r �ratio1 s zMessDetectorPlugin.ratio) �__name__� __module__� __qualname__�__doc__�str�boolr r r �property�floatr! r r r r r s r c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )� TooManySymbolOrPunctuationPluginN)r c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count� _symbol_count�_character_count�_last_printable_charZ_frenzy_symbol_in_word)r r r r �__init__; s z)TooManySymbolOrPunctuationPlugin.__init__)r r c C s |� � S )N)� isprintable)r r r r r r C s z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkrf|tkrft|�r8| jd7 _n.|�� dkrft|�rft|�dkrf| jd7 _|| _d S )Nr F� ) r- r. r r r+ �isdigitr r r, )r r r r r r F s z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S )Nr )r+ r- r, )r r r r r X s z&TooManySymbolOrPunctuationPlugin.resetc C s0 | j dkrdS | j| j | j }|dkr,|S dS )Nr g g333333�?)r- r+ r, )r Zratio_of_punctuationr r r r! ] s z&TooManySymbolOrPunctuationPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r* : s r* c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )�TooManyAccentuatedPluginN)r c C s d| _ d| _d S )Nr )r- �_accentuated_count)r r r r r/ j s z!TooManyAccentuatedPlugin.__init__)r r c C s |� � S )N)�isalpha)r r r r r r n s z!TooManyAccentuatedPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r- r r4 )r r r r r r q s z TooManyAccentuatedPlugin.feedc C s d| _ d| _d S )Nr )r- r4 )r r r r r w s z TooManyAccentuatedPlugin.resetc C s* | j dkrdS | j| j }|dkr&|S dS )Nr g gffffff�?)r- r4 )r Zratio_of_accentuationr r r r! { s z TooManyAccentuatedPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r3 i s r3 c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )�UnprintablePluginN)r c C s d| _ d| _d S )Nr )�_unprintable_countr- )r r r r r/ � s zUnprintablePlugin.__init__)r r c C s dS )NTr )r r r r r r � s zUnprintablePlugin.eligiblec C s@ |� � dkr.|�� dkr.|dkr.| jd7 _| jd7 _d S )NF�r )�isspacer0 r7 r- )r r r r r r � s zUnprintablePlugin.feedc C s d| _ d S )Nr )r7 )r r r r r � s zUnprintablePlugin.resetc C s | j dkrdS | jd | j S )Nr g � )r- r7 )r r r r r! � s zUnprintablePlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r6 � s r6 c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )� SuspiciousDuplicateAccentPluginN)r c C s d| _ d| _d | _d S )Nr )�_successive_countr- �_last_latin_character)r r r r r/ � s z(SuspiciousDuplicateAccentPlugin.__init__)r r c C s |� � ot|�S )N)r5 r )r r r r r r � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd k rjt|�rjt| j�rj|�� rJ| j�� rJ| jd7 _t|�t| j�krj| jd7 _|| _d S )Nr )r- r= r �isupperr< r )r r r r r r � s z$SuspiciousDuplicateAccentPlugin.feedc C s d| _ d| _d | _d S )Nr )r< r- r= )r r r r r � s z%SuspiciousDuplicateAccentPlugin.resetc C s | j dkrdS | jd | j S )Nr g r1 )r- r< )r r r r r! � s z%SuspiciousDuplicateAccentPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r; � s r; c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )�SuspiciousRangeN)r c C s d| _ d| _d | _d S )Nr )�"_suspicious_successive_range_countr- �_last_printable_seen)r r r r r/ � s zSuspiciousRange.__init__)r r c C s |� � S )N)r0 )r r r r r r � s zSuspiciousRange.eligiblec C sx | j d7 _ |�� s&t|�s&|tkr0d | _d S | jd krD|| _d S t| j�}t|�}t||�rn| jd7 _|| _d S )Nr )r- r9 r r rA r � is_suspiciously_successive_ranger@ )r r �unicode_range_a�unicode_range_br r r r � s zSuspiciousRange.feedc C s d| _ d| _d | _d S )Nr )r- r@ rA )r r r r r � s zSuspiciousRange.resetc C s. | j dkrdS | jd | j }|dk r*dS |S )Nr g r1 g�������?)r- r@ )r Z ratio_of_suspicious_range_usager r r r! � s zSuspiciousRange.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r? � s r? c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )�SuperWeirdWordPluginN)r c C s: d| _ d| _d| _d| _d| _d| _d| _d| _d| _d S )Nr F� ) � _word_count�_bad_word_count�_foreign_long_count�_is_current_word_bad�_foreign_long_watchr- �_bad_character_count�_buffer�_buffer_accent_count)r r r r r/ � s z SuperWeirdWordPlugin.__init__)r r c C s dS )NTr )r r r r r r s z SuperWeirdWordPlugin.eligiblec C s� |� � r�d�| j|g�| _t|�r0| jd7 _| jdkr�t|�dksNt|�r�t|�dkr�t|�dkr�t |�dkr�t |�dkr�t |�dkr�d| _d S | js�d S |� � s�t |�s�t|��r�| j�r�| jd7 _t| j�}| j|7 _|dk�r:| j| dk�rd| _t| jd ��r:| jd �� �r:| jd7 _d| _|dk�r`| j�r`| jd7 _d| _| j�r�| jd7 _| jt| j�7 _d| _d| _d| _d | _n6|d k�r�|�� dk�r�t|��r�d| _| j|7 _d S ) NrF r FT� g�(\���?���� r > �<�>�|�=�-�_�~)r5 �joinrM r rN rK r r r r r r r9 r r rG �lenr- rJ r> rI rH rL r2 r )r r Z buffer_lengthr r r r sR zSuperWeirdWordPlugin.feedc C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )NrF Fr )rM rJ rK rH rG r- rL rI )r r r r r D s zSuperWeirdWordPlugin.resetc C s$ | j dkr| jdkrdS | j| j S )N� r g )rG rI rL r- )r r r r r! N s zSuperWeirdWordPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r rE � s 6 rE c @ s^ e Zd ZdZdd�dd�Zeed�dd�Zedd�d d �Zdd�d d �Z e e d�d d��Z dS )�CjkInvalidStopPluginu� GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and can be easily detected. Searching for the overuse of '丅' and '丄'. N)r c C s d| _ d| _d S )Nr )�_wrong_stop_count�_cjk_character_count)r r r r r/ \ s z CjkInvalidStopPlugin.__init__)r r c C s dS )NTr )r r r r r r ` s z CjkInvalidStopPlugin.eligiblec C s4 |dkr| j d7 _ d S t|�r0| jd7 _d S )N> � 丅� 丄r )r] r r^ )r r r r r r c s zCjkInvalidStopPlugin.feedc C s d| _ d| _d S )Nr )r] r^ )r r r r r j s zCjkInvalidStopPlugin.resetc C s | j dk rdS | j| j S )N� g )r^ r] )r r r r r! n s zCjkInvalidStopPlugin.ratio) r" r# r$ r% r/ r& r' r r r r( r) r! r r r r r\ V s r\ c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d d �Ze e d�d d ��Z dS )�ArchaicUpperLowerPluginN)r c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr- �_last_alpha_seen�_current_ascii_only)r r r r r/ v s z ArchaicUpperLowerPlugin.__init__)r r c C s dS )NTr )r r r r r r � s z ArchaicUpperLowerPlugin.eligiblec C s$ |� � ot|�}|dk}|r�| jdkr�| jdkrV|�� dkrV| jdkrV| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdkr�t |�dkr�d| _| jd k r�|� � r�| j� � s�|� � r�| j� � r�| jdkr�| jd7 _d| _q�d| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r Tr1 ) r5 r rd r2 rh rf re rg rc r- r r> �islower)r r Z is_concernedZ chunk_sepr r r r � s8 z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r- rd re rf rg rc rh )r r r r r � s z ArchaicUpperLowerPlugin.resetc C s | j dkrdS | j| j S )Nr g )r- rf )r r r r r! � s z ArchaicUpperLowerPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r rb u s * rb )rC rD r c C s~ | dks|dkrdS | |kr dS d| kr4d|kr4dS d| ksDd|krHdS d| ksXd|krld| kshd|krldS | � d�|� d� }}x"|D ]}|tkr�q�||kr�dS q�W | dk|dk }}|s�|r�d | ks�d |kr�dS |r�|r�dS d | ks�d |k�r"d | k�sd |k�r dS | d k�s |d k�r"dS d | k�sJd |k�sJ| d k�rz|d k�rzd | k�s^d |k�rbdS d| k�svd|k�rzdS dS )za Determine if two Unicode range seen next to each other can be considered as suspicious. NTFZLatinZ EmoticonsZ Combining� )�Hiragana�KatakanaZCJKZHangulz Basic Latin)rm rl Z PunctuationZForms)�splitr )rC rD Zkeywords_range_aZkeywords_range_b�elZrange_a_jp_charsZrange_b_jp_charsr r r rB � sP rB i )�maxsize皙�����?F)�decoded_sequence�maximum_threshold�debugr c C s� dd� t �� D �}t| �d }d}|dk r0d}n|dkr>d}nd }x|t| d t|��D ]f\}}x |D ]} | �|�rd| �|� qdW |d kr�|| d ks�||d krVtd d � |D ��}||krVP qVW |r�x|D ]} t| j | j � q�W t |d�S )zw Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier. c S s g | ] }|� �qS r r )�.0Zmd_classr r r � <listcomp> s z mess_ratio.<locals>.<listcomp>r g i � i ri � � r c s s | ] }|j V qd S )N)r! )ru �dtr r r � <genexpr>& s z mess_ratio.<locals>.<genexpr>� ) r �__subclasses__rZ �zip�ranger r �sum�print� __class__r! �round) rr rs rt Z detectors�lengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcr �indexZdetectorrz r r r � mess_ratio s* r� N)rq F)&� functoolsr �typingr r Zconstantr r �utilsr r r r r r r r r r r r r r r r r* r3 r6 r; r? rE r\ rb r& r' rB r) r� r r r r �<module> s" D"/ %6Z M D