관리-도구
편집 파일: utils.cpython-37.opt-1.pyc
B �b\$ � @ s, y d dl ZW n ek r( d dlZY nX d dlZd dlZd dlmZ d dlmZ d dl m Z d dl m Z d dl mZmZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ e ed �e e d �d d ��Z e ed �e e d �dd��Z e ed �e ee d �dd��Z e ed �e e d �dd��Z!e e d �dd�Z"e ed �e e d �dd��Z#e ed �e e d �dd��Z$e ed �e e d �dd��Z%e ed �e e d �d d ��Z&e ed �e e d �d d ��Z'e e d �d d!�Z(e ed �e e d �d"d#��Z)e ed �e e d �d$d%��Z*e ed �e e d �d&d'��Z+e ed �e e d �d(d)��Z,e ed �e e d �d*d+��Z-e e.e�d �e e d,�d-d.��Z/dNe0e1ee d0�d1d2�Z2e d3d �e e d4�d5d6��Z3e0eee e0f d7�d8d9�Z4e e d:�d;d<�Z5dOe e e d>�d?d@�Z6e ee dA�dBdC�Z7e e e8dD�dEdF�Z9e e e dD�dGdH�Z:dIej;dJfe e1e ddK�dLdM�Z<dS )P� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize)� character�returnc C sT yt �| �}W n tk r" dS X d|kpRd|kpRd|kpRd|kpRd|kpRd|kS )NFz WITH GRAVEz WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz WITH TILDE)� unicodedata�name� ValueError)r � description� r �I/opt/alt/python37/lib/python3.7/site-packages/charset_normalizer/utils.py�is_accentuated s r c C s. t �| �}|s| S |�d�}tt|d d��S )N� r � )r Z decomposition�split�chr�int)r Z decomposedZcodesr r r � remove_accent* s r" c C s. t | �}x t�� D ]\}}||kr|S qW dS )zK Retrieve the Unicode range official name from a single character. N)�ordr �items)r Z character_ord� range_nameZ ord_ranger r r � unicode_range5 s r& c C s, yt �| �}W n tk r" dS X d|kS )NFZLATIN)r r r )r r r r r �is_latinC s r' c C s( y| � d� W n tk r" dS X dS )N�asciiFT)�encode�UnicodeEncodeError)r r r r �is_asciiL s r+ c C s2 t �| �}d|krdS t| �}|d kr*dS d|kS )N�PTFZ Punctuation)r �categoryr& )r �character_category�character_ranger r r �is_punctuationT s r0 c C s: t �| �}d|ksd|kr dS t| �}|d kr2dS d|kS )N�S�NTFZForms)r r- r& )r r. r/ r r r � is_symbolc s r3 c C s t | �}|d krdS d|kS )NFZ Emoticons)r& )r r/ r r r � is_emoticonr s r4 c C s& | � � s| dkrdS t�| �}d|kS )N> �+�;�<�>� ｜�,T�Z)�isspacer r- )r r. r r r � is_separator| s r= c C s | � � | �� kS )N)�islower�isupper)r r r r �is_case_variable� s r@ c C s t �| �}|dkS )NZCo)r r- )r r. r r r �is_private_use_only� s rA c C s, yt �| �}W n tk r" dS X d|kS )NFZCJK)r r r )r �character_namer r r �is_cjk� s rC c C s, yt �| �}W n tk r" dS X d|kS )NFZHIRAGANA)r r r )r rB r r r � is_hiragana� s rD c C s, yt �| �}W n tk r" dS X d|kS )NFZKATAKANA)r r r )r rB r r r � is_katakana� s rE c C s, yt �| �}W n tk r" dS X d|kS )NFZHANGUL)r r r )r rB r r r � is_hangul� s rF c C s, yt �| �}W n tk r" dS X d|kS )NFZTHAI)r r r )r rB r r r �is_thai� s rG )r% r c s t � fdd�tD ��S )Nc 3 s | ]}|� kV qd S )Nr )�.0�keyword)r% r r � <genexpr>� s z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr )r% r )r% r �is_unicode_range_secondary� s rL � )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS xJ|D ]B}|�� � dd�}x,t � � D ] \}}||kr�|S ||krl|S qlW qNW dS )zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. Nr( �ignore)�errorsr �-�_) � isinstance�bytes� TypeError�lenr r �min�decode�lower�replacer r$ )rN rO Zseq_lenZresultsZspecified_encoding�encoding_alias� encoding_ianar r r �any_specified_encoding� s r^ � )r r c C s | dkp t t�d�| ��jt�S )zQ Verify is a specific encoding is a multi byte one based on it IANA name > � utf_16_be� utf_8_sig�utf_32�utf_16� utf_16_le� utf_32_be�utf_7� utf_32_le�utf_8z encodings.{})� issubclass� importlib� import_module�formatr r )r r r r �is_multi_byte_encoding� s rm )rN r c C sJ xDt D ]<}t | }t|t�r"|g}x |D ]}| �|�r(||fS q(W qW dS )z9 Identify and extract SIG/BOM in given sequence. )N� )r rT rU � startswith)rN � iana_encodingZmarksZmarkr r r �identify_sig_or_bom� s rq )rp r c C s | dkS )N> rb rc r )rp r r r �should_strip_sig_or_bom s rr T)�cp_name�strictr c C sL | � � �dd�} x$t�� D ]\}}| ||gkr|S qW |rHtd�| ���| S )NrR rS z Unable to retrieve IANA for '{}')rZ r[ r r$ r rl )rs rt r\ r] r r r � iana_name s ru )�decoded_sequencer c C s8 t � }x(| D ] }t|�}|d kr"q |�|� q W t|�S )N)�setr& �add�list)rv Zrangesr r/ r r r � range_scan s rz )� iana_name_a� iana_name_br c C s� t | �st |�rdS t�d�| ��j}t�d�|��j}|dd�}|dd�}d}x6td�D ]*}t|g�}|�|�|�|�krZ|d7 }qZW |d S ) Ng z encodings.{}rP )rQ r � r � )rm rj rk rl r �rangerU rY ) r{ r| Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ to_be_decodedr r r � cp_similarity+ s r� c C s | t ko|t | kS )z� Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. )r )r{ r| r r r � is_cp_similar@ s r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r �level� format_stringr c C s: t �| �}|�|� t �� }|�t �|�� |�|� d S )N)�loggingZ getLoggerZsetLevelZ StreamHandlerZ setFormatter� FormatterZ addHandler)r r� r� Zlogger�handlerr r r �set_logging_handlerK s r� )rM )T)=Z unicodedata2r � ImportErrorrj r� �codecsr Zencodings.aliasesr � functoolsr �rer �typingr r r r r Z_multibytecodecr Zconstantr r r r r r �str�boolr r" r& r' r+ r0 r3 r4 r= r@ rA rC rD rE rF rG rW rL rU r! r^ rm rq rr ru rz �floatr� r� �INFOr� r r r r �<module> sr