관리-도구
편집 파일: utils.cpython-38.pyc
U *�ev. � @ sl d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m Z d dl m Z m Z mZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ eed �ee d �d d ��Z eed �eed �dd��Z eed �eee d �dd��Z eed �ee d �dd��Z eed �ee d �dd��Z!eed �ee d �dd��Z"eed �ee d �dd��Z#eed �ee d �dd��Z$eed �ee d �d d ��Z%eed �ee d �d d ��Z&eed �ee d �d d!��Z'eed �ee d �d"d#��Z(eed �ee d �d$d%��Z)eed �ee d �d&d'��Z*eed �ee d �d(d)��Z+eed �ee d �d*d+��Z,ee-e�d �ee d,�d-d.��Z.eed �ee d �d/d0��Z/dSe0e1ee d2�d3d4�Z2ed5d �ee d6�d7d8��Z3e0eee e0f d9�d:d;�Z4ee d<�d=d>�Z5dTee ed@�dAdB�Z6ee e dC�dDdE�Z7eee8dF�dGdH�Z9eee dF�dIdJ�Z:dKej;dLfee1eddM�dNdO�Z<dUe0ee=e1e e e0e ee e eddf dP� dQdR�Z>dS )V� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)� Generator�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize)� character�returnc C sf zt �| �}W n tk r$ Y dS X d|kpdd|kpdd|kpdd|kpdd|kpdd|kpdd|kpdd |kS ) NFz WITH GRAVEz WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz WITH TILDEz WITH MACRONzWITH RING ABOVE�� unicodedata�name� ValueError�r � description� r �H/opt/hc_python/lib64/python3.8/site-packages/charset_normalizer/utils.py�is_accentuated s& �������r c C s. t �| �}|s| S |�d�}tt|d d��S )N� r � )r � decomposition�split�chr�int)r Z decomposed�codesr r r � remove_accent( s r' c C s. t | �}t�� D ]\}}||kr| S qdS )zK Retrieve the Unicode range official name from a single character. N)�ordr �items)r Z character_ord� range_nameZ ord_ranger r r � unicode_range3 s r+ c C s. zt �| �}W n tk r$ Y dS X d|kS )NFZLATINr r r r r �is_latinA s r, c C s2 t �| �}d|krdS t| �}|d kr*dS d|kS )N�PTF� Punctuation�r �categoryr+ �r �character_category�character_ranger r r �is_punctuationJ s r4 c C sB t �| �}d|ksd|kr dS t| �}|d kr2dS d|ko@|dkS )N�S�NTFZFormsZLor/ r1 r r r � is_symbolY s r7 c C s$ t | �}|d krdS d|kp"d|kS )NFZ EmoticonsZ Pictographs)r+ )r r3 r r r � is_emoticonh s r8 c C s. | � � s| dkrdS t�| �}d|kp,|dkS )N> � ｜�+�>�<T�Z> �Pc�Pd�Po)�isspacer r0 )r r2 r r r � is_separatorr s rB c C s | � � | �� kS �N)�islower�isupper�r r r r �is_case_variable| s rG c C s. zt �| �}W n tk r$ Y dS X d|kS )NF�CJKr �r Zcharacter_namer r r �is_cjk� s rJ c C s. zt �| �}W n tk r$ Y dS X d|kS )NFZHIRAGANAr rI r r r � is_hiragana� s rK c C s. zt �| �}W n tk r$ Y dS X d|kS )NFZKATAKANAr rI r r r � is_katakana� s rL c C s. zt �| �}W n tk r$ Y dS X d|kS )NFZHANGULr rI r r r � is_hangul� s rM c C s. zt �| �}W n tk r$ Y dS X d|kS )NFZTHAIr rI r r r �is_thai� s rN c C s. zt �| �}W n tk r$ Y dS X d|kS )NF�ARABICr rI r r r � is_arabic� s rP c C s6 zt �| �}W n tk r$ Y dS X d|ko4d|kS )NFrO z ISOLATED FORMr rI r r r �is_arabic_isolated_form� s rQ )r* r c s t � fdd�tD ��S )Nc 3 s | ]}|� kV qd S rC r )�.0�keyword�r* r r � <genexpr>� s z-is_unicode_range_secondary.<locals>.<genexpr>)�anyr rT r rT r �is_unicode_range_secondary� s rW c C s( | � � dko&| �� dko&| dko&| dkS )NF�u )rA � isprintablerF r r r �is_unprintable� s ���rZ � )�sequence� search_zoner c C s� t | t�st�t| �}tt| dt||�� jddd��}t|�dkrHdS |D ]N}|�� � dd�}t � � D ]0\}}||kr�| S ||krh| S qhqLdS )zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. N�ascii�ignore��errorsr �-�_) � isinstance�bytes� TypeError�lenr r �min�decode�lower�replacer r) )r\ r] Zseq_len�resultsZspecified_encoding�encoding_alias� encoding_ianar r r �any_specified_encoding� s" � ro � )r r c C s | dkp t t�d�| ��jt�S )zQ Verify is a specific encoding is a multi byte one based on it IANA name > �utf_32�utf_7� utf_8_sig� utf_16_le� utf_16_be� utf_32_be� utf_32_le�utf_16�utf_8� encodings.{})� issubclass� importlib� import_module�formatr r )r r r r �is_multi_byte_encoding� s ��r )r\ r c C sJ t D ]@}t | }t|t�r |g}|D ] }| �|�r$||f S q$qdS )z9 Identify and extract SIG/BOM in given sequence. )N� )r rd re � startswith)r\ � iana_encodingZmarks�markr r r �identify_sig_or_bom s r� )r� r c C s | dkS )N> rq rx r )r� r r r �should_strip_sig_or_bom s r� T)�cp_name�strictr c C sL | � � �dd�} t�� D ] \}}| ||fkr| S q|rHtd�| ���| S )Nrb rc z Unable to retrieve IANA for '{}')rj rk r r) r r~ )r� r� rm rn r r r � iana_name" s r� )�decoded_sequencer c C s4 t � }| D ] }t|�}|d kr q |�|� q t|�S rC )�setr+ �add�list)r� �rangesr r3 r r r � range_scan2 s r� )� iana_name_a� iana_name_br c C s� t | �st |�rdS t�d�| ��j}t�d�|��j}|dd�}|dd�}d}td�D ]*}t|g�}|�|�|�|�krX|d7 }qX|d S ) Ng rz r_ r` r � r � )r r| r} r~ r �rangere ri ) r� r� Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ to_be_decodedr r r � cp_similarity@ s �� r� c C s | t ko|t | kS )z� Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. )r )r� r� r r r � is_cp_similarX s �r� Zcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s)r �level� format_stringr c C s: t �| �}|�|� t �� }|�t �|�� |�|� d S rC )�logging� getLogger�setLevel� StreamHandler� setFormatter� Formatter� addHandler)r r� r� �logger�handlerr r r �set_logging_handlerc s r� ) � sequencesrn �offsets� chunk_size�bom_or_sig_available�strip_sig_or_bom� sig_payload�is_multi_byte_decoder�decoded_payloadr c c s* |r6|dkr6|D ]"} || | | � } | s, q4| V qn�|D ]�} | | } | t | �d krXq:| | | | � } |r||dkr||| } | j||r�dndd�} |�r | dk�r t|d�} |�r | d | � |k�r t| | d d �D ]H}| || � } |r�|dkr�|| } | j|dd�} | d | � |kr� �q q�| V q:d S ) NF� r_ r� r` r r! � ���)rg ri rh r� )r� rn r� r� r� r� r� r� r� r� �chunkZ chunk_endZ cut_sequenceZchunk_partial_size_chk�jr r r �cut_sequence_chunksp s> � �� r� )r[ )T)N)?r| r� r �codecsr Zencodings.aliasesr � functoolsr �rer �typingr r r r r r Z_multibytecodecr Zconstantr r r r r r �str�boolr r' r+ r, r4 r7 r8 rB rG rJ rK rL rM rN rP rQ rg rW rZ re r% ro r r� r� r� r� �floatr� r� �INFOr� r� r� r r r r �<module> s� �� � �