관리-도구
편집 파일: lexer.cpython-38.pyc
U &?�f� � @ s� d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m Z m Z m Z m Z mZ ddlmZmZmZmZmZmZ ddlmZ dd d d d d dddddddg Ze�d�ZdddddgZe dd � �Z G d d � d e �Z G d d� de d �Z G d!d � d e �Z!G d"d � d e"�Z#G d#d$� d$�Z$e$� Z%G d%d&� d&e&�Z'G d'd(� d(�Z(d)d� Z)G d*d+� d+�Z*e*� Z+d,d� Z,G d-d� d�Z-G d.d� de�Z.G d/d0� d0e �Z/G d1d � d e e/d �Z0G d2d � d �Z1G d3d � d e0�Z2d4d5� Z3G d6d7� d7e/�Z4G d8d9� d9e0e4d �Z5dS ):z� pygments.lexer ~~~~~~~~~~~~~~ Base lexer classes. :copyright: Copyright 2006-2024 by the Pygments team, see AUTHORS. :license: BSD, see LICENSE for details. � N)� apply_filters�Filter)�get_filter_by_name)�Error�Text�Other� Whitespace� _TokenType)� get_bool_opt� get_int_opt� get_list_opt�make_analysator�Future� guess_decode)� regex_opt�Lexer� RegexLexer�ExtendedRegexLexer�DelegatingLexer� LexerContext�include�inherit�bygroups�using�this�default�words�line_rez.*? )s �utf-8)s �� zutf-32)s ��zutf-32be)s ��zutf-16)s ��zutf-16bec C s dS )N� � ��xr r �H/opt/hc_python/lib/python3.8/site-packages/pip/_vendor/pygments/lexer.py�<lambda>" � r$ c @ s e Zd ZdZdd� ZdS )� LexerMetaz� This metaclass automagically converts ``analyse_text`` methods into static methods which always return float values. c C s( d|krt |d �|d< t�| |||�S )N� analyse_text)r �type�__new__)�mcs�name�bases�dr r r# r) + s zLexerMeta.__new__N)�__name__� __module__� __qualname__�__doc__r) r r r r# r& % s r& c @ sn e Zd ZdZdZg Zg Zg Zg ZdZ dZ dZ dZ dd� Z dd� Zdd � Zd d � Zd d � Zddd�Zdd� ZdS )r a" Lexer for a specific language. See also :doc:`lexerdevelopment`, a high-level guide to writing lexers. Lexer classes have attributes used for choosing the most appropriate lexer based on various criteria. .. autoattribute:: name :no-value: .. autoattribute:: aliases :no-value: .. autoattribute:: filenames :no-value: .. autoattribute:: alias_filenames .. autoattribute:: mimetypes :no-value: .. autoattribute:: priority Lexers included in Pygments should have two additional attributes: .. autoattribute:: url :no-value: .. autoattribute:: version_added :no-value: Lexers included in Pygments may have additional attributes: .. autoattribute:: _example :no-value: You can pass options to the constructor. The basic options recognized by all lexers and processed by the base `Lexer` class are: ``stripnl`` Strip leading and trailing newlines from the input (default: True). ``stripall`` Strip all leading and trailing whitespace from the input (default: False). ``ensurenl`` Make sure that the input ends with a newline (default: True). This is required for some lexers that consume input linewise. .. versionadded:: 1.3 ``tabsize`` If given and greater than 0, expand tabs in the input (default: 0). ``encoding`` If given, must be an encoding name. This encoding will be used to convert the input string to Unicode, if it is not already a Unicode string (default: ``'guess'``, which uses a simple UTF-8 / Locale / Latin1 detection. Can also be ``'chardet'`` to use the chardet library, if it is installed. ``inencoding`` Overrides the ``encoding`` if given. Nr c K s� || _ t|dd�| _t|dd�| _t|dd�| _t|dd�| _|�dd �| _|�d �pZ| j| _g | _ t |d d �D ]}| � |� qpd S )a� This constructor takes arbitrary options as keyword arguments. Every subclass must first process its own options and then call the `Lexer` constructor, since it processes the basic options like `stripnl`. An example looks like this: .. sourcecode:: python def __init__(self, **options): self.compress = options.get('compress', '') Lexer.__init__(self, **options) As these options must all be specifiable as strings (due to the command line usage), there are various utility functions available to help with that, see `Utilities`_. �stripnlT�stripallF�ensurenl�tabsizer �encoding�guess� inencoding�filtersr N) �optionsr r2 r3 r4 r r5 �getr6 r9 r � add_filter)�selfr: �filter_r r r# �__init__� s zLexer.__init__c C s2 | j r d| jj� d| j �d�S d| jj� d�S d S )Nz<pygments.lexers.z with �>)r: � __class__r. �r= r r r# �__repr__� s zLexer.__repr__c K s& t |t�st|f|�}| j�|� dS )z8 Add a new stream filter to this lexer. N)� isinstancer r r9 �append)r= r> r: r r r# r< � s zLexer.add_filterc C s dS )a� A static method which is called for lexer guessing. It should analyse the text and return a float in the range from ``0.0`` to ``1.0``. If it returns ``0.0``, the lexer will not be selected as the most probable one, if it returns ``1.0``, it will be selected immediately. This is used by `guess_lexer`. The `LexerMeta` metaclass automatically wraps this function so that it works like a static method (no ``self`` or ``cls`` parameter) and the return value is automatically converted to `float`. If the return value is an object that is boolean `False` it's the same as if the return values was ``0.0``. Nr )�textr r r# r' � s zLexer.analyse_textc C s� t |t�s�| jdkr"t|�\}}n�| jdkr�z td��W n, tk rd } ztd�|�W 5 d}~X Y nX d}tD ].\}}|�|�rn|t|�d� �|d�} q�qn|dkr�t � |dd� �}|�|� d�p�d d�}|}n&|�| j�}|�d �r�|td �d� }n |�d ��r|td �d� }|� d d �}|� d d �}| j �rB|�� }n| j�rT|�d �}| jdk�rl|�| j�}| j�r�|�d ��s�|d 7 }|S )zVApply preprocessing such as decoding the input, removing BOM and normalizing newlines.r7 �chardetz chardet is not vendored by pipzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/N�replacei r6 r u z � � r )rD �strr6 r � ImportError� _encoding_map� startswith�len�decoderG �detectr; rH r3 �stripr2 r5 � expandtabsr4 �endswith)r= rF �_�e�decoded�bomr6 �encr r r# �_preprocess_lexer_input� sJ � � z Lexer._preprocess_lexer_inputFc s4 � � ���� �fdd�}|� }|s0t|� j� �}|S )ae This method is the basic interface of a lexer. It is called by the `highlight()` function. It must process the text and return an iterable of ``(tokentype, value)`` pairs from `text`. Normally, you don't need to override this method. The default implementation processes the options recognized by all lexers (`stripnl`, `stripall` and so on), and then yields all tokens from `get_tokens_unprocessed()`, with the ``index`` dropped. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. c 3 s$ � � ��D ]\} }}||fV q d S �N)�get_tokens_unprocessed)rU �t�v�r= rF r r# �streamer s z"Lexer.get_tokens.<locals>.streamer)rZ r r9 )r= rF � unfilteredr` �streamr r_ r# � get_tokens� s zLexer.get_tokensc C s t �dS )aS This method should process the text and return an iterable of ``(index, tokentype, value)`` tuples where ``index`` is the starting position of the token within the input text. It must be overridden by subclasses. It is recommended to implement it as a generator to maximize effectiveness. N)�NotImplementedErrorr_ r r r# r\ s z Lexer.get_tokens_unprocessed)F)r. r/ r0 r1 r+ �aliases� filenames�alias_filenames� mimetypes�priority�urlZ version_addedZ_exampler? rC r< r' rZ rc r\ r r r r# r 1 s" ; 1 )� metaclassc @ s$ e Zd ZdZefdd�Zdd� ZdS )r a This lexer takes two lexer as arguments. A root lexer and a language lexer. First everything is scanned using the language lexer, afterwards all ``Other`` tokens are lexed using the root lexer. The lexers from the ``template`` lexer package use this base lexer. c K s0 |f |�| _ |f |�| _|| _tj| f|� d S r[ )� root_lexer�language_lexer�needler r? )r= � _root_lexer�_language_lexer�_needler: r r r# r? - s zDelegatingLexer.__init__c C s� d}g }g }| j �|�D ]H\}}}|| jkrP|rF|�t|�|f� g }||7 }q|�|||f� q|rx|�t|�|f� t|| j�|��S )N� )rm r\ rn rE rO � do_insertionsrl )r= rF �buffered� insertions� lng_buffer�ir] r^ r r r# r\ 3 s �z&DelegatingLexer.get_tokens_unprocessedN)r. r/ r0 r1 r r? r\ r r r r# r # s c @ s e Zd ZdZdS )r zI Indicates that a state should include rules from another state. N�r. r/ r0 r1 r r r r# r J s c @ s e Zd ZdZdd� ZdS )�_inheritzC Indicates the a state should inherit from its superclass. c C s dS )Nr r rB r r r# rC U s z_inherit.__repr__N)r. r/ r0 r1 rC r r r r# ry Q s ry c @ s e Zd ZdZdd� Zdd� ZdS )�combinedz: Indicates a state combined from multiple states. c G s t �| |�S r[ )�tupler) )�cls�argsr r r# r) ` s zcombined.__new__c G s d S r[ r )r= r} r r r# r? c s zcombined.__init__N)r. r/ r0 r1 r) r? r r r r# rz [ s rz c @ sF e Zd ZdZdd� Zddd�Zddd�Zdd d �Zd d � Zd d� Z dS )� _PseudoMatchz: A pseudo match object constructed from a string. c C s || _ || _d S r[ )�_text�_start)r= �startrF r r r# r? m s z_PseudoMatch.__init__Nc C s | j S r[ )r� �r= �argr r r# r� q s z_PseudoMatch.startc C s | j t| j� S r[ )r� rO r r� r r r# �endt s z_PseudoMatch.endc C s |r t d��| jS )Nz No such group)� IndexErrorr r� r r r# �groupw s z_PseudoMatch.groupc C s | j fS r[ )r rB r r r# �groups| s z_PseudoMatch.groupsc C s i S r[ r rB r r r# � groupdict s z_PseudoMatch.groupdict)N)N)N) r. r/ r0 r1 r? r� r� r� r� r� r r r r# r~ h s r~ c s d� fdd� }|S )zL Callback that yields multiple actions for each group in the match. Nc 3 s� t � �D ]�\}}|d kr qqt|�tkrR|�|d �}|r�|�|d �||fV q|�|d �}|d k r|r||�|d �|_|| t|�|d �|�|�D ]}|r�|V q�q|r�|�� |_d S )N� )� enumerater( r r� r� �posr~ r� )�lexer�match�ctxrw �action�data�item�r} r r# �callback� s& � zbygroups.<locals>.callback)Nr )r} r� r r� r# r � s c @ s e Zd ZdZdS )�_ThiszX Special singleton used for indicating the caller class. Used by ``using``. Nrx r r r r# r� � s r� c sj i �d�kr:�� d�}t|ttf�r.|�d< n d|f�d< � tkrTd��fdd� }nd � ��fdd� }|S ) a� Callback that processes the match with a different lexer. The keyword arguments are forwarded to the lexer, except `state` which is handled separately. `state` specifies the state that the new lexer will start in, and can be an enumerable such as ('root', 'inline', 'string') or a simple string which is assumed to be on top of the root state. Note: For that to work, `_other` must not be an `ExtendedRegexLexer`. �state�stack�rootNc 3 sj �r �� | j� | jf ��}n| }|�� }|j|�� f� �D ]\}}}|| ||fV q<|rf|�� |_d S r[ )�updater: rA r� r\ r� r� r� �r� r� r� �lx�srw r] r^ )� gt_kwargs�kwargsr r# r� � s zusing.<locals>.callbackc 3 s^ �� | j� � f ��}|�� }|j|�� f��D ]\}}}|| ||fV q0|rZ|�� |_d S r[ )r� r: r� r\ r� r� r� r� ��_otherr� r� r r# r� � s )N)N)�poprD �listr{ r )r� r� r� r� r r� r# r � s c @ s e Zd ZdZdd� ZdS )r z� Indicates a state or state action (e.g. #pop) to apply. For example default('#pop') is equivalent to ('', Token, '#pop') Note that state tuples may be used as well. .. versionadded:: 2.0 c C s || _ d S r[ )r� )r= r� r r r# r? � s zdefault.__init__N)r. r/ r0 r1 r? r r r r# r � s c @ s" e Zd ZdZddd�Zdd� ZdS ) r z� Indicates a list of literal words that is transformed into an optimized regex that matches any of the words. .. versionadded:: 2.0 rr c C s || _ || _|| _d S r[ )r �prefix�suffix)r= r r� r� r r r# r? � s zwords.__init__c C s t | j| j| jd�S )N�r� r� )r r r� r� rB r r r# r; � s z words.getN)rr rr )r. r/ r0 r1 r? r; r r r r# r � s c @ sJ e Zd ZdZdd� Zdd� Zdd� Zdd � Zdd d �Zd d� Z dd� Z d S )�RegexLexerMetazw Metaclass for RegexLexer, creates the self._tokens attribute from self.tokens on the first instantiation. c C s t |t�r|�� }t�||�jS )zBPreprocess the regular expression component of a token definition.)rD r r; �re�compiler� )r| �regex�rflagsr� r r r# �_process_regex� s z RegexLexerMeta._process_regexc C s& t |�tks"t|�s"td|����|S )z5Preprocess the token component of a token definition.z0token type must be simple type or callable, not )r( r �callable�AssertionError)r| �tokenr r r# �_process_token s �z RegexLexerMeta._process_tokenc C s t |t�rf|dkrdS ||kr$|fS |dkr0|S |dd� dkrRt|dd� � S dsdtd|����n�t |t�r�d | j }| jd 7 _g }|D ].}||ks�td |����|�| �|||�� q�|||< |fS t |t��r|D ] }||ks�|d ks�td| ��q�|S d�std |����dS )z=Preprocess the state transition action of a token definition.�#pop����#pushN� z#pop:Fzunknown new state z_tmp_%dr� zcircular state ref )r� r� zunknown new state def ) rD rK �intr� rz �_tmpname�extend�_process_stater{ )r| � new_state� unprocessed� processed� tmp_state�itokens�istater r r# �_process_new_state s> � ��z!RegexLexerMeta._process_new_statec C s� t |t�std|����|d dks2td|����||krB|| S g }||< | j}|| D �]>}t |t�r�||ks�td|����|�| �||t|��� q\t |t�r�q\t |t�r�| � |j ||�}|� t � d�jd|f� q\t|�tks�td|����z| �|d ||�}W nJ tk �rV } z*td |d �d |�d | �d | � ��| �W 5 d} ~ X Y nX | �|d �} t|�dk�rzd}n| � |d ||�}|� || |f� q\|S )z%Preprocess a single state definition.zwrong state name r �#zinvalid state name zcircular state reference rr Nzwrong rule def zuncompilable regex z in state z of z: r� � )rD rK r� �flagsr r� r� ry r r� r� rE r� r� r� r( r{ r� � Exception� ValueErrorr� rO ) r| r� r� r� �tokensr� �tdefr� �rex�errr� r r r# r� ) sB � 8 �z RegexLexerMeta._process_stateNc C s< i }| j |< |p| j| }t|�D ]}| �|||� q$|S )z-Preprocess a dictionary of token definitions.)� _all_tokensr� r� r� )r| r+ � tokendefsr� r� r r r# �process_tokendefT s z RegexLexerMeta.process_tokendefc C s� i }i }| j D ]�}|j�di �}|�� D ]�\}}|�|�}|dkr||||< z|�t�}W n tk rp Y q(Y nX |||< q(|�|d�}|dkr�q(||||d �< z|�t�} W n tk r� Y q(X || ||< q(q|S )a Merge tokens from superclasses in MRO order, returning a single tokendef dictionary. Any state that is not defined by a subclass will be inherited automatically. States that *are* defined by subclasses will, by default, override that state in the superclass. If a subclass wishes to inherit definitions from a superclass, it can use the special value "inherit", which will cause the superclass' state definition to be included at that point in the state. r� Nr� )�__mro__�__dict__r; �items�indexr r� r� ) r| r� � inheritable�c�toksr� r� �curitems� inherit_ndx� new_inh_ndxr r r# � get_tokendefs\ s0 z RegexLexerMeta.get_tokendefsc O sL d| j kr:i | _d| _t| d�r(| jr(n| �d| �� �| _tj | f|�|�S )z:Instantiate cls after preprocessing its token definitions.�_tokensr �token_variantsrr ) r� r� r� �hasattrr� r� r� r� r( �__call__)r| r} �kwdsr r r# r� � s zRegexLexerMeta.__call__)N) r. r/ r0 r1 r� r� r� r� r� r� r� r r r r# r� � s #+ 1r� c @ s$ e Zd ZdZejZi Zddd�ZdS )r z� Base for simple stateful regular expression-based lexers. Simplifies the lexing process so that you need only provide a list of states and regular expressions. �r� c c s� d}| j }t|�}||d }|D �]*\}}} |||�} | r"|dk rrt|�tkrb||| �� fV n|| | �E dH | �� }| dk �rJt| t�r�| D ]D} | dkr�t|�dkr�|� � q�| dkr�|� |d � q�|� | � q�ndt| t ��rt | �t|�k�r|dd�= n || d�= n.| dk�r*|� |d � nd�s>t d| ����||d } q q"zP|| d k�r�d g}|d }|td fV |d7 }W q |t|| fV |d7 }W q tk �r� Y �q�Y q X q dS ) z~ Split ``text`` into (tokentype, text) pairs. ``stack`` is the initial stack (default: ``['root']``) r r� Nr� r� r� F�wrong state def: rI r� )r� r� r( r r� r� rD r{ rO r� rE r� �absr� r r r� ) r= rF r� r� r� � statestack� statetokens�rexmatchr� r� �mr� r r r# r\ � sR z!RegexLexer.get_tokens_unprocessedN)r� ) r. r/ r0 r1 r� � MULTILINEr� r� r\ r r r r# r � s c @ s" e Zd ZdZddd�Zdd� ZdS )r z9 A helper object that holds lexer position data. Nc C s* || _ || _|pt|�| _|p"dg| _d S )Nr� )rF r� rO r� r� )r= rF r� r� r� r r r# r? s zLexerContext.__init__c C s d| j �d| j�d| j�d�S )Nz LexerContext(z, �))rF r� r� rB r r r# rC s zLexerContext.__repr__)NN)r. r/ r0 r1 r? rC r r r r# r � s c @ s e Zd ZdZddd�ZdS )r zE A RegexLexer that uses a context object to store its state. Nc c s> | j }|s t|d�}|d }n|}||jd }|j}|D �]b\}}}|||j|j�} | r:|dk r�t|�tkr�|j|| �� fV | �� |_n$|| | |�E dH |s�||jd }|dk �r�t |t ��r |D ]P} | dkr�t |j�dkr�|j� � q�| dk�r|j� |jd � q�|j� | � q�nnt |t��rZt|�t |j�k�rL|jdd�= n |j|d�= n2|dk�rx|j� |jd � nd�s�td |����||jd } q6q:zz|j|jk�r�W �q:||j d k�r�dg|_|d }|jtd fV | jd7 _W q6|jt||j fV | jd7 _W q6 tk �r6 Y �q:Y q6X q6dS ) z Split ``text`` into (tokentype, text) pairs. If ``context`` is given, use this lexer context instead. r r� r� Nr� r� r� Fr� rI )r� r r� rF r� r� r( r r� rD r{ rO r� rE r� r� r� r r r� ) r= rF �contextr� r� r� r� r� r� r� r� r r r# r\ s` z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r. r/ r0 r1 r\ r r r r# r s c c s� t | �} zt| �\}}W n tk r8 |E dH Y dS X d}d}|D ]�\}}}|dkr\|}d} |�r|t|� |k�r|| || � } | r�||| fV |t| �7 }|D ]"\} } } || | fV |t| �7 }q�|| } zt| �\}}W q` tk �r d}Y �qY q`X q`| t|�k rF|||| d� fV |t|�| 7 }qF|�r�|�pHd}|D ]$\}}}|||fV |t|�7 }�qNzt| �\}}W n tk �r� d}Y �q�Y nX �q:dS )ag Helper for lexers which must combine the results of several sublexers. ``insertions`` is a list of ``(index, itokens)`` pairs. Each ``itokens`` iterable should be inserted at position ``index`` into the token stream given by the ``tokens`` argument. The result is a combined token stream. TODO: clean up the code here. NTr F)�iter�next� StopIterationrO )ru r� r� r� �realpos�insleftrw r] r^ �oldi�tmpval�it_index�it_token�it_value�pr r r# rs S sN rs c @ s e Zd ZdZdd� ZdS )�ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |t�r t|j|j|jd��n|�t��|��tjf� ���fdd� }|S )Nr� c s` � j d ���fddg�}t�� }��| ||�}t�� }|d d7 < |d || 7 < |S )Nr� r r r� )� _prof_data� setdefault�timer� )rF r� �endpos�info�t0�res�t1�r| �compiledr� r� r r# � match_func� s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) rD r r r� r� r� r� �sys�maxsize)r| r� r� r� r� r r� r# r� � s � z&ProfilingRegexLexerMeta._process_regexN)r. r/ r0 r1 r� r r r r# r� � s r� c @ s" e Zd ZdZg ZdZddd�ZdS )�ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.� r� c # s� � j j�i � t�� ||�E d H � j j�� }tdd� |�� D �� fdd�dd�}tdd� |D ��}t � t d� j j t |�|f � t d � t d d � t d � |D ]}t d | � q�t d � d S )Nc s sN | ]F\\}}\}}|t |��d ��dd�dd� |d| d| | fV qdS )zu'z\\�\N�A i� )�reprrR rH )�.0r� �r�nr] r r r# � <genexpr>� s � �z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s | � j S r[ )�_prof_sort_indexr! rB r r# r$ � r% z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)�key�reversec s s | ]}|d V qdS )� Nr )r r" r r r# r � s z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r� r� zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) rA r� rE r r\ r� �sortedr� �sum�printr. rO )r= rF r� �rawdatar� � sum_totalr- r rB r# r\ � s( � �� z*ProfilingRegexLexer.get_tokens_unprocessedN)r� )r. r/ r0 r1 r� r r\ r r r r# r� � s r� )6r1 r� r� r� �pip._vendor.pygments.filterr r � pip._vendor.pygments.filtersr �pip._vendor.pygments.tokenr r r r r �pip._vendor.pygments.utilr r r r r r � pip._vendor.pygments.regexoptr �__all__r� r rM � staticmethod�_default_analyser( r& r r rK r ry r r{ rz r~ r r� r r r r r� r r r rs r� r� r r r r# �<module> sd � � s' 2 (aH@