LEX(1)                     POSIX Programmer's Manual                    LEX(1)



PPRROOLLOOGG
       This manual page is part of the POSIX Programmer's Manual.  The OS
       implementation of this interface may differ (consult the corresponding
       OS manual page for details of OS behavior), or the interface may not be
       implemented on OS.

NNAAMMEE
       lex -- generate programs for lexical tasks (DDEEVVEELLOOPPMMEENNTT)

SSYYNNOOPPSSIISS
       lex [[-t]] [[-n|-v]] [[_f_i_l_e...]]

DDEESSCCRRIIPPTTIIOONN
       The _l_e_x utility shall generate C programs to be used in lexical
       processing of character input, and that can be used as an interface to
       _y_a_c_c.  The C programs shall be generated from _l_e_x source code and
       conform to the ISO C standard, without depending on any undefined,
       unspecified, or implementation-defined behavior, except in cases where
       the code is copied directly from the supplied source, or in cases that
       are documented by the implementation. Usually, the _l_e_x utility shall
       write the program it generates to the file lleexx..yyyy..cc; the state of this
       file is unspecified if _l_e_x exits with a non-zero exit status. See the
       EXTENDED DESCRIPTION section for a complete description of the _l_e_x
       input language.

OOPPTTIIOONNSS
       The _l_e_x utility shall conform to the Base Definitions volume of
       POSIX.1-2017, _S_e_c_t_i_o_n _1_2_._2, _U_t_i_l_i_t_y _S_y_n_t_a_x _G_u_i_d_e_l_i_n_e_s, except for
       Guideline 9.

       The following options shall be supported:

       --nn        Suppress the summary of statistics usually written with the
                 --vv option. If no table sizes are specified in the _l_e_x source
                 code and the --vv option is not specified, then --nn is implied.

       --tt        Write the resulting program to standard output instead of
                 lleexx..yyyy..cc.

       --vv        Write a summary of _l_e_x statistics to the standard output.
                 (See the discussion of _l_e_x table sizes in _D_e_f_i_n_i_t_i_o_n_s _i_n
                 _l_e_x.)  If the --tt option is specified and --nn is not specified,
                 this report shall be written to standard error. If table
                 sizes are specified in the _l_e_x source code, and if the --nn
                 option is not specified, the --vv option may be enabled.

OOPPEERRAANNDDSS
       The following operand shall be supported:

       _f_i_l_e      A pathname of an input file. If more than one such _f_i_l_e is
                 specified, all files shall be concatenated to produce a
                 single _l_e_x program. If no _f_i_l_e operands are specified, or if
                 a _f_i_l_e operand is ''--'', the standard input shall be used.

SSTTDDIINN
       The standard input shall be used if no _f_i_l_e operands are specified, or
       if a _f_i_l_e operand is ''--''.  See INPUT FILES.

IINNPPUUTT FFIILLEESS
       The input files shall be text files containing _l_e_x source code, as
       described in the EXTENDED DESCRIPTION section.

EENNVVIIRROONNMMEENNTT VVAARRIIAABBLLEESS
       The following environment variables shall affect the execution of _l_e_x:

       _L_A_N_G      Provide a default value for the internationalization
                 variables that are unset or null. (See the Base Definitions
                 volume of POSIX.1-2017, _S_e_c_t_i_o_n _8_._2, _I_n_t_e_r_n_a_t_i_o_n_a_l_i_z_a_t_i_o_n
                 _V_a_r_i_a_b_l_e_s for the precedence of internationalization
                 variables used to determine the values of locale categories.)

       _L_C___A_L_L    If set to a non-empty string value, override the values of
                 all the other internationalization variables.

       _L_C___C_O_L_L_A_T_E
                 Determine the locale for the behavior of ranges, equivalence
                 classes, and multi-character collating elements within
                 regular expressions. If this variable is not set to the POSIX
                 locale, the results are unspecified.

       _L_C___C_T_Y_P_E  Determine the locale for the interpretation of sequences of
                 bytes of text data as characters (for example, single-byte as
                 opposed to multi-byte characters in arguments and input
                 files), and the behavior of character classes within regular
                 expressions. If this variable is not set to the POSIX locale,
                 the results are unspecified.

       _L_C___M_E_S_S_A_G_E_S
                 Determine the locale that should be used to affect the format
                 and contents of diagnostic messages written to standard
                 error.

       _N_L_S_P_A_T_H   Determine the location of message catalogs for the processing
                 of _L_C___M_E_S_S_A_G_E_S.

AASSYYNNCCHHRROONNOOUUSS EEVVEENNTTSS
       Default.

SSTTDDOOUUTT
       If the --tt option is specified, the text file of C source code output of
       _l_e_x shall be written to standard output.

       If the --tt option is not specified:

        *  Implementation-defined informational, error, and warning messages
           concerning the contents of _l_e_x source code input shall be written
           to either the standard output or standard error.

        *  If the --vv option is specified and the --nn option is not specified,
           _l_e_x statistics shall also be written to either the standard output
           or standard error, in an implementation-defined format. These
           statistics may also be generated if table sizes are specified with
           a ''%%'' operator in the _D_e_f_i_n_i_t_i_o_n_s section, as long as the --nn option
           is not specified.

SSTTDDEERRRR
       If the --tt option is specified, implementation-defined informational,
       error, and warning messages concerning the contents of _l_e_x source code
       input shall be written to the standard error.

       If the --tt option is not specified:

        1. Implementation-defined informational, error, and warning messages
           concerning the contents of _l_e_x source code input shall be written
           to either the standard output or standard error.

        2. If the --vv option is specified and the --nn option is not specified,
           _l_e_x statistics shall also be written to either the standard output
           or standard error, in an implementation-defined format. These
           statistics may also be generated if table sizes are specified with
           a ''%%'' operator in the _D_e_f_i_n_i_t_i_o_n_s section, as long as the --nn option
           is not specified.

OOUUTTPPUUTT FFIILLEESS
       A text file containing C source code shall be written to lleexx..yyyy..cc, or
       to the standard output if the --tt option is present.

EEXXTTEENNDDEEDD DDEESSCCRRIIPPTTIIOONN
       Each input file shall contain _l_e_x source code, which is a table of
       regular expressions with corresponding actions in the form of C program
       fragments.

       When lleexx..yyyy..cc is compiled and linked with the _l_e_x library (using the
       --ll ll operand with _c_9_9), the resulting program shall read character
       input from the standard input and shall partition it into strings that
       match the given expressions.

       When an expression is matched, these actions shall occur:

        *  The input string that was matched shall be left in _y_y_t_e_x_t as a
           null-terminated string; _y_y_t_e_x_t shall either be an external
           character array or a pointer to a character string. As explained in
           _D_e_f_i_n_i_t_i_o_n_s _i_n _l_e_x, the type can be explicitly selected using the
           %%aarrrraayy or %%ppooiinntteerr declarations, but the default is implementation-
           defined.

        *  The external iinntt _y_y_l_e_n_g shall be set to the length of the matching
           string.

        *  The expression's corresponding program fragment, or action, shall
           be executed.

       During pattern matching, _l_e_x shall search the set of patterns for the
       single longest possible match. Among rules that match the same number
       of characters, the rule given first shall be chosen.

       The general format of _l_e_x source shall be:

              _D_e_f_i_n_i_t_i_o_n_s %%%% _R_u_l_e_s %%%% _U_s_e_rSubroutines

       The first ""%%%%"" is required to mark the beginning of the rules (regular
       expressions and actions); the second ""%%%%"" is required only if user
       subroutines follow.

       Any line in the _D_e_f_i_n_i_t_i_o_n_s section beginning with a <blank> shall be
       assumed to be a C program fragment and shall be copied to the external
       definition area of the lleexx..yyyy..cc file. Similarly, anything in the
       _D_e_f_i_n_i_t_i_o_n_s section included between delimiter lines containing only
       ""%%{{"" and ""%%}}"" shall also be copied unchanged to the external definition
       area of the lleexx..yyyy..cc file.

       Any such input (beginning with a <blank> or within ""%%{{"" and ""%%}}""
       delimiter lines) appearing at the beginning of the _R_u_l_e_s section before
       any rules are specified shall be written to lleexx..yyyy..cc after the
       declarations of variables for the _y_y_l_e_x() function and before the first
       line of code in _y_y_l_e_x().  Thus, user variables local to _y_y_l_e_x() can be
       declared here, as well as application code to execute upon entry to
       _y_y_l_e_x().

       The action taken by _l_e_x when encountering any input beginning with a
       <blank> or within ""%%{{"" and ""%%}}"" delimiter lines appearing in the _R_u_l_e_s
       section but coming after one or more rules is undefined. The presence
       of such input may result in an erroneous definition of the _y_y_l_e_x()
       function.

       C-language code in the input shall not contain C-language trigraphs.
       The C-language code within ""%%{{"" and ""%%}}"" delimiter lines shall not
       contain any lines consisting only of ""%%}}"", or only of ""%%%%"".

   DDeeffiinniittiioonnss iinn lleexx
       _D_e_f_i_n_i_t_i_o_n_s appear before the first ""%%%%"" delimiter. Any line in this
       section not contained between ""%%{{"" and ""%%}}"" lines and not beginning
       with a <blank> shall be assumed to define a _l_e_x substitution string.
       The format of these lines shall be:


           _n_a_m_e _s_u_b_s_t_i_t_u_t_e

       If a _n_a_m_e does not meet the requirements for identifiers in the ISO C
       standard, the result is undefined. The string _s_u_b_s_t_i_t_u_t_e shall replace
       the string {_n_a_m_e} when it is used in a rule. The _n_a_m_e string shall be
       recognized in this context only when the braces are provided and when
       it does not appear within a bracket expression or within double-quotes.

       In the _D_e_f_i_n_i_t_i_o_n_s section, any line beginning with a <percent-sign>
       (''%%'') character and followed by an alphanumeric word beginning with
       either ''ss'' or ''SS'' shall define a set of start conditions. Any line
       beginning with a ''%%'' followed by a word beginning with either ''xx'' or
       ''XX'' shall define a set of exclusive start conditions. When the
       generated scanner is in a %%ss state, patterns with no state specified
       shall be also active; in a %%xx state, such patterns shall not be active.
       The rest of the line, after the first word, shall be considered to be
       one or more <blank>-separated names of start conditions. Start
       condition names shall be constructed in the same way as definition
       names. Start conditions can be used to restrict the matching of regular
       expressions to one or more states as described in _R_e_g_u_l_a_r _E_x_p_r_e_s_s_i_o_n_s
       _i_n _l_e_x.

       Implementations shall accept either of the following two mutually-
       exclusive declarations in the _D_e_f_i_n_i_t_i_o_n_s section:

       %%aarrrraayy    Declare the type of _y_y_t_e_x_t to be a null-terminated character
                 array.

       %%ppooiinntteerr  Declare the type of _y_y_t_e_x_t to be a pointer to a null-
                 terminated character string.

       The default type of _y_y_t_e_x_t is implementation-defined. If an application
       refers to _y_y_t_e_x_t outside of the scanner source file (that is, via an
       eexxtteerrnn), the application shall include the appropriate %%aarrrraayy or
       %%ppooiinntteerr declaration in the scanner source file.

       Implementations shall accept declarations in the _D_e_f_i_n_i_t_i_o_n_s section
       for setting certain internal table sizes. The declarations are shown in
       the following table.

                        TTaabbllee:: TTaabbllee SSiizzee DDeeccllaarraattiioonnss iinn _l_e_x

         +------------+------------------------------------+---------------+
         |DDeeccllaarraattiioonn |            DDeessccrriippttiioonn             | MMiinniimmuumm VVaalluuee |
         +------------+------------------------------------+---------------+
         |%pp _n        | Number of positions                |     2500      |
         |%nn _n        | Number of states                   |      500      |
         |%aa _n        | Number of transitions              |     2000      |
         |%ee _n        | Number of parse tree nodes         |     1000      |
         |%kk _n        | Number of packed character classes |     1000      |
         |%oo _n        | Size of the output array           |     3000      |
         +------------+------------------------------------+---------------+
       In the table, _n represents a positive decimal integer, preceded by one
       or more <blank> characters. The exact meaning of these table size
       numbers is implementation-defined. The implementation shall document
       how these numbers affect the _l_e_x utility and how they are related to
       any output that may be generated by the implementation should
       limitations be encountered during the execution of _l_e_x.  It shall be
       possible to determine from this output which of the table size values
       needs to be modified to permit _l_e_x to successfully generate tables for
       the input language. The values in the column Minimum Value represent
       the lowest values conforming implementations shall provide.

   RRuulleess iinn lleexx
       The rules in _l_e_x source files are a table in which the left column
       contains regular expressions and the right column contains actions (C
       program fragments) to be executed when the expressions are recognized.


           _E_R_E _a_c_t_i_o_n
           _E_R_E _a_c_t_i_o_n
           ...

       The extended regular expression (ERE) portion of a row shall be
       separated from _a_c_t_i_o_n by one or more <blank> characters. A regular
       expression containing <blank> characters shall be recognized under one
       of the following conditions:

        *  The entire expression appears within double-quotes.

        *  The <blank> characters appear within double-quotes or square
           brackets.

        *  Each <blank> is preceded by a <backslash> character.

   UUsseerr SSuubbrroouuttiinneess iinn lleexx
       Anything in the user subroutines section shall be copied to lleexx..yyyy..cc
       following _y_y_l_e_x().

   RReegguullaarr EExxpprreessssiioonnss iinn lleexx
       The _l_e_x utility shall support the set of extended regular expressions
       (see the Base Definitions volume of POSIX.1-2017, _S_e_c_t_i_o_n _9_._4, _E_x_t_e_n_d_e_d
       _R_e_g_u_l_a_r _E_x_p_r_e_s_s_i_o_n_s), with the following additions and exceptions to
       the syntax:

       "..."     Any string enclosed in double-quotes shall represent the
                 characters within the double-quotes as themselves, except
                 that <backslash>-escapes (which appear in the following
                 table) shall be recognized. Any <backslash>-escape sequence
                 shall be terminated by the closing quote. For example,
                 ""\\0011""""11"" represents a single string: the octal value 1
                 followed by the character ''11''.

       <_s_t_a_t_e>_r, <_s_t_a_t_e_1_,_s_t_a_t_e_2_,...>_r
                 The regular expression _r shall be matched only when the
                 program is in one of the start conditions indicated by _s_t_a_t_e,
                 _s_t_a_t_e_1, and so on; see _A_c_t_i_o_n_s _i_n _l_e_x.  (As an exception to
                 the typographical conventions of the rest of this volume of
                 POSIX.1-2017, in this case <_s_t_a_t_e> does not represent a
                 metavariable, but the literal angle-bracket characters
                 surrounding a symbol.) The start condition shall be
                 recognized as such only at the beginning of a regular
                 expression.

       _r/_x       The regular expression _r shall be matched only if it is
                 followed by an occurrence of regular expression _x (_x is the
                 instance of trailing context, further defined below). The
                 token returned in _y_y_t_e_x_t shall only match _r.  If the trailing
                 portion of _r matches the beginning of _x, the result is
                 unspecified. The _r expression cannot include further trailing
                 context or the ''$$'' (match-end-of-line) operator; _x cannot
                 include the ''^^'' (match-beginning-of-line) operator, nor
                 trailing context, nor the ''$$'' operator. That is, only one
                 occurrence of trailing context is allowed in a _l_e_x regular
                 expression, and the ''^^'' operator only can be used at the
                 beginning of such an expression.

       {_n_a_m_e}    When _n_a_m_e is one of the substitution symbols from the
                 _D_e_f_i_n_i_t_i_o_n_s section, the string, including the enclosing
                 braces, shall be replaced by the _s_u_b_s_t_i_t_u_t_e value. The
                 _s_u_b_s_t_i_t_u_t_e value shall be treated in the extended regular
                 expression as if it were enclosed in parentheses. No
                 substitution shall occur if {_n_a_m_e} occurs within a bracket
                 expression or within double-quotes.

       Within an ERE, a <backslash> character shall be considered to begin an
       escape sequence as specified in the table in the Base Definitions
       volume of POSIX.1-2017, _C_h_a_p_t_e_r _5, _F_i_l_e _F_o_r_m_a_t _N_o_t_a_t_i_o_n (''\\\\'', ''\\aa'',
       ''\\bb'', ''\\ff'', ''\\nn'', ''\\rr'', ''\\tt'', ''\\vv'').  In addition, the escape sequences
       in the following table shall be recognized.

       A literal <newline> cannot occur within an ERE; the escape sequence
       ''\\nn'' can be used to represent a <newline>.  A <newline> shall not be
       matched by a period operator.

                           TTaabbllee:: EEssccaappee SSeeqquueenncceess iinn _l_e_x

          +---------+--------------------------+--------------------------+
          | EEssccaappee  |                          |                          |
          |SSeeqquueennccee |       DDeessccrriippttiioonn        |         MMeeaanniinngg          |
          +---------+--------------------------+--------------------------+
          |\_d_i_g_i_t_s  | A <backslash> character  | The character whose      |
          |         | followed by the longest  | encoding is represented  |
          |         | sequence of one, two, or | by the one, two, or      |
          |         | three octal-digit        | three-digit octal        |
          |         | characters (01234567).   | integer. Multi-byte      |
          |         | If all of the digits are | characters require       |
          |         | 0 (that is,              | multiple, concatenated   |
          |         | representation of the    | escape sequences of this |
          |         | NUL character), the      | type, including the      |
          |         | behavior is undefined.   | leading <backslash> for  |
          |         |                          | each byte.               |
          +---------+--------------------------+--------------------------+
          |\x_d_i_g_i_t_s | A <backslash> character  | The character whose      |
          |         | followed by the longest  | encoding is represented  |
          |         | sequence of hexadecimal- | by the hexadecimal       |
          |         | digit characters         | integer.                 |
          |         | (01234567abcdefABCDEF).  |                          |
          |         | If all of the digits are |                          |
          |         | 0 (that is,              |                          |
          |         | representation of the    |                          |
          |         | NUL character), the      |                          |
          |         | behavior is undefined.   |                          |
          +---------+--------------------------+--------------------------+
          |\c       | A <backslash> character  | The character ''cc'',       |
          |         | followed by any          | unchanged.               |
          |         | character not described  |                          |
          |         | in this table or in the  |                          |
          |         | table in the Base        |                          |
          |         | Definitions volume of    |                          |
          |         | POSIX.1-2017, _C_h_a_p_t_e_r _5, |                          |
          |         | _F_i_l_e _F_o_r_m_a_t _N_o_t_a_t_i_o_n     |                          |
          |         | (''\\\\'', ''\\aa'', ''\\bb'', ''\\ff'', |                          |
          |         | ''\\nn'', ''\\rr'', ''\\tt'', ''\\vv''). |                          |
          +---------+--------------------------+--------------------------+
       NNoottee::     If a ''\\xx'' sequence needs to be immediately followed by a
                 hexadecimal digit character, a sequence such as ""\\xx11""""11"" can
                 be used, which represents a character containing the value 1,
                 followed by the character ''11''.

       The order of precedence given to extended regular expressions for _l_e_x
       differs from that specified in the Base Definitions volume of
       POSIX.1-2017, _S_e_c_t_i_o_n _9_._4, _E_x_t_e_n_d_e_d _R_e_g_u_l_a_r _E_x_p_r_e_s_s_i_o_n_s.  The order of
       precedence for _l_e_x shall be as shown in the following table, from high
       to low.

       NNoottee::     The escaped characters entry is not meant to imply that these
                 are operators, but they are included in the table to show
                 their relationships to the true operators. The start
                 condition, trailing context, and anchoring notations have
                 been omitted from the table because of the placement
                 restrictions described in this section; they can only appear
                 at the beginning or ending of an ERE.

                            TTaabbllee:: EERREE PPrreecceeddeennccee iinn _l_e_x

             +----------------------------------+----------------------+
             |   EExxtteennddeedd RReegguullaarr EExxpprreessssiioonn    |      PPrreecceeddeennccee      |
             +----------------------------------+----------------------+
             |_c_o_l_l_a_t_i_o_n_-_r_e_l_a_t_e_d _b_r_a_c_k_e_t _s_y_m_b_o_l_s | [= =]  [: :]  [. .]  |
             |_e_s_c_a_p_e_d _c_h_a_r_a_c_t_e_r_s                | \<_s_p_e_c_i_a_l _c_h_a_r_a_c_t_e_r> |
             |_b_r_a_c_k_e_t _e_x_p_r_e_s_s_i_o_n                | [ ]                  |
             |_q_u_o_t_i_n_g                           | "..."                |
             |_g_r_o_u_p_i_n_g                          | ( )                  |
             |_d_e_f_i_n_i_t_i_o_n                        | {_n_a_m_e}               |
             |_s_i_n_g_l_e_-_c_h_a_r_a_c_t_e_r _R_E _d_u_p_l_i_c_a_t_i_o_n   | * + ?                |
             |_c_o_n_c_a_t_e_n_a_t_i_o_n                     |                      |
             |_i_n_t_e_r_v_a_l _e_x_p_r_e_s_s_i_o_n               | {m,n}                |
             |_a_l_t_e_r_n_a_t_i_o_n                       | |                    |
             +----------------------------------+----------------------+
       The ERE anchoring operators ''^^'' and ''$$'' do not appear in the table.
       With _l_e_x regular expressions, these operators are restricted in their
       use: the ''^^'' operator can only be used at the beginning of an entire
       regular expression, and the ''$$'' operator only at the end. The operators
       apply to the entire regular expression. Thus, for example, the pattern
       ""((^^aabbcc))||((ddeeff$$))"" is undefined; it can instead be written as two separate
       rules, one with the regular expression ""^^aabbcc"" and one with ""ddeeff$$"",
       which share a common action via the special ''||'' action (see below). If
       the pattern were written ""^^aabbcc||ddeeff$$"", it would match either ""aabbcc"" or
       ""ddeeff"" on a line by itself.

       Unlike the general ERE rules, embedded anchoring is not allowed by most
       historical _l_e_x implementations. An example of embedded anchoring would
       be for patterns such as ""((^^|| ))ffoooo(( ||$$))"" to match ""ffoooo"" when it exists
       as a complete word. This functionality can be obtained using existing
       _l_e_x features:


           ^foo/[ \n]      |
           " foo"/[ \n]    /* Found foo as a separate word. */

       Note also that ''$$'' is a form of trailing context (it is equivalent to
       ""//\\nn"") and as such cannot be used with regular expressions containing
       another instance of the operator (see the preceding discussion of
       trailing context).

       The additional regular expressions trailing-context operator ''//'' can be
       used as an ordinary character if presented within double-quotes, ""//"";
       preceded by a <backslash>, ""\\//""; or within a bracket expression, ""[[//]]"".
       The start-condition ''<<'' and ''>>'' operators shall be special only in a
       start condition at the beginning of a regular expression; elsewhere in
       the regular expression they shall be treated as ordinary characters.

   AAccttiioonnss iinn lleexx
       The action to be taken when an ERE is matched can be a C program
       fragment or the special actions described below; the program fragment
       can contain one or more C statements, and can also include special
       actions. The empty C statement '';;'' shall be a valid action; any string
       in the lleexx..yyyy..cc input that matches the pattern portion of such a rule
       is effectively ignored or skipped. However, the absence of an action
       shall not be valid, and the action _l_e_x takes in such a condition is
       undefined.

       The specification for an action, including C statements and special
       actions, can extend across several lines if enclosed in braces:


           _E_R_E <_o_n_e _o_r _m_o_r_e _b_l_a_n_k_s> { _p_r_o_g_r_a_m _s_t_a_t_e_m_e_n_t
                                      _p_r_o_g_r_a_m _s_t_a_t_e_m_e_n_t }

       The program statements shall not contain unbalanced curly brace
       preprocessing tokens.

       The default action when a string in the input to a lleexx..yyyy..cc program is
       not matched by any expression shall be to copy the string to the
       output. Because the default behavior of a program generated by _l_e_x is
       to read the input and copy it to the output, a minimal _l_e_x source
       program that has just ""%%%%"" shall generate a C program that simply
       copies the input to the output unchanged.

       Four special actions shall be available:


           |   ECHO;   REJECT;   BEGIN

       |         The action ''||'' means that the action for the next rule is the
                 action for this rule.  Unlike the other three actions, ''||''
                 cannot be enclosed in braces or be <semicolon>-terminated;
                 the application shall ensure that it is specified alone, with
                 no other actions.

       EECCHHOO;;     Write the contents of the string _y_y_t_e_x_t on the output.

       RREEJJEECCTT;;   Usually only a single expression is matched by a given string
                 in the input.  RREEJJEECCTT means ``continue to the next expression
                 that matches the current input'', and shall cause whatever
                 rule was the second choice after the current rule to be
                 executed for the same input. Thus, multiple rules can be
                 matched and executed for one input string or overlapping
                 input strings. For example, given the regular expressions
                 ""xxyyzz"" and ""xxyy"" and the input ""xxyyzz"", usually only the regular
                 expression ""xxyyzz"" would match. The next attempted match would
                 start after zz..  If the last action in the ""xxyyzz"" rule is
                 RREEJJEECCTT, both this rule and the ""xxyy"" rule would be executed.
                 The RREEJJEECCTT action may be implemented in such a fashion that
                 flow of control does not continue after it, as if it were
                 equivalent to a ggoottoo to another part of _y_y_l_e_x().  The use of
                 RREEJJEECCTT may result in somewhat larger and slower scanners.

       BBEEGGIINN     The action:


                     BEGIN _n_e_w_s_t_a_t_e;

                 switches the state (start condition) to _n_e_w_s_t_a_t_e.  If the
                 string _n_e_w_s_t_a_t_e has not been declared previously as a start
                 condition in the _D_e_f_i_n_i_t_i_o_n_s section, the results are
                 unspecified. The initial state is indicated by the digit ''00''
                 or the token IINNIITTIIAALL.

       The functions or macros described below are accessible to user code
       included in the _l_e_x input. It is unspecified whether they appear in the
       C code output of _l_e_x, or are accessible only through the --ll ll operand
       to _c_9_9 (the _l_e_x library).

       iinntt _y_y_l_e_x(vvooiidd)
             Performs lexical analysis on the input; this is the primary
             function generated by the _l_e_x utility. The function shall return
             zero when the end of input is reached; otherwise, it shall return
             non-zero values (tokens) determined by the actions that are
             selected.

       iinntt _y_y_m_o_r_e(vvooiidd)
             When called, indicates that when the next input string is
             recognized, it is to be appended to the current value of _y_y_t_e_x_t
             rather than replacing it; the value in _y_y_l_e_n_g shall be adjusted
             accordingly.

       iinntt _y_y_l_e_s_s(iinntt _n)
             Retains _n initial characters in _y_y_t_e_x_t, NUL-terminated, and
             treats the remaining characters as if they had not been read; the
             value in _y_y_l_e_n_g shall be adjusted accordingly.

       iinntt _i_n_p_u_t(vvooiidd)
             Returns the next character from the input, or zero on end-of-
             file. It shall obtain input from the stream pointer _y_y_i_n,
             although possibly via an intermediate buffer. Thus, once scanning
             has begun, the effect of altering the value of _y_y_i_n is undefined.
             The character read shall be removed from the input stream of the
             scanner without any processing by the scanner.

       iinntt _u_n_p_u_t(iinntt _c)
             Returns the character ''cc'' to the input; _y_y_t_e_x_t and _y_y_l_e_n_g are
             undefined until the next expression is matched. The result of
             using _u_n_p_u_t() for more characters than have been input is
             unspecified.

       The following functions shall appear only in the _l_e_x library accessible
       through the --ll ll operand; they can therefore be redefined by a
       conforming application:

       iinntt _y_y_w_r_a_p(vvooiidd)
             Called by _y_y_l_e_x() at end-of-file; the default _y_y_w_r_a_p() shall
             always return 1. If the application requires _y_y_l_e_x() to continue
             processing with another source of input, then the application can
             include a function _y_y_w_r_a_p(), which associates another file with
             the external variable FFIILLEE ** _y_y_i_n and shall return a value of
             zero.

       iinntt _m_a_i_n(iinntt _a_r_g_c, cchhaarr **_a_r_g_v[])
             Calls _y_y_l_e_x() to perform lexical analysis, then exits. The user
             code can contain _m_a_i_n() to perform application-specific
             operations, calling _y_y_l_e_x() as applicable.

       Except for _i_n_p_u_t(), _u_n_p_u_t(), and _m_a_i_n(), all external and static names
       generated by _l_e_x shall begin with the prefix yyyy or YYYY.

EEXXIITT SSTTAATTUUSS
       The following exit values shall be returned:

        0    Successful completion.

       >0    An error occurred.

CCOONNSSEEQQUUEENNCCEESS OOFF EERRRROORRSS
       Default.

       _T_h_e _f_o_l_l_o_w_i_n_g _s_e_c_t_i_o_n_s _a_r_e _i_n_f_o_r_m_a_t_i_v_e_.

AAPPPPLLIICCAATTIIOONN UUSSAAGGEE
       Conforming applications are warned that in the _R_u_l_e_s section, an ERE
       without an action is not acceptable, but need not be detected as
       erroneous by _l_e_x.  This may result in compilation or runtime errors.

       The purpose of _i_n_p_u_t() is to take characters off the input stream and
       discard them as far as the lexical analysis is concerned. A common use
       is to discard the body of a comment once the beginning of a comment is
       recognized.

       The _l_e_x utility is not fully internationalized in its treatment of
       regular expressions in the _l_e_x source code or generated lexical
       analyzer. It would seem desirable to have the lexical analyzer
       interpret the regular expressions given in the _l_e_x source according to
       the environment specified when the lexical analyzer is executed, but
       this is not possible with the current _l_e_x technology. Furthermore, the
       very nature of the lexical analyzers produced by _l_e_x must be closely
       tied to the lexical requirements of the input language being described,
       which is frequently locale-specific anyway. (For example, writing an
       analyzer that is used for French text is not automatically useful for
       processing other languages.)

EEXXAAMMPPLLEESS
       The following is an example of a _l_e_x program that implements a
       rudimentary scanner for a Pascal-like syntax:


           %{
           /* Need this for the call to atof() below. */
           #include <math.h>
           /* Need this for printf(), fopen(), and stdin below. */
           #include <stdio.h>
           %}

           DIGIT    [0-9]
           ID       [a-z][a-z0-9]*

           %%

           {DIGIT}+ {
               printf("An integer: %s (%d)\n", yytext,
                   atoi(yytext));
               }

           {DIGIT}+"."{DIGIT}*        {
               printf("A float: %s (%g)\n", yytext,
                   atof(yytext));
               }

           if|then|begin|end|procedure|function        {
               printf("A keyword: %s\n", yytext);
               }

           {ID}    printf("An identifier: %s\n", yytext);

           "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);

           "{"[^}\n]*"}"    /* Eat up one-line comments. */

           [ \t\n]+        /* Eat up white space. */

           .  printf("Unrecognized character: %s\n", yytext);

           %%

           int main(int argc, char *argv[])
           {
               ++argv, --argc;  /* Skip over program name. */
               if (argc > 0)
                   yyin = fopen(argv[0], "r");
               else
                   yyin = stdin;

               yylex();
           }

RRAATTIIOONNAALLEE
       Even though the --cc option and references to the C language are retained
       in this description, _l_e_x may be generalized to other languages, as was
       done at one time for EFL, the Extended FORTRAN Language. Since the _l_e_x
       input specification is essentially language-independent, versions of
       this utility could be written to produce Ada, Modula-2, or Pascal code,
       and there are known historical implementations that do so.

       The current description of _l_e_x bypasses the issue of dealing with
       internationalized EREs in the _l_e_x source code or generated lexical
       analyzer. If it follows the model used by _a_w_k (the source code is
       assumed to be presented in the POSIX locale, but input and output are
       in the locale specified by the environment variables), then the tables
       in the lexical analyzer produced by _l_e_x would interpret EREs specified
       in the _l_e_x source in terms of the environment variables specified when
       _l_e_x was executed. The desired effect would be to have the lexical
       analyzer interpret the EREs given in the _l_e_x source according to the
       environment specified when the lexical analyzer is executed, but this
       is not possible with the current _l_e_x technology.

       The description of octal and hexadecimal-digit escape sequences agrees
       with the ISO C standard use of escape sequences.

       Earlier versions of this standard allowed for implementations with
       bytes other than eight bits, but this has been modified in this
       version.

       There is no detailed output format specification. The observed behavior
       of _l_e_x under four different historical implementations was that none of
       these implementations consistently reported the line numbers for error
       and warning messages. Furthermore, there was a desire that _l_e_x be
       allowed to output additional diagnostic messages. Leaving message
       formats unspecified avoids these formatting questions and problems with
       internationalization.

       Although the %%xx specifier for _e_x_c_l_u_s_i_v_e start conditions is not
       historical practice, it is believed to be a minor change to historical
       implementations and greatly enhances the usability of _l_e_x programs
       since it permits an application to obtain the expected functionality
       with fewer statements.

       The %%aarrrraayy and %%ppooiinntteerr declarations were added as a compromise between
       historical systems.  The System V-based _l_e_x copies the matched text to
       a _y_y_t_e_x_t array. The _f_l_e_x program, supported in BSD and GNU systems,
       uses a pointer. In the latter case, significant performance
       improvements are available for some scanners. Most historical programs
       should require no change in porting from one system to another because
       the string being referenced is null-terminated in both cases. (The
       method used by _f_l_e_x in its case is to null-terminate the token in place
       by remembering the character that used to come right after the token
       and replacing it before continuing on to the next scan.) Multi-file
       programs with external references to _y_y_t_e_x_t outside the scanner source
       file should continue to operate on their historical systems, but would
       require one of the new declarations to be considered strictly portable.

       The description of EREs avoids unnecessary duplication of ERE details
       because their meanings within a _l_e_x ERE are the same as that for the
       ERE in this volume of POSIX.1-2017.

       The reason for the undefined condition associated with text beginning
       with a <blank> or within ""%%{{"" and ""%%}}"" delimiter lines appearing in the
       _R_u_l_e_s section is historical practice. Both the BSD and System V _l_e_x
       copy the indented (or enclosed) input in the _R_u_l_e_s section (except at
       the beginning) to unreachable areas of the _y_y_l_e_x() function (the code
       is written directly after a _b_r_e_a_k statement). In some cases, the System
       V _l_e_x generates an error message or a syntax error, depending on the
       form of indented input.

       The intention in breaking the list of functions into those that may
       appear in lleexx..yyyy..cc _v_e_r_s_u_s those that only appear in lliibbll..aa is that only
       those functions in lliibbll..aa can be reliably redefined by a conforming
       application.

       The descriptions of standard output and standard error are somewhat
       complicated because historical _l_e_x implementations chose to issue
       diagnostic messages to standard output (unless --tt was given).
       POSIX.1-2008 allows this behavior, but leaves an opening for the more
       expected behavior of using standard error for diagnostics.  Also, the
       System V behavior of writing the statistics when any table sizes are
       given is allowed, while BSD-derived systems can avoid it. The
       programmer can always precisely obtain the desired results by using
       either the --tt or --nn options.

       The OPERANDS section does not mention the use of -- as a synonym for
       standard input; not all historical implementations support such usage
       for any of the _f_i_l_e operands.

       A description of the _t_r_a_n_s_l_a_t_i_o_n _t_a_b_l_e was deleted from early proposals
       because of its relatively low usage in historical applications.

       The change to the definition of the _i_n_p_u_t() function that allows
       buffering of input presents the opportunity for major performance gains
       in some applications.

       The following examples clarify the differences between _l_e_x regular
       expressions and regular expressions appearing elsewhere in this volume
       of POSIX.1-2017. For regular expressions of the form ""rr//xx"", the string
       matching _r is always returned; confusion may arise when the beginning
       of _x matches the trailing portion of _r.  For example, given the regular
       expression ""aa**bb//cccc"" and the input ""aaaaaabbcccc"", _y_y_t_e_x_t would contain the
       string ""aaaaaabb"" on this match. But given the regular expression ""xx**//xxyy""
       and the input ""xxxxxxyy"", the token xxxxxx, not xxxx, is returned by some
       implementations because xxxxxx matches ""xx**"".

       In the rule ""aabb**//bbcc"", the ""bb**"" at the end of _r extends _r's match into
       the beginning of the trailing context, so the result is unspecified. If
       this rule were ""aabb//bbcc"", however, the rule matches the text ""aabb"" when it
       is followed by the text ""bbcc"".  In this latter case, the matching of _r
       cannot extend into the beginning of _x, so the result is specified.

FFUUTTUURREE DDIIRREECCTTIIOONNSS
       None.

SSEEEE AALLSSOO
       _c_9_9, _e_d, _y_a_c_c

       The Base Definitions volume of POSIX.1-2017, _C_h_a_p_t_e_r _5, _F_i_l_e _F_o_r_m_a_t
       _N_o_t_a_t_i_o_n, _C_h_a_p_t_e_r _8, _E_n_v_i_r_o_n_m_e_n_t _V_a_r_i_a_b_l_e_s, _C_h_a_p_t_e_r _9, _R_e_g_u_l_a_r
       _E_x_p_r_e_s_s_i_o_n_s, _S_e_c_t_i_o_n _1_2_._2, _U_t_i_l_i_t_y _S_y_n_t_a_x _G_u_i_d_e_l_i_n_e_s

CCOOPPYYRRIIGGHHTT
       Portions of this text are reprinted and reproduced in electronic form
       from IEEE Std 1003.1-2017, Standard for Information Technology --
       Portable Operating System Interface (POSIX), The Open Group Base
       Specifications Issue 7, 2018 Edition, Copyright (C) 2018 by the
       Institute of Electrical and Electronics Engineers, Inc and The Open
       Group.  In the event of any discrepancy between this version and the
       original IEEE and The Open Group Standard, the original IEEE and The
       Open Group Standard is the referee document. The original Standard can
       be obtained online at http://www.opengroup.org/unix/online.html .

       Any typographical or formatting errors that appear in this page are
       most likely to have been introduced during the conversion of the source
       files to man page format. To report such errors, see
       https://www.kernel.org/doc/man-pages/reporting_bugs.html .



IEEE/The Open Group                  2017                               LEX(1)
