11 Multiple Input Buffers

Some scanners (such as those which support “include” files) require reading from several input streams. As flex scanners do a large amount of buffering, one cannot control where the next input will be read from by simply writing a yyread() which is sensitive to the scanning context. yyread() is only called when the scanner reaches the end of its buffer, which may be a long time after scanning a statement such as an include statement which requires switching the input source.

To negotiate these sorts of problems, flex provides a mechanism for creating and switching between multiple input buffers. An input buffer is created by using:

Function: yybuffer yy_create_buffer ( FILE *file, int size ) ¶

which takes a FILE pointer and a size and creates a buffer associated with the given file and large enough to hold size characters (when in doubt, use YY_BUF_SIZE for the size). It returns a yybuffer handle, which may then be passed to other routines (see below).

In target languages other than C/C++, this prototype will look different. The input-stream type won’t be FILE *, but you can expect the same semantics expressed using the target language’s native types.

The yybuffer type is a reference to an opaque buffer state structure, so you may safely initialize yybuffer variables to ((yybuffer) NULL) if you wish, and also refer to the opaque structure in order to correctly declare input buffers in source files other than that of your scanner. Note that the FILE pointer in the call to yy_create_buffer is only used as the value of yyin seen by yyread(). If you redefine yyread() so it no longer uses yyin, then you can safely pass a NULL FILE pointer to yy_create_buffer. You select a particular buffer to scan from using:

Function: void yy_switch_to_buffer ( yybuffer new_buffer ) ¶

The above function switches the scanner’s input buffer so subsequent tokens will come from new_buffer. Note that yy_switch_to_buffer() may be used by yywrap() to set things up for continued scanning, instead of opening a new file and pointing yyin at it. If you are looking for a stack of input buffers, then you want to use yypush_buffer_state() instead of this function. Note also that switching input sources via either yy_switch_to_buffer() or yywrap() does not change the start condition.

Function: void yy_delete_buffer ( yybuffer buffer ) ¶

is used to reclaim the storage associated with a buffer. (buffer can be NULL, in which case the routine does nothing.) You can also clear the current contents of a buffer using:

Function: void yypush_buffer_state ( yybuffer buffer ) ¶

This function pushes the new buffer state onto an internal stack. The pushed state becomes the new current state. The stack is maintained by flex and will grow as required. This function is intended to be used instead of yy_switch_to_buffer, when you want to change states, but preserve the current state for later use.

Function: void yypop_buffer_state ( ) ¶

This function removes the current state from the top of the stack, and deletes it by calling yy_delete_buffer. The next state on the stack, if any, becomes the new current state.

Function: void yy_flush_buffer ( yybuffer buffer ) ¶

This function discards the buffer’s contents, so the next time the scanner attempts to match a token from the buffer, it will first fill the buffer anew using yyread().

Function: yybuffer yy_new_buffer ( FILE *file, int size ) ¶

is an alias for yy_create_buffer(), provided for compatibility with the C++ use of new and delete for creating and destroying dynamic objects.

Finally, yy_current_buffer() returns a yybuffer handle to the current buffer. It should not be used as an lvalue, because it can return NULL to indicate no buffer is current.

Here are two examples of using these features for writing a scanner which expands include files (the <<EOF>> feature is discussed below).

This first example uses yypush_buffer_state and yypop_buffer_state. Flex maintains the stack internally.

/* the "incl" state is used for picking up the name
 * of an include file
 */
%x incl
%%
include             yybegin(incl);

[a-z]+              yyecho();
[^a-z\n]*\n?        yyecho();

<incl>[ \t]*      /* eat the whitespace */
<incl>[^ \t\n]+   { /* got the include file name */
        yyin = fopen( yytext, "r" );

        if ( ! yyin )
            error( ... );

        yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE ));

        yybegin(INITIAL);
}

<<EOF>> {
    yypop_buffer_state();

    if ( !yy_current_buffer() ) {
            yyterminate();
    }
}

The second example, below, does the same thing as the previous example did, but manages its own input buffer stack manually (instead of letting flex do it).

/* the "incl" state is used for picking up the name
 * of an include file
 */
%x incl

%{
#define MAX_INCLUDE_DEPTH 10
yybuffer include_stack[MAX_INCLUDE_DEPTH];
int include_stack_ptr = 0;
%}

%%
include             yybegin(incl);

[a-z]+              yyecho();
[^a-z\n]*\n?        yyecho();

<incl>[ \t]*      /* eat the whitespace */
<incl>[^ \t\n]+   { /* got the include file name */
        if ( include_stack_ptr >= MAX_INCLUDE_DEPTH ) {
            fprintf( stderr, "Includes nested too deeply" );
            exit( 1 );
        }

        include_stack[include_stack_ptr++] =
            yy_current_buffer();

        yyin = fopen( yytext, "r" );

        if ( ! yyin )
            error( ... );

        yy_switch_to_buffer(
            yy_create_buffer( yyin, YY_BUF_SIZE ) );

        yybegin(INITIAL);
}

<<EOF>> {
        if ( --include_stack_ptr == 0 ) {
            yyterminate();
        } else {
            yy_delete_buffer( yy_current_buffer() );
            yy_switch_to_buffer(
                 include_stack[include_stack_ptr] );
        }
}

The following routines are available for setting up input buffers for scanning in-memory strings instead of files. All of them create a new input buffer for scanning the string, and return a corresponding yybuffer handle (which you should delete with yy_delete_buffer() when done with it). They also switch to the new buffer using yy_switch_to_buffer(), so the next call to yylex() will start scanning the string.

Function: yybuffer yy_scan_string ( char *str ) ¶

scans a string. This declaration is correct for C/C++, in which strings are simply character sequences terminated by a NUL. It is expected that each target language will use the most appropriate native string type instead.

Function: yybuffer yy_scan_bytes ( const char *bytes, int len ) ¶

scans len bytes (including possibly NULs) starting at location bytes. It is expected that each target language will use the most appropriate native type instead of char*, such as a reference to a byte array or slice.

Note that both of these functions create and scan a copy of the string or bytes. (This may be desirable, since yylex() modifies the contents of the buffer it is scanning.) You can avoid the copy by using:

Function: yybuffer yy_scan_buffer (char *base, yy_size_t size) ¶

which scans in place the buffer starting at base, consisting of size bytes, the last two bytes of which must be YY_END_OF_BUFFER_CHAR (ASCII NUL). These last two bytes are not scanned; thus, scanning consists of base[0] through base[size-2], inclusive. It is expected that each target language will use the most appropriate native type instead of char*, such as an owned byte array.

If you fail to set up base in this manner (i.e., forget the final two YY_END_OF_BUFFER_CHAR bytes), then yy_scan_buffer() returns a NULL pointer (and/or an error) instead of creating a new input buffer.

Data type: yy_size_t ¶

is an integral type to which you can cast an integer expression reflecting the size of the buffer.