4 Some Simple Examples

First some simple examples to get the flavor of how one uses flex.

The following flex input specifies a scanner which, when it encounters the string ‘username’ will replace it with the user’s login name:

    %%
    username    printf( "%s", getlogin() );

By default, any text not matched by a flex scanner is copied to the output, so the net effect of this scanner is to copy its input file to its output with each occurrence of ‘username’ expanded. In this input, there is just one rule. ‘username’ is the pattern and the ‘printf’ is the action. The ‘%%’ symbol marks the beginning of the rules.

Here’s another simple example:

/* basic example - flawed reentrant version with global */
%{
        int num_lines = 0, num_chars = 0;
%}
%option reentrant noyywrap
%%
\n      ++num_lines; ++num_chars;
.       ++num_chars;

%%

int main() {
        yyscan_t scanner;

        yylex_init ( &scanner );
        yylex ( scanner );
        yylex_destroy ( scanner );

        printf( "# of lines = %d, # of chars = %d\n",
                num_lines, num_chars );
}

If you have looked at older versions of the Flex manual, you might have seen a version of the above example that looked more like this:

/* basic example - non-reentrant version */
%{
	int num_lines = 0, num_chars = 0;
%}
%option noyywrap
%%
\n      ++num_lines; ++num_chars;
.       ++num_chars;

%%

int main() {
        yylex();
        printf( "# of lines = %d, # of chars = %d\n",
                num_lines, num_chars );
}

Both versions count the number of characters and the number of lines in its input. Both produces no output other than the final report on the character and line counts. The first code line declares two globals, num_lines and num_chars, which are accessible both inside yylex() and in the main() routine declared after the second ‘%%’. There are two rules, one which matches a newline (‘\n’) and increments both the line count and the character count, and one which matches any character other than a newline (indicated by the ‘.’ regular expression).

The difference between these two variants is that the first uses Flex’s reentrant interface, which bundles the scanner state into a yyscan_t structure; the second uses the non-reentrant interface, in which the scanner’s state is exposed through global variables.

The non-reentrant interface is a relic from the early 1970s when Lex, the ancestor of Flex, was designed. Modern programming practice frowns on hidden global variables; thus when Flex generates a scanner in any language other than the original C/C++ non-reentrancy is not even an option. Most likely it will make you some kind of scanner class that you instantiate, with methods and fields rather than exposed globals.

Thus it’s a good idea to get used to not relying on the exposed globals of the original interface from the beginning of your Flex programming. This is so even though the reentrant example above is a rather poor one; it avoids exposing the scanner state in globals but creates globals of its own. There is a mechanism for including user-defined fields in the scanner structure which will be explained in detail at See Extra Data. For now, consider this:

/* basic example, fully reentrant thread-safe version */
%{
 struct stats {
     int num_lines;
     int num_chars;
 };
%}
%option reentrant noyywrap
%option extra-type="struct stats"
%%
\n	{
     		struct stats ns = yyget_extra(yyscanner);
		++ns.num_lines; ++ns.num_chars;
		yyset_extra(ns, yyscanner);
	}
.       {
     		struct stats ns = yyget_extra(yyscanner);
		++ns.num_chars;
		yyset_extra(ns, yyscanner);
	}

%%

int main() {
        yyscan_t scanner;
	struct stats ns;

        yylex_init ( &scanner );
        yylex ( scanner );

	ns = yyget_extra(scanner);
        printf( "# of lines = %d, # of chars = %d\n",
                ns.num_lines, ns.num_chars);
        yylex_destroy ( scanner );
}

While it requires a bit more ceremony, several instances of this scanner can be run concurrently without stepping on each others’ storage.

(The %option noyywrap in these examples is helpful in making them run standalone, but does not change the behavior of the scsnner.)

A somewhat more complicated example:

/* scanner for a toy Pascal-like language */

%{
/* need this for the call to atof() below */
#include <math.h>
%}

DIGIT    [0-9]
ID       [a-z][a-z0-9]*

%%

{DIGIT}+    {
            printf( "An integer: %s (%d)\n", yytext,
                    atoi( yytext ) );
            }

{DIGIT}+"."{DIGIT}*        {
            printf( "A float: %s (%g)\n", yytext,
                    atof( yytext ) );
            }

if|then|begin|end|procedure|function        {
            printf( "A keyword: %s\n", yytext );
            }

{ID}        printf( "An identifier: %s\n", yytext );

"+"|"-"|"*"|"/"   printf( "An operator: %s\n", yytext );

"{"[^{}\n]*"}"     /* eat up one-line comments */

[ \t\n]+          /* eat up whitespace */

.           printf( "Unrecognized character: %s\n", yytext );

%%

int main( int argc, char **argv ) {
    ++argv, --argc;  /* skip over program name */
    if ( argc > 0 ) {
            yyin = fopen( argv[0], "r" );
    } else {
            yyin = stdin;
    }
    yylex();
}

This is the beginnings of a simple scanner for a language like Pascal. It identifies different types of tokens and reports on what it has seen.

The details of this example will be explained in the following sections.