9 The Generated Scanner

The output of flex is a file with the name lex.yy, which contains the scanning routine yylex(), a number of tables used by it for matching tokens, and a number of auxiliary routines. By default in C, yylex() is declared as follows:

    int yylex(void) {
        ... various definitions and the actions in here ...
    }

This definition may be changed with the the yydecl option. For example, you could put this in among your directives:

%option yydecl="float lexscan(float a, float b)"

to give the scanning routine the name lexscan, returning a float, and taking two floats as arguments.

flex generates ‘C99’ function definitions by default. Flex used to have the ability to generate obsolete, er, ‘traditional’, function definitions. This was to support bootstrapping gcc on old systems. Unfortunately, traditional definitions prevent us from using any standard data types smaller than int (such as short, char, or bool) as function arguments. Furthermore, traditional definitions support added extra complexity in the skeleton file. For this reason, current versions of flex generate standard C99 code only, leaving K&R-style functions to the historians.

In other languages, yylex() will be generated as a reentrant function with a scanner context argument added. This can be enabled in C as well, and specifying your C scanners to be reentrant is recommended for portability.

Whenever yylex() is called, it scans tokens from the global input file yyin (which defaults to stdin). It continues until it either reaches an end-of-file (at which point it returns the value 0) or one of its actions executes a return statement.

If the scanner reaches an end-of-file, subsequent calls are undefined unless either yyin is pointed at a new input file (in which case scanning continues from that file), or yyrestart() is called. yyrestart() takes one argument, an input stream, and initializes yyin for scanning from that stream. Essentially there is no difference between just assigning yyin to a new input stream or using yyrestart() to do so; the latter is available for compatibility with previous versions of flex, and because it can be used to switch input files in the middle of scanning. It can also be used to throw away the current input buffer, by calling it with an argument of yyin; but it would be better to use yy_flush_current_buffer() (see Actions). Note that yyrestart() does not reset the start condition to INITIAL (see Start Conditions).

In C, an input stream is a a FILE * pointer. This pointer can be NULL, if you’ve set up a %yyread() hook to scan from a source other than yyin.

If yylex() stops scanning due to executing a return statement in one of the actions, the scanner may then be called again and it will resume scanning where it left off.

By default (and for purposes of efficiency), scanners use block-reads rather than simple getc() calls to read characters from yyin. The nature of how it gets its input can be controlled by redefining the yyread function used to fill the scanner buffer. The calling sequence for yyread() is yyread(buf,max_size). Its action is to place up to max_size characters in the character array buf and return either the number of characters read or the constant YY_NULL (0 on Unix systems) to indicate ‘EOF’. The default yyread() reads from the global file-pointer yyin.

#cindex %noyyread Here is a sample redefinition of yyread() (in the definitions section of the input file):

int yyread(char *buf, size_t max_size) {
    int c = getchar();
    return (c == EOF) ? YY_NULL : (buf[0] = c, 1);
}

This definition will change the input processing to occur one character at a time.

When Flex sees the %noyyread option, it omits the default definition from the boilerplate in the rest of the parser. Your yyread() function then replaces it.

When the scanner receives an end-of-file indication from yyread(), it then checks the yywrap() function. If yywrap() returns false (zero), then it is assumed that the function has gone ahead and set up yyin to point to another input file, and scanning continues. If it returns true (non-zero), then the scanner terminates, returning 0 to its caller. Note that in either case, the start condition remains unchanged; it does not revert to INITIAL.

If you do not supply your own version of yywrap(), then you must either use %option noyywrap (in which case the scanner behaves as though yywrap() returned 1), or you must link with ‘-lfl’ to obtain the default version of the routine, which always returns 1.

For scanning from in-memory buffers (e.g., scanning strings), see Scanning Strings. See Multiple Input Buffers.

The scanner writes its yyecho() output to the yyout global (default, stdout), which may be redefined by the user simply by assigning it to some other sream - in C/C++, a FILE pointer.