24 Limitations

Some trailing context patterns cannot be properly matched and generate warning messages (‘dangerous trailing context’). These are patterns where the ending of the first part of the rule matches the beginning of the second part, such as ‘zx*/xy*’, where the ’x*’ matches the ’x’ at the beginning of the trailing context. (Note that the POSIX draft states that the text matched by such patterns is undefined.) For some trailing context rules, parts which are actually fixed-length are not recognized as such, leading to the abovementioned performance loss. In particular, parts using ‘|’ or ‘{n}’ (such as ‘foo{3}’) are always considered variable-length. Combining trailing context with the special ‘|’ action can result in fixed trailing context being turned into the more expensive variable trailing context. For example, in the following:

    %%
    abc      |
    xyz/def

Some caveats are specific to the C/C++ back end: Use of yyunput() invalidates yytext and yyleng, unless the %array directive or the ‘-l’ option has been used. Pattern-matching of NULs is substantially slower than matching other characters. Dynamic resizing of the input buffer is slow, as it entails rescanning all the text matched so far by the current (generally huge) token. Due to both buffering of input and read-ahead, you cannot intermix calls to <stdio.h> routines (such as getchar()) with flex rules and expect it to work. Call yyinput() instead.

The total table entries listed by the ‘-v’ flag excludes the number of table entries needed to determine what rule has been matched. The number of entries is equal to the number of DFA states if the scanner does not use yyreject(), and somewhat greater than the number of states if it does. yyreject() cannot be used with the ‘-f’ or ‘-F’ options.

The flex internal algorithms need documentation.