parser: bug: handing of lex state in pushback tokens.

This is fairly obscure. A repro test case is a file which
contains:

 3"foo"

When the 3 is parsed, the " is also scanned as a lookahead
token, and when that happens, the lexer shifts into the
STRLIT state. At that point the parse job finishes for
that top-level form.

The next time the parser is called, it will prime the token
stream by pushing the " token into it. But, the lex state is
not put into the STRLIT. State.  The result is that the parser
obtains the " token, and then foo is lexically analyzed in the
wrong state as a symbol. A syntax error occurs: symbol token
in the middle of a string literal, instead of just a sequence
of LITCHAR tokens, as expected.

What we can do is associate a lex state with pushback tokens.
If a pushback token has a nonzero lex state which is different
from the current YYSTATE, then when that pushback token is
consumed, we push that state also.

* parser.h (struct yy_token): New member, yy_lex_state.

* parser.c (parser_common_init): Initialize the new
yy_lex_state member of every token member of the parser
structure.

* parser.l (yylex): When feeding a pushed token to the parser,
if that token has a nonzero state, and the state is different
from YYSTATE, we push that state. So for instance a pushed
back " token will carry the STRLIT state, which is different
from the NESTED state that will be in effect at the start of
the parse job, and so it will be pushed, as if the " character
had been scanned.  Also, when we call the real yylex_impl,
when we are storing the recenty seen token in recent_tok, we
also store the current YYSTATE along with it. That's how
tokens get associated with a state. The artificial tokens that
are used for priming parsing like SECRET_ESCAPE_E are never
associated with a nonzero state.

* tests/012/syntax.tl: Some test cases that didn't pass
before this.

* lex.yy.c.shipped: Regenerated.

0 files changed, 0 insertions, 0 deletions