From 295568eae0dea8dec20fe836a399dcffc5a15abb Mon Sep 17 00:00:00 2001 From: Kaz Kylheku Date: Wed, 12 May 2021 21:42:34 -0700 Subject: parser: bug: handing of lex state in pushback tokens. This is fairly obscure. A repro test case is a file which contains: 3"foo" When the 3 is parsed, the " is also scanned as a lookahead token, and when that happens, the lexer shifts into the STRLIT state. At that point the parse job finishes for that top-level form. The next time the parser is called, it will prime the token stream by pushing the " token into it. But, the lex state is not put into the STRLIT. State. The result is that the parser obtains the " token, and then foo is lexically analyzed in the wrong state as a symbol. A syntax error occurs: symbol token in the middle of a string literal, instead of just a sequence of LITCHAR tokens, as expected. What we can do is associate a lex state with pushback tokens. If a pushback token has a nonzero lex state which is different from the current YYSTATE, then when that pushback token is consumed, we push that state also. * parser.h (struct yy_token): New member, yy_lex_state. * parser.c (parser_common_init): Initialize the new yy_lex_state member of every token member of the parser structure. * parser.l (yylex): When feeding a pushed token to the parser, if that token has a nonzero state, and the state is different from YYSTATE, we push that state. So for instance a pushed back " token will carry the STRLIT state, which is different from the NESTED state that will be in effect at the start of the parse job, and so it will be pushed, as if the " character had been scanned. Also, when we call the real yylex_impl, when we are storing the recenty seen token in recent_tok, we also store the current YYSTATE along with it. That's how tokens get associated with a state. The artificial tokens that are used for priming parsing like SECRET_ESCAPE_E are never associated with a nonzero state. * tests/012/syntax.tl: Some test cases that didn't pass before this. * lex.yy.c.shipped: Regenerated. --- parser.c | 2 ++ 1 file changed, 2 insertions(+) (limited to 'parser.c') diff --git a/parser.c b/parser.c index 48928741..51739f95 100644 --- a/parser.c +++ b/parser.c @@ -135,9 +135,11 @@ void parser_common_init(parser_t *p) p->scanner = convert(scanner_t *, yyscan); yyset_extra(p, p->scanner); p->recent_tok.yy_char = 0; + p->recent_tok.yy_lex_state = 0; p->recent_tok.yy_lval.val = 0; for (i = 0; i < 4; i++) { p->tok_pushback[i].yy_char = 0; + p->tok_pushback[i].yy_lex_state = 0; p->tok_pushback[i].yy_lval.val = 0; } p->tok_idx = 0; -- cgit v1.2.3