summaryrefslogtreecommitdiffstats
path: root/lex.yy.c.shipped
Commit message (Collapse)AuthorAgeFilesLines
* parser: bug: handing of lex state in pushback tokens.Kaz Kylheku2021-05-121-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is fairly obscure. A repro test case is a file which contains: 3"foo" When the 3 is parsed, the " is also scanned as a lookahead token, and when that happens, the lexer shifts into the STRLIT state. At that point the parse job finishes for that top-level form. The next time the parser is called, it will prime the token stream by pushing the " token into it. But, the lex state is not put into the STRLIT. State. The result is that the parser obtains the " token, and then foo is lexically analyzed in the wrong state as a symbol. A syntax error occurs: symbol token in the middle of a string literal, instead of just a sequence of LITCHAR tokens, as expected. What we can do is associate a lex state with pushback tokens. If a pushback token has a nonzero lex state which is different from the current YYSTATE, then when that pushback token is consumed, we push that state also. * parser.h (struct yy_token): New member, yy_lex_state. * parser.c (parser_common_init): Initialize the new yy_lex_state member of every token member of the parser structure. * parser.l (yylex): When feeding a pushed token to the parser, if that token has a nonzero state, and the state is different from YYSTATE, we push that state. So for instance a pushed back " token will carry the STRLIT state, which is different from the NESTED state that will be in effect at the start of the parse job, and so it will be pushed, as if the " character had been scanned. Also, when we call the real yylex_impl, when we are storing the recenty seen token in recent_tok, we also store the current YYSTATE along with it. That's how tokens get associated with a state. The artificial tokens that are used for priming parsing like SECRET_ESCAPE_E are never associated with a nonzero state. * tests/012/syntax.tl: Some test cases that didn't pass before this. * lex.yy.c.shipped: Regenerated.
* parser: allow non-UTF-8 bytes in literals and regexes.Kaz Kylheku2021-04-081-2981/+2936
| | | | | | | | | | * parser.l (grammar): Just like we do in SREGEX, allow an arbitrary byte in REGEX, mapping it to the DCxx range. Do the same inside string literals of all types. * lex.yy.c.shipped: Updated. * tests/012/parse.tl: New tests.
* parser: check in .shipped materials.Kaz Kylheku2021-04-081-425/+440
| | | | | | | | | | | This picks up the changes introduced by the previous three commits. * lex.yy.c.shipped: Updated. * y.tab.c.shipped: Likewise. * y.tab.h.shipped: Likewise.
* parser: fix bad precedence of @ token.Kaz Kylheku2021-01-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Whereas @a..@b parses and transforms to (rcons @a @a), @(a)..@(a) goes to @(rcons a @(a)). * parser.l (grammar): Under 248 compatibility or lower, the @ character now produces the OLD_AT token. Otherwise it produces the '@' character, as before. * parser.y (OLD_AT): New token replaces the '@' at the old low precedence position. '@' is now at the highest precedence, together with OLD_DOTDOT. (We don't care about interactions between '@' and OLD_DOTDOT, because OLD_DOTDOT only exists in 185 compatibility, in which '@' is OLD_AT). (meta): The two rules have to be unfortunately duplicated for OLD_AT, since there is no BNF OR operator in Yacc. * txr.1: Compat note added. * lex.yy.c.shipped: Updated. * y.tab.c.shipped, y.tab.h.shipped: Likewise.
* Copyright year bump 2021.Kaz Kylheku2021-01-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * METALICENSE: 2020 copyrights bumped to 2021. Added note about SHA-256 routines from Colin Percival. * LICENSE, LICENSE-CYG, Makefile, alloca.h, args.c, args.h, arith.c, arith.h, buf.c, buf.h, cadr.c, cadr.h, chksum.c, chksum.h, chksums/crc32.c, chksums/crc32.h, combi.c, combi.h, configure, debug.c, debug.h, eval.c, eval.h, ffi.c, ffi.h, filter.c, filter.h, ftw.c, ftw.h, gc.c, gc.h, glob.c, glob.h, hash.c, hash.h, itypes.c, itypes.h, jmp.S, lex.yy.c.shipped, lib.c, lib.h, linenoise/linenoise.c, linenoise/linenoise.h, lisplib.c, lisplib.h, match.c, match.h, parser.c, parser.h, parser.l, parser.y, protsym.c, rand.c, rand.h, regex.c, regex.h, share/txr/stdlib/asm.tl, share/txr/stdlib/awk.tl, share/txr/stdlib/build.tl, share/txr/stdlib/cadr.tl, share/txr/stdlib/compiler.tl, share/txr/stdlib/conv.tl, share/txr/stdlib/copy-file.tl, share/txr/stdlib/debugger.tl, share/txr/stdlib/defset.tl, share/txr/stdlib/doloop.tl, share/txr/stdlib/each-prod.tl, share/txr/stdlib/error.tl, share/txr/stdlib/except.tl, share/txr/stdlib/ffi.tl, share/txr/stdlib/getopts.tl, share/txr/stdlib/getput.tl, share/txr/stdlib/hash.tl, share/txr/stdlib/ifa.tl, share/txr/stdlib/keyparams.tl, share/txr/stdlib/op.tl, share/txr/stdlib/package.tl, share/txr/stdlib/param.tl, share/txr/stdlib/path-test.tl, share/txr/stdlib/place.tl, share/txr/stdlib/pmac.tl, share/txr/stdlib/quips.tl, share/txr/stdlib/save-exe.tl, share/txr/stdlib/socket.tl, share/txr/stdlib/stream-wrap.tl, share/txr/stdlib/struct.tl, share/txr/stdlib/tagbody.tl, share/txr/stdlib/termios.tl, share/txr/stdlib/trace.tl, share/txr/stdlib/txr-case.tl, share/txr/stdlib/type.tl, share/txr/stdlib/vm-param.tl, share/txr/stdlib/with-resources.tl, share/txr/stdlib/with-stream.tl, share/txr/stdlib/yield.tl, signal.c, signal.h, socket.c, socket.h, stream.c, stream.h, struct.c, struct.h, strudel.c, strudel.h, sysif.c, sysif.h, syslog.c, syslog.h, termios.c, termios.h, time.c, time.h, tree.c, tree.h, txr.1, txr.c, txr.h, unwind.c, unwind.h, utf8.c, utf8.h, vm.c, vm.h, vmop.h, win/cleansvg.txr, y.tab.c.shipped: Copyright year bumped to 2021.
* shipped lexer: fix misleading indentation.Kaz Kylheku2020-12-221-5/+5
| | | | | * lex.yy.c.shippped (YY_DECL): Fix some bad indentation, most likely caused by using a mixture of tabs and spaces.
* parser: commit Bison-2.5-generated code.Kaz Kylheku2020-09-241-0/+6805
* Makefile (BS_LIC_FROM, BS_LIC_TO): Variables removed. (y.tab.c): Remove all filtering hacks. Don't remove the license from y.tab.c. Don't remove yyparse declaration from y.tab.h. Provide a pattern rule for producing any missing file X from X.shipped. That's how y.tab.c and y.tab.h get produced from y.tab.c.shipped and y.tab.h.shipped, respectively, in user mode. * y.tab.c.shipped, y.tab.h.shipped: New files, generated using Bison 2.5.