| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
| |
* lib.c (obj_print_impl): Do not print (rcons X Y)
as X..Y if X looks like (rcons ...). This
causes the problem that (rcons (rcons 1 2) 3)
prints as 1..2..3, a notation which unambiguously
means (rcons 1 (rcons 2 3)).
* tests/012/syntax.tl: New test cases.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.l (remove_char): New static function.
(DIGSEP, XDIGSEP, NUMSEP, FLOSEP, XNUMSEP, ONUMSEP,
BNUMSEP, ONUM, BNUM): New named lex patterns.
(FLODOT): Use DIGSEP instead of DIG.
(ONUM): Use ODIG instead of [0-7].
(BNUM): Use BDIG instead of [0-1].
(grammar): New rule for producing NUMBER from decimal
token with commas based on BNUMSEP instead of BNUM.
This is a copy and paste so that the BNUM rule doesn't
deal with the comma removal, not to slow it down.
For the octal, binary and hex, we just switch to
BNUMSEP, ONUMSEP and XNUMSEP, so they all go through
one case.
Floating point numbers are also handled with a copy
pasted case using FLOSEP.
* tests/012/syntax.tl: New test cases.
* txr.1: Documented.
* genvim.txr (alpha-noe, digsep, hexsep, octsep, binsep): New
variables.
(txr_pnum, txr_xnum, txr_onum, txr_bnum, txr_num): Integrate
separating commas. Some bugs fixed in txr_num, some simplifications,
better txr_badnum pattern.
* lex.yy.c.shipped: Updated.
|
| |
|
|
| |
* tests/012/syntax.tl: New tests, some of which fail.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is fairly obscure. A repro test case is a file which
contains:
3"foo"
When the 3 is parsed, the " is also scanned as a lookahead
token, and when that happens, the lexer shifts into the
STRLIT state. At that point the parse job finishes for
that top-level form.
The next time the parser is called, it will prime the token
stream by pushing the " token into it. But, the lex state is
not put into the STRLIT. State. The result is that the parser
obtains the " token, and then foo is lexically analyzed in the
wrong state as a symbol. A syntax error occurs: symbol token
in the middle of a string literal, instead of just a sequence
of LITCHAR tokens, as expected.
What we can do is associate a lex state with pushback tokens.
If a pushback token has a nonzero lex state which is different
from the current YYSTATE, then when that pushback token is
consumed, we push that state also.
* parser.h (struct yy_token): New member, yy_lex_state.
* parser.c (parser_common_init): Initialize the new
yy_lex_state member of every token member of the parser
structure.
* parser.l (yylex): When feeding a pushed token to the parser,
if that token has a nonzero state, and the state is different
from YYSTATE, we push that state. So for instance a pushed
back " token will carry the STRLIT state, which is different
from the NESTED state that will be in effect at the start of
the parse job, and so it will be pushed, as if the " character
had been scanned. Also, when we call the real yylex_impl,
when we are storing the recenty seen token in recent_tok, we
also store the current YYSTATE along with it. That's how
tokens get associated with a state. The artificial tokens that
are used for priming parsing like SECRET_ESCAPE_E are never
associated with a nonzero state.
* tests/012/syntax.tl: Some test cases that didn't pass
before this.
* lex.yy.c.shipped: Regenerated.
|
|
|
This is motivated by the recent crash regression in the #;
comment out mechanism. The parser doesn't have adequate
coverage in the test suite.
* tests/012/syntax.tl: New file, for testing syntax.
A problem was found #;.expr did not work inside a list,
only at top level. It required a space before the dot.
* parser.y (listacc): A couple of productions to handle
hash-semicolon immediately followed by a dot without
any whitespace, and then by an expression.
* y.tab.c.shipped: Regenerated.
|
