| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
* parser.c (parser_common_init): Use local variable to capture
output of yylex_init, rather than yyscan structure member.
* parser.h (struct parser): Member yyscan removed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This bug causes the parser associated with a stream to suddenly
disappear while reading forms from the stream. Parsing continues,
but with a new parser which does not carry the lookahead token
from the previous parse. So for instance, we miss the opening
parenthesis of the next form.
* parser.c (parse_init): The stream_parser_hash must be
a hash table with weak keys, but not weak values.
We want the association to go away only if the stream becomes
unreachable, not if the parser becomes unreachable
while the stream is still reachable.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
GC now exports a function for marking something that might not
be an object, instead of a function for testing.
The previous way wasn't integrated with Valgrind properly,
and didn't observe the FREE flag.
* gc.c (gc_is_heap_obj): Function removed.
(mark_obj_maybe): New static function, with body consisting
of code moved from mark_mem_region.
(mark_mem_region): Moved code replaced by call to mark_obj_maybe.
(gc_conservative_mark): New function, wraps mark_obj_maybe.
* gc.h (gc_conservative_mark): Declared.
(gc_is_heap_obj): Declaration removed.
* parser.c (yy_tok_mark): Use gc_conservative_mark
instead of gc_is_heap_obj check and gc_mark.
|
| |
|
|
|
| |
* parser.c (read_eval_stream): Simplify hash bang code and avoid
creating a string that might not end up being used.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.h (enum prime_parser): New enum.
(prime_parser, prime_scanner, parse): Declarations updated with new
argument.
* parser.c (prime_parser): New argument of enum prime_parser type
Select appropriate secret token for regex and Lisp case. Pass prime
selector down to prime_scanner.
(regex_parse): Do not prepend secret escape to string. Do not use
parse_once function; instead do the parser init and cleanup here and
use the parse function.
(lisp_parse): Pass new argument to parse, configuring the parser to be
primed for Lisp parsing.
* parser.l (grammar): Rule producing SECRET_ESCAPE_R removed.
(prime_scanner): New argument. Pop the scanner state down to INITIAL.
Then unconditionally switch to appopriate state based on priming
configuration.
* parser.y (parse): New argument for priming selection, passed down to
prime parser.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The method of inserting a character sequence which generates a
SECRET_TOKEN_E token is being replaced with a purely token based
method.
Because we don't manipulate the input stream, the lexer is not
involved. We don't have to flush its state and deal with the carry-over
of the yy_hold_char.
This comes about because recent changes expose a weakness in the old
scheme. Now that a top-level expression can have the form expr.expr, it
means that the Yacc parser reads one token ahead, to see whether there
is a dot or something else. This lookahead token is discarded. We must
re-create it when we call yyparse again. This re-creation is done by
creating a custom yylex function, which can maintain pushback tokens.
We can prime this array of pushback tokens to generate the
SECRET_TOKEN_E, as well as to re-inject the lookahead symbol that was
thrown away by the previous yyparse. To know which lookahead symbol to
re-inject is simple: the scanner just keeps a copy of the most recent
token that it returns to the parser. When the parser returns, that
token must be the lookahead one.
The tokens we keep now in the parser structure are subject to garbage
collection, and so we must mark them. Since the YYSTYPE union has no
type field, a new API is opened up into the garbage collector to help
implement a conservative GC technique.
* gc.c (gc_is_heap_obj): New function.
* gc.h (gc_is_heap_obj): Declared.
* match.c: Include y.tab.h. This is now needed by any module
that needs to instantiate a parser_t structure, because members
of type YYSTYPE occur in the structure. (parser.h can still be included
without y.tab.h, but only an incomplete declaration for the parser
strucure is then given, and a few functions are not declared.)
* parser.c (yy_tok_mark): New static function.
(parser_mark): Mark the recent token and the pushback tokens.
(parser_common_init): Initialize the recent token, the
pushback tokens, and the pushback stack index.
(pushback_token): New static function.
(prime_parser): hold_byte argument removed. Body considerably
simplified. The catenated stream trick is no longer required.
All we do here is set up two pushback tokens and prime the scanner,
if necessary, so it is in the right start state for Lisp.
* parser.l (YY_DECL): Take over definition of scanning function, renaming
to yylex_impl, so we can implement yylex.
(grammar): Rule which produces SECRET_ESCAPE_E token removed.
(reset_scanner): Function removed.
(yylex): New function.
* parser.h (struct parser): Now only forward-declared unless y.tab.h
has been included. New members, recent_tok, tok_pushback and tok_idx.
(yyset_hold_char): Declared.
(reset_scanner): Declaration removed.
(yylex): Declared (if y.tab.h included).
(prime_parser): Declaration updated.
(prime_scanner): Declared.
* Makefile: express new dependency on existence of y.tab.h of txr.o,
match.o and parser.o.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.y (r_exprs): Use unique object in the terminating cons to
indicate the empty spot where the dotted cdr item will go. Check for
misplaced consing dot. (misplaced_consing_dot_check): New static
function. Checks for the terminator atom spot being taken already.
Thus, the spot may be taken only by the very last reduction, such that
the next reduction is r_exprs -> n_exprs where the terminating atom is
processed.
* parser.c (unique_s): New global variable.
(parse_init): Initialize unique_s.
* parser.h (unique_s): Declared.
* share/txr/stdlib/place.tl (sys:placelet-1): We have a misplaced
consing dot here! It was working correctly by "terminating atom
propagation" behavior, which allowed (a . b c d) to produce
(a c d . b). If a single terminating atom occurred in the middle of a list, it
was promoted to the end.
|
| |
|
|
|
| |
stream should be the resolved name, not the abstract original,
so that error messages correlate to the file.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* parser.c (prime_parser): Take name as argument,
and install it into parser.
(lisp_parser): Pass name to parse, instead of setting it
in the parser object.
* parser.y (parse): Take name as argument and pass
down to prime_parser.
* parser.h (prime_parser, parse): Declarations updated.
|
| |
|
|
|
| |
* parser.c (read_eval_stream): Get name of original stream,
and pass that down to lisp_parse as the name argument.
|
| |
|
|
|
| |
* parser.c (read_eval_stream): Add terminating newline to
the first line that was examined for hash bang.
|
| |
|
|
|
|
|
|
|
|
|
| |
* parser.c (parser_mark, parser_common_init): Remove reference to primer.
(parser): Don't take primer argument, remove reference to member.
(ensure_parser): Don't take primer argument, don't pass to parser
function.
(lisp_parse): Don't pass primer string to ensure_parser.
* parser.h (primer_t): Remove primer member.
(parser): Declaration updated.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Test case: file containing 4(prinl 3). Scanner consumes 4 and (.
The ( is lost when the scanner is reset for the next call to yyparse,
resulting in jut prinl being read and interpreted as a variable.
* parser.c (prime_parser): If present, append hold byte to priming
string. Takes parser_t * instead of parser, and returns void now.
* parser.l (reset_scanner): Now returns int value, the value
of the scanner's yy_hold_char variable which is nonzero when
the scanner is hanging on to an unmatched byte of input.
* parser.h (reset_scanner, prime_parser): Declarations updated.
* parser.y (parse): Pass hold byte returned by reset_scanner to
prime_parser.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.c (parser_destroy): New GC finalizer static function.
(parser_ops): Register parser_destroy.
(parser_common_init): New function, shared by parse and parse_once.
Initializes embedded scanner.
(parser_cleanup): New function, shared by parse_once and
parser_destroy.
(parser): Use parser_common_init.
* parser.h (parser_t): New member, yyscan.
(reset_scanner, parser_common_init): Declared.
* parser.l (reset_scanner): New function.
* parser.y (parse_once): Use parser_common_init, and
thus perform only a few initializations. Do not
define scanner as a local variable.
(parse): Call reset_scanner instead of
yylex_init since the scanner is being reused,
and for the same reason do not call yylex_destroy.
GC will do that now.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.c (read_eval_stream): New boolean argument
to request hash bang support.
* parser.h (read_eval_stream): Declaration updated.
* eval.c (sys_load): Pass new thid argument to read_eval_stream,
to decline hash bang support.
* match.c (v_load): Likewise.
* txr.c (txr_main): Request hash bang support from
read_eval_stream. Thus files referenced from the txr
command line can have a #! line, which is ignored.
|
| |
|
|
|
| |
its value is nao. Introduced on 2015-06-10, "Error handling improvement
in read".
|
| |
|
|
|
|
|
|
|
|
|
| |
* match.c (v_load): Check txr_lisp_p flag coming out of
open_txr_file and handle the Lisp case usin read_eval_stream.
* parser.c (read_eval_stream, get_parser, parser_errors):
New functions.
* parser.h (read_eval_stream, get_parser, parser_errors):
Declared.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.c (open_txr_file): Rewritten to take new argument
which indicates whether to treat an unsuffixed file as
TXR or TXR Lisp, and is updated to indicate which is the
case by looking at the suffix.
* parser.h (open_txr_file): Declaration updated.
* match.c (v_load): Follow change in open_txr_file.
* txr.c (txr_main): Likewise.
|
| |
|
|
|
|
|
|
|
|
| |
* parser.y (spec): New grammar production to handle the cases
that SECRET_ESCAPE_E is not followed by anything (the input
ends before any object is scanned, or there is no input
token which starts an object).
* parser.c (lisp_parse): Deal with EOF indication from parser
(the syntax_tree member of parser_t set to nao).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(parser_mark): Mark parser and primer members.
(parser, ensure_parser): new argument: primer.
(get_parser_impl, ensure_parser): New static
functions.
(prime_parser): New function.
(lisp_parse): Multiple calls to this function on the same stream
now logically continue the parse, not resetting the line
number to 1.
(parse_init): Initialize and gc-protect stream_parser_hash.
* parser.h (parser_t): New members, primer and parser.
(prime_parser): Declared.
(parser): Declaration updated.
* parser.y (parse): Now responsible for calling prime_parser.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* parser.c (regex_parse): Likewise.
* txr.c (txr_main): Likewise.
* parser.h (parse): Declaration updated.
(parse_once): Declared.
* parser.y (parse_once): New function, same as old parse implementation.
(parse): Becomes one argument function which works with a previously
initialized parser and continues the parse.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
so line numbers don't change.
* eval.c (eval_init): Fix registrations of lisp-parse and read.
* lisplib.c (place_instantiate): Give name to parsed string stream
using new lisp_parse argument.
* parser.c (lisp_parse): Takes new argument to override name.
* parser.h (lisp_parse): Declaration updated.
* txr.c (txr_main): Call lisp_parse with four args, defaulting
the new one.
* txr.1: Documented new argument.
|
| |
|
|
|
|
|
| |
moved here from parser.l.
* parser.l (open_txr_file, regex_parse, lisp_parse): Functions
moved from here to parser.c.
|
|
|
* parser.h (parser_s): Declared.
(parse_init): Declaration removed.
(parser_l_init): Declared.
* parser.l (parse_init): Function renamed to parser_l_init.
* parser.c: New file.
|
