Infrastructure for storing line number information

outside of the code, in hash tables.

* filter.c (make_trie, trie_add): Update to three-argument
make_hash.

* hash.c (struct hash): New members, hash_fun, assoc_fun
acons_new_l_fun.
(ll_hash): Renamed to equal_hash.
(eql_hash): New static function.
(cobj_hash_op): Follows ll_hash rename.
(hash_grow): Use new function indirection to call hashing function.
(make_hash): New argument to specify type of hashing. Initialize new
members of struct hash.
(gethash_l, gethash, remhash): Use function indirection for hashing and
chain search and update.
(pushhash): New function.

* hash.h (make_hash): Declaration updated with new parameter.
(pushhash): Declared.

* lib.c (eql_f): New global variable.
(eql, assq, aconsq_new, aconsq_new_l): New functions.
(make_package): Updated to new three-argument make_hash.
(obj_init): gc-protect and initialize new variable eql_f.

* lib.h (eql, assq, aconsq_new, aconsq_new_l): Declared.

* match.c (dir_tables_init): Updated to there-argument make_hash.

* parser.h (form_to_ln_hash, ln_to_forms_hash): Global variables
declared.

* parser.l (form_to_ln_hash, ln_to_forms_hash): New global variables.
(grammar): Set yylval.lineno for tokens that are classified to
that type in parser.y.
(parse_init): Initialize and gc-protect new global variables.

* parser.y (rl): New static helper function.
(%union): New member, lineno.
(ALL, SOME, NONE, MAYBE, CASES, CHOOSE, GATHER,
AND, OR, END, COLLECT, UNTIL, COLL, OUTPUT, REPEAT,
REP, SINGLE, FIRST, LAST, EMPTY, DEFINE,
TRY, CATCH, FINALLY, ERRTOK, '('): Reclassified as lineno type.
In the grammar, these keywords can thus provide a stable line number
from the lexer.
(grammar): Numerous rules updated to add constructs to the
line number hash tables via the rl helper.

* dep.mk: Updated.

* Makefile (depend): Use the installed, stable txr in the
system path to update dependencies rather than locally built ./txr, to
prevent the problem that txr is broken because out out-of-date
dependencies, and thus cannot regenerate dependencies.

12 files changed, 299 insertions, 82 deletions

diff --git a/ChangeLog b/ChangeLog
index 6082d9bb..f3630225 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,61 @@
+2011-11-12  Kaz Kylheku  <kaz@kylheku.com>
+
+	Infrastructure for storing line number information
+	outside of the code, in hash tables.
+
+	* filter.c (make_trie, trie_add): Update to three-argument
+	make_hash.
+
+	* hash.c (struct hash): New members, hash_fun, assoc_fun
+	acons_new_l_fun.
+	(ll_hash): Renamed to equal_hash.
+	(eql_hash): New static function.
+	(cobj_hash_op): Follows ll_hash rename.
+	(hash_grow): Use new function indirection to call hashing function.
+	(make_hash): New argument to specify type of hashing. Initialize new
+	members of struct hash.
+	(gethash_l, gethash, remhash): Use function indirection for hashing and
+	chain search and update.
+	(pushhash): New function.
+
+	* hash.h (make_hash): Declaration updated with new parameter.
+	(pushhash): Declared.
+
+	* lib.c (eql_f): New global variable.
+	(eql, assq, aconsq_new, aconsq_new_l): New functions.
+	(make_package): Updated to new three-argument make_hash.
+	(obj_init): gc-protect and initialize new variable eql_f.
+
+	* lib.h (eql, assq, aconsq_new, aconsq_new_l): Declared.
+
+	* match.c (dir_tables_init): Updated to there-argument make_hash.
+
+	* parser.h (form_to_ln_hash, ln_to_forms_hash): Global variables
+	declared.
+
+	* parser.l (form_to_ln_hash, ln_to_forms_hash): New global variables.
+	(grammar): Set yylval.lineno for tokens that are classified to
+	that type in parser.y.
+	(parse_init): Initialize and gc-protect new global variables.
+
+	* parser.y (rl): New static helper function.
+	(%union): New member, lineno.
+	(ALL, SOME, NONE, MAYBE, CASES, CHOOSE, GATHER,
+	AND, OR, END, COLLECT, UNTIL, COLL, OUTPUT, REPEAT,
+	REP, SINGLE, FIRST, LAST, EMPTY, DEFINE,
+	TRY, CATCH, FINALLY, ERRTOK, '('): Reclassified as lineno type.
+	In the grammar, these keywords can thus provide a stable line number
+	from the lexer.
+	(grammar): Numerous rules updated to add constructs to the
+	line number hash tables via the rl helper.
+
+	* dep.mk: Updated.
+
+	* Makefile (depend): Use the installed, stable txr in the
+	system path to update dependencies rather than locally built ./txr, to
+	prevent the problem that txr is broken because out out-of-date
+	dependencies, and thus cannot regenerate dependencies.
+
 2011-11-10  Kaz Kylheku  <kaz@kylheku.com>
 
 	Bug #34799: errors in horizontal functions
diff --git a/Makefile b/Makefile
index fa62e9fb..feb25d2e 100644
--- a/Makefile
+++ b/Makefile
@@ -72,7 +72,7 @@ distclean: clean
 
 .PHONY: depend
 depend:
-	$(PROG) $(top_srcdir)/depend.txr $(OBJS) > $(top_srcdir)/dep.mk
+	txr $(top_srcdir)/depend.txr $(OBJS) > $(top_srcdir)/dep.mk
 
 TESTS := $(patsubst $(top_srcdir)/%.txr,./%.ok,\
                     $(shell find $(top_srcdir)/tests -name '*.txr' | sort))
diff --git a/dep.mk b/dep.mk
index 5a8d334f..703a8d3b 100644
--- a/dep.mk
+++ b/dep.mk
@@ -1,6 +1,6 @@
 txr.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/stream.h $(top_srcdir)/gc.h $(top_srcdir)/unwind.h $(top_srcdir)/parser.h $(top_srcdir)/match.h $(top_srcdir)/utf8.h $(top_srcdir)/txr.h
-lex.yy.o: config.h $(top_srcdir)/lib.h y.tab.h $(top_srcdir)/gc.h $(top_srcdir)/stream.h $(top_srcdir)/utf8.h $(top_srcdir)/unwind.h $(top_srcdir)/parser.h
-y.tab.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/regex.h $(top_srcdir)/utf8.h $(top_srcdir)/match.h $(top_srcdir)/parser.h
+lex.yy.o: config.h $(top_srcdir)/lib.h y.tab.h $(top_srcdir)/gc.h $(top_srcdir)/stream.h $(top_srcdir)/utf8.h $(top_srcdir)/unwind.h $(top_srcdir)/hash.h $(top_srcdir)/parser.h
+y.tab.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/regex.h $(top_srcdir)/utf8.h $(top_srcdir)/match.h $(top_srcdir)/hash.h $(top_srcdir)/parser.h
 match.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/gc.h $(top_srcdir)/unwind.h $(top_srcdir)/regex.h $(top_srcdir)/stream.h $(top_srcdir)/parser.h $(top_srcdir)/txr.h $(top_srcdir)/utf8.h $(top_srcdir)/filter.h $(top_srcdir)/hash.h $(top_srcdir)/match.h
 lib.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/gc.h $(top_srcdir)/hash.h $(top_srcdir)/unwind.h $(top_srcdir)/stream.h $(top_srcdir)/utf8.h $(top_srcdir)/filter.h
 regex.o: config.h $(top_srcdir)/lib.h $(top_srcdir)/unwind.h $(top_srcdir)/regex.h $(top_srcdir)/txr.h
diff --git a/filter.c b/filter.c
index 770a4b4e..29378f48 100644
--- a/filter.c
+++ b/filter.c
@@ -42,7 +42,7 @@ val upcase_k, downcase_k, fun_k;
 
 static val make_trie(void)
 {
-  return make_hash(nil, nil);
+  return make_hash(nil, nil, nil);
 }
 
 static val trie_add(val trie, val key, val value)
@@ -54,7 +54,7 @@ static val trie_add(val trie, val key, val value)
     val newnode_p;
     val *loc = gethash_l(node, ch, &newnode_p);
     if (newnode_p)
-      *loc = make_hash(nil, nil);
+      *loc = make_hash(nil, nil, nil);
     node = *loc;
   }
 
@@ -577,7 +577,7 @@ void filter_init(void)
 {
   protect(&filters, (val *) 0);
 
-  filters = make_hash(nil, nil);
+  filters = make_hash(nil, nil, nil);
   filter_k = intern(lit("filter"), keyword_package);
   lfilt_k = intern(lit("lfilt"), keyword_package);
   rfilt_k = intern(lit("rfilt"), keyword_package);
diff --git a/hash.c b/hash.c
index 8ee7a072..7ee39d43 100644
--- a/hash.c
+++ b/hash.c
@@ -52,6 +52,9 @@ struct hash {
   cnum modulus;
   cnum count;
   val userdata;
+  cnum (*hash_fun)(val);
+  val (*assoc_fun)(val list, val key);
+  val *(*acons_new_l_fun)(val *list, val key, val *new_p);
 };
 
 struct hash_iter {
@@ -84,7 +87,7 @@ static long hash_c_str(const wchar_t *str)
   return h;
 }
 
-static cnum ll_hash(val obj)
+static cnum equal_hash(val obj)
 {
   if (obj == nil)
     return NUM_MAX;
@@ -93,7 +96,7 @@ static cnum ll_hash(val obj)
   case LIT:
     return hash_c_str(litptr(obj));
   case CONS:
-    return (ll_hash(obj->c.car) + ll_hash(obj->c.cdr)) & NUM_MAX;
+    return (equal_hash(obj->c.car) + equal_hash(obj->c.cdr)) & NUM_MAX;
   case STR:
     return hash_c_str(obj->st.str);
   case CHR:
@@ -110,23 +113,23 @@ static cnum ll_hash(val obj)
     }
     break;
   case FUN:
-    return ((cnum) obj->f.f.interp_fun + ll_hash(obj->f.env)) & NUM_MAX;
+    return ((cnum) obj->f.f.interp_fun + equal_hash(obj->f.env)) & NUM_MAX;
   case VEC:
     {
       val fill = obj->v.vec[vec_fill];
-      cnum i, h = ll_hash(obj->v.vec[vec_fill]);
+      cnum i, h = equal_hash(obj->v.vec[vec_fill]);
       cnum len = c_num(fill);
 
       for (i = 0; i < len; i++)
-        h = (h + ll_hash(obj->v.vec[i])) & NUM_MAX;
+        h = (h + equal_hash(obj->v.vec[i])) & NUM_MAX;
 
       return h;
     }
   case LCONS:
-    return (ll_hash(car(obj)) + ll_hash(cdr(obj))) & NUM_MAX;
+    return (equal_hash(car(obj)) + equal_hash(cdr(obj))) & NUM_MAX;
   case LSTR:
     lazy_str_force(obj);
-    return ll_hash(obj->ls.prefix);
+    return equal_hash(obj->ls.prefix);
   case COBJ:
     return obj->co.ops->hash(obj);
   }
@@ -134,6 +137,16 @@ static cnum ll_hash(val obj)
   internal_error("unhandled case in equal function");
 }
 
+static cnum eql_hash(val obj)
+{
+    switch (sizeof (mem_t *)) {
+    case 4:
+      return (((cnum) obj) & NUM_MAX) >> 4;
+    case 8: default:
+      return (((cnum) obj) & NUM_MAX) >> 5;
+    }
+}
+
 cnum cobj_hash_op(val obj)
 {
   return ((cnum) obj) & NUM_MAX;
@@ -141,7 +154,7 @@ cnum cobj_hash_op(val obj)
 
 val hash_obj(val obj)
 {
-  return num(ll_hash(obj));
+  return num(equal_hash(obj));
 }
 
 static void hash_mark(val hash)
@@ -219,8 +232,7 @@ static void hash_grow(struct hash *h)
       val entry = car(conses);
       val next = cdr(conses);
       val key = car(entry);
-      val *pchain = vecref_l(new_table,
-                                num(ll_hash(key) % new_modulus));
+      val *pchain = vecref_l(new_table, num(h->hash_fun(key) % new_modulus));
       *cdr_l(conses) = *pchain;
       *pchain = conses;
       conses = next;
@@ -231,7 +243,7 @@ static void hash_grow(struct hash *h)
   h->table = new_table;
 }
 
-val make_hash(val weak_keys, val weak_vals)
+val make_hash(val weak_keys, val weak_vals, val equal_based)
 {
   int flags = ((weak_vals != nil) << 1) | (weak_keys != nil);
   struct hash *h = (struct hash *) chk_malloc(sizeof *h);
@@ -247,15 +259,19 @@ val make_hash(val weak_keys, val weak_vals)
   h->table = table;
   h->userdata = nil;
 
+  h->hash_fun = equal_based ? equal_hash : eql_hash;
+  h->assoc_fun = equal_based ? assoc : assq;
+  h->acons_new_l_fun = equal_based ? acons_new_l : aconsq_new_l;
+
   return hash;
 }
 
 val *gethash_l(val hash, val key, val *new_p)
 {
   struct hash *h = (struct hash *) hash->co.handle;
-  val *pchain = vecref_l(h->table, num(ll_hash(key) % h->modulus));
+  val *pchain = vecref_l(h->table, num(h->hash_fun(key) % h->modulus));
   val old = *pchain;
-  val *place = acons_new_l(pchain, key, new_p);
+  val *place = h->acons_new_l_fun(pchain, key, new_p);
   if (old != *pchain && ++h->count > 2 * h->modulus)
     hash_grow(h);
   return place;
@@ -264,16 +280,16 @@ val *gethash_l(val hash, val key, val *new_p)
 val gethash(val hash, val key)
 {
   struct hash *h = (struct hash *) hash->co.handle;
-  val chain = *vecref_l(h->table, num(ll_hash(key) % h->modulus));
-  val found = assoc(chain, key);
+  val chain = *vecref_l(h->table, num(h->hash_fun(key) % h->modulus));
+  val found = h->assoc_fun(chain, key);
   return cdr(found);
 }
 
 val gethash_f(val hash, val key, val *found)
 {
   struct hash *h = (struct hash *) hash->co.handle;
-  val chain = *vecref_l(h->table, num(ll_hash(key) % h->modulus));
-  *found = assoc(chain, key);
+  val chain = *vecref_l(h->table, num(h->hash_fun(key) % h->modulus));
+  *found = h->assoc_fun(chain, key);
   return cdr(*found);
 }
 
@@ -284,10 +300,17 @@ val sethash(val hash, val key, val value)
   return new_p;
 }
 
+val pushhash(val hash, val key, val value)
+{
+  val new_p;
+  push(value, gethash_l(hash, key, &new_p));
+  return new_p;
+}
+
 val remhash(val hash, val key)
 {
   struct hash *h = (struct hash *) hash->co.handle;
-  val *pchain = vecref_l(h->table, num(ll_hash(key) % h->modulus));
+  val *pchain = vecref_l(h->table, num(h->hash_fun(key) % h->modulus));
   *pchain = alist_remove1(*pchain, key);
   h->count--;
   bug_unless (h->count >= 0);
diff --git a/hash.h b/hash.h
index 67e56f10..3f4cd2d0 100644
--- a/hash.h
+++ b/hash.h
@@ -25,11 +25,12 @@
  */
 
 val hash_obj(val);
-val make_hash(val weak_keys, val weak_vals);
+val make_hash(val weak_keys, val weak_vals, val equal_based);
 val *gethash_l(val hash, val key, val *new_p);
 val gethash(val hash, val key);
 val gethash_f(val hash, val key, val *found);
 val sethash(val hash, val key, val value);
+val pushhash(val hash, val key, val value);
 val remhash(val hash, val key);
 val hash_count(val hash);
 val get_hash_userdata(val hash);
diff --git a/lib.c b/lib.c
index 72cbd551..7b3f9765 100644
--- a/lib.c
+++ b/lib.c
@@ -76,7 +76,7 @@ val null_string;
 val nil_string;
 val null_list;
 
-val identity_f, equal_f, eq_f, car_f;
+val identity_f, equal_f, eql_f, eq_f, car_f;
 
 val prog_string;
 
@@ -440,6 +440,14 @@ val flatten(val list)
 
 cnum c_num(val num);
 
+val eql(val left, val right)
+{
+  /* eql is same as eq for now, but when we get bignums,
+     eql will compare different bignum objects which are
+     the same number as equal. */
+  return eq(left, right);
+}
+
 val equal(val left, val right)
 {
   /* Bitwise equality is equality, period. */
@@ -1300,7 +1308,7 @@ val make_package(val name)
     val obj = make_obj();
     obj->pk.type = PKG;
     obj->pk.name = name;
-    obj->pk.symhash = make_hash(nil, nil);
+    obj->pk.symhash = make_hash(nil, nil, lit("t")); /* don't have t yet! */
 
     push(cons(name, obj), &packages);
     return obj;
@@ -2073,6 +2081,18 @@ val assoc(val list, val key)
   return nil;
 }
 
+val assq(val list, val key)
+{
+  while (list) {
+    val elem = car(list);
+    if (eql(car(elem), key))
+      return elem;
+    list = cdr(list);
+  }
+
+  return nil;
+}
+
 val acons(val list, val car, val cdr)
 {
   return cons(cons(car, cdr), list);
@@ -2107,6 +2127,36 @@ val *acons_new_l(val *list, val key, val *new_p)
   }
 }
 
+val aconsq_new(val list, val key, val value)
+{
+  val existing = assq(list, key);
+
+  if (existing) {
+    *cdr_l(existing) = value;
+    return list;
+  } else {
+    return cons(cons(key, value), list);
+  }
+}
+
+val *aconsq_new_l(val *list, val key, val *new_p)
+{
+  val existing = assq(*list, key);
+
+  if (existing) {
+    if (new_p)
+      *new_p = nil;
+    return cdr_l(existing);
+  } else {
+    val nc = cons(key, nil);
+    *list = cons(nc, *list);
+    if (new_p)
+      *new_p = t;
+    return cdr_l(nc);
+  }
+}
+
+
 static val alist_remove_test(val item, val key)
 {
   return eq(car(item), key);
@@ -2346,7 +2396,7 @@ static void obj_init(void)
 
   protect(&packages, &system_package, &keyword_package,
           &user_package, &null_string, &nil_string,
-          &null_list, &equal_f, &eq_f, &car_f,
+          &null_list, &equal_f, &eq_f, &eql_f, &car_f,
           &identity_f, &prog_string, &env_list,
           (val *) 0);
 
@@ -2451,6 +2501,7 @@ static void obj_init(void)
 
   equal_f = func_n2(equal);
   eq_f = func_n2(eq);
+  eql_f = func_n2(eql);
   identity_f = func_n1(identity);
   car_f = func_n1(car);
   prog_string = string(progname);
diff --git a/lib.h b/lib.h
index b0bd50a8..a395b76b 100644
--- a/lib.h
+++ b/lib.h
@@ -300,6 +300,7 @@ val tree_find(val obj, val tree, val testfun);
 val some_satisfy(val list, val pred, val key);
 val all_satisfy(val list, val pred, val key);
 val none_satisfy(val list, val pred, val key);
+val eql(val left, val right);
 val equal(val left, val right);
 mem_t *chk_malloc(size_t size);
 mem_t *chk_realloc(mem_t *, size_t size);
@@ -419,9 +420,12 @@ mem_t *cobj_handle(val cobj, val cls_sym);
 val cptr(mem_t *ptr);
 mem_t *cptr_get(val cptr);
 val assoc(val list, val key);
+val assq(val list, val key);
 val acons(val list, val car, val cdr);
 val acons_new(val list, val key, val value);
 val *acons_new_l(val *list, val key, val *new_p);
+val aconsq_new(val list, val key, val value);
+val *aconsq_new_l(val *list, val key, val *new_p);
 val alist_remove(val list, val keys);
 val alist_remove1(val list, val key);
 val alist_nremove(val list, val keys);
diff --git a/match.c b/match.c
index d4bd86de..38fc8ec7 100644
--- a/match.c
+++ b/match.c
@@ -3138,8 +3138,8 @@ static void syms_init(void)
 
 static void dir_tables_init(void)
 {
-  h_directive_table = make_hash(nil, nil);
-  v_directive_table = make_hash(nil, nil);
+  h_directive_table = make_hash(nil, nil, nil);
+  v_directive_table = make_hash(nil, nil, nil);
 
   protect(&h_directive_table, &v_directive_table, (val *) 0);
 
diff --git a/parser.h b/parser.h
index 5c096720..7567cefb 100644
--- a/parser.h
+++ b/parser.h
@@ -29,6 +29,8 @@ extern int errors;
 extern val yyin_stream;
 extern const wchar_t *spec_file;
 extern val spec_file_str;
+extern val form_to_ln_hash;
+extern val ln_to_forms_hash;
 int yyparse(void);
 val get_spec(void);
 void yyerror(const char *s);
diff --git a/parser.l b/parser.l
index 897c6532..4fe0e521 100644
--- a/parser.l
+++ b/parser.l
@@ -43,6 +43,7 @@
 #include "stream.h"
 #include "utf8.h"
 #include "unwind.h"
+#include "hash.h"
 #include "parser.h"
 
 #define YY_INPUT(buf, result, max_size)           \
@@ -66,6 +67,9 @@ int opt_loglevel = 1;   /* 0 - quiet; 1 - normal; 2 - verbose */
 
 int errors;
 
+val form_to_ln_hash;
+val ln_to_forms_hash;
+
 static val prepared_error_message;
 
 void yyerror(const char *s)
@@ -210,122 +214,146 @@ UONLY   {U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}
 
 <SPECIAL>\({WS}all{WS}\)        {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return ALL;
                                 }
 
 <SPECIAL>\({WS}some{WS}\)       {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return SOME;
                                 }
 
 <SPECIAL>\({WS}none{WS}\)       {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return NONE;
                                 }
 
 <SPECIAL>\({WS}maybe{WS}\)      {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return MAYBE;
                                 }
 
 <SPECIAL>\({WS}cases{WS}\)      {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return CASES;
                                 }
 
 <SPECIAL>\({WS}choose/{ID_END}  {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return CHOOSE;
                                 }
 
 <SPECIAL>\({WS}gather/{ID_END}  {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return GATHER;
                                 }
 
 <SPECIAL>\({WS}and{WS}\)        {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return AND;
                                 }
 
 <SPECIAL>\({WS}or{WS}\)         {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return OR;
                                 }
 
 <SPECIAL>\({WS}end{WS}\)        {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return END;
                                 }
 
 <SPECIAL>\({WS}collect/{ID_END} {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return COLLECT;
                                 }
 
 <SPECIAL>\({WS}coll/{ID_END}    {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return COLL;
                                 }
 
 <SPECIAL>\({WS}until{WS}\)      {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return UNTIL;
                                 }
 
 <SPECIAL>\({WS}output/{ID_END}  {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return OUTPUT;
                                 }
 
 <SPECIAL>\({WS}repeat{WS}\)     {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return REPEAT;
                                 }
 
 
 <SPECIAL>\({WS}rep{WS}\)        {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return REP;
                                 }
 
 <SPECIAL>\({WS}single{WS}\)     {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return SINGLE;
                                 }
 
 <SPECIAL>\({WS}first{WS}\)      {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return FIRST;
                                 }
 
 <SPECIAL>\({WS}last{WS}\)       {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return LAST;
                                 }
 
 <SPECIAL>\({WS}empty{WS}\)      {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return EMPTY;
                                 }
 
 <SPECIAL>\({WS}define/{ID_END}  {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return DEFINE;
                                 }
 
 <SPECIAL>\({WS}try{WS}\)        {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return TRY;
                                 }
 
 <SPECIAL>\({WS}catch/{ID_END}   {
                                   yy_push_state(NESTED);
+                                  yylval.lineno = lineno;
                                   return CATCH;
                                 }
 
 <SPECIAL>\({WS}finally{WS}\)    {
                                   yy_pop_state();
+                                  yylval.lineno = lineno;
                                   return FINALLY;
                                 }
 
@@ -339,6 +367,7 @@ UONLY   {U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}
                             yylval.chr = '(';
                             return METAPAR;
                           }
+                          yylval.lineno = lineno;
                           return yytext[0];
                         }
 
@@ -594,5 +623,9 @@ void end_of_regex(void)
 
 void parse_init(void)
 {
-  protect(&yyin_stream, &prepared_error_message, (val *) 0);
+  protect(&yyin_stream, &prepared_error_message,
+          &form_to_ln_hash, &ln_to_forms_hash, (val *) 0);
+
+  form_to_ln_hash = make_hash(t, nil, nil);
+  ln_to_forms_hash = make_hash(nil, t, nil);
 }
diff --git a/parser.y b/parser.y
index e15b9f5d..fd4fae34 100644
--- a/parser.y
+++ b/parser.y
@@ -37,6 +37,7 @@
 #include "regex.h"
 #include "utf8.h"
 #include "match.h"
+#include "hash.h"
 #include "parser.h"
 
 int yylex(void);
@@ -49,21 +50,29 @@ val lit_char_helper(val litchars);
 
 static val parsed_spec;
 
+static val rl(val form, val lineno)
+{
+  sethash(form_to_ln_hash, form, lineno);
+  pushhash(ln_to_forms_hash, lineno, form);
+  return form;
+}
+
+
 %}
 
 %union {
   wchar_t *lexeme;
   union obj *val;
   wchar_t chr;
-  cnum num;
+  cnum num, lineno;
 }
 
 %token <lexeme> SPACE TEXT IDENT KEYWORD METAVAR
-%token <lexeme> ALL SOME NONE MAYBE CASES CHOOSE GATHER
-%token <lexeme> AND OR END COLLECT
-%token <lexeme> UNTIL COLL OUTPUT REPEAT REP SINGLE FIRST LAST EMPTY DEFINE
-%token <lexeme> TRY CATCH FINALLY
-%token <lexeme> ERRTOK /* deliberately not used in grammar */
+%token <lineno> ALL SOME NONE MAYBE CASES CHOOSE GATHER
+%token <lineno> AND OR END COLLECT
+%token <lineno> UNTIL COLL OUTPUT REPEAT REP SINGLE FIRST LAST EMPTY DEFINE
+%token <lineno> TRY CATCH FINALLY
+%token <lineno> ERRTOK /* deliberately not used in grammar */
 
 %token <num> NUMBER
 
@@ -84,6 +93,7 @@ static val parsed_spec;
 %type <val> regterm regclass regclassterm regrange
 %type <val> strlit chrlit quasilit quasi_items quasi_item litchars
 %type <chr> regchar
+%type <lineno> '('
 
 %nonassoc LOW /* used for precedence assertion */
 %nonassoc ALL SOME NONE MAYBE CASES CHOOSE AND OR END COLLECT UNTIL COLL
@@ -133,7 +143,8 @@ clause : all_clause             { $$ = list(num(lineno - 1), $1, nao); }
                                   yyerror("repeat outside of output"); }
        ;
 
-all_clause : ALL newl clause_parts      { $$ = list(all_s, $3, nao); }
+all_clause : ALL newl clause_parts      { $$ = list(all_s, $3, nao);
+                                          rl($$, num($1)); }
            | ALL newl error             { $$ = nil;
                                           yybadtoken(yychar,
                                                      lit("all clause")); }
@@ -142,7 +153,8 @@ all_clause : ALL newl clause_parts      { $$ = list(all_s, $3, nao); }
 
            ;
 
-some_clause : SOME newl clause_parts    { $$ = list(some_s, $3, nao); }
+some_clause : SOME newl clause_parts    { $$ = list(some_s, $3, nao);
+                                          rl($$, num($1)); }
             | SOME newl error           { $$ = nil;
                                           yybadtoken(yychar,
                                                      lit("some clause")); }
@@ -150,7 +162,8 @@ some_clause : SOME newl clause_parts    { $$ = list(some_s, $3, nao); }
                                           yyerror("empty some clause"); }
             ;
 
-none_clause : NONE newl clause_parts    { $$ = list(none_s, $3, nao); }
+none_clause : NONE newl clause_parts    { $$ = list(none_s, $3, nao);
+                                          rl($$, num($1)); }
             | NONE newl error           { $$ = nil;
                                           yybadtoken(yychar,
                                                      lit("none clause")); }
@@ -158,7 +171,8 @@ none_clause : NONE newl clause_parts    { $$ = list(none_s, $3, nao); }
                                           yyerror("empty none clause"); }
             ;
 
-maybe_clause : MAYBE newl clause_parts  { $$ = list(maybe_s, $3, nao); }
+maybe_clause : MAYBE newl clause_parts  { $$ = list(maybe_s, $3, nao);
+                                          rl($$, num($1)); }
              | MAYBE newl error         { $$ = nil;
                                           yybadtoken(yychar,
                                                      lit("maybe clause")); }
@@ -166,7 +180,8 @@ maybe_clause : MAYBE newl clause_parts  { $$ = list(maybe_s, $3, nao); }
                                           yyerror("empty maybe clause"); }
              ;
 
-cases_clause : CASES newl clause_parts  { $$ = list(cases_s, $3, nao); }
+cases_clause : CASES newl clause_parts  { $$ = list(cases_s, $3, nao);
+                                          rl($$, num($1)); }
              | CASES newl error         { $$ = nil;
                                           yybadtoken(yychar,
                                                      lit("cases clause")); }
@@ -175,7 +190,8 @@ cases_clause : CASES newl clause_parts  { $$ = list(cases_s, $3, nao); }
              ;
 
 choose_clause : CHOOSE exprs_opt ')'
-                newl clause_parts       { $$ = list(choose_s, $5, $2, nao); }
+                newl clause_parts       { $$ = list(choose_s, $5, $2, nao);
+                                          rl($$, num($1)); }
               | CHOOSE exprs_opt ')'
                 newl error              { $$ = nil;
                                           yybadtoken(yychar,
@@ -189,7 +205,8 @@ gather_clause : GATHER exprs_opt ')'
                 newl clause_parts       { $$ = list(gather_s, 
                                                     append2(mapcar(curry_12_1(func_n2(cons), nil),
                                                                    first($5)), rest($5)),
-                                                    $2, nao); }
+                                                    $2, nao); 
+                                          rl($$, num($1)); }
               | GATHER exprs_opt ')'
                 newl error              { $$ = nil;
                                           yybadtoken(yychar,
@@ -203,11 +220,13 @@ gather_clause : GATHER exprs_opt ')'
 collect_clause : COLLECT exprs_opt ')' newl
                  clauses END newl                { $$ = list(collect_s,
                                                              $5, nil, $2,
-                                                             nao); }
+                                                             nao);
+                                                   rl($$, num($1)); }
                | COLLECT exprs_opt ')'
                  newl clauses until_last
                  newl clauses END newl    { $$ = list(collect_s, $5,
-                                                      cons($6, $8), $2, nao); }
+                                                      cons($6, $8), $2, nao);
+                                            rl($$, num($1)); }
                | COLLECT exprs_opt ')'
                  newl error             { $$ = nil;
                                           if (yychar == UNTIL ||
@@ -228,7 +247,7 @@ clause_parts : clauses additional_parts { $$ = cons($1, $2); }
 
 additional_parts : END newl                             { $$ = nil; }
                  | AND newl clauses additional_parts    { $$ = cons($3, $4); }
-                 | OR newl clauses additional_parts    { $$ = cons($3, $4); }
+                 | OR newl clauses additional_parts     { $$ = cons($3, $4); }
                  ;
 
 line : elems_opt '\n'           { $$ = $1; }
@@ -244,38 +263,44 @@ elems : elem                    { $$ = cons($1, nil); }
                                   yyerror("rep outside of output"); }
       ;
 
-elem : TEXT                     { $$ = string_own($1); }
+elem : TEXT                     { $$ = rl(string_own($1), num(lineno)); }
      | SPACE                    { if ($1[0] == ' ' && $1[1] == 0)
                                   { val spaces = list(oneplus_s, 
                                                       chr(' '), nao);
                                     $$ = cons(regex_compile(spaces), spaces);
+                                    rl($$, num(lineno));
                                     free($1); }
                                   else
-                                  { $$ = string_own($1); }}
-     | var                      { $$ = $1; }
+                                  { $$ = rl(string_own($1), num(lineno)); }}
+     | var                      { $$ = rl($1, num(lineno)); }
      | list                     { $$ = $1; }
      | regex                    { $$ = cons(regex_compile(rest($1)),
-                                            rest($1)); }
-     | COLL exprs_opt ')' elems END     { $$ = list(coll_s, $4, nil, $2, nao); }
+                                            rest($1));
+                                  rl($1, num(lineno)); }
+     | COLL exprs_opt ')' elems END     { $$ = list(coll_s, $4, nil, $2, nao);
+                                          rl($$, num($1)); }
      | COLL exprs_opt ')' elems
        until_last elems END     { $$ = list(coll_s, $4, cons($5, $6), 
-                                            $2, nao); }
+                                            $2, nao);
+                                  rl($$, num($1)); }
      | COLL error               { $$ = nil;
                                   yybadtoken(yychar, lit("coll clause")); }
-     | ALL clause_parts_h       { $$ = list(all_s, t, $2, nao); }
+     | ALL clause_parts_h       { $$ = rl(list(all_s, t, $2, nao), num($1)); }
      | ALL END                  { yyerror("empty all clause"); }
-     | SOME clause_parts_h      { $$ = list(some_s, t, $2, nao); }
+     | SOME clause_parts_h      { $$ = rl(list(some_s, t, $2, nao), num($1)); }
      | SOME END                 { yyerror("empty some clause"); }
-     | NONE clause_parts_h      { $$ = list(none_s, t, $2, nao); }
+     | NONE clause_parts_h      { $$ = rl(list(none_s, t, $2, nao), num($1)); }
      | NONE END                 { yyerror("empty none clause"); }
-     | MAYBE clause_parts_h     { $$ = list(maybe_s, t, $2, nao); }
+     | MAYBE clause_parts_h     { $$ = rl(list(maybe_s, t, $2, nao), num($1)); }
      | MAYBE END                { yyerror("empty maybe clause"); }
-     | CASES clause_parts_h     { $$ = list(cases_s, t, $2, nao); }
+     | CASES clause_parts_h     { $$ = rl(list(cases_s, t, $2, nao), num($1)); }
      | CASES END                { yyerror("empty cases clause"); }
      | CHOOSE exprs_opt ')'
-       clause_parts_h           { $$ = list(choose_s, t, $4, $2, nao); }
+       clause_parts_h           { $$ = list(choose_s, t, $4, $2, nao);
+                                  rl($$, num($1)); }
      | CHOOSE exprs_opt ')' END { yyerror("empty cases clause"); }
-     | DEFINE exprs ')' elems END       { $$ = list(define_s, t, $4, $2, nao); }
+     | DEFINE exprs ')' elems END       { $$ = list(define_s, t, $4, $2, nao);
+                                          rl($$, num($1)); }
      ;
 
 clause_parts_h : elems additional_parts_h { $$ = cons($1, $2); }
@@ -288,10 +313,12 @@ additional_parts_h : END                                { $$ = nil; }
 
 define_clause : DEFINE exprs ')' newl
                 clauses_opt
-                END newl        { $$ = list(define_s, $2, $5, nao); }
+                END newl        { $$ = list(define_s, $2, $5, nao);
+                                  rl($$, num($1)); }
               | DEFINE ')' newl
                 clauses_opt
-                END newl        { $$ = list(define_s, nil, $4, nao); }
+                END newl        { $$ = list(define_s, nil, $4, nao);
+                                  rl($$, num($1)); }
               | DEFINE error    { $$ = nil;
                                   yybadtoken(yychar, lit("list expression")); }
               | DEFINE exprs ')' newl
@@ -307,7 +334,8 @@ try_clause : TRY newl
              END newl           { $$ = list(try_s,
                                             flatten(mapcar(func_n1(second),
                                                            $4)),
-                                            $3, $4, nao); }
+                                            $3, $4, nao);
+                                  rl($$, num($1)); }
            | TRY newl
              error              { $$ = nil;
                                   if (yychar == END || yychar == CATCH ||
@@ -324,15 +352,18 @@ try_clause : TRY newl
 catch_clauses_opt : CATCH ')' newl
                     clauses_opt
                     catch_clauses_opt   { $$ = cons(list(catch_s, cons(t, nil),
-                                                         $4, nao), $5); }
+                                                         $4, nao), $5);
+                                          rl($$, num($1)); }
                   | CATCH exprs ')' newl
                     clauses_opt
                     catch_clauses_opt   { $$ = cons(list(catch_s, $2, $5, nao),
-                                                    $6); }
+                                                    $6);
+                                          rl($$, num($1)); }
                   | FINALLY newl
                     clauses_opt         { $$ = cons(list(finally_s, nil,
                                                          $3, nao),
-                                                    nil); }
+                                                    nil);
+                                          rl($$, num($1)); }
                   |                     { $$ = nil; }
                   | CATCH ')' newl
                     error               { $$ = nil;
@@ -354,13 +385,15 @@ output_clause : OUTPUT ')' o_elems '\n'
                 END newl        { $$ = nil;
                                   yyerror("obsolete output syntax: trailing material"); }
               | OUTPUT ')' newl
-                END newl        { $$ = list(output_s, nao); }
+                END newl        { $$ = rl(list(output_s, nao), num($1)); }
               | OUTPUT ')' newl
                 out_clauses
-                END newl        { $$ = list(output_s, $4, nao); }
+                END newl        { $$ = rl(list(output_s, $4, nao), num($1)); }
               | OUTPUT exprs ')' newl
                 out_clauses
-                END newl        { $$ = list(output_s, $5, $2, nao); }
+                END newl        { $$ = list(output_s, $5, $2, nao);
+                                  rl($$, num($1)); }
+
               | OUTPUT exprs ')' o_elems '\n'
                 out_clauses
                 END newl        { $$ = nil;
@@ -414,7 +447,8 @@ out_clause : repeat_clause              { $$ = list(num(lineno - 1), $1, nao); }
 repeat_clause : REPEAT newl
                 out_clauses
                 repeat_parts_opt
-                END newl                { $$ = repeat_rep_helper(repeat_s, $3, $4); }
+                END newl                { $$ = repeat_rep_helper(repeat_s, $3, $4);
+                                          rl($$, num($1)); }
               | REPEAT newl
                 error           { $$ = nil;
                                   yybadtoken(yychar, lit("repeat clause")); }
@@ -422,16 +456,20 @@ repeat_clause : REPEAT newl
 
 repeat_parts_opt : SINGLE newl
                    out_clauses_opt
-                   repeat_parts_opt     { $$ = cons(cons(single_s, $3), $4); }
+                   repeat_parts_opt     { $$ = cons(cons(single_s, $3), $4);
+                                          rl($$, num($1)); }
                  | FIRST newl
                    out_clauses_opt
-                   repeat_parts_opt     { $$ = cons(cons(first_s, $3), $4); }
+                   repeat_parts_opt     { $$ = cons(cons(first_s, $3), $4);
+                                          rl($$, num($1)); }
                  | LAST newl
                    out_clauses_opt
-                   repeat_parts_opt     { $$ = cons(cons(last_s, $3), $4); }
+                   repeat_parts_opt     { $$ = cons(cons(last_s, $3), $4);
+                                          rl($$, num($1)); }
                  | EMPTY newl
                    out_clauses_opt
-                   repeat_parts_opt     { $$ = cons(cons(empty_s, $3), $4); }
+                   repeat_parts_opt     { $$ = cons(cons(empty_s, $3), $4);
+                                          rl($$, num($1)); }
                  | /* empty */          { $$ = nil; }
                  ;
 
@@ -464,19 +502,24 @@ o_elem : TEXT                   { $$ = string_own($1); }
 rep_elem : REP o_elems
            rep_parts_opt END    { $$ = repeat_rep_helper(rep_s, 
                                                          o_elems_transform($2),
-                                                         $3); }
+                                                         $3);
+                                  rl($$, num($1)); }
          | REP error            { $$ = nil;
                                   yybadtoken(yychar, lit("rep clause")); }
          ;
 
 rep_parts_opt : SINGLE o_elems_opt2
-                rep_parts_opt           { $$ = cons(cons(single_s, $2), $3); }
+                rep_parts_opt           { $$ = cons(cons(single_s, $2), $3);
+                                          rl($$, num($1)); }
               | FIRST o_elems_opt2
-                rep_parts_opt           { $$ = cons(cons(first_s, $2), $3); }
+                rep_parts_opt           { $$ = cons(cons(first_s, $2), $3);
+                                          rl($$, num($1)); }
               | LAST o_elems_opt2
-                rep_parts_opt           { $$ = cons(cons(last_s, $2), $3); }
+                rep_parts_opt           { $$ = cons(cons(last_s, $2), $3);
+                                          rl($$, num($1)); }
               | EMPTY o_elems_opt2
-                rep_parts_opt           { $$ = cons(cons(empty_s, $2), $3); }
+                rep_parts_opt           { $$ = cons(cons(empty_s, $2), $3);
+                                          rl($$, num($1)); }
               | /* empty */             { $$ = nil; }
               ;
 
@@ -537,7 +580,7 @@ o_var : IDENT                   { $$ = list(var_s, intern(string_own($1), nil),
 var_op : '*'                    { $$ = list(t, nao); }
        ;
 
-list : '(' exprs ')'            { $$ = $2; }
+list : '(' exprs ')'            { $$ = rl($2, num($1)); }
      | '(' ')'                  { $$ = nil; }
      | '(' error                { $$ = nil;
                                   yybadtoken(yychar, lit("list expression")); }
@@ -572,7 +615,8 @@ expr : IDENT                    { $$ = intern(string_own($1), nil); }
      | quasilit                 { $$ = $1; }
      ;
 
-regex : '/' regexpr '/'         { $$ = cons(regex_s, $2); end_of_regex(); }
+regex : '/' regexpr '/'         { $$ = cons(regex_s, $2); end_of_regex();
+                                  rl($$, num(lineno)); }
       | '/' error               { $$ = nil;
                                   yybadtoken(yychar, lit("regex"));
                                   end_of_regex(); }
@@ -664,7 +708,8 @@ chrlit : '\'' '\''              { $$ = nil;
        ;
 
 quasilit : '`' '`'              { $$ = null_string; }
-         | '`' quasi_items '`'  { $$ = cons(quasi_s, o_elems_transform($2)); }
+         | '`' quasi_items '`'  { $$ = cons(quasi_s, o_elems_transform($2));
+                                  rl($$, num(lineno)); }
          | '`' error            { $$ = nil;
                                   yybadtoken(yychar, lit("string literal")); }
          ;