New: stack overflow protection.

* configure: detect getrlimit, producing HAVE_RLIMIT in
config.h.

* eval.c (do_eval, do_expand): Call gc_stack_check inline
function to check stack pointer against limit.

* gc.c (gc_stack_bottom): Static becomes extern, so inline
function in gc.h can refer to it.
(gc_stack_limit): New global variable.
(gc_init): If we have rlimit, then probe RLIMIT_STACK.
If the stack is sufficiently large, then enable the stack
overflow protection, which kicks in when the stack pointer
appears to be within a certain percentage of the limit.
(set_stack_limit, get_stack_limit): New static functions.
(gc_late_init): Register set-stack-limit and get-stack-limit
intrinsics.
(gc_stack_overflow): New function.

* gc.h (gc_stack_bottom, gc_stack_limit, gc_stack_overflow):
Declared.
(gc_stack_check): New inline function.

* lib.c (stack_overflow_s): New symbol variable.
(obj_print_impl): Call gc_stack_check to protect recursive
printing againts overflow.

* lib.h (stack_overflow_s): Declared.

* unwind.c (uw_init): Register stack-overflow symbol as a an
exception symbol subtyped from error.
(uw_unwind_to_exit_point): When dealing with an unhandled
exception, turn off the stack limit, so we can print the
messages without triggering it in a loop.

* vm.c (vm_execute_closure, vm_funcall_common): Insert
gc_stack_check to the top of the execution of every VM
function.

* txr.1: Documented.

* share/txr/stdlib/doc-syms.tl: Updated.

1 files changed, 3 insertions, 0 deletions

diff --git a/unwind.c b/unwind.c
index 3cb47a9a..13089c3c 100644
--- a/unwind.c
+++ b/unwind.c
@@ -120,6 +120,8 @@ static void uw_unwind_to_exit_point(void)
     val sym = unhandled_ex.ca.sym;
     val args = unhandled_ex.ca.args;
 
+    gc_stack_limit = 0;
+
     dyn_env = nil;
 
     if (opt_loglevel >= 1) {
@@ -1256,6 +1258,7 @@ void uw_init(void)
   uw_register_subtype(process_error_s, error_s);
   uw_register_subtype(system_error_s, error_s);
   uw_register_subtype(alloc_error_s, error_s);
+  uw_register_subtype(stack_overflow_s, error_s);
   uw_register_subtype(timeout_error_s, error_s);
   uw_register_subtype(assert_s, error_s);
   uw_register_subtype(syntax_error_s, error_s);