bugfix: fixnum crackdown.

The purpose of this commit is to address certain situations in
which code is wrongly relying on a cnum value being in the
fixnum range (NUM_MIN to NUM_MAX), so that num_fast can safely
be used on it.

One wrong pattern is that c_num is applied to some Lisp value,
and that value (or one derived from it arithmetically) is then
passed to num_fast. The problem is that c_num succeeds on
integers outside of the fixnum range.  Some bignum values
convert to a cnum successfully. Thus either num has to be used
instead of num_fast, or else the original c_num attempt must
be replaced with something that will fail if the original
value isn't a fixnum. (In the latter case, any arithmetic on
the fixnum cannot produce value outside of that range).

* buf.c (buf_put_bytes): The size argument here is not
guaranteed to be in fixnum range: use num.

* combi.c (perm_init_common): Throw if the sequence length
isn't a fixnum. Thus the num_fast in perm_while_fun is
correct, since the ci value is bounded by k, which is bounded
by n.

* hash.c (hash_grow): Remove dubious assertion which aborts
the run-time if the hash table doubling overflows.  Simply
don't allow the modulus to grow beyond NUM_MAX.  If doubling
it makes it larger than NUM_MAX, then just don't grow the
table. We need the modulus to be in fixnum range, so that
uses of num_fast on the modulus value elsewhere are correct.
(group_by, group_reduce): Use c_fixnum rather than c_num to
extract a value that is later assumed to be a fixnum.

* lib.c (c_fixnum): New function.
(nreverse, reverse, remove_if, less, window_map_list,
sort_vec, unique): Use c_fixnum rather than c_num to extract a
value that is later assumed to be a fixnum.
(string_extend): Use c_fixnum rather than c_num to extract a
value that is later assumed to be a fixnum. Cap the string
allocation size to fixnum range rather than INT_PTR_MAX.
(cmp_str): The wcscmp function could return values outside
of the fixnum range, so we must use num, not num_fast.

* lib.h (c_fixnum): Declared.

1 files changed, 4 insertions, 0 deletions

diff --git a/combi.c b/combi.c
index 51bbdc9b..c749b300 100644
--- a/combi.c
+++ b/combi.c
@@ -100,6 +100,10 @@ static val perm_init_common(val p, val k_null)
   val n = length(p);
   val k = or2(k_null, n);
 
+  if (!fixnump(n))
+    uw_throwf(error_s, lit("perm: sequence length ~s is out of fixnum range"),
+              n, nao);
+
   if (gt(k, n)) {
     return nil;
   } else {