mpi: hardening; bug found.

The flaw in the rand function caused by mp_count_ones is
serious. For instance calls to (rand 3000000000) are expected
to produce evenly distributed values in the range zero
to one less than 3 billion. Due to this flaw, only values
in the range [0, 2147483648) are produced. The function is
wrongly informed that 3000000000 is a power of two, and
so it subtracts one from the number of bits needed for the
clamping bit mask, causing raw random values to be clamped
to 31 bits instead of 32.

* arith.c (logcount): Use mp_err to capture return value from
mp_count_ones, rather than mp_size. If the result is less than
zero, throw an internal error. (The only non-internal error
indication that could come from this function is a memory
allocation, and we already use chk_malloc under MPI).

* mpi/mpi.c (mp_count_ones): Switch return type to mp_err,
because this function returns error codes, which are negative.
(mp_is_pow_two): Fix broken test that is always true.
This affects our random number module; it's used in the random
function for bignum moduli.
(mp_toradix_case): Use unsigned char temporary for exchanging
two unsigned chars, rather than a char temporary.
(s_mp_div_d): Assert that the partial quotient t  and
remainder w, of mp_word_type, have values that fit into the
mp_digit variables to which they are being assigned.
(s_mp_ispow2d): The result of s_highest_bit(d) is unsigned,
so if we subtract 1 from 0, we create a large value which
likely converts to -1. Let's ensure that clearly with a cast
to (int) of the return value.

* mpi/mpi.h (mp_count_ones): Declaration updated.

* mpi/mplogic.c (mpl_rsh, mpl_lsh): Fix two places where we
shift the constant 1 (of type int) left by a number of bits
that can exceed the type int. Note, that these functions are
currently not called anywhere in TXR.

1 files changed, 1 insertions, 1 deletions

diff --git a/mpi/mpi.h b/mpi/mpi.h
index 1106abde..c1b50296 100644
--- a/mpi/mpi.h
+++ b/mpi/mpi.h
@@ -177,7 +177,7 @@ mp_err mp_to_unsigned_bin(mp_int *mp, unsigned char *str);
 mp_err mp_to_unsigned_buf(mp_int *mp, unsigned char *str, size_t size);
 
 mp_size mp_count_bits(mp_int *mp);
-mp_size mp_count_ones(mp_int *mp);
+mp_err mp_count_ones(mp_int *mp);
 mp_size mp_is_pow_two(mp_int *mp);
 
 #if MP_COMPAT_MACROS