| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is motivated by the desired to be able to
specify gzip compression: e.g.
(file-put-string "file.gz" "abc" "z")
However, it has other uses. For instance with "x",
we can exclusively create a file with content in one
call.
* stdlib/getput.tl (file-get, file-put, file-append,
file-get-string, file-put-string, file-append-string,
file-put-lines, file-append-lines, file-get-buf,
file-put-buf, file-place-buf, file-append-buf, file-get-json,
file-put-json, file-append-json, file-get-jsons,
file-put-jsons, file-append-jsons): New option argument
which combines with the open-file mode.
* txr.1: Documented.
|
| |
|
|
|
| |
* stream.c (open_fileno): Use w_gzdopen_mode and make_gzio_stream
to make a gzio stream if the gzip flag is present.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* gzio.c (w_gzopen_mode): Use w_open_mode if available, in order
to support most of the flags (including "x" which Zlib has,
but which we are not passing through).
* stream.c (w_open_mode): New function formed from w_fopen_mode
content.
(w_fopen_mode): Call w_open_mode.
* stream.c (w_open_mode): Declared.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Makefile (OBJS): Conditionally include new gzio.o object file.
* configure: Detect external zlib, setting up new have_zlib
variable in config.make, HAVE_ZLIB in config.h, and also -lz
in conf_ldlibs.
* gzio.[ch]: New files, implementing the stream abstraction
over the gzip file I/O routines in zlib.
* stream.h (struct stdio_mode): New gzip flag and gzlevel
bitfield to hold a value 0 to 9.
(stdio_mode_init_blank, stdio_mode_init_r, stdio_mode_init_rpb):
Update intializers to cover new bitfield members.
* stream.c: Include <zlib.h> and "gzio.h" if HAVE_ZLIB.
(do_parse_mode): Recognize new mode modifier letter "z",
setting the gzip flag in the mode structure. If it's followed
by a digit, set the gziplevel to that value.
(format_mode): Don't output "b" letter for binary mode if
gzip is set, because gzopen interprets "b" differently.
Don't put out "t" if gzip is set. If gzip mode is specified,
do put out the level. If gzip is set, and gziplevel is nonzero
then encode the level: gzopen will understand it.
(open_file): If gzip mode is requested, then open the file
using gzopen mode, a new function in gzio.c. The return a
gzio stream based on the returned gzip file handle. However,
if we are reading, and the gzip stream indicates that it's
not decompressing anything, then we close it and open the
file using an ordinary stream.
(stream_init): Call gzio_init if HAVE_ZLIB is true. This is done
here because the module is integrated with stream.c, and also
so that lib.c doesn't have to know about HAVE_ZLIB and <zlib.h>.
|
| |
|
|
|
|
| |
* stream.c (normalize_mode): Include the self symbol in the
error message being thrown. This was not forgotten in the
sister function normalize_mode_no_bin.
|
| |
|
|
|
|
| |
* stream.c (stdio_set_prop): When setting the :name property,
return t, to indicate it was recognized and stored, as is
documented.
|
| |
|
|
|
| |
* stream.c (stdio_maybe_read_error): Access errno once, and
refer to the value.
|
| |
|
|
|
|
| |
* configure: do not treat the failure to detect mmap as
a fatal condition. This was just debugging commands accidentally
left behind.
|
| |
|
|
|
|
| |
* tests/019/load-search.tl: skip a certain test if it is run as
superuser; it fails because superuser is not affected by denied
directory search and execute permissions.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* RELNOTES: Updated.
* configure (txr_ver): Bumped version.
* stdlib/ver.tl (lib-version): Bumped.
* txr.1: Bumped version and date.
* txr.vim, tl.vim: Regenerated.
* protsym.c: Likewise.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We've already taken care of imitating the situation that GNU
C allows __attribute__((aligned(n))) to weaken the alignment
of a bitfield, contrary to it being documented that align only
strengthens alignment. Even a value of n == 1 is meaningful
in that it can cause the bitfield to start allocating from
a new byte.
This patch corrects a newly discovered nuance: when a bitfield
is attributed with a weaker alignment than its underlying
type (e.g. uint32_t field marked with 2 byte alignment),
the original type's alignment is still in effect for calculating
the alignment of the structure, and the padding.
* ffi.c (struct txr_ffi_type): New member oalign, for keeping
track of the type's original alignment, prior to adjustment.
(make_ffi_type_struct): For a named bitfield, take the oalign
value into account when determining the most strict member
alignment.
(ffi_type_compile): When marking a type as aligned, the
we remember the original alignment in atft->oalign.
* tests/017/bitfields.tl: New test case, struct s16.
* txr.1: Documented.
|
| | |
|
| |
|
|
|
| |
* txr.1: pack with alignment values greater than 1 doesn't
directly correspond to a single GNU C feature.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (make_ffi_type_struct): Add check for impossible condition.
The bits_alloc variable could only exceed bits_type (and thus
cause the room variable to have a nonsensical, large value)
if the bitfield allocation tried to continue allocating bits into
an aligned unit, whose alignment exceeds the size of the underlying
type. But in that case, tft->aligned would have to be true, and
so the offset would have been aligned prior to this code, rendering
bits_alloc zero.
* tests/017/bitfields.tl: New tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* struct.c (alloc_seen): Fix under-sized alloca and memset,
where the size being used is the number of elements,
but should be the number of bytes. This showed up as a
failure in the tests/012/oop-mi.tl on PPC64, uncovered by
a completely unrelated change in eval.c. The unrelated code
never executes during the failed test case; it just affects
some memory contents of the stack somehow. The under-allocated
array is used during structure type initialization to hold
a bitmask which helps to suppress the duplicate initiation of
base structures which are inherited multiple times (duplicate
bases). I think why this shows up on PPC64 is the big endian
order. The sizes involved are small. The test case has a couple
of duplicate bases only, but a full 64 bit mask needs to be
allocated for that since the mask is an array of ucnum. On
little endian, it doesn't matter than only a single byte is
allocated: that is enough for 8 duplicate bases, and because of
the byte order, those bases land into the byte that is allocated.
On big endian, the bases will be on the high part of the 64
bit word, and thus over in the area we didn't allocate.
To reproduce an issue on little endian, you need an inheritance
hierarchy with more duplicate bases.
|
| |
|
|
|
|
| |
* ffi.c (swap_get32, swap_put32, swap_get64, swap_put64):
correct implementations for operating on little-endian
words on big endian.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The bitfield allocation rules are wrong. Some of it is due
to the recent changes which are based on incorrect analysis,
but reverting things doesn't fix it.
The idea that we compare the current member's alignment
with the previous is wrong; it is not borne out by empirical
tests with gcc. So we do a straight revert of that.
In GNU C, an __attribute__((aligned (N))) attribute applied
to a bitfield member will perform the requested alignment if,
evidently, the bit field is already being placed into a new
byte. (If the bit field is about to be packed into an existing
byte, then there is a warning about the align attribute being
ignored). Because we don't have alignment as a member attribute,
but only as a type attribute, we must implement a flag which
indicates that a type has had align applied to it (even if
the alignment didn't change) so we can then honor this in the
right place in the bitfield allocation code.
* ffi.c (struct txr_ffi_type): New attribute flag, aligned.
(make_ffi_type_struct): Remove the prev_align variable and
all related logic. Consolidate all alignment into one place,
which is done before we allocate the bitfield or regular member.
We align if the new member isn't a bitfield, or even if it is
a bitfield if it has the aligned attribute, or if the bitfield
is changing endian compared to the previous member (our local
rule, not from GNU C).
(ffi_type_compile): The align and pack operators now set the
aligned attribute, except in the (pack 1 ...) case which
semantically denotes lack of alignment.
* tests/017/bitfields.tl: New file.
* txr.1: Documented.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is very nice: you can now declaratively match a structure
that uses bitfields, and comes from a different endian system.
* ffi.c (ffi_kind_t): Replace FFI_KIND_NUM type with FFI_KIND_INT,
FFI_KIND_UINT and FFI_KIND_FLO. Now using the tft->kind value,
we can distinguish integer and floating types, and determine
signedness.
(struct txr_ffi_type): New flag, bigendian, which is set to 1
for all big types that are big endian. This is not just the endian
types like be-int32, but natural types like int, if the underlying
platform is big endian.
(swap_get32, swap_put32, swap_get64, swap_put64): New static functions.
(ffi_generic_swap_fat_ubit_put, ffi_generic_swap_fat_sbit_put,
ffi_generic_swap_fat_ubit_get, ffi_generic_swap_fat_sbit_get,
ffi_generic_swap_ubit_put, ffi_generic_swap_sbit_put,
ffi_generic_swap_ubit_get, ffi_generic_swap_sbit_get): New
static functions.
(ffi_make_type_builtin): On big endian, set the bigendian flag on every
type. For the endian types, this will get adjusted as an additional
construction step.
(make_ffi_type_endian): New static function. Calls ffi_make_type_builtin,
and then initializes the bigendian flag with the given value.
(make_ffi_type_struct, make_ffi_type_union): Because bitfields can
have endiannness now, we can no longer refer to the machine's own
endianness in laying them out; we have to look at the mtft->bigendian
value. Furthermore, we need an additional rule: when a bitfield member
follows a member that has different endian, a new allocation unit
has to begin.
(ffi_type_compile): the sbit and ubit types must set the type to
FFI_KIND_INT and FFI_KIND_UINT now. For the big operator, we can
simplify the error checking: instead of exhaustively comparing the
type to all the supported integer types, we just now check whether
it is FFI_KIND_INT or FFI_KIND_UINT. Here, if we detect that an
endian bitfield has opposite byte order from the machine, then we
instantiate the bitfield with the ffi_generic_swap_* virtual
functions. These perform the required byte swapping accesses to
the bitfield storage cells, so then the bit field manipulation
code just works using the local integer representation (shifting
and masking). Of course, the shift amount depends on the endian;
and that is calculated at type creation time in make_ffi_type_struct.
(ffi_init_types): Replace FFI_KIND_NUM with the appropriate constant
for each affected type. In some cases, we have to calculate whether
to use the INT or UINT one, for the types whose signedness is
not specified. We switch all the endian types to new constructor
make_ffi_type_endian, passing the required value of the bigendian
flag.
* txr.1: Documented.
|
| |
|
|
|
|
|
| |
* ffi.c (make_ffi_type_struct, make_ffi_type_union): We must
take fat bitfields into account when calculating mtft->shift
on big endian. We must subtract from bits_llint not bits_int
for big endian.
|
| |
|
|
|
|
|
|
|
| |
* ffi.c (ffi_init_types): The be-float, be-double, le-float and
le-double types wrongly have integer indicated as their Lisp
type symbol; it should be float. This doesn't fix any actual bug;
the Lisp type is only important for structures/unions. The
conversion of be-double is driven by the behavior of the type's
virtual functions, which don't look at the Lisp type symbol.
|
| |
|
|
|
|
|
|
| |
This was developed together with what became the May 12 commit
1162a735b61c1c5086fb6055471ee35cc8ed62a4; I just forgot to
git add the file.
* tests/011/macros-4.tl
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_flex_struct_in): Function renamed to ffi_flex_array_len,
because its responsibility is determining the length of a flexible
array that is not null terminated. We don't pass in the structure's
type's descriptor any more, but the member descriptor.
(ffi_struct_in, ffi_struct_get): Follow rename and changed parameter
conventions.
* tests/017/flexstruct.tl: Added test case with nested flexible
structure.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_flex_struct_in): Check for the last member being
an array, and not null-terminated. We now check the character
conversion disposition of the array. If it has character
conversion, then we store the length right into the slot that
will become the string. In the no-conversion case, we assume
that if the member exists, it's a vector we can resize.
Otherwise we plant a vector of the required size.
(ffi_varray_put): Only call ffi_varray_dynsize if the Lisp
object is a vector. If the Lisp objecct is a number, then use
that as the size. Otherwise the size is zero.
* tests/017/flexstruct.tl: New file.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After "years of trouble-free operation" a bug in the UTF-8
decoder was found, which violates its property that any
sequence of bytes will decode to some kind of string, which
will encode to the original bytes.
When the UTF-8 data prematurely ends in the middle of a valid
character, the decoder just drops that data as if it didn't
exist. So for instance the two-byte sequence E6 BC should
decode to "\xDCE6\xDCBC", since it is a fragment of a three-byte
UTF-8 sequence. It actually decodes to the empty string.
* utf8.c (utf8_bfom_buffer): When the buffer is exhausted, if we are
not in the utf8_init state, it means we were in the middle of a
UTF-8 sequence. Walk the bytes from the backtrack point to the end
of the buffer and store them into the string as U+DCxx codes.
* tests/012/buf.tl: Tests added for this via buf-str, str-buf.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_transform_pack): Fix: return the original syntax in
the situation when no cases are recognized, rather than
the cdr of the syntax. When the struct/union syntax has no
members, return the original syntax to indicate no transformation
took place.
* txr.1: Document the feature that pack on a typedef name or struct
name with no members will do the alignment adjustment only, without
the syntactic transformation.
* tests/017/pack-align.tl: New file.
|
| |
|
|
|
|
|
| |
* eval.c (error_trace): Don't mention that --backtrace can be
used to enable backtraces. This is not only a nuisance, but
appears in packaged programs which do not support such an option,
making it highly misleading.
|
| |
|
|
|
| |
* txr.1: fix broken quoting which results in stray "
character appearing in rendered output.
|
| |
|
|
|
|
|
| |
* txr.1: The documentation for array and zarray wrongly states
that if the dimension is omitted, the type may not be used
as a structure member. In fact, it may be used as the last
member of a flexible structure.
|
| |
|
|
| |
* txr.1: two instances of "an incomplete arrays" in FFI doc.
|
| |
|
|
|
|
|
| |
* ffi.c (ffi_type_compile): Allow align to weaken (lower) alignment
if compatibility 275 or lower is requested.
* txr.1: Compat note added.
|
| |
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_pack_members): Static function removed.
(ffi_transform_pack): New static function.
(ffi_type_compile): Rely on ffi_transform_pack to recognize
and perform all necessary transformations.
Cosmetic issue: when a struct is compiled, and the individual
member types undergo transformation during member compilation,
the syntax for the struct is nevertheless the original one
with the untransformed members.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* stream.c (trim_path_seps): New function.
(stream_init): trim-path-seps intrinsic registered.
* stream.c (trim_path_seps): Declared.
* tests/018/path.tl: New tests.
* txr.1: Documented.
* stdlib/doc-syms.tl: Updated.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_type_copy, ffi_type_copy_new_ops): The cloned txr_ffi_type
structure must have a self member which points to the new cobj,
not the original one. Otherwise things are inconsistent. For instance
if the clone is being made for the purposes of adjusting alignment,
any operation which chases the self pointer will be accessing incorrect
attributes. One example of this is (alignof foo.bar) where if bar
is the clone of a type, this will incorrectly report the alignment of
the original from which bar was cloned, and the original alignment,
not the adjusted alignment is reported.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (pack_s): New symbol variable.
(ffi_type_compile): Handle new pack type operator together with
align. Allow a one-argument form of align and pack in which
the value is defaulted. The behavior of align changes: align
can only increase alignment now, never decrease, so for
instance (align 1 ...) does nothing. pack must be used to
decrease alignment. Furthermore, for certain argument types,
pack performs transformations of the type syntax, rather than
compiling the argument type and producing a variant of it with
altered alignment.
* txr.1: Documented.
* stdlib/doc-syms.tl: Updated.
|
| |
|
|
|
|
|
|
|
| |
* ffi.c (make_ffi_type_struct): When a bitfield is declared for
a type whose alignment is different from that of the previous
member, then we start the new bitfield on a fresh alignment.
As usual, if the previous member is a bitfield which left a
partially filled byte, we skip to the next byte and then do the
alignment. This seems to match GCC behavior.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In gcc, bitfields can be declared with overridden alignment;
for instance when the entire structure is packed.
We must modify the algorithm to make this work.
Firstly, we must must pretend that we are filling from the start
of the containing alignment unit, not size based unit. For
instance, if the cell is an 8 byte type, but the alignment is 1,
then we are just filling the current byte, not the current 8 byte
multiple. However, the number of bits left (room) is still
calculated from the size: so if the byte has been filled with 3 bits,
we have 4 left in that byte, and then 7 more bytes, and so 61 bits.
Secondly, the case of the zero-bit field is special: it does
perform an alignment calculation in terms of multiples of the
size, ignoring the declared alignment of the field.
Thirdly, when the bits do not fit into the existing room, the
offset does not move by the size of the bitfield, but by the
alignment unit. For instance, if we have 63 bits left in an 8
byte field that has alignment 1, and we want to place 64 bits
into another 8 byte field, then we will just move to the next
byte, not to the next 64 bit cell. Starting at the next byte,
we pretend we have a fres 64 bit cell, albeit misaligned.
* ffi.c (make_ffi_type_struct): Make the above adjustments to
the algorithm. Some common values size, alignment, mask)
are declared in a wider scope and adjustments made.
|
| |
|
|
|
|
|
|
| |
* ffi.c (make_ffi_type_struct, make_ffi_type_union): Initialize
most_align local variable to 1, so the lower bound of alignment
is that, rather than zero.
* tests/017/ffi-misc.tl: Tests added.
|
| |
|
|
|
|
|
| |
* ffi.c (make_ffi_type_struct, make_ffi_type_union): Do not fall
back from the fat (64 bit) bitfield case to the regular (32 bit)
case when the number of bits is less than 32. This is completely
wrong.
|
| |
|
|
|
|
|
|
|
| |
* ffi.c (ffi_type_compile): Show the original bitfield syntax
in the invalid size diagnostics. In the case of the (bit n type)
syntax, restrict the maximum bits to the width of the type,
not just to 32 or 64. I realize the 8 * tft->size calculation
can overflow for huge array types, but it makes no sense to use
them as bitfields.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (struct txr_ffi_type): Replace unsigned mask member
with a union m which holds unsigned mask and 64-bit fmask (fat
mask).
(ffi_sbit_put, ffi_sbit_get, ffi_ubit_put, ffi_ubit_get):
Refer to m.mask.
(ffi_fat_sbit_put, ffi_fat_sbit_get, ffi_fat_ubit_put,
ffi_fat_ubit_get): New static functions.
(ffi_generic_fat_sbit_put, ffi_generic_fat_sbit_get,
ffi_generic_fat_ubit_put, ffi_generic_fat_ubit_get):
Likewise.
(make_ffi_type_struct, make_ffi_type_union): Set up fat
mask for bitfields that are wider than int.
(ffi_type_compile): Refer to m.mask for the int and unsigned
int based bitfields declared with sbit and ubit that don't
mention a type. The bit operator now allows int64 and uint64
to be valid types for a bitfield. In this case, the "fat"
get and put functions are selected which use 64 bit operations.
Thus there is no efficiency impact on non-fat bitfields which
continue to use code with 32 bit operands.
(ffi_offsetof): Use the bitfield flag in the member's type
structure to detect bitfields, rather than the mask.
|
| |
|
|
|
|
| |
ffi.c (ffi_init_extra_types): Use longlong_t and ulonglong_t, subject
to HAVE_LONGLONG_T. If there is no intmax_t and no longlong_t,
don't define intmax-t and uintmax-t.
|
| |
|
|
|
|
| |
* ffi.c (ffi_generic_sbit_put, ffi_generic_sbit_get,
ffi_generic_ubit_put, ffi_generic_ubit_get): Replace int-sized
alloca with declared int object.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Because the varray behavior for undimensioned arrays was introduced
in dubious commit 7880c9b565ab438e1bf0250a967acdbf8d04cb42
in 2017, which used make_ffi_type_pointer to register the type,
claiming that the C representation is pointer (which was not true
in that commit, nor ever since).
As a result, though, undimensioned arrays received the alignment
of pointers, rather than deriving it from the element type.
Thus (array char) has 4 or 8 byte alignment whereas (array 4 char)
correctly has 1 byte alignment.
* ffi.c (ffi_type_compile): Use make_ffi_type_array for the two-element
array syntax, just like for the dimensioned case with three elements.
Then override some of the functions with the varray versions.
* tests/017/ffi-misc.tl: Fix the test case which exposed this.
In the type (struct flex (a char) (b (zarray char)), the array
b must be at offset 1. I didn't notice that the offset of 4
being confirmed by the test case was wrong, but this showed up
when running the test case on a platform with 8 byte pointers.
|
| |
|
|
|
|
|
| |
* tests/017/ffi-misc.tl: Fix incorrect test whose loop body
does not execute. A remaining issue here is why the diagnostics
about unbound functions and variables in the loop body get
swept under the rug.
|
| |
|
|
|
|
|
|
|
| |
* stdlib/copy-file.tl (copy-path-rec): When from-path has
a trailing slash, the starts-with test fails, and the operation
fails without doing anything. Let's just get rid of that test.
It's probably only there because of the ad-hoc path munging
that is calculating rel-path. Why don't we just use the rel-path
function for that.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* eval.c (eval_init): Fix up registration of split-str to
account for new parameter.
* lib.c (split_str_keep): Implement new optional count
argument.
(spl): Pass nil value to split_str_keep for new argument.
I'd like this function to benefit from this argument also,
but the design isn't settled.
(split_str): Pass nil argument to split_str_keep.
* lib.h (split_str_keep): Declaration updated.
* tests/015/split.tl: New tests.
* txr.1: Documented.
|
| |
|
|
| |
* txr.1: agreement between "body" and "consist".
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* ffi.c (ffi_char_array_get, ffi_zchar_array_get, ffi_wchar_array_get,
ffi_bchar_array_get): Rearrange so that we test for tft->null_term
first, and not nelem == 0. If nelem happens to be zero, but we are
supposed to decode a null-terminated string, we will do the wrong
thing and return the null string.
(ffi_varray_in): The body can't be conditional on vec being non-nil,
because then we do nothing if we don't have a Lisp object, which means
we skip the cases when we should decode a null-terminated array.
Now if vec is nil, we must guard against calling ffi_varray_dynsize.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* txr.c (help): Mention new options.
(do_compile_opt, do_in_package_opt): New static functions.
(txr_main): Implement options.
* Makefile (COMPILE_TL): Use the options instead of -e.
* txr.1: Document.
|
