diff options
Diffstat (limited to 'txr.1')
| -rw-r--r-- | txr.1 | 23 |
1 files changed, 19 insertions, 4 deletions
@@ -410,13 +410,28 @@ On some platforms, wide characters may be restricted to 16 bits, so that can only work with characters in the BMP (Basic Multilingual Plane) subset of Unicode. +.B txr +does not use the localization features of the system library; +its handling of extended characters is not affected by environment variables +like LANG and L_CTYPE. The program reads and writes only the UTF-8 encoding. + If .B txr encounters an invalid bytes in the UTF-8 input, what happens depends on the -context in which this occurs. Invalid bytes in a query are reported as errors. -Invalid bytes in data are currently treated in an unspecified way. In -the future, invalid bytes in data will be mapped to the Unicode codes -U+DC00 through U+DCFF. +context in which this occurs. In a query, comments are read without regard +for encoding, so invalid encoding bytes are not detected. A comment is +simply a sequence of bytes terminated by a newline. Invalid +encoding bytes in signficant query text are diagnosed as syntax errors. +When the scanner is faced with input that isn't a valid multibyte character, it +issues an error message, skips one byte, and resumes scanning. + +Invalid bytes in data are treated as follows: when an invalid byte is +encountered in the middle of a multibyte character, or if the input +ends in the middle of a multibyte character, the UTF-8 decoder returns +to the starting byte of the ill-formed multibyte character, and decodes just +that byte, by mapping it to the Unicode character range U+DC00 through U+DCFF. +The decoding resumes at the following character, expecting that byte to be the +start of another multibyte character. .SS Variables |