Bug 103214 - 'man rand' gives 'iconv: illegal input sequence at position 1789'
'man rand' gives 'iconv: illegal input sequence at position 1789'
Status: CLOSED RAWHIDE
Product: Red Hat Linux
Classification: Retired
Component: man-pages (Show other bugs)
9
All Linux
medium Severity medium
: ---
: ---
Assigned To: Eido Inoue
Ben Levenson
:
: 103122 104542 104570 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-08-27 15:18 EDT by P Fudd
Modified: 2007-04-18 12:57 EDT (History)
6 users (show)

See Also:
Fixed In Version: 1.60-3
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-09-17 19:18:02 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description P Fudd 2003-08-27 15:18:37 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1) Gecko/20030225

Description of problem:
The 'iconv' program (part of glibc-common) is croaking on the accented character
in the name 'Nicolas Lichtmaier' in the comments of the man page for rand.3 and
srand.3.

I can read the man page on RH8.0 but not RH9.

Version-Release number of selected component (if applicable):
glibc-common-2.3.2-27.9

How reproducible:
Always

Steps to Reproduce:
1.man rand
2.
3.
    

Actual Results:  iconv: illegal input sequence at position 1789

and the man page is not displayed.

Expected Results:  man page should have been displayed.

Additional info:

setenv LANG C
man rand    - fail
setenv LANG en_US
man rand    - fail
(I haven't tried any other language besides these two)
Comment 1 Jakub Jelinek 2003-08-27 15:27:06 EDT
That's a bug in the man page. It should be UTF-8 encoded, not ISO-8859-1 encoded.
Then it will work with any locale with charset which has small a with acute letter
(like en_US, de_DE, cs_CZ, en_US.UTF-8), but not in others (like C, ru_RU, etc.).
Given that it is in a comment, I'd say the acute should go.
Comment 2 P Fudd 2003-08-27 18:00:05 EDT
Here are further man(3) pages that generate the same error message, cutting off
the man page at the error:

DBD::mysql iconv: illegal input sequence at position 42954
DBI::Shell iconv: illegal input sequence at position 9095
QDial iconv: illegal input sequence at position 12877
Unicode::Collate iconv: illegal input sequence at position 19426
encrypt iconv: illegal input sequence at position 115
fclose iconv: illegal input sequence at position 2275
fcloseall iconv: illegal input sequence at position 2279
fflush iconv: illegal input sequence at position 2275
fontconfig iconv: illegal input sequence at position 203
fprintf iconv: illegal input sequence at position 13854
lockf iconv: illegal input sequence at position 115
logwtmp iconv: illegal input sequence at position 115
qdial iconv: illegal input sequence at position 12877
rand iconv: illegal input sequence at position 1787
setkey iconv: illegal input sequence at position 115
snprintf iconv: illegal input sequence at position 13854
sprintf iconv: illegal input sequence at position 13854
srand iconv: illegal input sequence at position 1787
strtok iconv: illegal input sequence at position 1281
strtok_r iconv: illegal input sequence at position 1281
tolower iconv: illegal input sequence at position 1309
toupper iconv: illegal input sequence at position 1309
updwtmp iconv: illegal input sequence at position 115
vfprintf iconv: illegal input sequence at position 13854
vprintf iconv: illegal input sequence at position 13854
vsnprintf iconv: illegal input sequence at position 13854
vsprintf iconv: illegal input sequence at position 13854

There were other kinds of errors, but these were all the iconv errors.  To
reproduce, type 'man 3 <keyword>'.
Comment 3 Need Real Name 2003-09-02 21:11:50 EDT
There are lots of man page problems like this in at least sections 2 and 3
in RH9.  Horking up the man pages for programmers has become a chronic
annoyance for me with Red Hat releases.

All the pages I've gone back and retested work okay if
LANG=en_US.UTF-8.  Sections 2 and 3 are for C programmers, so they
must work if LANG=C (or LANG=POSIX, or LANG=XOPEN, or with LANG unset).

Suggestions:

(1) recheck all the man pages as part of standard testing

Just write a script that unsets LANG and runs a "man SECT" on each
file in /usr/share/man/manSECT/, looking for "iconv" in stderr.

Having "man" return a non-zero exit status if it fails would make
this testing even easier. :-)

(2) Educate the benighted managers in marketing and product test.

LANG=POSIX (or LANG=C or ...) is supposed to be the default, at least for
testing.  These (LANG=POSIX, etc.) are the *only* settings that have a
well-defined and portable behavior.  They're what programmers count on.

Even if nothing else works, these need to
to let programmers work.  The other stuff's fine, but it isn't required
by the basic standards.
Comment 4 Eido Inoue 2003-09-03 12:18:53 EDT
comment 3:

> LANG=POSIX (or LANG=C or ...) is supposed to be the default, at least for
> testing.  These (LANG=POSIX, etc.) are the *only* settings that have a
> well-defined and portable behavior.  They're what programmers count on.

No it's not. Even if we play devil's advocate and assume that the whole world
speaks English and is happy to get English pages as the default, setting the
locale to POSIX/C will still fail for some English man pages that have accent
and other characters from ISO-8859-1 that aren't in ASCII (usually the author's
names, but not always). glibc interprets the POSIX/C locale as being a seven bit
(ASCII only) locale, and certain functions/operations will reject strings that
have the eighth bit set. Previous versions of glibc would let chars>127 slide in
C/POSIX, but newer versions are more strict.

Thus if we made C/POSIX the default, some applications (I'm talking about
complicated apps, not shell utilities that don't differentiate between text and
binary data) would lose functionality (display/accept accented characters,
non-traditional punctuation, etc) even for users of English.

The man pages will be filtered so that non-ASCII is either transliterated or
converted to ASCII. Given that the behavior of man is to fall back to the
"POSIX"  locale version of a man page if a man page for the desired language is
not available, the POSIX manpages should probably be encoded using a charset
that is a subset of all charset/encodings in use (ASCII); while encoding in
UTF-8 is the right thing to do as Jakub mentions, users of non-English terminals
that can't display the entire Unicode charset will still want to see POSIX pages
without having to change the charset/enc/font of their terminal.
Comment 5 Eido Inoue 2003-09-03 12:35:50 EDT
comment 4:

> the POSIX manpages should probably be encoded using a charset
> that is a subset of all charset/encodings in use (ASCII)

By "encoded using ... ASCII", I mean that any manpage that is currently in the
"POSIX" locale that uses non-ASCII should:

1) be copied and converted from ISO-8859-1 to UTF-8 into a en_US locale-specific
man page area, much like how all the other non-English man pages are currently done

and

2) be "transliterated" for the POSIX locale. That is, convert the "acute a"s and
the umlauts into plain ASCII "a" and "u" respectively. Yes, I know that an
umlaut and a "u" are entirely different things and this is going to upset some
people who will get their name mangled, but...

a) Users prefer man pages in their native language first. But if they can't have
that they'd rather have an English man page than no man page at all. A realistic
example would be a Japanese user: if the man page isn't available in Japanese,
they'd rather have the ASCII/POSIX version as the fallback than no page
whatsoever. Currently, because the "fallback" pages are not in ASCII, they
sometimes fail with an iconv error because the "a acutes" and the "umlauts"
aren't representable in EUC-JP. (Later versions of RHL will use Unicode for CJK
just like the Western European languages)

b) Users can still see the "pristine source" man pages by setting their LANG to
en_US, which is a Unicode locale in RHL 8 and beyond.
Comment 6 Need Real Name 2003-09-03 20:15:40 EDT
Parts of the last two comments are right on the money.  Parts
confuse "language" with "locale".

Here, I think, is the scoop, or at least a sketch thereof:

LANG=POSIX does not "assume the whole world speaks English" any more than the
marking "Andante" on a page of music assumes the whole world speaks Italian.
Similarly, in C, "if" and "else" are keywords, not English -- though
their syntax and semantics were probably influenced by Dennis Ritchie's native
language. :-)

The POSIX (and C, and XOPEN) standards specify the way one particular locale
behaves.

Just one, actually.

Functions and utilities defined in those standards are required to behave in a
specific way when LANG (and LC_*) are set in to the values C, POSIX, XOPEN, or
unset.  These are all, approximately, synonyms -- and with good reason.

Unless things have changed recently, all other locales remain undefined and
implementation-dependent.  When $LANG is set to en_US.UTF-8, RedHat may,
legitimately, have a different collation sequence from, say, Slackware,
Microsoft, or IBM.  I happen to think this isn't good, but it's still true.
(Used to be, at least, that even the string describing the locale was
implementation-defined; however, language_COUNTRY.CHARSET seems to be a
de-facto standard.)

Standardizing at least one behavior lets programmers write code whose behavior
they can predict.  (Indeed, if need be, the programmer can force that behavior,
setting one of those locales within the program.)

The choice of how to behave in this locale was made based on historical
precedent: traditional Unix/C implementations.

Yes, this assumes we're not using EUC-J.  Yes it assumes ISO-646IRV.  No, it's
not "everyone speaks English" chauvinism: it also assumes we're not running in
EBCDIC or CDC-ASCII or using APL terminals.  It wasn't a value judgement, it
was a way to maximize portability of existing Unix code and programmers.

Anyway, once LANG is unset, "ls" is guaranteed to list files in ASCII collating
sequence, and "ln -s [a-z]* subdir" will link files that start with lowercase
letters, but not "Makefile", into the subdirectory.

If a "complex app" is in one of these standards, it must behave conventionally
and successfully with LANG unset.  If a programmer works in one of the standard
locales, his programs should behave in standard ways.

If you create, say, a man page with unusual data, no matter what the contents
are -- accented characters, binary, Swahili -- then folks who don't understand
those data will just have to uiopreqwjopdfr 389y493tqh, but the utilities that
process those pages should still run, and predictably, when LANG is unset.

And, because even English words have diacriticals, though rarely, there should
legitimately be provision for man pages in "English," which are picked up when
the local is en_*, just like man pages in French, or German, or Arabic.  The
ones that have only ASCII may be links to the default pages.  What man, or col,
(or nroff, or ...) do on these is, I think, undefined, though I don't have my
dot-2 standard at hand.

But "man", when LANG is unset should work, no matter what, and must be tested
in that locale.  Even if no other locale works.
Ditto for all the other standard utilities.

That a C programmer should get only an error message from "man fclose",
when LANG=C, seems like, oh, a bug. :-)
Comment 7 Need Real Name 2003-09-04 11:00:41 EDT
Parts of the last two comments are right on the money.  Parts
confuse "language" with "locale".

Here, I think, is the scoop, or at least a sketch thereof:

LANG=POSIX does not "assume the whole world speaks English" any more than the
marking "Andante" on a page of music assumes the whole world speaks Italian.
Similarly, in C, "if" and "else" are keywords, not English -- though
their syntax and semantics were probably influenced by Dennis Ritchie's native
language. :-)

The POSIX (and C, and XOPEN) standards specify the way one particular locale
behaves.

Just one, actually.

Functions and utilities defined in those standards are required to behave in a
specific way when LANG (and LC_*) are set in to the values C, POSIX, XOPEN, or
unset.  These are all, approximately, synonyms -- and with good reason.

Unless things have changed recently, all other locales remain undefined and
implementation-dependent.  When $LANG is set to en_US.UTF-8, RedHat may,
legitimately, have a different collation sequence from, say, Slackware,
Microsoft, or IBM.  I happen to think this isn't good, but it's still true.
(Used to be, at least, that even the string describing the locale was
implementation-defined; however, language_COUNTRY.CHARSET seems to be a
de-facto standard.)

Standardizing at least one behavior lets programmers write code whose behavior
they can predict.  (Indeed, if need be, the programmer can force that behavior,
setting one of those locales within the program.)

The choice of how to behave in this locale was made based on historical
precedent: traditional Unix/C implementations.

Yes, this assumes we're not using EUC-J.  Yes it assumes ISO-646IRV.  No, it's
not "everyone speaks English" chauvinism: it also assumes we're not running in
EBCDIC or CDC-ASCII or using APL terminals.  It wasn't a value judgement, it
was a way to maximize portability of existing Unix code and programmers.

Anyway, once LANG is unset, "ls" is guaranteed to list files in ASCII collating
sequence, and "ln -s [a-z]* subdir" will link files that start with lowercase
letters, but not "Makefile", into the subdirectory.

If a "complex app" is in one of these standards, it must behave conventionally
and successfully with LANG unset.  If a programmer works in one of the standard
locales, his programs should behave in standard ways.

If you create, say, a man page with unusual data, no matter what the contents
are -- accented characters, binary, Swahili -- then folks who don't understand
those data will just have to uiopreqwjopdfr 389y493tqh, but the utilities that
process those pages should still run, and predictably, when LANG is unset.

And, because even English words have diacriticals, though rarely, there should
legitimately be provision for man pages in "English," which are picked up when
the local is en_*, just like man pages in French, or German, or Arabic.  The
ones that have only ASCII may be links to the default pages.  What man, or col,
(or nroff, or ...) do on these is, I think, undefined, though I don't have my
dot-2 standard at hand.

But "man", when LANG is unset should work, no matter what, and must be tested
in that locale.  Even if no other locale works.
Ditto for all the other standard utilities.

That a C programmer should get only an error message from "man fclose",
when LANG=C, seems like, oh, a bug. :-)
Comment 8 Eido Inoue 2003-09-04 13:03:03 EDT
> LANG=POSIX does not "assume the whole world speaks English"
True, but it does mean "everything except for ASCII is undefined" with respect
to text data. Thus it is reasonable for non-trivial text handlers (such as
groff, used to man, to balk and reject text data that it can't tell if it's
non-spacing, part of an escape sequence, one part of a multi-byte character, etc.)

> Unless things have changed recently, all other locales remain undefined and
> implementation-dependent.  When $LANG is set to en_US.UTF-8, RedHat may,
> legitimately, have a different collation sequence from, say, Slackware,

Things HAVE changed recently. LSB-i (formerly the LI18NUX spec) does exercise
distros through conformance tests to make sure the collation behavior is the
same for certain locales. Those distros that have a collation sequence that is
different from what the LSB-i states for a given locale will flunk those I18N
standards conformance tests.

Current versions of Red Hat Linux have passed level 1 LSB-i/LI18NUX conformance.

> If you create, say, a man page with unusual data, [snip], but the utilities
> that process those pages should still run, and predictably, when LANG is unset.

> But "man", when LANG is unset should work, no matter what, and must be tested
> in that locale.  Even if no other locale works.
> Ditto for all the other standard utilities.

You're confusing the data (man-pages) with the program(s) (man and groff).
There's no standard that specifies that groff/man must work with any arbitrary
data. Man/groff will reject data it can't understand within a certain context;
especially because man pages are non-trivial marked up pages.

I do not agree with the statement "Ditto for all the other standard utilities".
How utilities deal with unknown/invalid data is either specified in specs or
undefined. There is no blanket requirement that when LANG is unset "standard
utilities" should accept any arbitrary 8-bit data (which is what I interpret you
mean with the phrase "should work, no matter what").

When in the POSIX locale ("LANG is unset"), I18N sensitive glibc functions, such
as those in <ctype.h>, will fail/report-as-undefined for non-ASCII data; for
example, iscntrl() will fail to identify the C1 control characters from 0x80 to
0x9F, functions to determine the width of a character, crucial for groff
word-wrapping, will not work.

> That a C programmer should get only an error message from "man fclose",
> when LANG=C, seems like, oh, a bug. :-)

I'm not claiming that the current state is correct; which is why this bug is
assigned and not closed. As mentioned above the man-page package will be
corrected so that pages with non-ASCII will be ASCII-ized for the sake of
graceful fallback, and will also be UTF-ed and moved into en for all pages that
contain characters > 0x7E.
Comment 9 Need Real Name 2003-09-04 16:58:48 EDT
Believe it or not, I now think we're in complete agreement.
At this point, I'm convinced
everywhere that I (or you) thought we disagreed was
misunderstanding what the other person was saying.

The problem's in the data, and it's a bug.
Comment 10 Eido Inoue 2003-09-04 23:02:03 EDT
*** Bug 103122 has been marked as a duplicate of this bug. ***
Comment 11 Eido Inoue 2003-09-17 11:23:27 EDT
*** Bug 104542 has been marked as a duplicate of this bug. ***
Comment 12 Eido Inoue 2003-09-17 11:24:19 EDT
*** Bug 104570 has been marked as a duplicate of this bug. ***
Comment 13 Need Real Name 2003-09-17 11:35:59 EDT
Even if all Red Hat's man pages are UTF-8, what happens when I install a
third-party man page?  Man should be able to recover from an illegal UTF-8
sequence, not treat is as a fatal error.
Comment 14 Eido Inoue 2003-09-17 19:18:02 EDT
ascii transliteration will be done according to glibc localedata with respect to
LANG=en
Comment 15 Eido Inoue 2003-09-24 10:38:48 EDT
*** Bug 104991 has been marked as a duplicate of this bug. ***
Comment 16 Bojan Smojver 2003-10-23 00:59:24 EDT
Just a note that this is still a problem in Fedora Core 1, Test 3. RPM
man-pages-1.60-4 (I'm guessing this is the same as Rawhide).
Comment 17 Bojan Smojver 2003-10-23 01:02:07 EDT
Sorry, forgot to mention that LANG is set to en_AU. with LANG=C, it goes away...
Comment 18 Eido Inoue 2003-10-23 12:08:19 EDT
> Sorry, forgot to mention that LANG is set to en_AU. with LANG=C, it goes away...

this is not a man-page problem... for some reason the iconv logic is getting
confused with respect to the locale "en_AU".

Are you running in ISO-8859-1 or the UTF-8 character set? (if you're running RHL
8.0 and beyond, en_AU should be UTF-8 by default)

Note You need to log in before you can comment on or make changes to this bug.