Red Hat Bugzilla – Full Text Bug Listing
|Summary:||unexpand: non-ASCII data corruption|
|Product:||[Fedora] Fedora||Reporter:||Ville Skyttä <scop>|
|Component:||coreutils||Assignee:||Tim Waugh <twaugh>|
|Status:||CLOSED NOTABUG||QA Contact:|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2005-04-21 08:29:26 EDT||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
Description Ville Skyttä 2005-04-19 14:45:40 EDT
In coreutils-5.2.1-44 (also earlier versions): when fed eg. ISO-8859-1 text on a UTF-8 system, unexpand corrupts non-ASCII chars. $ echo $LANG en_US.UTF-8 $ echo SkyttÃ¤ \ | iconv -f utf-8 -t iso-8859-1 \ | unexpand \ | iconv -f iso-8859-1 -t utf-8 SkyttS expand(1) does not seem to suffer from this.
Comment 1 Tim Waugh 2005-04-20 04:25:33 EDT
Why are you converting charset in the first place? Doesn't expand work in UTF-8? If so, that's the real bug.
Comment 2 Ville Skyttä 2005-04-20 12:36:08 EDT
Created attachment 113419 [details] Test file The charset conversion here is just for the purpose of providing a oneliner that easily reproduces the bug. The full practical reproducer is: - I have some source code that contains non-ASCII ISO-8859-1 accented characters. ISO-8859-1 is fine for these files, and must not be changed. - I want to convert spaces to tabs in those files using unexpand. - When I run unexpand on these files, the spaces get converted to tabs as expected, but as a side effect, all the ISO-8859-1 accented chars turn into gibberish. Please also note that it's unexpand(1) that has the bug. Not expand(1), it seems to be doing the right thing. Try it out with the attached file: gunzip 155382.txt.gz unexpand 155382.txt > test.out # compare 155382.txt and test.out, they should be equivalent, but are not
Comment 3 Tim Waugh 2005-04-21 06:25:50 EDT
Works fine here, when the locale is set correctly. Your test case is incorrect, and should be using 'LC_CTYPE=en_US unexpand' as in: $ echo $LANG en_US.UTF-8 $ echo SkyttÃ¤ \ | iconv -f utf-8 -t iso-8859-1 \ | LC_ALL=en_US unexpand \ | iconv -f iso-8859-1 -t utf-8 Otherwise unexpand will be expecting UTF-8 input.
Comment 4 Ville Skyttä 2005-04-21 08:20:02 EDT
Well, I still think this is a bug because unlike unexpand, expand is doing the right thing, not changing anything but the spaces or tabs without any LC_* environment fiddling in this case. Reopening based on that. If you disagree, I'd like to hear why expand and unexpand are expected to behave differently regarding these accented characters with the same environment settings and same input.
Comment 5 Tim Waugh 2005-04-21 08:29:26 EDT
No, you *must* set the encoding correctly, otherwise all bets are off. You can't expect expand or unexpand to be able to do their job if they cannot make sense of the input. Garbage in, garbage out.