Hide Forgot
In coreutils-5.2.1-44 (also earlier versions): when fed eg. ISO-8859-1 text on a UTF-8 system, unexpand corrupts non-ASCII chars. $ echo $LANG en_US.UTF-8 $ echo Skyttä \ | iconv -f utf-8 -t iso-8859-1 \ | unexpand \ | iconv -f iso-8859-1 -t utf-8 SkyttS expand(1) does not seem to suffer from this.
Why are you converting charset in the first place? Doesn't expand work in UTF-8? If so, that's the real bug.
Created attachment 113419 [details] Test file The charset conversion here is just for the purpose of providing a oneliner that easily reproduces the bug. The full practical reproducer is: - I have some source code that contains non-ASCII ISO-8859-1 accented characters. ISO-8859-1 is fine for these files, and must not be changed. - I want to convert spaces to tabs in those files using unexpand. - When I run unexpand on these files, the spaces get converted to tabs as expected, but as a side effect, all the ISO-8859-1 accented chars turn into gibberish. Please also note that it's unexpand(1) that has the bug. Not expand(1), it seems to be doing the right thing. Try it out with the attached file: gunzip 155382.txt.gz unexpand 155382.txt > test.out # compare 155382.txt and test.out, they should be equivalent, but are not
Works fine here, when the locale is set correctly. Your test case is incorrect, and should be using 'LC_CTYPE=en_US unexpand' as in: $ echo $LANG en_US.UTF-8 $ echo Skyttä \ | iconv -f utf-8 -t iso-8859-1 \ | LC_ALL=en_US unexpand \ | iconv -f iso-8859-1 -t utf-8 Otherwise unexpand will be expecting UTF-8 input.
Well, I still think this is a bug because unlike unexpand, expand is doing the right thing, not changing anything but the spaces or tabs without any LC_* environment fiddling in this case. Reopening based on that. If you disagree, I'd like to hear why expand and unexpand are expected to behave differently regarding these accented characters with the same environment settings and same input.
No, you *must* set the encoding correctly, otherwise all bets are off. You can't expect expand or unexpand to be able to do their job if they cannot make sense of the input. Garbage in, garbage out.