Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 155382 - unexpand: non-ASCII data corruption
unexpand: non-ASCII data corruption
Product: Fedora
Classification: Fedora
Component: coreutils (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Tim Waugh
Depends On:
  Show dependency treegraph
Reported: 2005-04-19 14:45 EDT by Ville Skyttä
Modified: 2007-11-30 17:11 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2005-04-21 08:29:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Test file (38 bytes, application/octet-stream)
2005-04-20 12:36 EDT, Ville Skyttä
no flags Details

  None (edit)
Description Ville Skyttä 2005-04-19 14:45:40 EDT
In coreutils-5.2.1-44 (also earlier versions): when fed eg. ISO-8859-1 text on a
UTF-8 system, unexpand corrupts non-ASCII chars.

  $ echo $LANG
  $ echo Skyttä \
  | iconv -f utf-8 -t iso-8859-1 \
  | unexpand \
  | iconv -f iso-8859-1 -t utf-8

expand(1) does not seem to suffer from this.
Comment 1 Tim Waugh 2005-04-20 04:25:33 EDT
Why are you converting charset in the first place?  Doesn't expand work in
UTF-8?  If so, that's the real bug.
Comment 2 Ville Skyttä 2005-04-20 12:36:08 EDT
Created attachment 113419 [details]
Test file

The charset conversion here is just for the purpose of providing a oneliner
that easily reproduces the bug.

The full practical reproducer is:

- I have some source code that contains non-ASCII ISO-8859-1 accented 
  characters.  ISO-8859-1 is fine for these files, and must not be changed.
- I want to convert spaces to tabs in those files using unexpand.
- When I run unexpand on these files, the spaces get converted to tabs as 
  expected, but as a side effect, all the ISO-8859-1 accented chars turn into

Please also note that it's unexpand(1) that has the bug.  Not expand(1), it
seems to be doing the right thing.

Try it out with the attached file:

  gunzip 155382.txt.gz
  unexpand 155382.txt > test.out
  # compare 155382.txt and test.out, they should be equivalent, but are not
Comment 3 Tim Waugh 2005-04-21 06:25:50 EDT
Works fine here, when the locale is set correctly.  Your test case is incorrect,
and should be using 'LC_CTYPE=en_US unexpand' as in:

$ echo $LANG
  $ echo Skyttä \
  | iconv -f utf-8 -t iso-8859-1 \
  | LC_ALL=en_US unexpand \
  | iconv -f iso-8859-1 -t utf-8

Otherwise unexpand will be expecting UTF-8 input.
Comment 4 Ville Skyttä 2005-04-21 08:20:02 EDT
Well, I still think this is a bug because unlike unexpand, expand is doing the
right thing, not changing anything but the spaces or tabs without any LC_*
environment fiddling in this case.

Reopening based on that.  If you disagree, I'd like to hear why expand and
unexpand are expected to behave differently regarding these accented characters
with the same environment settings and same input.
Comment 5 Tim Waugh 2005-04-21 08:29:26 EDT
No, you *must* set the encoding correctly, otherwise all bets are off.

You can't expect expand or unexpand to be able to do their job if they cannot
make sense of the input.  Garbage in, garbage out.

Note You need to log in before you can comment on or make changes to this bug.