Bug 155382

Summary: unexpand: non-ASCII data corruption
Product: [Fedora] Fedora Reporter: Ville Skyttä <scop>
Component: coreutilsAssignee: Tim Waugh <twaugh>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: rawhide   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-04-21 08:29:26 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Description Flags
Test file none

Description Ville Skyttä 2005-04-19 14:45:40 EDT
In coreutils-5.2.1-44 (also earlier versions): when fed eg. ISO-8859-1 text on a
UTF-8 system, unexpand corrupts non-ASCII chars.

  $ echo $LANG
  $ echo Skyttä \
  | iconv -f utf-8 -t iso-8859-1 \
  | unexpand \
  | iconv -f iso-8859-1 -t utf-8

expand(1) does not seem to suffer from this.
Comment 1 Tim Waugh 2005-04-20 04:25:33 EDT
Why are you converting charset in the first place?  Doesn't expand work in
UTF-8?  If so, that's the real bug.
Comment 2 Ville Skyttä 2005-04-20 12:36:08 EDT
Created attachment 113419 [details]
Test file

The charset conversion here is just for the purpose of providing a oneliner
that easily reproduces the bug.

The full practical reproducer is:

- I have some source code that contains non-ASCII ISO-8859-1 accented 
  characters.  ISO-8859-1 is fine for these files, and must not be changed.
- I want to convert spaces to tabs in those files using unexpand.
- When I run unexpand on these files, the spaces get converted to tabs as 
  expected, but as a side effect, all the ISO-8859-1 accented chars turn into

Please also note that it's unexpand(1) that has the bug.  Not expand(1), it
seems to be doing the right thing.

Try it out with the attached file:

  gunzip 155382.txt.gz
  unexpand 155382.txt > test.out
  # compare 155382.txt and test.out, they should be equivalent, but are not
Comment 3 Tim Waugh 2005-04-21 06:25:50 EDT
Works fine here, when the locale is set correctly.  Your test case is incorrect,
and should be using 'LC_CTYPE=en_US unexpand' as in:

$ echo $LANG
  $ echo Skyttä \
  | iconv -f utf-8 -t iso-8859-1 \
  | LC_ALL=en_US unexpand \
  | iconv -f iso-8859-1 -t utf-8

Otherwise unexpand will be expecting UTF-8 input.
Comment 4 Ville Skyttä 2005-04-21 08:20:02 EDT
Well, I still think this is a bug because unlike unexpand, expand is doing the
right thing, not changing anything but the spaces or tabs without any LC_*
environment fiddling in this case.

Reopening based on that.  If you disagree, I'd like to hear why expand and
unexpand are expected to behave differently regarding these accented characters
with the same environment settings and same input.
Comment 5 Tim Waugh 2005-04-21 08:29:26 EDT
No, you *must* set the encoding correctly, otherwise all bets are off.

You can't expect expand or unexpand to be able to do their job if they cannot
make sense of the input.  Garbage in, garbage out.