Bug 783912 - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 44: ordinal not in range(128)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 44: ordi...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: rpmlint (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tom "spot" Callaway
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-01-23 03:35 EST by Kamil Páral
Modified: 2013-01-17 06:08 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-17 06:08:57 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Kamil Páral 2012-01-23 03:35:57 EST
Description of problem:
Download this package:
http://kojipkgs.fedoraproject.org/packages/autofs/5.0.6/5.fc16/i686/autofs-5.0.6-5.fc16.i686.rpm

If you run rpmlint on it with stdout printed out into the terminal, everything works fine. But if you redirect stdout to a file, rpmlint crashes:

$ rpmlint autofs-5.0.6-5.fc16.i686.rpm > out
Traceback (most recent call last):
  File "/usr/bin/rpmlint", line 379, in <module>
    main()
  File "/usr/bin/rpmlint", line 169, in main
    runChecks(pkg)
  File "/usr/bin/rpmlint", line 229, in runChecks
    check.check(pkg)
  File "/usr/share/rpmlint/TagsCheck.py", line 685, in check
    self.check_summary(pkg, lang, ignored_words)
  File "/usr/share/rpmlint/TagsCheck.py", line 890, in check_summary
    printWarning(pkg, 'summary-not-capitalized', lang, summary)
  File "/usr/share/rpmlint/Filter.py", line 38, in printWarning
    _print("W", pkg, reason, details)
  File "/usr/share/rpmlint/Filter.py", line 80, in _print
    __print(s)
  File "/usr/share/rpmlint/Filter.py", line 32, in __print
    print(s.encode(locale.getpreferredencoding(), "replace"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 44: ordinal not in range(128)


Locale is set fine:

$ locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=


I tried to add a printout before the offending line /usr/share/rpmlint/Filter.py:32, and locale.getpreferredencoding() returns 'UTF-8', so that should be fine.

Version-Release number of selected component (if applicable):
rpmlint-1.4-1.fc16.noarch

How reproducible:
always
Comment 1 Toshio Ernie Kuratomi 2012-01-23 12:26:13 EST
Here's the relevant line in the output:

autofs.i686: W: summary-not-capitalized tr autofs sunucu süreci

And here's an explanation of the problem:
http://packages.python.org/kitchen/unicode-frustrations.html#frustration-3-inconsistent-treatment-of-output

python uses the locale settings when it prints to a terminal but it ignores the locale settings when it prints to a file.

Let me see if I can come up with something to fix this... note that I probably can't fix it globally unless rpmlint upstream is willing to grow a kitchen dependency and use kitchen.text.converters.getwriter().
Comment 2 Toshio Ernie Kuratomi 2012-01-23 12:53:34 EST
Ah -- So this is actually simpler than the problem I was quoting.

What's really happening is that the function in Filter.py is only capable of handling unicode strings correctly.  When handed a byte str, it breaks.


if sys.stdout.isatty():
    def __print(s):
        print(s)
else:
    def __print(s):
        print(s.encode(locale.getpreferredencoding(), "replace"))

When we redirect to a file, we attempt to call the .encode() method of "s".  If s is a unicode string then this works fine.  If s is a byte str, then python first has to turn the byte str into a unicode string.  Then it can call the .encode() method on that to turn it back into a byte str.  It's when python is turning the byte str into a unicode string that python uses the ASCII codec and throws the exception.

Here's the minimally invasive fix:


if sys.stdout.isatty():
    def __print(s):
        print(s)
else:
    def __print(s):
        if isinstance(s, unicode):
            s = s.encode(locale.getpreferredencoding(), "replace")    
        print(s)
Comment 3 Toshio Ernie Kuratomi 2012-01-23 13:41:55 EST
Reported upstream and built for rawhide.

http://rpmlint.zarb.org/cgi-bin/trac.cgi/ticket/308

http://koji.fedoraproject.org/koji/taskinfo?taskID=3726180

spot and Ville, if this looks good to you feel free to push it to older Fedora releases as well.
Comment 4 Kamil Páral 2012-01-24 06:00:54 EST
I can confirm that the new koji build fixes the problem.
Comment 5 Ville Skyttä 2012-01-24 16:05:59 EST
Fix looks fine to me, applied upstream but I'll leave updating the Fedora rpmlint package to others.
Comment 6 Fedora End Of Life 2013-01-16 14:17:03 EST
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 7 Kamil Páral 2013-01-17 06:08:57 EST
Verified fixed in rpmlint-1.4-11.fc17.noarch.

Note You need to log in before you can comment on or make changes to this bug.