Bug 963810 - Could not get update details: Error Type: <type 'exceptions.UnicodeDecodeError'>
Could not get update details: Error Type: <type 'exceptions.UnicodeDecodeError'>
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: PackageKit (Show other bugs)
19
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Richard Hughes
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F19Beta/F19BetaBlocker 964352
  Show dependency treegraph
 
Reported: 2013-05-16 11:17 EDT by Michael Schwendt
Modified: 2013-05-21 21:59 EDT (History)
9 users (show)

See Also:
Fixed In Version: PackageKit-0.8.9-1.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-05-21 21:59:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
console output of gpk-update-viewer -v (26.18 KB, text/plain)
2013-05-16 11:17 EDT, Michael Schwendt
no flags Details
gpk-u-v output on F18 (16.24 KB, text/plain)
2013-05-17 10:15 EDT, Kamil Páral
no flags Details
output of pkcon (66.53 KB, text/plain)
2013-05-19 19:16 EDT, Reartes Guillermo
no flags Details
test patch (9.40 KB, patch)
2013-05-20 07:27 EDT, Richard Hughes
no flags Details | Diff

  None (edit)
Description Michael Schwendt 2013-05-16 11:17:32 EDT
Created attachment 748898 [details]
console output of gpk-update-viewer -v

$ rpm -qf $(which gpk-update-viewer)
gnome-packagekit-3.8.1-1.fc19.x86_64

Reproducibility: currently always

An error dialog is displayed, but there is no hint about which package (perhaps a %changelog entry) it refers to. Verbose output attached.


Could not get update details

Failed to process request

Error Type: <type 'exceptions.UnicodeDecodeError'>
Error Value: 'ascii' codec can't decode byte 0xc2 in position 49: ordinal not in range(128)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3590, in <module>
    main()
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3587, in main
    backend.dispatcher(sys.argv[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 722, in dispatcher
    self.dispatch_command(args[0], args[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 582, in dispatch_command
    self.get_details(package_ids)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2546, in get_details
    self._show_details_pkg(pkg)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2572, in _show_details_pkg
    self.details(package_id, license, group, desc, url, size)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 303, in details
    PackageKitBaseBackend.details(self, package_id, package_license, group, desc, url, bytes)
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 239, in details
    print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, _to_utf8(group), _to_utf8(desc), url, bytes), file=sys.stdout)
Comment 1 Richard Hughes 2013-05-17 07:48:56 EDT
Hmm, I can't reproduce. What's the output of "echo $LANG" on your machine (logged in as your user)?
Comment 2 Michael Schwendt 2013-05-17 09:36:07 EDT
It trips over the %description of the policycoreutils package, which uses a 0xc2 0xae Registered Trade Mark symbol character. In packagekit/backend.py, line 239, in details that is the "desc" variable.

>>> s = 'Linux® kernel'
>>> print isinstance(s,str)
True
>>> print isinstance(s,unicode)
False
>>> print s.encode('utf-8',errors='replace')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5: ordinal not in range(128)

$ echo $LANG
en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Comment 3 Richard Hughes 2013-05-17 10:09:02 EDT
Brilliant, thanks for narrowing it down. I don't think s.encode is what's tripping up PK tho, I think is a more reliable testcase:

---
#!/usr/bin/python2
# -*- coding: utf-8 -*-

def _to_utf8(txt, errors='replace'):
    if isinstance(txt, str):
        return txt
    if isinstance(txt, unicode):
        return txt.encode('utf-8', errors=errors)
    return str(txt)

s = 'Linux® kernel'
print isinstance(s,str)
print isinstance(s,unicode)
print _to_utf8(s)
---

This prints for me:

True
False
Linux® kernel

Just to check the obvious, what version of PackageKit is this? Thanks.
Comment 4 Kamil Páral 2013-05-17 10:15:16 EDT
Created attachment 749354 [details]
gpk-u-v output on F18

I see the same traceback on F18:
PackageKit-0.8.8-1.fc18.x86_64
gnome-packagekit-3.6.2-1.fc18.x86_64

I have cs_CZ.UTF-8 locale, but it also fails with en_US. No policycoreutils update is available. I don't see any valuable info in pkmon or on stdout.
Comment 5 Kamil Páral 2013-05-17 10:26:23 EDT
When I try to update anyway, it fails:

16:21:37	GnomePackageKit     status download
16:23:02	GnomePackageKit     status sig-check
16:23:02	GnomePackageKit     status test-commit
16:23:04	GnomePackageKit     status update
16:23:04	GnomePackageKit     already active
16:23:04	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:06	GnomePackageKit     status finished
16:23:06	GnomePackageKit     status check-executable-files
16:23:06	GnomePackageKit     status finished
16:23:06	GnomePackageKit     Unknown error, please report a bug at https://bugs.freedesktop.org/.
More information is available in the detailed report.
16:23:06	GnomePackageKit     Unknown error
16:23:06	GnomePackageKit     (null): There are unfinished transactions remaining. Please run yum-complete-transaction as root.
Comment 6 Michael Schwendt 2013-05-17 11:51:37 EDT
$ rpm -q PackageKit
PackageKit-0.8.8-2.fc19.x86_64

> This prints for me:
> 
> True
> False
> Linux® kernel

The same here, of course, because _to_utf8() returns the unchanged string in the first if-statement.
Comment 7 Richard Hughes 2013-05-17 11:58:57 EDT
Hey Kamil:

Can you directly edit:

/usr/lib/python2.7/site-packages/packagekit/backend.py

and add before line 239:

print package_id
print package_license
print group
print desc
print url
print bytes

and try to work out:

a) which package it is (so I have a hope of reproducing it locally)
b) which parameter is encoded in a weird way (uncommenting them one by one)

Thanks dude,

Richard
Comment 8 Michael Schwendt 2013-05-17 11:59:43 EDT
Oh, last comment is confusing, of course. What I mean is that in packagekit backend.py, the isinstance() results for "desc" are the opposite. This is what I get during debugging:

  str = 0
  unicode = 1
  Security-enhanced Linux is a feature of the Linux® kernel [...]

In turn, encode() is called and throws an exception.
Comment 9 Michael Schwendt 2013-05-17 12:13:03 EDT
Restructuring the code shows that print() throws the exception for the utf-8 encoded strings:

    uni_desc = _to_utf8(desc)
    uni_group = _to_utf8(group)
    print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout)
Comment 10 Adam Williamson 2013-05-18 19:44:27 EDT
KK tells me https://bugzilla.redhat.com/show_bug.cgi?id=964352 is the same as this. If so, this is preventing installation of updates on F19 systems currently, and leaving the yum database in an inconsistent and apparently unrecoverable state. Serious issue, Beta blocker.
Comment 11 Reartes Guillermo 2013-05-19 12:04:28 EDT
> leaving the yum database in an inconsistent and apparently 
> unrecoverable state. Serious issue

I performed a 'yum clean all' then a 'yum-complete-transaction --cleanup-only' then a 'yum-complete-transaction' with redirection to a file, i leaved it some hours and produced a +200mb output file (!). I interrupted it with ctrl+c. As Adam said, it really looks bad. (KDE Plasma Workspaces + apper).
Comment 12 Adam Williamson 2013-05-19 13:54:04 EDT
*** Bug 964352 has been marked as a duplicate of this bug. ***
Comment 13 Reartes Guillermo 2013-05-19 19:16:19 EDT
Created attachment 750427 [details]
output of pkcon

Ok, another try.
I reinstalled the guest, KDE Plasma Workspaces Base Environment without Add-ONs.
I will try packagekit command line 'pkcon' utility instead of KDE 'apper'.

Ssh from another host:

# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

# localectl 
   System Locale: LANG=en_US.UTF-8
       VC Keymap: es
      X11 Layout: es

# pkcon --version
0.8.8

The attached file contains the output of:

# pkcon -p -v get-updates
# pkcon -p -v update

Something to highlight from the pkcon's output is this:

20:04:19        PackageKit          cannot translate 'cleanup', please report!
cleanup
20:04:19        PackageKit          cannot translate 'obsoleting', please report!
obsoleting

I don't know if this is related to the issue, there are a lot of this messages.

The command 'pkcon -p -v update' ends in:

Status:         Waiting in queue
20:06:15        PackageKit          emit transaction-list-changed
Status:         Starting
Percentage:     0
Status:         Running
Status:         Resolving dependencies
Percentage:     10
Status:         Downloading packages
Percentage:     40
Status:         Checking signatures

Software source signature required
 Package: vim-minimal-2:7.3.944-1.fc19.x86_64
 Software source name: updates-testing
 Key URL: /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-x86_64
 Key user: Fedora (19) <fedora@fedoraproject.org>
 Key ID: FB4B18E6
 Key fingerprint: ca81b2c85e4f4d4a1a3f723407477e65fb4b18e6
 Key Timestamp: Sat Dec  1 00:31:25 2012
Do you accept this signature? [N/y] y

20:39:23        PackageKit          emit transaction-list-changed
20:39:23        PackageKit          notify::connected
The daemon crashed mid-transaction!

From this point, a 'yum-complete-transaction --cleanup-only' is required, if executed, all packages look like they were installed while there were not. :-(

A regular 'yum-complete-transaction' might work but i aborted it when its output
exceeded 200mb after several hours (i was using redirection to a file), so maybe
leaving it all night might work.
Comment 14 Richard Hughes 2013-05-20 04:31:20 EDT
Can anyone who can reproduce this bug successfully run:

pkcon get-details libselinux

It should display:

description: Security-enhanced Linux is a feature of the Linux® kernel..

I'm struggling to work out why printing a UTF-8 string should suddenly start throwing exceptions.

Richard
Comment 15 Richard Hughes 2013-05-20 04:53:06 EDT
I'm going to need the help of an experienced python developer here. Something funky is going on. It's nothing to do with the daemon, and everything to do with the yum backend. Since Nils got reassigned, we don't actually have an official yum backend maintainer anymore. Ideas welcome.

Richard
Comment 16 Kamil Páral 2013-05-20 04:56:03 EDT
(In reply to Richard Hughes from comment #7)

Richard, it doesn't happen with PackageKit-0.8.7-1.fc18.x86_64, but it happens with PackageKit-0.8.8-1.fc18.x86_64.

The problematic package is "opencv;2.4.3-5.fc18;x86_64;updates-testing" and its "desc" attribute.
> Error Value: 'ascii' codec can't encode character u'\xae' in position 18: ordinal not in range(128)

This is the value, retrieved by repr():
u'OpenCV means Intel\xae Open Source Computer Vision Library. It is a collection of;C functions and a few C++ classes that implement some popular Image Processing;and Computer Vision algorithms.'
Comment 17 Michael Schwendt 2013-05-20 05:01:40 EDT
Just to answer comment 14, it reproduces the issue in the same way (except that for me it displays my modifications to packagekit/backend.py in the traceback).

Also, one cannot simply print() in there for debugging purposes, since there's some redirection taking place.

type(desc) = <type 'unicode'>
sys.stdout.encoding = 'None'

[...]

$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing]
2. libselinux-2.1.13-15.fc19.i686 [fedora]
3. libselinux-2.1.13-15.fc19.i686 [updates-testing]
Please choose the correct package: 3
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
Fatal error: Error Type: <type 'exceptions.UnicodeDecodeError'>
Error Value: 'ascii' codec can't decode byte 0xc2 in position 49: ordinal not in range(128)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3590, in <module>
    main()
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3587, in main
    backend.dispatcher(sys.argv[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 760, in dispatcher
    self.dispatch_command(args[0], args[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 620, in dispatch_command
    self.get_details(package_ids)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2546, in get_details
    self._show_details_pkg(pkg)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2572, in _show_details_pkg
    self.details(package_id, license, group, desc, url, size)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 303, in details
    PackageKitBaseBackend.details(self, package_id, package_license, group, desc, url, bytes)
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 268, in details
    print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout)
Comment 18 Kamil Páral 2013-05-20 05:02:36 EDT
(In reply to Richard Hughes from comment #14)
> Can anyone who can reproduce this bug successfully run:
> 
> pkcon get-details libselinux

This is throwing the usual exception with PackageKit-0.8.8-1.fc18.x86_64 and PackageKit-0.8.8-2.fc19.x86_64.

With PackageKit-0.8.7-1.fc18.x86_64 it doesn't crash, but doesn't print anything:
$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.12-7.3.fc18.x86_64 [installed:updates]
2. libselinux-2.1.12-7.3.fc18.i686 [updates]
Please choose the correct package: 2
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
$
Comment 19 Michael Schwendt 2013-05-20 05:11:17 EDT
Re: comment 16

> u'OpenCV means Intel\xae Open Source

0xae is ISO 8859-1 for the Trade Mark symbol.
The f18 spec file contains UTF-8 0xc2 0xae.
Comment 20 Michael Schwendt 2013-05-20 05:21:41 EDT
PackageKit 0.8.8 changelog mentions "Do not use _UTF8Writer when using python3 (Richard Hughes)". Actually using an UTF8Writer based on eithers codecs.getwriter() or kitchen.text.converters is something I've tried while taking a first look at the problem. Hasn't worked, and that's where I discovered the 'sys.stdout.encoding = None', too.
Comment 21 Richard Hughes 2013-05-20 07:27:34 EDT
Created attachment 750552 [details]
test patch

Okay, so I've got something I think works (basically, rip out all the print() encoding stuff).

Can someone try doing:

/usr/lib/python2.7/site-packages/packagekit
sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch

If that fixes things, I'll commit upstream and roll a new update.

Thanks,

Richard.
Comment 22 Michael Schwendt 2013-05-20 07:59:04 EDT
Works for me. I've also run gpk-update-viewer and applied the pending updates.

$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing]
2. libselinux-2.1.13-15.fc19.i686 [fedora]
3. libselinux-2.1.13-15.fc19.i686 [updates-testing]
Please choose the correct package: 3
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
Package description
  package:     libselinux-2.1.13-15.fc19.i686
  license:     Public Domain
  group:       other
  description: Security-enhanced Linux is a feature of the Linux® kernel and a number
of utilities with enhanced security functionality designed to add
mandatory access controls to Linux.  The Security-enhanced Linux
kernel contains new architectural components originally developed to
improve the security of the Flask operating system. These
architectural components provide general support for the enforcement
of many kinds of mandatory access control policies, including those
based on the concepts of Type Enforcement®, Role-based Access
Control, and Multi-level Security.

libselinux provides an API for SELinux applications to get and set
process and file security contexts and to obtain security policy
decisions.  Required for any applications that use the SELinux API.
  size:        141244 bytes
  url:         http://oss.tresys.com/git/selinux.git
Comment 23 Kamil Páral 2013-05-20 09:00:19 EDT
(In reply to Richard Hughes from comment #21)
> /usr/lib/python2.7/site-packages/packagekit
> sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch

Fixes the unicode problem in F18. I still can't perform updates because of some other bug, but that was present before. I'll file that separately. The encoding issue seems to be gone.

"pkcon get-details libselinux" also works.
Comment 24 Fedora Update System 2013-05-20 10:30:45 EDT
PackageKit-0.8.9-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/PackageKit-0.8.9-1.fc19
Comment 25 Adam Williamson 2013-05-20 12:49:14 EDT
Discussed at 2013-05-20 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-20/f19beta-blocker-review-7.2013-05-20-16.07.log.txt . Accepted as a blocker per criterion https://fedoraproject.org/wiki/Fedora_19_Alpha_Release_Criteria#Updates  : "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops."
Comment 26 Rex Dieter 2013-05-20 13:32:13 EDT
While this backend problem causes the the apper crash ( bug #964352 ), dannti thinks we can fix apper/kded separately (undup'ing)
Comment 27 Fedora Update System 2013-05-20 16:06:23 EDT
Package PackageKit-0.8.9-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing PackageKit-0.8.9-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-8637/PackageKit-0.8.9-1.fc19
then log in and leave karma (feedback).
Comment 28 Kamil Páral 2013-05-21 03:39:10 EDT
Richard, I need an F18 build as well. Thanks.

On F19, "pkcon get-details libselinux" works well with the new version. I can't test real policycoreutils update, because there is no version to downgrade to.
Comment 29 Fedora Update System 2013-05-21 21:59:14 EDT
PackageKit-0.8.9-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.