Hmm, I can't reproduce. What's the output of "echo $LANG" on your machine (logged in as your user)?

It trips over the %description of the policycoreutils package, which uses a 0xc2 0xae Registered Trade Mark symbol character. In packagekit/backend.py, line 239, in details that is the "desc" variable.

>>> s = 'Linux® kernel'
>>> print isinstance(s,str)
True
>>> print isinstance(s,unicode)
False
>>> print s.encode('utf-8',errors='replace')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 5: ordinal not in range(128)

$ echo $LANG
en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Brilliant, thanks for narrowing it down. I don't think s.encode is what's tripping up PK tho, I think is a more reliable testcase:

---
#!/usr/bin/python2
# -*- coding: utf-8 -*-

def _to_utf8(txt, errors='replace'):
    if isinstance(txt, str):
        return txt
    if isinstance(txt, unicode):
        return txt.encode('utf-8', errors=errors)
    return str(txt)

s = 'Linux® kernel'
print isinstance(s,str)
print isinstance(s,unicode)
print _to_utf8(s)
---

This prints for me:

True
False
Linux® kernel

Just to check the obvious, what version of PackageKit is this? Thanks.

Created attachment 749354 [details]
gpk-u-v output on F18

I see the same traceback on F18:
PackageKit-0.8.8-1.fc18.x86_64
gnome-packagekit-3.6.2-1.fc18.x86_64

I have cs_CZ.UTF-8 locale, but it also fails with en_US. No policycoreutils update is available. I don't see any valuable info in pkmon or on stdout.

When I try to update anyway, it fails:

16:21:37	GnomePackageKit     status download
16:23:02	GnomePackageKit     status sig-check
16:23:02	GnomePackageKit     status test-commit
16:23:04	GnomePackageKit     status update
16:23:04	GnomePackageKit     already active
16:23:04	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:05	GnomePackageKit     already active
16:23:06	GnomePackageKit     status finished
16:23:06	GnomePackageKit     status check-executable-files
16:23:06	GnomePackageKit     status finished
16:23:06	GnomePackageKit     Unknown error, please report a bug at https://bugs.freedesktop.org/.
More information is available in the detailed report.
16:23:06	GnomePackageKit     Unknown error
16:23:06	GnomePackageKit     (null): There are unfinished transactions remaining. Please run yum-complete-transaction as root.

$ rpm -q PackageKit
PackageKit-0.8.8-2.fc19.x86_64

> This prints for me:
> 
> True
> False
> Linux® kernel

The same here, of course, because _to_utf8() returns the unchanged string in the first if-statement.

Hey Kamil:

Can you directly edit:

/usr/lib/python2.7/site-packages/packagekit/backend.py

and add before line 239:

print package_id
print package_license
print group
print desc
print url
print bytes

and try to work out:

a) which package it is (so I have a hope of reproducing it locally)
b) which parameter is encoded in a weird way (uncommenting them one by one)

Thanks dude,

Richard

Oh, last comment is confusing, of course. What I mean is that in packagekit backend.py, the isinstance() results for "desc" are the opposite. This is what I get during debugging:

  str = 0
  unicode = 1
  Security-enhanced Linux is a feature of the Linux® kernel [...]

In turn, encode() is called and throws an exception.

Restructuring the code shows that print() throws the exception for the utf-8 encoded strings:

    uni_desc = _to_utf8(desc)
    uni_group = _to_utf8(group)
    print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout)

KK tells me https://bugzilla.redhat.com/show_bug.cgi?id=964352 is the same as this. If so, this is preventing installation of updates on F19 systems currently, and leaving the yum database in an inconsistent and apparently unrecoverable state. Serious issue, Beta blocker.

> leaving the yum database in an inconsistent and apparently 
> unrecoverable state. Serious issue

I performed a 'yum clean all' then a 'yum-complete-transaction --cleanup-only' then a 'yum-complete-transaction' with redirection to a file, i leaved it some hours and produced a +200mb output file (!). I interrupted it with ctrl+c. As Adam said, it really looks bad. (KDE Plasma Workspaces + apper).

*** Bug 964352 has been marked as a duplicate of this bug. ***

Created attachment 750427 [details]
output of pkcon

Ok, another try.
I reinstalled the guest, KDE Plasma Workspaces Base Environment without Add-ONs.
I will try packagekit command line 'pkcon' utility instead of KDE 'apper'.

Ssh from another host:

# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

# localectl 
   System Locale: LANG=en_US.UTF-8
       VC Keymap: es
      X11 Layout: es

# pkcon --version
0.8.8

The attached file contains the output of:

# pkcon -p -v get-updates
# pkcon -p -v update

Something to highlight from the pkcon's output is this:

20:04:19        PackageKit          cannot translate 'cleanup', please report!
cleanup
20:04:19        PackageKit          cannot translate 'obsoleting', please report!
obsoleting

I don't know if this is related to the issue, there are a lot of this messages.

The command 'pkcon -p -v update' ends in:

Status:         Waiting in queue
20:06:15        PackageKit          emit transaction-list-changed
Status:         Starting
Percentage:     0
Status:         Running
Status:         Resolving dependencies
Percentage:     10
Status:         Downloading packages
Percentage:     40
Status:         Checking signatures

Software source signature required
 Package: vim-minimal-2:7.3.944-1.fc19.x86_64
 Software source name: updates-testing
 Key URL: /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-x86_64
 Key user: Fedora (19) <fedora>
 Key ID: FB4B18E6
 Key fingerprint: ca81b2c85e4f4d4a1a3f723407477e65fb4b18e6
 Key Timestamp: Sat Dec  1 00:31:25 2012
Do you accept this signature? [N/y] y

20:39:23        PackageKit          emit transaction-list-changed
20:39:23        PackageKit          notify::connected
The daemon crashed mid-transaction!

From this point, a 'yum-complete-transaction --cleanup-only' is required, if executed, all packages look like they were installed while there were not. :-(

A regular 'yum-complete-transaction' might work but i aborted it when its output
exceeded 200mb after several hours (i was using redirection to a file), so maybe
leaving it all night might work.

Can anyone who can reproduce this bug successfully run:

pkcon get-details libselinux

It should display:

description: Security-enhanced Linux is a feature of the Linux® kernel..

I'm struggling to work out why printing a UTF-8 string should suddenly start throwing exceptions.

Richard

I'm going to need the help of an experienced python developer here. Something funky is going on. It's nothing to do with the daemon, and everything to do with the yum backend. Since Nils got reassigned, we don't actually have an official yum backend maintainer anymore. Ideas welcome.

Richard

(In reply to Richard Hughes from comment #7)

Richard, it doesn't happen with PackageKit-0.8.7-1.fc18.x86_64, but it happens with PackageKit-0.8.8-1.fc18.x86_64.

The problematic package is "opencv;2.4.3-5.fc18;x86_64;updates-testing" and its "desc" attribute.
> Error Value: 'ascii' codec can't encode character u'\xae' in position 18: ordinal not in range(128)

This is the value, retrieved by repr():
u'OpenCV means Intel\xae Open Source Computer Vision Library. It is a collection of;C functions and a few C++ classes that implement some popular Image Processing;and Computer Vision algorithms.'

Just to answer comment 14, it reproduces the issue in the same way (except that for me it displays my modifications to packagekit/backend.py in the traceback).

Also, one cannot simply print() in there for debugging purposes, since there's some redirection taking place.

type(desc) = <type 'unicode'>
sys.stdout.encoding = 'None'

[...]

$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing]
2. libselinux-2.1.13-15.fc19.i686 [fedora]
3. libselinux-2.1.13-15.fc19.i686 [updates-testing]
Please choose the correct package: 3
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
Fatal error: Error Type: <type 'exceptions.UnicodeDecodeError'>
Error Value: 'ascii' codec can't decode byte 0xc2 in position 49: ordinal not in range(128)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3590, in <module>
    main()
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 3587, in main
    backend.dispatcher(sys.argv[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 760, in dispatcher
    self.dispatch_command(args[0], args[1:])
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 620, in dispatch_command
    self.get_details(package_ids)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2546, in get_details
    self._show_details_pkg(pkg)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 2572, in _show_details_pkg
    self.details(package_id, license, group, desc, url, size)
  File : /usr/share/PackageKit/helpers/yum/yumBackend.py, line 303, in details
    PackageKitBaseBackend.details(self, package_id, package_license, group, desc, url, bytes)
  File : /usr/lib/python2.7/site-packages/packagekit/backend.py, line 268, in details
    print("details\t$s\t$s\t$s\t$s\t$s\t$ld" $ (package_id, package_license, uni_group, uni_desc, url, bytes), file=sys.stdout)

(In reply to Richard Hughes from comment #14)
> Can anyone who can reproduce this bug successfully run:
> 
> pkcon get-details libselinux

This is throwing the usual exception with PackageKit-0.8.8-1.fc18.x86_64 and PackageKit-0.8.8-2.fc19.x86_64.

With PackageKit-0.8.7-1.fc18.x86_64 it doesn't crash, but doesn't print anything:
$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.12-7.3.fc18.x86_64 [installed:updates]
2. libselinux-2.1.12-7.3.fc18.i686 [updates]
Please choose the correct package: 2
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
$

Re: comment 16

> u'OpenCV means Intel\xae Open Source

0xae is ISO 8859-1 for the Trade Mark symbol.
The f18 spec file contains UTF-8 0xc2 0xae.

PackageKit 0.8.8 changelog mentions "Do not use _UTF8Writer when using python3 (Richard Hughes)". Actually using an UTF8Writer based on eithers codecs.getwriter() or kitchen.text.converters is something I've tried while taking a first look at the problem. Hasn't worked, and that's where I discovered the 'sys.stdout.encoding = None', too.

Created attachment 750552 [details]
test patch

Okay, so I've got something I think works (basically, rip out all the print() encoding stuff).

Can someone try doing:

/usr/lib/python2.7/site-packages/packagekit
sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch

If that fixes things, I'll commit upstream and roll a new update.

Thanks,

Richard.

Works for me. I've also run gpk-update-viewer and applied the pending updates.

$ pkcon get-details libselinux
More than one package matches:
1. libselinux-2.1.13-15.fc19.x86_64 [installed:updates-testing]
2. libselinux-2.1.13-15.fc19.i686 [fedora]
3. libselinux-2.1.13-15.fc19.i686 [updates-testing]
Please choose the correct package: 3
Getting details               [=========================]         
Waiting in queue              [=========================]         
Starting                      [=========================]         
Getting information           [=========================]         
Package description
  package:     libselinux-2.1.13-15.fc19.i686
  license:     Public Domain
  group:       other
  description: Security-enhanced Linux is a feature of the Linux® kernel and a number
of utilities with enhanced security functionality designed to add
mandatory access controls to Linux.  The Security-enhanced Linux
kernel contains new architectural components originally developed to
improve the security of the Flask operating system. These
architectural components provide general support for the enforcement
of many kinds of mandatory access control policies, including those
based on the concepts of Type Enforcement®, Role-based Access
Control, and Multi-level Security.

libselinux provides an API for SELinux applications to get and set
process and file security contexts and to obtain security policy
decisions.  Required for any applications that use the SELinux API.
  size:        141244 bytes
  url:         http://oss.tresys.com/git/selinux.git

(In reply to Richard Hughes from comment #21)
> /usr/lib/python2.7/site-packages/packagekit
> sudo patch -p0 < 0001-Do-not-rely-on-Python2-to-write-UTF-8-strings.patch

Fixes the unicode problem in F18. I still can't perform updates because of some other bug, but that was present before. I'll file that separately. The encoding issue seems to be gone.

"pkcon get-details libselinux" also works.

PackageKit-0.8.9-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/PackageKit-0.8.9-1.fc19

Discussed at 2013-05-20 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-20/f19beta-blocker-review-7.2013-05-20-16.07.log.txt . Accepted as a blocker per criterion https://fedoraproject.org/wiki/Fedora_19_Alpha_Release_Criteria#Updates  : "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops."

While this backend problem causes the the apper crash ( bug #964352 ), dannti thinks we can fix apper/kded separately (undup'ing)

Package PackageKit-0.8.9-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing PackageKit-0.8.9-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-8637/PackageKit-0.8.9-1.fc19
then log in and leave karma (feedback).

Richard, I need an F18 build as well. Thanks.

On F19, "pkcon get-details libselinux" works well with the new version. I can't test real policycoreutils update, because there is no version to downgrade to.

PackageKit-0.8.9-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.