Bug 622683 - Missing .decoce("utf-8") in yum/history.py ?
Missing .decoce("utf-8") in yum/history.py ?
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
13
All Linux
low Severity high
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-10 03:13 EDT by ARNAL
Modified: 2014-01-21 18:16 EST (History)
8 users (show)

See Also:
Fixed In Version: yum-3.2.28-3.fc14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-17 01:47:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description ARNAL 2010-08-10 03:13:18 EDT
Description of problem:

Receive the following error when doing a yum groupupdate "..." after upgrade to fedora 13.

Total                                                                                                                                                                        1.0 MB/s |  43 MB     00:42
Lancement de rpm_check_debug
Lancement de la transaction de test
Transaction de test réussie
Lancement de la transaction
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 258, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 196, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 544, in doTransaction
    resultobject = self.runTransaction(cb=cb)
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1334, in runTransaction
    self.skipped_packages, rpmdb_problems, cmdline)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 503, in beg
    self._trans_cmdline(cmdline)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 470, in _trans_cmdline
    VALUES (?, ?)""", (self._tid, cmdline))
  File "/usr/lib/python2.6/site-packages/yum/sqlutils.py", line 171, in executeSQLQmark
    return cursor.execute(query.decode("utf-8"), params)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.



Version-Release number of selected component (if applicable): yum 3.2.28


How reproducible:
Always reproducible

Steps to Reproduce:
1. yum groupupdate "Base"
  
Actual results:
  yum end with error

Expected results:
  yum may update the group package

Additional info:

It seems that there's a missing .decode("utf-8") in the following file:
    "/usr/lib/python2.6/site-packages/yum/history.py"

Changing line 467-470:

    467     res = executeSQL(cur,
    468                          """INSERT INTO trans_cmdline
    469                          (tid, cmdline)
    470                          VALUES (?, ?)""", (self._tid, 
                                                     cmdline))
by

    467     res = executeSQL(cur,
    468                          """INSERT INTO trans_cmdline
    469                          (tid, cmdline)
    470                          VALUES (?, ?)""", (self._tid, 
                                                     cmdline.decode("utf-8")))

fix the problems.


Regards,
       Laurent.
Comment 1 seth vidal 2010-08-10 11:02:12 EDT
yum 3.2.28-1 or 3.2.28-2?


-2 is in testing and may well fix this issue.
Comment 2 James Antill 2010-08-10 11:38:29 EDT
nah, this is another one ... just went upstream:

http://yum.baseurl.org/gitweb?p=yum.git;a=commitdiff;h=7bd63b84d1202e06bc78c92d27fb7167707fa2af
Comment 3 Fedora Update System 2010-08-12 16:15:21 EDT
yum-3.2.28-3.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/yum-3.2.28-3.fc12
Comment 4 Fedora Update System 2010-08-12 16:17:50 EDT
yum-3.2.28-3.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/yum-3.2.28-3.fc13
Comment 5 Fedora Update System 2010-08-12 16:20:14 EDT
yum-3.2.28-3.fc14 has been submitted as an update for Fedora 14.
http://admin.fedoraproject.org/updates/yum-3.2.28-3.fc14
Comment 6 Fedora Update System 2010-08-13 17:18:21 EDT
yum-3.2.28-3.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update yum'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/yum-3.2.28-3.fc12
Comment 7 Fedora Update System 2010-08-17 01:45:30 EDT
yum-3.2.28-3.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 8 Fedora Update System 2010-08-19 22:19:12 EDT
yum-3.2.28-3.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 9 josh 2010-08-21 10:23:51 EDT
The problem still exists in yum-3.2.28-3.fc13.
Comment 10 josh 2010-08-21 17:09:44 EDT
Actually, it's a slightly different problem, one that's been around for a while and I realize I'd had to patch one of the previous times I had to patch yum.  The problem is that line in history.py:

    if checksum == sql_checksum:

is failing because checksum is not unicode, which results in the following error output:

/usr/lib/python2.6/site-packages/yum/history.py:321: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if checksum == sql_checksum:
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 258, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 196, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 544, in doTransaction
    resultobject = self.runTransaction(cb=cb)
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1343, in runTransaction
    self.skipped_packages, rpmdb_problems, cmdline)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 494, in beg
    pid   = self.pkg2pid(txmbr.po)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 352, in pkg2pid
    return self._ipkg2pid(po)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 349, in _ipkg2pid
    return self._pkgtup2pid(po.pkgtup, csum)
  File "/usr/lib/python2.6/site-packages/yum/history.py", line 332, in _pkgtup2pid
    checksum))
  File "/usr/lib/python2.6/site-packages/yum/sqlutils.py", line 168, in executeSQLQmark
    return cursor.execute(query, params)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Not sure if this is the correct solution, but adding the line:

    checksum = to_unicode(checksum)

before the if sequence fixes the problem.

It may be related to having upgraded from FC11 to FC12 to FC13, and having non-unicode data in the sqlite DB that isn't being converted properly...
Comment 11 James Antill 2010-08-21 18:20:19 EDT
A checksum should never be outside the 7-bit range, and the only times I've seen that it was FS corruption. We should probably check for this somewhere though, as various bits of yum don't like it.
Comment 12 josh 2010-08-21 18:29:52 EDT
In that case, I wonder if the problem is that sql_checksum is erroneously getting converted to unicode elsewhere, which is what causes the failure.  Purely speculation on my part, since I'm fairly clueless regarding Python (and what I know of it isn't my cup of tea, but that's neither here nor there with regards to yum)...
Comment 13 Fedora Update System 2010-08-23 21:36:52 EDT
yum-3.2.28-3.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 Billy Coutsis 2010-09-27 23:15:38 EDT
I am still getting the same sqlite3.ProgrammingError when I try and do "yum update". I am currently running yum-3.2.28-4.fc13.noarch.
Comment 15 Greg 2011-05-21 23:54:17 EDT
I'm getting this error. Yum versions below. I've done a 'yum clean all', and got it down to only failing on samba packages (don't know if that is relavant). I have three 4 installs of Fedora and this is only happening on one of them.

Versions
  yum-3.2.28-6.fc14.noarch
  kernel is 2.6.35.12-88.fc14.x86_64.



# yum update
Loaded plugins: fastestmirror, kmdl, langpacks, presto, refresh-packagekit
Adding en_US to language list
Determining fastest mirrors
fedora/metalink                                                                           | 2.9 kB     00:00
updates/metalink                                                                          | 2.9 kB     00:00
 * fedora: ucmirror.canterbury.ac.nz
 * updates: ucmirror.canterbury.ac.nz
atrpms                                                                                    | 3.5 kB     00:00
atrpms/primary_db                                                                         | 665 kB     00:02
fedora                                                                                    | 4.3 kB     00:00
fedora/primary_db                                                                         |  14 MB     00:10
updates                                                                                   | 4.7 kB     00:00
updates/primary_db                                                                        | 5.7 MB     00:04
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package samba-client.x86_64 0:3.5.8-76.fc14 set to be updated
---> Package samba-common.x86_64 0:3.5.8-76.fc14 set to be updated
---> Package samba-winbind-clients.x86_64 0:3.5.8-76.fc14 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================
 Package                            Arch                Version                       Repository            Size
=================================================================================================================
Updating:
 samba-client                       x86_64              3.5.8-76.fc14                 updates               11 M
 samba-common                       x86_64              3.5.8-76.fc14                 updates               13 M
 samba-winbind-clients              x86_64              3.5.8-76.fc14                 updates              1.1 M

Transaction Summary
=================================================================================================================
Upgrade       3 Package(s)

Total download size: 26 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates/prestodelta                                                                       | 1.0 MB     00:00
Processing delta metadata
Download delta size: 3.6 M
(1/2): samba-common-3.5.8-74.fc14_3.5.8-76.fc14.x86_64.drpm                               | 3.6 MB     00:02
(2/2): samba-winbind-clients-3.5.8-74.fc14_3.5.8-76.fc14.x86_64.drpm                      |  50 kB     00:00
Finishing rebuild of rpms, from deltarpms
<delta rebuild>                                                                           |  15 MB     00:16
Presto reduced the update size by 76% (from 15 M to 3.6 M).
Package(s) data still to download: 11 M
samba-client-3.5.8-76.fc14.x86_64.rpm                                                     |  11 MB     00:08
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
/usr/lib/python2.7/site-packages/yum/history.py:321: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if checksum == sql_checksum:
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 258, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 196, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 544, in doTransaction
    resultobject = self.runTransaction(cb=cb)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1350, in runTransaction
    self.skipped_packages, rpmdb_problems, cmdline)
  File "/usr/lib/python2.7/site-packages/yum/history.py", line 494, in beg
    pid   = self.pkg2pid(txmbr.po)
  File "/usr/lib/python2.7/site-packages/yum/history.py", line 352, in pkg2pid
    return self._ipkg2pid(po)
  File "/usr/lib/python2.7/site-packages/yum/history.py", line 349, in _ipkg2pid
    return self._pkgtup2pid(po.pkgtup, csum)
  File "/usr/lib/python2.7/site-packages/yum/history.py", line 332, in _pkgtup2pid
    checksum))
  File "/usr/lib/python2.7/site-packages/yum/sqlutils.py", line 168, in executeSQLQmark
    return cursor.execute(query, params)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
Comment 16 James Antill 2011-05-24 13:33:57 EDT
Greg, that's a different bug ... and whenever I've seen it it's due to filesystem corruption. Check Eg. "yumdb info 'samba*'".

I don't think we deal with it well yet, but again it's not 100% obvious we should due to the nature of the problem.
Comment 17 Greg 2011-05-24 22:12:34 EDT
Yes thank you. I can confirm some inconsistent data in the yumdb.

samba-client-3.5.8-74.fc14.x86_64
     changed_by = 501
     checksum_data = gqmP
▒▒.,6▒▒▒MK)6▒rNUW▒*▒
     checksum_type = sha256
     command_line = update
     from_repo = updates
     from_repo_revision = 1301977831
     from_repo_timestamp = 1301979362
     installed_by = 4294967295
     reason = user
     releasever = 14

I worked around this by putting a sha256 sum into the file (/var/lib/yum/yumdb/s/...).

For me there seems to be an issue that something has corrupted the file, then the code consuming the data hasn't handled the unexpected data very well.  I couldn't find a tool or command to rebuild the data automatically.  I couldn't find a RPM query format to give the SHA256 sum of the rpm (RPMTAG_FILEMD5S & RPMTAG_FILEDIGESTS didn't seem to match other yumdb entries).

Note: The machine in question is in an uncontrolled environment with no UPS. It is entirely possible user error has generated the corrupt file.

http://plone.lucidsolutions.co.nz/linux/rpm/unicodewarning-unicode-equal-comparison-failed-to-convert-both-arguments-to-unicode-interpreting-them-as-being-unequal
Comment 18 James Antill 2011-05-25 15:03:55 EDT
Greg: yumdb sync-force samba-client

Note You need to log in before you can comment on or make changes to this bug.