Bug 831918 - repoquery -C causes URLGrabError: Check uncompressed DB failed
repoquery -C causes URLGrabError: Check uncompressed DB failed
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: yum-utils (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Fedora Packaging Toolset Team
Fedora Extras Quality Assurance
:
: 832615 835491 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-14 01:03 EDT by Scott Tsai
Modified: 2014-01-21 18:22 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-01 14:47:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
repoquery.strace (384.12 KB, application/octet-stream)
2012-06-14 04:33 EDT, Scott Tsai
no flags Details
0001-misc.decompress-compare-mtime-without-sub-second-pre.patch (1.16 KB, patch)
2012-06-14 06:39 EDT, Scott Tsai
no flags Details | Diff

  None (edit)
Description Scott Tsai 2012-06-14 01:03:08 EDT
Description of problem:
repoquery -C stopped working after a recent yum update.

Version-Release number of selected component (if applicable):
yum-3.4.3-26.fc17.noarch
yum-utils-1.1.31-4.fc17.noarch

How reproducible:
Always.

Steps to Reproduce:
1. sudo yum clean all
2. sudo repoquery -l zsh
3. sudo repoquery -C -l zsh
  
Actual results:
Traceback (most recent call last):
  File "/bin/repoquery", line 1510, in <module>
    main(sys.argv)
  File "/bin/repoquery", line 1504, in main
    repoq.runQuery(regexs)
  File "/bin/repoquery", line 982, in runQuery
    pkgs = self.matchPkgs(items, plain_pkgs=plain_pkgs)
  File "/bin/repoquery", line 903, in matchPkgs
    pkgs = self.returnPkgList(patterns=items)
  File "/bin/repoquery", line 856, in returnPkgList
    pkgs = self.pkgSack.returnNewestByNameArch(**kwargs)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1013, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 777, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 337, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 201, in populate
    raise URLGrabError(-1, 'Check uncompressed DB failed')
urlgrabber.grabber.URLGrabError: [Errno -1] Check uncompressed DB failed


Additional info:
I suspect this is caused by a recent yum update, from /var/log/yum.log:
Jun 13 10:02:53 Updated: yum-3.4.3-26.fc17.noarch
Comment 1 Zdeněk Pavlas 2012-06-14 03:59:21 EDT
Hi, thanks for the report!  I'm not able to reproduce this.  The first repoquery (without -C) works fine, but with -C it (repeatedly?) crashes, right?

Does 'yum -C list zsh' produce the same traceback?

Can you run 'sudo strace -e trace=file repoquery -C -l zsh >/dev/null' and paste last 10 or so lines?  Thanks!
Comment 2 Scott Tsai 2012-06-14 04:32:44 EDT
(In reply to comment #1)
Hi Zdeněk,

Following your instructions and looking at the strace log lead to me believe you need to have the rpmfusion repositories enabled (see http://rpmfusion.org/Configuration) to trigger this bug. Details follow.


> The first repoquery (without -C) works fine, but with -C it (repeatedly?) crashes, right?

Correct. I get the same traceback everytime.

> Does 'yum -C list zsh' produce the same traceback?

It produces the same error message:
[Errno -1] Check uncompressed DB failed
but the exception is apparently handled so no traceback is produced.

> Can you run 'sudo strace -e trace=file repoquery -C -l zsh >/dev/null'

I'll attach the full repoquery.strace file to this bug.
The last few lines turned out to be not very useful, but:

$ grep cache repoquery.strace  | tail -n 5
open("/var/cache/yum/x86_64/17/rpmfusion-free-updates/0ff8840bfce4d9edf341fa327d0b57f2e9dc4971645a52f213f7e7437b653ad4-primary.sqlite.bz2", O_RDONLY) = 7
stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/0ff8840bfce4d9edf341fa327d0b57f2e9dc4971645a52f213f7e7437b653ad4-primary.sqlite.bz2", {st_mode=S_IFREG|0644, st_size=23069, ...}) = 0
stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/gen", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/0ff8840bfce4d9edf341fa327d0b57f2e9dc4971645a52f213f7e7437b653ad4-primary.sqlite.bz2", {st_mode=S_IFREG|0644, st_size=23069, ...}) = 0
stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/gen/primary_db.sqlite", {st_mode=S_IFREG|0644, st_size=124928, ...}) = 0

This lead me to try disabling the rpmfusion repos in /etc/yum.repos.d and disabling them does indeed make the traceback go away.

I'd really like yum and friends to be fixed to work with rpmfusion even if they're using an older 'createrepo(8)' though.
Comment 3 Scott Tsai 2012-06-14 04:33:14 EDT
Created attachment 591773 [details]
repoquery.strace
Comment 4 Zdeněk Pavlas 2012-06-14 04:59:26 EDT
Thanks! Seems the very last thing Yum did before traceback was:

stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/0ff8840bfce4d9edf341fa327d0b57f2e9dc4971645a52f213f7e7437b653ad4-primary.sqlite.bz2", {st_mode=S_IFREG|0644, st_size=23069, ...}) = 0
stat("/var/cache/yum/x86_64/17/rpmfusion-free-updates/gen/primary_db.sqlite", {st_mode=S_IFREG|0644, st_size=124928, ...}) = 0

We compare timestamps of these 2 files, and if they differ, we err out.  This check was added in -26.fc17, to detect invalid or partially decompressed files.
Could you check their mtime, please? (i guess they will differ).

Still, have no clue WHY they should be different.. The .bz2 file is decompressed, and then the mtime is set..  You have plenty of space in /var, I assume..
Comment 5 Scott Tsai 2012-06-14 06:38:45 EDT
(In reply to comment #4)
> Could you check their mtime, please? (i guess they will differ)

Good suggestion. One look and the problem becomes pretty easy to guess:

$ stat --print '%y %n\n'  ... ...

2012-06-14 16:22:03.839396409 +0800 /var/cache/yum/x86_64/17/rpmfusion-free-updates/0ff8840bfce4d9edf341fa327d0b57f2e9dc4971645a52f213f7e7437b653ad4-primary.sqlite.bz2
2012-06-14 16:22:03.839396000 +0800 /var/cache/yum/x86_64/17/rpmfusion-free-updates/gen/primary_db.sqlite

Note that the decompressed file lost its nano second mtime precision.

I googled around and found Python issue 14127: http://bugs.python.org/issue14127
Basically on Linux filesystems with nano second timestamps,
Python's os.stat() and os.utime() doesn't guarantee timestamps would compare equal when read back. New APIs that no longer encode timestamps as a float in seconds and would use syscalls with nano second precisions on Linux were only added to the not yet released Python 3.3+.

I'll attach a patch to yum that limits the mtime comparison precision to seconds.

On a separate note, I think this bug should be taken as a data point that we'd want to port yum to Python 3 eventually for better Linux support in the Python standard library.
Comment 6 Scott Tsai 2012-06-14 06:39:31 EDT
Created attachment 591802 [details]
0001-misc.decompress-compare-mtime-without-sub-second-pre.patch
Comment 7 Zdeněk Pavlas 2012-06-14 07:44:18 EDT
Thanks, merged.  I was thinking about some epsilon-check or rounding instead, but truncating to zero seems to be exactly what the OS does in case one of the files would reside on a FS that does not support ns timestamps.
Comment 8 Zdeněk Pavlas 2012-06-18 05:06:41 EDT
*** Bug 832615 has been marked as a duplicate of this bug. ***
Comment 9 Pratyush Sahay 2012-06-18 05:33:30 EDT
An accidental / intentional press a tab in a yum command is causing this crash. E.g: yum info naut<tab>

Package: yum-3.4.3-26.fc17
OS Release: Fedora release 17 (Beefy Miracle)
Comment 10 William Bader 2012-06-25 01:38:58 EDT
It happened shortly after I ran
  yum install redhat-lsb
as one of the prerequisites for installing google chrome.


Package: yum-3.4.3-26.fc17
OS Release: Fedora release 17 (Beefy Miracle)
Comment 11 Fedora Update System 2012-06-25 04:31:28 EDT
yum-3.4.3-28.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/yum-3.4.3-28.fc17
Comment 12 William Bader 2012-06-25 23:15:10 EDT
It happened shortly after a yum command.

Package: yum-3.4.3-26.fc17
OS Release: Fedora release 17 (Beefy Miracle)
Comment 13 Damien Grassart 2012-06-26 00:12:57 EDT
Tried to tab complete a "yum info" command to find gdevilspie.

Package: yum-3.4.3-26.fc17
OS Release: Fedora release 17 (Beefy Miracle)
Comment 14 Zdeněk Pavlas 2012-06-26 08:00:08 EDT
*** Bug 835491 has been marked as a duplicate of this bug. ***
Comment 15 Fedora Update System 2012-06-27 23:41:04 EDT
yum-3.4.3-28.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 Gerard Ryan 2012-07-25 14:45:46 EDT
This happened when I tried to tab to autocomplete a package name (at least I think that's what caused it)

Package: yum-3.4.3-28.fc17
OS Release: Fedora release 17 (Beefy Miracle)
Comment 17 Fedora End Of Life 2013-07-04 03:04:00 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 18 Fedora End Of Life 2013-08-01 14:47:51 EDT
Fedora 17 changed to end-of-life (EOL) status on 2013-07-30. Fedora 17 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.