Bug 591488

Summary: yum doesn't lock the rpmdb RW constantly, so rpm can remove installed packages that yum knows about (yum traceback)
Product: Red Hat Enterprise Linux 6 Reporter: Petr Sklenar <psklenar>
Component: yumAssignee: James Antill <james.antill>
Status: CLOSED WONTFIX QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1CC: herrold, jhutar, pmatilai, slukasik
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 476195 Environment:
Last Closed: 2010-11-23 19:24:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 476195    
Bug Blocks: 582655    

Description Petr Sklenar 2010-05-12 11:54:26 UTC
I can see this bug also on rhel6 with yum-3.2.27-8.el6.noarch

test procedure:
1. run in one terminal: 
for i in `seq 1 200`; do rpm -e test-bbb;rpm -ivh test-rpmbuild-*/RPMS/*/test-bbb*rpm ; done

2. run in second terminal :
yum --disablerepo=\* --enablerepo=myrepo -y install test-aaa\*; yum --disablerepo=\* --enablerepo=myrepo -y remove test-aaa\*

There is rhts test which can reproduce it: /CoreOS/yum/Regression/bz476195-lock-rpmdb




:: [   PASS   ] :: Running 'yum --disablerepo='*' --enablerepo=myrepo --showduplicates list test-aaa*'

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Install packages
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

error: package test-bbb is not installed
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
Preparing...                ########################################### [100%]
   1:test-bbb               ########################################### [100%]
  File "/usr/bin/yum", line 29, in <module>
  File "/usr/share/yum-cli/yummain.py", line 254, in user_main
  File "/usr/share/yum-cli/yummain.py", line 174, in main
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1142, in _rpmdb_warn_checks
  File "/usr/lib/python2.6/site-packages/yum/rpmsack.py", line 1247, in check_dependencies
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 508, in <lambda>
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1147, in returnPrco
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1161, in _populatePrco
  File "/usr/lib/python2.6/site-packages/yum/rpmsack.py", line 66, in _get_hdr
:: [   FAIL   ] :: There is Traceback (Assert: expected 0, got 1)





+++ This bug was initially created as a clone of Bug #476195 +++

Description of problem:
during rpm installation, I can see yum traceback sometimes

Version-Release number of selected component (if applicable):

[root@dhcp-lab-158 rpm_test1]# rpm -qa yum*
yum-security-1.1.16-13.el5
yum-3.2.19-18.el5
yum-rhn-plugin-0.5.3-30.el5
yum-updatesd-0.9-2.el5
yum-utils-1.1.16-13.el5
yum-metadata-parser-1.1.2-2.el5
[root@dhcp-lab-158 rpm_test1]# rpm -qa rpm*
rpm-4.4.2.3-9.el5
rpm-libs-4.4.2.3-9.el5
rpm-devel-4.4.2.3-9.el5
rpm-python-4.4.2.3-9.el5
rpm-build-4.4.2.3-9.el5
rpm-apidocs-4.4.2.3-9.el5
rpm-debuginfo-4.4.2.3-9.el5
[root@dhcp-lab-158 rpm_test1]# uname -a
Linux dhcp-lab-158.englab.brq.redhat.com 2.6.18-125.el5 #1 SMP Mon Dec 1 17:46:51 EST 2008 ppc64 ppc64 ppc64 GNU/Linux


How reproducible:
sometimes


Steps to Reproduce:
1. in background:  for i in `seq 1 1000` do; rpm -e foo-package.rpm;rpm -i foo-package.rpm;done 
2. yum update

  
Actual results:
#  yum update
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
--> Running transaction check
---> Package ppc64-utils.ppc 0:0.11-10.el5 set to be updated
---> Package kernel.ppc64 0:2.6.18-126.el5 set to be installed
---> Package kernel-headers.ppc 0:2.6.18-126.el5 set to be updated
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 229, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 145, in main
    (result, resultmsgs) = base.buildTransaction() 
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 647, in buildTransaction
    (rescode, restring) = self.resolveDeps()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 704, in resolveDeps
    for po, dep in self._checkFileRequires():
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 902, in _checkFileRequires
    for name, flag, evr in pkg.requires:
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 411, in <lambda>
    requires = property(fget=lambda self: self.returnPrco('requires'))
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 1002, in returnPrco
    self._populatePrco()
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 1016, in _populatePrco
    hdr = self._get_hdr()
  File "/usr/lib/python2.4/site-packages/yum/rpmsack.py", line 57, in _get_hdr
    return mi.next()
StopIteration


Expected results:
yum update should wait for rpm lock and wait, then install packages successfully

Additional info:
I tried that only on ppc with RHEL 5U3 snapshot5

--- Additional comment from james.antill on 2008-12-12 10:49:28 EST ---

Is this only happening on one machine?
Why do you think it's something to do with the rpm lock?
Can you put "print self" just before the above return value?

AFAICS this is saying that we got a particular installed rpm X from index XI, but then when we later try to get some more data from index XI ... rpm says it's bad.

What does "yum --version" say?

Panu, any ideas?

--- Additional comment from james.antill on 2008-12-12 10:57:23 EST ---

I'd guess that this is related to:

 https://bugzilla.redhat.com/show_bug.cgi?id=476188

--- Additional comment from psklenar on 2008-12-12 12:02:47 EST ---

it happens on every machine what I have now (all ppc, maybe all arch has same bug).

I thing that bug 476188 has nothing to common with this - because I cannot reproduce bug 476188 on my ppc machines.


:anyway now I tried this 476195 on 
Linux squad7-lp1.rhts.bos.redhat.com 2.6.18-126.el5 #1 SMP Mon Dec 8 18:35:09 EST 2008 ppc64 ppc64 ppc64 GNU/Linux

one my terminal: for i in `seq 1 1000`; do rpm -e rhts-rh-devel ;rpm -ivh /root/rhts-rh-devel-3.4-20081205.0.noarch.rpm;done

second terminal:
I have old version of package file, and yum should upgrade to latest file:

# yum update
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
--> Running transaction check
---> Package file.ppc 0:4.17-15 set to be updated
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 229, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 145, in main
    (result, resultmsgs) = base.buildTransaction() 
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 647, in buildTransaction
    (rescode, restring) = self.resolveDeps()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 704, in resolveDeps
    for po, dep in self._checkFileRequires():
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 902, in _checkFileRequires
    for name, flag, evr in pkg.requires:
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 411, in <lambda>
    requires = property(fget=lambda self: self.returnPrco('requires'))
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 1002, in returnPrco
    self._populatePrco()
  File "/usr/lib/python2.4/site-packages/yum/packages.py", line 1016, in _populatePrco
    hdr = self._get_hdr()
  File "/usr/lib/python2.4/site-packages/yum/rpmsack.py", line 57, in _get_hdr
    return mi.next()
StopIteration
[root@squad7-lp1 ~]#
--------------------------
try it: squad7-lp1.rhts.bos.redhat.com

--- Additional comment from james.antill on 2008-12-12 13:29:23 EST ---

> one my terminal: for i in `seq 1 1000`; do rpm -e rhts-rh-devel ;rpm -ivh
> /root/rhts-rh-devel-3.4-20081205.0.noarch.rpm;done

> second terminal:
> I have old version of package file, and yum should upgrade to latest file:

 Ok, this makes sense now ... what is happening is that you insert FOO into the DB, and it has index 123, then rpm removes that from the DB, then yum requests the data at index 123 (but it isn't there anymore).

 I'm not sure we can fix this for 5.3 ... a cheap/obvious fix is to lock the rpmdb as soon as we have loaded a single package from it, until we've finished with it completely. This however would be a major "regression" in lots of ways (for instance C-c wouldn't work again) (I'm not 100% sure which other yum releases had that bug ... did you try 3.2.8? -- I'm also pretty sure we said we'd fix C-c at various points).
 Doing something else requires realizing that the package has disappeared and doing something about it ... doing what is non-obvious though.

--- Additional comment from james.antill on 2008-12-12 13:48:57 EST ---

 Ok, here's a simple testcase:

#! /usr/bin/python -tt

from rpmUtils.transaction import initReadOnlyTransaction
import time

__pkg__ = 'xgalaxy'

ts =  initReadOnlyTransaction(root='/')

mi = ts.dbMatch('name', __pkg__)
hdr = mi.next()
print hdr['name'], hdr['epoch'], hdr['version'], hdr['release'], hdr['arch']
idx = mi.instance()
print "Index:", idx
while True:
    time.sleep(1)
    mi = ts.dbMatch(0, idx)
    hdr = mi.next()
    print hdr['name'], hdr['epoch'], hdr['version'], hdr['release'], hdr['arch']

...if you run the above you'll see you have a read only lock on the rpmdb (ts above), which is what yum has for most of it's lifetime ... however you can still run rpm -e in another window.

 Probably the best part about this is that if you run the testcase as a user, it doesn't fail!

--- Additional comment from james.antill on 2008-12-12 13:51:11 EST ---

 Panu, options?

--- Additional comment from james.antill on 2008-12-12 13:55:15 EST ---

 I know why this has started happening in 3.2.19 though ... before that we kept the original hdr around (which was a huge drag on memory).
 As a quick fix for 5.3 we could just undo that.

--- Additional comment from james.antill on 2008-12-12 14:02:51 EST ---

 Saying that I think it's very likely that with the old behavior we just fail silently in weird ways... for instance searching for the prco data will not find the data for the pkg that is gone. As will getting packages by name (unless we are "completely_loaded").
 I'd also be interested in what rpm does on the transaction if we tell it to upgrade things that aren't there anymore/etc.

--- Additional comment from jhutar on 2008-12-15 03:25:45 EST ---

I do not think we need to fix this in 5.3. It would be better to fix it upstream first.

I was able to reproduce on x86_64 F10 using original reproducer with:

yum-3.2.20-3.fc10.noarch
rpm-4.6.0-0.rc1.8.x86_64

--- Additional comment from pmatilai on 2008-12-15 05:31:52 EST ---

Welcome to the world of "rpmdb concurrent access"... Unless you grab a
write-lock on the rpmdb, you're not forbidding others from making changes
behind your back (and OTOH grabbing an exclusive lock has other side-effects).

Catch the exception from mi.next() (would be a good idea to do it anyway),
raise "Please kindly stop pulling the carpet from beneath me" PackageSackError
and abort?

> Probably the best part about this is that if you run the testcase as a user,
> it doesn't fail!

That's because it runs with "private" locking which is about as good as no
locking at all, and can/will run into much uglier problems, see bug 444044.

> I'd also be interested in what rpm does on the transaction if we tell it to
> upgrade things that aren't there anymore/etc.

It'll (quietly) notice the record it was supposed to erase was no longer there
and skip it. Return code from the transaction will show error(s) but that's
all, IIRC. That's essentially what happens in (otherwise unrelated) bug 348971.

--- Additional comment from james.antill on 2009-03-25 02:28:07 EDT ---

 The fix we'll be doing is just the simple try/except and raise PackageSackError.

--- Additional comment from errata-xmlrpc on 2009-04-17 16:04:20 EDT ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2009:8465-01
http://errata.devel.redhat.com/errata/show/8465

--- Additional comment from errata-xmlrpc on 2009-04-17 16:04:43 EDT ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2009:8465-01
http://errata.devel.redhat.com/errata/show/8465

--- Additional comment from jhutar on 2009-07-07 16:46:00 EDT ---

VERIFIED on PV x86_64 guest. Will try to prepare RHTS test for this bug.

--- Additional comment from errata-xmlrpc on 2009-07-30 05:08:15 EDT ---

Bug report changed to RELEASE_PENDING status by Errata System.
Advisory RHBA-2009:8465-01 has been changed to HOLD status.
http://errata.devel.redhat.com/errata/show/8465

--- Additional comment from rlandman on 2009-09-01 09:12:40 EDT ---


Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Because yum does not lock rpmdb, other applications can make changes to the package database at the same time that yum does. Previously, changes made to rpmdb by another application could cause yum to crash; for example, if an application removed data about a package and yum then attempted to retrieve this data. Now, if yum discovers data that is needs to complete a transaction is missing from the rpmdb, yum will exit safely and avoid crashing.

--- Additional comment from errata-xmlrpc on 2009-09-02 03:33:44 EDT ---

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1419.html

Comment 2 James Antill 2010-05-12 14:25:15 UTC
Given that it's a clone of a RHEL-5 bug, I'm going to remove the regression flag.

Also this is line 65-66 of rpmsack:

        except StopIteration:
            raise Errors.PackageSackError, 'Rpmdb changed underneath us'

...which is what we said we'd do on:

Additional comment from james.antill on 2009-03-25 02:28:07 EDT

Comment 3 James Antill 2010-05-25 16:35:39 UTC
We might fix this for 6.1, certainly not for 6.0 though.

Comment 4 Šimon Lukašík 2010-06-15 08:14:30 UTC
Just adding full (not greped) traceback. It's slightly different from bug 476195.

Loaded plugins: rhnplugin, versionlock
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package test-aaa-1.i386 0:0-0 set to be updated
---> Package test-aaa-2.i386 0:0-0 set to be updated
---> Package test-aaa-3.i386 0:0-0 set to be updated
---> Package test-aaa-4.i386 0:0-0 set to be updated
---> Package test-aaa-5.i386 0:0-0 set to be updated
---> Package test-aaa-6.i386 0:0-0 set to be updated
---> Package test-aaa-7.i386 0:0-0 set to be updated
---> Package test-aaa-8.i386 0:0-0 set to be updated
---> Package test-aaa-9.i386 0:0-0 set to be updated
Error: Rpmdb changed underneath us
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 254, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 174, in main
    if not base._rpmdb_warn_checks(out=verbose_logger.info, warn=False):
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1143, in _rpmdb_warn_checks
    probs.extend(self.rpmdb.check_dependencies())
  File "/usr/lib/python2.6/site-packages/yum/rpmsack.py", line 1295, in check_dependencies
    for rreq in pkg.requires:
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 508, in <lambda>
    requires = property(fget=lambda self: self.returnPrco('requires'))
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1147, in returnPrco
    self._populatePrco()
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1161, in _populatePrco
    hdr = self._get_hdr()
  File "/usr/lib/python2.6/site-packages/yum/rpmsack.py", line 66, in _get_hdr
    raise Errors.PackageSackError, 'Rpmdb changed underneath us'
yum.Errors.PackageSackError: Rpmdb changed underneath us


Also note that it happens with '--skip-broken' as well. And yes, it's not regression against rhel5.0. But it's regression against rhel5.5.

Comment 6 James Antill 2010-11-23 19:15:46 UTC
 Ok, so here are the problems:

1. We can't keep the headers around, because:

   i. It takes up way too much memory (why we dropped them in RHEL-5).
   ii. Very soon upstream rpm/yum should be moving to code where we don't even load the header once, for most packages.

2. We can't just "lock" the rpmdb, because then C-c doesn't work anymore (we'd need to hold the lock over downloads).

...which makes it "hard to fix". And even now it almost never happens.

 Combine that with the fact that yum will complain _very_ loudly in RHEL-6 if you start using rpm directly (even if yum isn't running at the same time).

 So given all of that, I'm going to just NAK this. If rpm adds a "lock the rpmdb from other users, but don't alter C-c" API ... then we could call that, feel free to reopen.

Comment 7 RHEL Program Management 2010-11-23 19:24:54 UTC
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.