This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 453037 - yum-updatesd leaves a lot of stale locks on RPMDB when auto-installing
yum-updatesd leaves a lot of stale locks on RPMDB when auto-installing
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: yum-updatesd (Show other bugs)
5.2
All Linux
high Severity high
: rc
: ---
Assigned To: James Antill
:
Depends On:
Blocks: 479695
  Show dependency treegraph
 
Reported: 2008-06-26 16:01 EDT by Lubomir Rintel
Modified: 2014-11-30 20:11 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 479695 491138 (view as bug list)
Environment:
Last Closed: 2009-01-20 16:43:19 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Remove rpmdb locks after automatic update (500 bytes, text/plain)
2008-06-26 16:01 EDT, Lubomir Rintel
no flags Details

  None (edit)
Description Lubomir Rintel 2008-06-26 16:01:50 EDT
Description of problem:

When yum-updatesd-helper is launched with --apply option and there are updates
to install available, a couple of locks are left on RPM db after the update is
finished.

This can lead to rpmdb being locked up for any use until db_recover is run when
the limit is reached. The default limit for concurrent lockers is 1000 and for
me it was reached after about 20 updates of 3 packages each; though it can be
considered a very specific situation, maybe it's not likely to happen very often
for most users.

Version-Release number of selected component (if applicable):

yum-updatesd-0.9-2.el5

How reproducible:

Allways reproducible.

Steps to Reproduce:
1. Downgrade some package to an older version
2. cd /var/lib/rpm; db_stat -c; # Check the number of current locks and lockers
3. /usr/libexec/yum-updatesd-helper --apply
4. Repeat step 2. and compare

Actual results:

RPM, when out of locker slots in db:

rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 1f94
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm

Additional info:

A fix is attached to this bug report.
Comment 1 Lubomir Rintel 2008-06-26 16:01:50 EDT
Created attachment 310370 [details]
Remove rpmdb locks after automatic update
Comment 2 James Antill 2008-08-25 17:19:26 EDT
 The above patch is now upstream, however yum-updatesd probably won't be updated soon ... however upstream yum-3.2.19 contains the fixes to make it just work without the patch (YumBase will remove the locks when it is destroyed). And so that fix should make it into a yum errata for 5.3.
Comment 5 Markus Benning 2008-12-04 08:25:46 EST
Since all Clients that have this Bug will stop checking in to RHN Satellite Server due to the rpmdb error (after having more than 1000 locks),i would recommend to set the severity to high. 


Every time a Client connects to the RHN Satellite Server, there will be two additional locks on the rpm db's. With an intervall of 60 minutes for automatic RHN Sattelite logins, this means 48 locks/day or in other words: Trouble after 20 days ;-)

This Bug causes a lot of trouble in large productive environments.
Comment 6 Jason Smith 2009-01-05 12:03:46 EST
I agree with Markus that this bug should be marked High or maybe even Urgent.  We have nearly 100 RHEL5 servers that are subscribed to the RHN, which now stop checking in to get updates every few weeks because of the broken rpm DB, and the only solution is to log in to each and every one and rebuild the DB.  This effectively makes our subscriptions to the RHN useless!  Why is it taking so long to fix this bug, which was first reported over 6 months ago, with a request for escalation of this issue 1 month ago?

~Jason

PS.  I believe there is an official problem report opened in RH's issue tracker for this bug also, #245304.
Comment 8 James Antill 2009-01-05 14:20:53 EST
Jason, you can use yum-cron for servers which will work around your problem. Also you could move to the 5.3 Beta version of yum, as that should fix this problem too.
Comment 9 Jason Smith 2009-01-05 15:16:46 EST
Hi James, unfortunately we are a Satellite customer and last I checked, the Beta channels were not available through a Satellite server (it would be nice if they were).  Also, going through all hundred servers and retro-fitting them with yum-cron just for a temporary bug fix does not sound that attractive either.  It would be nice if RedHat would recognize the seriousness of this bug and escalate the issue to make the official errata available ASAP, instead of making their customers wait months for a "quarterly" update.
Comment 10 csb sysadmin 2009-01-07 13:00:48 EST
I concur, we have 125+ RHEL5 systems and more than half will develop and re-develop this problem (after running --rebuilddb).
Comment 14 csb sysadmin 2009-01-09 15:14:09 EST
A similarity between all the systems we have here that don't have this problem is that they are all running yum-3.0.1-5.el5 and the systems that do have this problem are running yum-3.2.8-9.el5_2.1 .
Comment 21 errata-xmlrpc 2009-01-20 16:43:19 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0176.html
Comment 28 Chen Chang 2009-03-18 17:01:27 EDT
From: Campbell, Floyd 
Sent: Wednesday, March 18, 2009 4:15 PM
To: Chang, Chen
Subject: RE: bug report entry

The new version of yum doesn’t fix this problem. I built a new installation of RedHat Enterprise Linux release 5.2 in a lab environment and did the following.
 
With the old version of yum installed (yum-3.2.8-9.el5_2.1)
# rpm -q yum
yum-3.2.8-9.el5_2.1

 I ran /usr/libexec/yum-updatesd-helper –apply 1000 times.

# for i in $(seq 1 1000); do /usr/libexec/yum-updatesd-helper --apply; done

and it caused the rpm database to run out of available lockers.

rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: a55
error: db4 error(22) from db->close: Invalid argument
error: cannot open Providename index using db3 - Cannot allocate memory (12)

I checked the status of the rpm database and it showed that the lock limit was reached.

# /usr/lib/rpm/rpmdb_stat –c

2860    Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
500     Number of current locks
501     Maximum number of locks at any one time
1000    Number of current lockers
1000    Maximum number of lockers at any one time
1       Number of current lock objects
5       Maximum number of lock objects at any one time
271728  Total number of locks requested
271228  Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)

I then rebuilt the rpm database and upgraded to the recommended version of yum (yum-3.2.19-18.el5) then repeat the above steps.

Rebuild the rpm database.
# rm /var/lib/rpm/__db.00* 
# rpm –rebuilddb
# wget http://download.rhn.redhat.com/download/1237416700/a89607b277ab5c83c87306cf196e13166d090e05/1919334/0/rhn/public/NULL/yum/3.2.19-18.el5/noarch/yum-3.2.19-18.el5.noarch.rpm
--14:57:35--  http://download.rhn.redhat.com/download/1237416700/a89607b277ab5c83c87306cf196e13166d090e05/1919334/0/rhn/public/NULL/yum/3.2.19-18.el5/noarch/yum-3.2.19-18.el5.noarch.rpm
Resolving download.rhn.redhat.com... 209.132.177.120
Connecting to download.rhn.redhat.com|209.132.177.120|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 952992 (931K) [application/octet-stream]
Saving to: `yum-3.2.19-18.el5.noarch.rpm'

100%[=========================================================================================================================================>] 952,992      840K/s   in 1.1s

14:57:36 (840 KB/s) - `yum-3.2.19-18.el5.noarch.rpm' saved [952992/952992]

# rpm -q yum
yum-3.2.8-9.el5_2.1
rpm -Uvh --test yum-3.2.19-18.el5.noarch.rpm
Preparing...                ########################################### [100%]
# rpm -Uvh yum-3.2.19-18.el5.noarch.rpm
Preparing...                ########################################### [100%]
   1:yum                    ########################################### [100%]
# rpm -q yum
yum-3.2.19-18.el5
#

# for i in $(seq 1 1000); do /usr/libexec/yum-updatesd-helper --apply; done

The rpm database still ran out of available lockers. Checking the rpm db stats showed the following.

# /usr/lib/rpm/rpmdb_stat -c
3181    Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
500     Number of current locks
501     Maximum number of locks at any one time
1000    Number of current lockers
1000    Maximum number of lockers at any one time
1       Number of current lock objects
17      Maximum number of lock objects at any one time
313705  Total number of locks requested
309077  Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)
#

 I then rebooted, rebuilt the rpm database again and repeat the above steps, and again the rpm database ran out of available lockers.

Rebuild the rpm database.
# rm /var/lib/rpm/__db.00* 
# rpm –rebuilddb

# for i in $(seq 1 1000); do /usr/libexec/yum-updatesd-helper --apply; done

The rpm database still ran out of available lockers. Checking the rpm db stats showed the following.


# /usr/lib/rpm/rpmdb_stat -c
2546    Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
500     Number of current locks
501     Maximum number of locks at any one time
1000    Number of current lockers
1000    Maximum number of lockers at any one time
1       Number of current lock objects
7       Maximum number of lock objects at any one time
266618  Total number of locks requested
266100  Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)
#
Comment 29 James Antill 2009-03-19 10:46:02 EDT
 Ahh, could you try the following patch:

http://yum.baseurl.org/gitweb?p=yum.git;a=commitdiff;h=bf276cc6369a5d97113950ba7a10df1ea3e47032
Comment 30 Chen Chang 2009-03-19 14:20:30 EDT
Still experiencing the issue...

From: Campbell, Floyd 
Sent: Thursday, March 19, 2009 2:12 PM
To: Chang, Chen
Subject: RE: [Bug 453037] yum-updatesd leaves a lot of stale locks on RPMDB when auto-installing

Chen,

I applied the patch and ran the test again and the rpm database still ran out of available lockers. The output of the test follows.

# cd /usr/lib/python2.4/site-packages/yum/

# ls repos.py
repos.py

# cp repos.py repos.py.`date +%Y%m%d.%H%M%S`

# cat /tmp/patchfile
--- a/yum/repos.py
+++ b/yum/repos.py
@@ -43,9 +43,16 @@ class RepoStorage:
         # callbacks for handling gpg key imports for repomd.xml sig checks
         # need to be set from outside of the repos object to do anything
         # even quasi-useful
-        self.gpg_import_func = self.ayum.getKeyForRepo # defaults to what is probably sane-ish
+        # defaults to what is probably sane-ish
+        self.gpg_import_func = self._wrap_ayum_getKeyForRepo
         self.confirm_func = None

+    def _wrap_ayum_getKeyForRepo(repo, callback=None):
+        """ This is a wrapper for calling self.ayum.getKeyForRepo() because
+            otherwise we take a real reference through the bound method and
+            that is d00m. """
+        return self.ayum.getKeyForRepo(repo, callback)
+
     def doSetup(self, thisrepo = None):

         self.ayum.plugins.run('prereposetup')

# patch repos.py /tmp/patchfile
patching file repos.py
Hunk #1 succeeded at 43 with fuzz 2.

# diff repos.py.20090319.113903 repos.py
46c46,47
<         self.gpg_import_func = self.ayum.getKeyForRepo # defaults to what is probably sane-ish
---
>         # defaults to what is probably sane-ish
>         self.gpg_import_func = self._wrap_ayum_getKeyForRepo
48a50,55
>     def _wrap_ayum_getKeyForRepo(repo, callback=None):
>         """ This is a wrapper for calling self.ayum.getKeyForRepo() because
>             otherwise we take a real reference through the bound method and
>             that is d00m. """
>         return self.ayum.getKeyForRepo(repo, callback)
>

# rpm -q rpm
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: a21
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
package rpm is not installed


# rm -f /var/lib/rpm/__db.00*

# rpm --rebuilddb

# rpm -q rpm
rpm-4.4.2-48.el5

# cd /var/lib/rpm/

# /usr/lib/rpm/rpmdb_stat -c
4       Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
0       Number of current locks
5       Maximum number of locks at any one time
0       Number of current lockers
4       Maximum number of lockers at any one time
0       Number of current lock objects
5       Maximum number of lock objects at any one time
18      Total number of locks requested
18      Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)

# for i in $(seq 1 1000); do /usr/libexec/yum-updatesd-helper --apply; done
Loaded plugins: rhnplugin
.
.
.
Loaded plugins: rhnplugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: bca
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/libexec/yum-updatesd-helper", line 665, in ?
    main()
  File "/usr/libexec/yum-updatesd-helper", line 620, in main
    updd.doSetup()
  File "/usr/libexec/yum-updatesd-helper", line 374, in doSetup
    for repo in self.repos.listEnabled():
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 588, in <lambda>
    repos = property(fget=lambda self: self._getRepos(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 394, in _getRepos
    self._getConfig() # touch the config class first
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 189, in _getConfig
    startupconf.pluginconfpath,disabled_plugins,enabled_plugins)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 352, in doPluginSetup
    raise RuntimeError(_("plugins already initialised"))
RuntimeError: plugins already initialised
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'INIConfig'" in <bound method UpdatesDaemon.__del__ of <__main__.UpdatesDaemon object at 0x9b1908c>> ignored

# /usr/lib/rpm/rpmdb_stat -c
3020    Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
500     Number of current locks
501     Maximum number of locks at any one time
1000    Number of current lockers
1000    Maximum number of lockers at any one time
1       Number of current lock objects
5       Maximum number of lock objects at any one time
265544  Total number of locks requested
265044  Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
1       The number of region locks that required waiting (0%)

# rpm -q rpm
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: bcd
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
package rpm is not installed

I rebooted the server and ran the test again and it still ran out of available lockers.

# cd /var/lib/rpm

# ls
Basenames     Dirnames  Group       Name      Providename     Pubkeys      Requireversion  Sigmd5
Conflictname  Filemd5s  Installtid  Packages  Provideversion  Requirename  Sha1header      Triggername

# /usr/lib/rpm/rpmdb_stat –c
db_stat: DB_ENV->open: No such file or directory

# rpm --rebuilddb

# rpm -q rpm
rpm-4.4.2-48.el5

# /usr/lib/rpm/rpmdb_stat -c
4       Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
0       Number of current locks
5       Maximum number of locks at any one time
0       Number of current lockers
4       Maximum number of lockers at any one time
0       Number of current lock objects
5       Maximum number of lock objects at any one time
18      Total number of locks requested
18      Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)

# for i in $(seq 1 1000); do /usr/libexec/yum-updatesd-helper --apply; done
Loaded plugins: rhnplugin
.
.
.
Loaded plugins: rhnplugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 9d7
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/libexec/yum-updatesd-helper", line 665, in ?
    main()
  File "/usr/libexec/yum-updatesd-helper", line 620, in main
    updd.doSetup()
  File "/usr/libexec/yum-updatesd-helper", line 374, in doSetup
    for repo in self.repos.listEnabled():
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 588, in <lambda>
    repos = property(fget=lambda self: self._getRepos(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 394, in _getRepos
    self._getConfig() # touch the config class first
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 191, in _getConfig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 754, in readMainConfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 824, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'INIConfig'" in <bound method UpdatesDaemon.__del__ of <__main__.UpdatesDaemon object at 0x96cd08c>> ignored

# /usr/lib/rpm/rpmdb_stat -c
2520    Last allocated locker ID
0x7fffffff      Current maximum unused locker ID
5       Number of lock modes
1000    Maximum number of locks possible
1000    Maximum number of lockers possible
1000    Maximum number of lock objects possible
500     Number of current locks
501     Maximum number of locks at any one time
1000    Number of current lockers
1000    Maximum number of lockers at any one time
1       Number of current lock objects
5       Maximum number of lock objects at any one time
265522  Total number of locks requested
265022  Total number of locks released
0       Total number of lock requests failing because DB_LOCK_NOWAIT was set
0       Total number of locks not immediately available due to conflicts
0       Number of deadlocks
0       Lock timeout value
0       Number of locks that have timed out
0       Transaction timeout value
0       Number of transactions that have timed out
440KB   The size of the lock region
0       The number of region locks that required waiting (0%)
Comment 31 James Antill 2009-03-19 17:56:00 EDT
 Ok, I'm trying this now with:

# yum list installed yum\* rpm\* python glibc
Loaded plugins: aliases, rhnplugin
Installed Packages
glibc.i686                             2.5-34                        installed
python.i386                            2.4.3-24.el5                  installed
rpm.i386                               4.4.2.3-9.el5                 installed
rpm-build.i386                         4.4.2.3-9.el5                 installed
rpm-devel.i386                         4.4.2.3-9.el5                 installed
rpm-libs.i386                          4.4.2.3-9.el5                 installed
rpm-python.i386                        4.4.2.3-9.el5                 installed
yum.noarch                             3.2.19-18.el5                 installed
yum-aliases.noarch                     1.1.16-13.el5                 installed
yum-metadata-parser.i386               1.1.2-2.el5                   installed
yum-rhn-plugin.noarch                  0.5.3-30.el5                  installed
yum-updatesd.noarch                    1:0.9-2.el5                   installed
yum-utils.noarch                       1.1.16-13.el5                 installed

...and I'm running:

while true; do /usr/libexec/yum-updatesd-helper -a; echo done; sleep 2; done   

...and in another window:

/usr/lib/rpm/rpmdb_stat -c | fgrep 'current lock'

...but the lock numbers don't change. Can you see anything that's different?
Comment 32 James Antill 2009-03-19 17:56:39 EDT
> # rpm -q rpm
> rpm-4.4.2-48.el5

 Can you try with a more recent version of rpm?
Comment 33 Chen Chang 2009-03-20 11:31:47 EDT
James,

'current lock' isn't the issue.  The error condition surfaces when 'Maximum number of lockers at any one time' reaches 1000.

Try the following.  It will exit once you hit the error condition which is around the 500th iteration.

while rpm -q rpm > /dev/null; do /usr/libexec/yum-updatesd-helper -a; /usr/lib/rpm/rpmdb_stat -c 2> /dev/null| grep 'Maximum number of lockers'; done    

Here is the output for the last few iterations of the loop:

Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
988     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
990     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
992     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
994     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
996     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
998     Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000    Maximum number of lockers possible
1000    Maximum number of lockers at any one time
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 1190
error: db4 error(22) from db->close: Invalid argument
error: cannot open Name index using db3 - Cannot allocate memory (12)


Here is the output of the yum list installed command on our system:

[root@unxtssaf rpm]# yum list installed yum\* rpm\* python glibc
Loaded plugins: rhnplugin, security
Installed Packages
glibc.i686                                                                     2.5-24                                                              installed
python.i386                                                                    2.4.3-21.el5                                                        installed
rpm.i386                                                                       4.4.2-48.el5                                                        installed
rpm-build.i386                                                                 4.4.2-48.el5                                                        installed
rpm-devel.i386                                                                 4.4.2-48.el5                                                        installed
rpm-libs.i386                                                                  4.4.2-48.el5                                                        installed
rpm-python.i386                                                                4.4.2-48.el5                                                        installed
yum.noarch                                                                     3.2.19-18.el5                                                       installed
yum-metadata-parser.i386                                                       1.1.2-2.el5                                                         installed
yum-rhn-plugin.noarch                                                          0.5.3-30.el5                                                        installed
yum-security.noarch                                                            1.1.10-9.el5                                                        installed
yum-updatesd.noarch                                                            1:0.9-2.el5                                                         installed

I did not try upgrading rpm.  I want to ensure we are first both using the same measuring stick for the test case before I start introducing changes to our environment.

Thanks,
Chen
Comment 34 James Antill 2009-03-23 10:01:55 EDT
I've left your script running for a while now, I'm still getting:

Loaded plugins: rhnplugin
1000	Maximum number of lockers possible
29	Maximum number of lockers at any one time
Loaded plugins: rhnplugin
1000	Maximum number of lockers possible
29	Maximum number of lockers at any one time

...which is what it said at the start.
Comment 35 Floyd Campbell 2009-03-26 14:03:36 EDT
We built and installed 2 test servers with RedHat Enterprise 5.2.  We subscribed unxtssaf to RedHat network and subscribe unxtssam to our in house satellite server. The server unxtssaf that was subscribed to RedHat did not run out of available lockers after accessing the rpm database 887 times. Server unxtssam ran out of available lockers. On server unxtssam I then rebuilt the rpm database and upgraded the rpm package and all it’s dependencies, then re-ran the test and the server did not run out of available lockers. Which indicate that upgrading the rpm packages to the latest version will fix this problem in our environment. The output from the test follows. Please not that the server was also subscribed to our in house satellite server on all the previous test postings.


***********
UNXTSSAF
***********
[root@unxtssaf ~]#
[root@unxtssaf ~]# rhn_register
[root@unxtssaf ~]#

[root@unxtssaf ~]# cd /var/lib/rpm/
[root@unxtssaf rpm]#
[root@unxtssaf rpm]#
[root@unxtssaf rpm]# /usr/libexec/yum-updatesd-helper -a
Loading "rhnplugin" plugin
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186
Importing GPG key 0x37017186 "Red Hat, Inc. (release key) <security@redhat.com>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[root@unxtssaf rpm]# rpm -q rpm
rpm-4.4.2-48.el5
[root@unxtssaf rpm]#


[root@unxtssaf rpm]# count=0; while rpm -q rpm > /dev/null; do let count=count+1; echo $count;/usr/libexec/yum-updatesd-helper -c; /usr/lib/rpm/rpmdb_stat -c 2> /dev/null| grep 'Maximum number of lockers'; done
1
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
16      Maximum number of lockers at any one time
.
.
.
1000    Maximum number of lockers possible
18      Maximum number of lockers at any one time
887
Loading "rhnplugin" plugin
[root@unxtssaf rpm]#

[root@unxtssaf rpm]# yum list installed yum\* rpm\* python glibc
Loading "rhnplugin" plugin
Loading "security" plugin
rhel-i386-server-5        100% |=========================| 1.3 kB    00:00
Installed Packages
glibc.i686                               2.5-24                 installed
python.i386                              2.4.3-21.el5           installed
rpm.i386                                 4.4.2-48.el5           installed
rpm-build.i386                           4.4.2-48.el5           installed
rpm-devel.i386                           4.4.2-48.el5           installed
rpm-libs.i386                            4.4.2-48.el5           installed
rpm-python.i386                          4.4.2-48.el5           installed
yum.noarch                               3.2.8-9.el5            installed
yum-metadata-parser.i386                 1.1.2-2.el5            installed
yum-rhn-plugin.noarch                    0.5.3-6.el5            installed
yum-security.noarch                      1.1.10-9.el5           installed
yum-updatesd.noarch                      1:0.9-2.el5            installed
[root@unxtssaf rpm]#




***********
UNXTSSAM
***********
[root@unxtssam ~]# rhn_register
[root@unxtssam ~]# bkp /etc/sysconfig/rhn/up2date fcampbell
[root@unxtssam ~]# vi /etc/sysconfig/rhn/up2date
 [root@unxtssam ~]# rpm -Uvh --test http://satprapa.turner.com/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Retrieving http://satprapa.turner.com/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Preparing...                ########################################### [100%]
[root@unxtssam ~]# rpm -Uvh http://satprapa.turner.com/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Retrieving http://satprapa.turner.com/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:rhn-org-trusted-ssl-cer########################################### [100%]
[root@unxtssam ~]#
[root@unxtssam ~]# vi /etc/sysconfig/rhn/up2date
[root@unxtssam ~]# rhn_register
[root@unxtssam ~]#
[root@unxtssam ~]# cd /var/lib/rpm/
[root@unxtssam rpm]# diff /etc/sysconfig/rhn/up2date.20090325.133126.fcampbell /etc/sysconfig/rhn/up2date
19c19
< serverURL=https://xmlrpc.rhn.redhat.com/XMLRPC
---
> serverURL=https://satprapa.turner.com/XMLRPC
31c31
< sslCACert=/usr/share/rhn/RHNS-CA-CERT
---
> sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
 [root@unxtssam rpm]#
 [root@unxtssam rpm]#
[root@unxtssam rpm]# count=0; while rpm -q rpm > /dev/null; do let count=count+1; echo $count;/usr/libexec/yum-updatesd-helper -c; /usr/lib/rpm/rpmdb_stat -c 2> /dev/null| grep 'Maximum number of lockers'; done
1
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
15      Maximum number of lockers at any one time
.
.
.
496
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
998     Maximum number of lockers at any one time
497
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
1000    Maximum number of lockers at any one time
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 11ad
error: db4 error(22) from db->close: Invalid argument
error: cannot open Name index using db3 - Cannot allocate memory (12)
[root@unxtssam rpm]#

[root@unxtssam rpm]# pwd
/var/lib/rpm
[root@unxtssam rpm]# rm -f __db.00*
[root@unxtssam rpm]# rpm --rebuilddb
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]# yum list installed yum\* rpm\* python glibc
Loading "security" plugin
Loading "rhnplugin" plugin
rhel-i386-server-5        100% |=========================| 1.4 kB    00:00
Installed Packages
glibc.i686                               2.5-24                 installed
python.i386                              2.4.3-21.el5           installed
rpm.i386                                 4.4.2-48.el5           installed
rpm-build.i386                           4.4.2-48.el5           installed
rpm-devel.i386                           4.4.2-48.el5           installed
rpm-libs.i386                            4.4.2-48.el5           installed
rpm-python.i386                          4.4.2-48.el5           installed
yum.noarch                               3.2.8-9.el5            installed
yum-metadata-parser.i386                 1.1.2-2.el5            installed
yum-rhn-plugin.noarch                    0.5.3-6.el5            installed
yum-security.noarch                      1.1.10-9.el5           installed
yum-updatesd.noarch                      1:0.9-2.el5            installed
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]# yum update rpm-4.4.2.3-9.el5
Loading "security" plugin
Loading "rhnplugin" plugin
rhel-i386-server-5        100% |=========================| 1.4 kB    00:00
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
--> Running transaction check
--> Processing Dependency: rpm = 4.4.2-48.el5 for package: rpm-build
--> Processing Dependency: rpm = 4.4.2-48.el5 for package: rpm-devel
--> Processing Dependency: rpm = 4.4.2-48.el5 for package: rpm-libs
--> Processing Dependency: rpm = 4.4.2-48.el5 for package: rpm-python
---> Package rpm.i386 0:4.4.2.3-9.el5 set to be updated
--> Processing Dependency: popt = 1.10.2.3-9.el5 for package: rpm
--> Running transaction check
---> Package rpm-build.i386 0:4.4.2.3-9.el5 set to be updated
---> Package popt.i386 0:1.10.2.3-9.el5 set to be updated
---> Package rpm-libs.i386 0:4.4.2.3-9.el5 set to be updated
---> Package rpm-devel.i386 0:4.4.2.3-9.el5 set to be updated
--> Processing Dependency: nss-devel for package: rpm-devel
---> Package rpm-python.i386 0:4.4.2.3-9.el5 set to be updated
--> Running transaction check
---> Package nss-devel.i386 0:3.12.2.0-4.el5 set to be updated
--> Processing Dependency: nss = 3.12.2.0-4.el5 for package: nss-devel
--> Processing Dependency: nspr-devel >= 4.6.99-0 for package: nss-devel
--> Running transaction check
---> Package nss.i386 0:3.12.2.0-4.el5 set to be updated
--> Processing Dependency: nss = 3.11.99.5-2.el5 for package: nss-tools
---> Package nspr-devel.i386 0:4.7.3-2.el5 set to be updated
--> Processing Dependency: nspr = 4.7.3-2.el5 for package: nspr-devel
--> Running transaction check
---> Package nss-tools.i386 0:3.12.2.0-4.el5 set to be updated
---> Package nspr.i386 0:4.7.3-2.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Updating:
 nspr                    i386       4.7.3-2.el5      rhel-i386-server-5  119 k
 nss                     i386       3.12.2.0-4.el5   rhel-i386-server-5  1.1 M
 popt                    i386       1.10.2.3-9.el5   rhel-i386-server-5   73 k
 rpm                     i386       4.4.2.3-9.el5    rhel-i386-server-5  1.2 M
Installing for dependencies:
 nspr-devel              i386       4.7.3-2.el5      rhel-i386-server-5  115 k
 nss-devel               i386       3.12.2.0-4.el5   rhel-i386-server-5  226 k
Updating for dependencies:
 nss-tools               i386       3.12.2.0-4.el5   rhel-i386-server-5  1.2 M
 rpm-build               i386       4.4.2.3-9.el5    rhel-i386-server-5  303 k
 rpm-devel               i386       4.4.2.3-9.el5    rhel-i386-server-5  1.3 M
 rpm-libs                i386       4.4.2.3-9.el5    rhel-i386-server-5  931 k
 rpm-python              i386       4.4.2.3-9.el5    rhel-i386-server-5   57 k

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       9 Package(s)
Remove       0 Package(s)

Total download size: 6.6 M
Is this ok [y/N]: y
Downloading Packages:
(1/11): nspr-devel-4.7.3- 100% |=========================| 115 kB    00:00
(2/11): nss-devel-3.12.2. 100% |=========================| 226 kB    00:00
(3/11): rpm-python-4.4.2. 100% |=========================|  57 kB    00:00
(4/11): rpm-4.4.2.3-9.el5 100% |=========================| 1.2 MB    00:00
(5/11): rpm-devel-4.4.2.3 100% |=========================| 1.3 MB    00:00
(6/11): nspr-4.7.3-2.el5. 100% |=========================| 119 kB    00:00
(7/11): nss-3.12.2.0-4.el 100% |=========================| 1.1 MB    00:00
(8/11): rpm-libs-4.4.2.3- 100% |=========================| 931 kB    00:00
(9/11): popt-1.10.2.3-9.e 100% |=========================|  73 kB    00:00
(10/11): rpm-build-4.4.2. 100% |=========================| 303 kB    00:00
(11/11): nss-tools-3.12.2 100% |=========================| 1.2 MB    00:00
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186
Importing GPG key 0x37017186 "Red Hat, Inc. (release key) <security@redhat.com>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating  : nspr                         ####################### [ 1/20]
  Updating  : popt                         ####################### [ 2/20]
  Updating  : nss                          ####################### [ 3/20]
  Installing: nspr-devel                   ####################### [ 4/20]
  Installing: nss-devel                    ####################### [ 5/20]
  Updating  : nss-tools                    ####################### [ 6/20]
  Updating  : rpm-libs                     ####################### [ 7/20]
  Updating  : rpm                          ####################### [ 8/20]
  Updating  : rpm-build                    ####################### [ 9/20]
  Updating  : rpm-devel                    ####################### [10/20]
  Updating  : rpm-python                   ####################### [11/20]
  Cleanup   : nss-tools                    ####################### [12/20]
  Cleanup   : rpm-build                    ####################### [13/20]
  Cleanup   : popt                         ####################### [14/20]
  Cleanup   : rpm-libs                     ####################### [15/20]
  Cleanup   : nss                          ####################### [16/20]
  Cleanup   : nspr                         ####################### [17/20]
  Cleanup   : rpm-devel                    ####################### [18/20]
  Cleanup   : rpm                          ####################### [19/20]
  Cleanup   : rpm-python                   ####################### [20/20]

Dependency Installed: nspr-devel.i386 0:4.7.3-2.el5 nss-devel.i386 0:3.12.2.0-4.el5
Updated: nspr.i386 0:4.7.3-2.el5 nss.i386 0:3.12.2.0-4.el5 popt.i386 0:1.10.2.3-9.el5 rpm.i386 0:4.4.2.3-9.el5
Dependency Updated: nss-tools.i386 0:3.12.2.0-4.el5 rpm-build.i386 0:4.4.2.3-9.el5 rpm-devel.i386 0:4.4.2.3-9.el5 rpm-libs.i386 0:4.4.2.3-9.el5 rpm-python.i386 0:4.4.2.3-9.el5
Complete!
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]# yum list installed yum\* rpm\* python glibc
Loading "security" plugin
Loading "rhnplugin" plugin
rhel-i386-server-5        100% |=========================| 1.4 kB    00:00
Installed Packages
glibc.i686                               2.5-24                 installed
python.i386                              2.4.3-21.el5           installed
rpm.i386                                 4.4.2.3-9.el5          installed
rpm-build.i386                           4.4.2.3-9.el5          installed
rpm-devel.i386                           4.4.2.3-9.el5          installed
rpm-libs.i386                            4.4.2.3-9.el5          installed
rpm-python.i386                          4.4.2.3-9.el5          installed
yum.noarch                               3.2.8-9.el5            installed
yum-metadata-parser.i386                 1.1.2-2.el5            installed
yum-rhn-plugin.noarch                    0.5.3-6.el5            installed
yum-security.noarch                      1.1.10-9.el5           installed
yum-updatesd.noarch                      1:0.9-2.el5            installed
[root@unxtssam rpm]#
[root@unxtssam rpm]#
[root@unxtssam rpm]# count=0; while rpm -q rpm > /dev/null; do let count=count+1; echo $count;/usr/libexec/yum-updatesd-helper -c; /usr/lib/rpm/rpmdb_stat -c 2> /dev/null| grep 'Maximum number of lockers'; done
1
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
23      Maximum number of lockers at any one time
.
.
.
814
Loading "rhnplugin" plugin
1000    Maximum number of lockers possible
23      Maximum number of lockers at any one time
[root@unxtssam rpm]#

Note You need to log in before you can comment on or make changes to this bug.