685119 – openldap-servers upgrade hangs or do not upgrade the database

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 685119 - openldap-servers upgrade hangs or do not upgrade the database

Summary: openldap-servers upgrade hangs or do not upgrade the database

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	openldap
Sub Component:
Version:	6.1
Hardware:	Unspecified
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Jan Vcelak
QA Contact:	Ondrej Moriš
Docs Contact:
URL:
Whiteboard:
Depends On:	661682 664433
Blocks:
TreeView+	depends on / blocked

Reported:	2011-03-15 11:11 UTC by Ondrej Moriš
Modified:	2013-03-04 01:28 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openldap-2.4.23-14.el6
Doc Type:	Bug Fix
Doc Text:	- openldap database is corrupted and the slapd is shut down - doing openldap-servers package upgrade cause update process freeze as the openldap tool for exporting the database was not able to handle the situation - updated upgrade script to run database recovery before database export - the upgrade of openldap-servers will no longer cause upgrade process freeze when the database is corrupted
Clone Of:	664433
Environment:
Last Closed:	2011-05-19 14:00:03 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2011:0673	0	normal	SHIPPED_LIVE	openldap bug fix and enhancement update	2011-05-18 18:10:44 UTC

Description Ondrej Moriš 2011-03-15 11:11:17 UTC

+++ This bug was initially created as a clone of Bug #664433 +++

Cloning - I would like to investigate this problem more closely.

+++ This bug was initially created as a clone of Bug #661682 +++

Created attachment 467729 [details]
Anaconda logs

Description of problem:

Uprade from f13->f14 (i386) got stuck at middle of installing packages. I could switch to console, but not back to GUI, since GUI was stuck.

Version-Release number of selected component (if applicable):

I don't know the runtime version of anaconda, I did preupgrade from f13, whichever was the version it happened to use at the time of yesterday.

How reproducible:

I don't know.

Steps to Reproduce:
1. I updated F13
2. I run preupgrade to download the packages and rebooted
3. At boot I had to tell anaconda to use proxies while it asked for it
4. Just let run until got stuck
    
Actual results:

Upgrade GUI got stuck with the latest error message:

15:13:35,562 WARN anaconda: /usr/lib/python2.7/site-packages/pyanaconda/iw/progress_gui.py:67: GtkWarning: Failed to set text from markup due to error parsing markup: Error on line 2 char 32: Element 'markup' was closed, but the currently open element is 'NAME'
  self.infolabel.set_markup(txt)



Expected results:

Upgrade to finnish the package installs.

Additional info:

See attachments for more logs

--- Additional comment from clumens on 2010-12-09 15:16:02 CET ---

You appear to be hung up on the package scriptlet for this package:

Upgrading openldap-servers-2.4.23-4.fc14.i686
bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
Expect poor performance for suffix "dc=my-domain,dc=com".

How long did you let it sit?  There's really nothing anaconda can do here (except redo the UI to actually refresh, but ugh).

--- Additional comment from ilkka.tengvall on 2010-12-10 14:43:39 CET ---

It was sitting there over night. I haven't even configured openldap-servers in my system, too bad the default settings fail the upgrade.

--- Additional comment from jvcelak on 2011-01-06 14:19:57 EST ---

Ilkka,

was your F13 up-to-date before the upgrade? Do you know which version of openldap you had?

I haven't encountered any problem when upgrading from  openldap-2.4.21-11.fc13 to openldap-servers-2.4.23-4.fc14. I tried upgrading with database filled with quite a lot of data, with empty database and even with not-created database. Everything seems to be fine.

Jan

--- Additional comment from ilkka.tengvall on 2011-01-07 02:45:50 EST ---

I still have the /var lvm snapshot still available. It seems the last version of updated ldap was:

Aug 24 15:49:29 Updated: openldap-2.4.21-10.fc13.x86_64
Aug 24 15:50:21 Updated: openldap-clients-2.4.21-10.fc13.x86_64
Aug 24 15:50:45 Updated: openldap-2.4.21-10.fc13.i686
Aug 24 15:52:38 Updated: openldap-devel-2.4.21-10.fc13.x86_64

Hmmm, that's weird, I seem to have both 64/32 bit versions installed. I have Wind River environment on this PC, and it requires certain packages for 32 bit also. I don't see another reason for that.

and yes, the F13 was up-to-date.

And I had nothing in ldap database, it was unused. I installed ldap stuff to debug something, but I was not using the server for anything real, never did anything with the database.

--- Additional comment from jvcelak on 2011-01-07 07:14:20 EST ---

I managed to reproduce this by damaging the database a little bit before the upgrade. (You might not even know, if you weren't using openldap-servers. Unclean server shutdown is also possible.) In fact it is the same problem as here (bug 667768). It seems that slapd tools (slaptest, slapcat, slapadd, ...) get stuck if the database is inconsistent.

Unfortunately it can not be fixed as easily as the previous bug. There is no way to check that database is OK before doing the database export (which is done during the upgrade).

This is definitely a bug, not very serious one. I suppose that nobody with damaged ldap database will run package upgrade knowingly. I will investigate this more and try to find some solution.

Ilkka, Thanks for reporting. :)

--- Additional comment from ilkka.tengvall on 2011-01-07 07:35:17 EST ---

heh, thanks, good to know it wasn't only in between my ears after all :D !

--- Additional comment from jvcelak on 2011-01-20 14:02:42 EST ---

Can be fixed by adding 'db_recover' before database migration.

--- Additional comment from jvcelak on 2011-01-25 08:32:49 EST ---

Fixed in openldap-2.4.23-7.fc14

--- Additional comment from updates on 2011-01-25 08:35:06 EST ---

openldap-2.4.23-7.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/openldap-2.4.23-7.fc14

--- Additional comment from updates on 2011-01-25 16:02:01 EST ---

openldap-2.4.23-7.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update openldap'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/openldap-2.4.23-7.fc14

--- Additional comment from updates on 2011-02-02 07:43:15 EST ---

openldap-2.4.23-8.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/openldap-2.4.23-8.fc14

--- Additional comment from updates on 2011-03-01 07:41:58 EST ---

Package openldap-2.4.21-12.fc13:
* should fix your issue,
* was pushed to the Fedora 13 updates-testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openldap-2.4.21-12.fc13'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/openldap-2.4.21-12.fc13
then log in and leave karma (feedback).

--- Additional comment from updates on 2011-03-01 07:42:45 EST ---

Package openldap-2.4.23-9.fc14:
* should fix your issue,
* was pushed to the Fedora 14 updates-testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/openldap-2.4.23-9.fc14
then log in and leave karma (feedback).

--- Additional comment from ilkka.tengvall on 2011-03-02 04:52:36 EST ---

thanks for solving this. I don't though know what to put into karma, since I don't want to test it by re-installing old f13 over my existing f14 and upgrade it back to f14. I can test it though if you tell me what you did to break the db in purpose to get it stuck, and a way to repeat the error. Only if you want me to, I trust your tests though.

--- Additional comment from ilkka.tengvall on 2011-03-02 05:10:52 EST ---

Sorry, I was too hasty... The upgrade got stuck:

--&<-----------------------------------------------------------
[itengval@pikkud ~]$ sudo yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14
Loaded plugins: presto, refresh-packagekit
updates-testing/metalink                                 |  17 kB     00:00     
updates-testing                                          | 4.7 kB     00:00     
updates-testing/primary_db                               | 790 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-devel-2.4.23-4.fc14.i686
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-clients-2.4.23-4.fc14.i686
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-servers-2.4.23-4.fc14.i686
---> Package openldap.i686 0:2.4.23-9.fc14 set to be updated
--> Running transaction check
---> Package openldap-clients.i686 0:2.4.23-9.fc14 set to be updated
---> Package openldap-devel.i686 0:2.4.23-9.fc14 set to be updated
---> Package openldap-servers.i686 0:2.4.23-9.fc14 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch      Version             Repository            Size
================================================================================
Updating:
 openldap              i686      2.4.23-9.fc14       updates-testing      256 k
Updating for dependencies:
 openldap-clients      i686      2.4.23-9.fc14       updates-testing      157 k
 openldap-devel        i686      2.4.23-9.fc14       updates-testing      1.1 M
 openldap-servers      i686      2.4.23-9.fc14       updates-testing      2.0 M

Transaction Summary
================================================================================
Upgrade       4 Package(s)

Total download size: 3.5 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates-testing/prestodelta                              |  22 kB     00:00     
Processing delta metadata
Package(s) data still to download: 3.5 M
(1/4): openldap-2.4.23-9.fc14.i686.rpm                   | 256 kB     00:00     
(2/4): openldap-clients-2.4.23-9.fc14.i686.rpm           | 157 kB     00:00     
(3/4): openldap-devel-2.4.23-9.fc14.i686.rpm             | 1.1 MB     00:00     
(4/4): openldap-servers-2.4.23-9.fc14.i686.rpm           | 2.0 MB     00:00     
--------------------------------------------------------------------------------
Total                                           5.5 MB/s | 3.5 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : openldap-2.4.23-9.fc14.i686                              1/8 
  Updating       : openldap-clients-2.4.23-9.fc14.i686                      2/8 
  Updating       : openldap-servers-2.4.23-9.fc14.i686                      3/8 
--&<-----------------------------------------------------------

Looking simultaneously at the yum log

--&<-----------------------------------------------------------
Mar 01 14:41:05 Updated: ruby-libs-1.8.7.334-1.fc14.i686
Mar 01 14:41:05 Installed: nss-softokn-freebl-devel-3.12.9-5.fc14.i686
Mar 01 14:41:05 Updated: nss-softokn-devel-3.12.9-5.fc14.i686
Mar 01 14:41:05 Updated: nss-devel-3.12.9-8.fc14.i686
Mar 02 11:43:09 Updated: gpxe-roms-qemu-1.0.1-3.fc14.noarch
Mar 02 11:43:09 Updated: 1:NetworkManager-glib-0.8.3.996-1.fc14.i686
Mar 02 11:43:13 Updated: 1:NetworkManager-0.8.3.996-1.fc14.i686
Mar 02 11:43:24 Updated: 1:NetworkManager-gnome-0.8.3.996-1.fc14.i686
Mar 02 11:44:13 Updated: openldap-2.4.23-9.fc14.i686
Mar 02 11:44:13 Updated: openldap-clients-2.4.23-9.fc14.i686
--&<-----------------------------------------------------------

So there is no log of openldap-servers. It's been 20 minutes now stuck there. And DB should be rather empty, I don't use it for anything.

You can here see from ps awfux what it's doing:

--&<-----------------------------------------------------------
root      4191  0.0  0.0   8332  1596 pts/0    S+   11:43   0:00  |   \_ sudo yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14
root      4192  0.9  3.2 124196 100104 pts/0   S+   11:43   0:11  |       \_ /usr/bin/python /usr/bin/yum update --enablerepo=updates-testing openldap-2.4.23
root      4248  0.0  0.0   5084  1092 pts/0    S+   11:44   0:00  |           \_ /bin/sh /home/itengval/rpmbuild/rpm/tmp/rpm-tmp.5uZ0N8 2
root      4252  0.0  0.0   5748  1136 pts/0    S+   11:44   0:00  |               \_ runuser -m -s /usr/sbin/slapadd -- ldap -q -l /var/lib/ldap/upgrade.ldif
ldap      4253  0.0  0.1  13832  3300 pts/0    Sl+  11:44   0:00  |                   \_ slapadd -q -l /var/lib/ldap/upgrade.ldif
--&<-----------------------------------------------------------


and here is what the ldap is having open (lsof -p 4253):

--&<-----------------------------------------------------------
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
slapadd 4253 ldap  cwd    DIR  253,1     4096      2 /
slapadd 4253 ldap  rtd    DIR  253,1     4096      2 /
slapadd 4253 ldap  txt    REG  253,1  2241560  26958 /usr/sbin/slapd
slapadd 4253 ldap  mem    REG  253,1  1577412  26181 /lib/libdb-4.8.so
slapadd 4253 ldap  mem    REG  253,1   105200  18360 /lib/libresolv-2.13.so
slapadd 4253 ldap  mem    REG  253,1  1274636  14000 /usr/lib/libnss3.so
slapadd 4253 ldap  mem    REG  253,1   107528  70518 /usr/lib/libnssutil3.so
slapadd 4253 ldap  mem    REG  253,1    12164  70427 /lib/libplds4.so
slapadd 4253 ldap  mem    REG  253,1    16676  69396 /lib/libplc4.so
slapadd 4253 ldap  mem    REG  253,1   240412  17322 /lib/libnspr4.so
slapadd 4253 ldap  mem    REG  253,1   133344  18349 /lib/libpthread-2.13.so
slapadd 4253 ldap  mem    REG  253,1    19776  14990 /lib/libdl-2.13.so
slapadd 4253 ldap  mem    REG  253,1    84848  17743 /lib/libz.so.1.2.5
slapadd 4253 ldap  mem    REG  253,1   169060  14538 /usr/lib/libsmime3.so
slapadd 4253 ldap  mem    REG  253,1   216536  14705 /usr/lib/libssl3.so
slapadd 4253 ldap  mem    REG  253,1   298084  12682 /lib/libfreebl3.so
slapadd 4253 ldap  mem    REG  253,1    36132  19914 /lib/libcrypt-2.13.so
slapadd 4253 ldap  mem    REG  253,1   137160  14144 /lib/ld-2.13.so
slapadd 4253 ldap  mem    REG  253,1  1847224  16957 /lib/libc-2.13.so
slapadd 4253 ldap  mem    REG  253,1   102740  20866 /usr/lib/libsasl2.so.2.0.23
slapadd 4253 ldap  mem    REG  253,1    36852  61664 /usr/lib/libltdl.so.7.2.2
slapadd 4253 ldap    0r  FIFO    0,8      0t0 164117 pipe
slapadd 4253 ldap    1w   CHR    1,3      0t0   4032 /dev/null
slapadd 4253 ldap    2u   REG  253,1      138  17497 /tmp/tmpcAgZ4K
slapadd 4253 ldap    3r   REG  253,1        0 218516 /var/lib/ldap/upgrade.ldif
--&<-----------------------------------------------------------


this is the temp file content:

--&<-----------------------------------------------------------
$ sudo cat /tmp/tmpcAgZ4K
bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
Expect poor performance for suffix "dc=my-domain,dc=com".
--&<-----------------------------------------------------------

and notice the file upgrade.ldif is empty

Any additional info you would like to get from the environment (my work laptop)?

br,

 it

Comment 2 Jan Vcelak 2011-03-16 14:31:12 UTC

Renaming this bug because a very related issue was discovered:

Upgrade from old openldap-servers with embedded db4 library were unable to migrate the database. After upgrade, slapd exitted a few moments after startup. Recovery was possible only by downgrading the package, exporting the database, deleting it, upgrading the package, and reimporting the content.

Comment 15 Jan Vcelak 2011-04-20 17:29:18 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
- openldap database is corrupted and the slapd is shut down
- doing openldap-servers package upgrade cause update process freeze as the openldap tool for exporting the database was not able to handle the situation
- updated upgrade script to run database recovery before database export
- the upgrade of openldap-servers will no longer cause upgrade process freeze when the database is corrupted

Comment 16 errata-xmlrpc 2011-05-19 14:00:03 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0673.html

Note You need to log in before you can comment on or make changes to this bug.