Bug 664433 - openldap-servers upgrade hangs or do not upgrade the database
Summary: openldap-servers upgrade hangs or do not upgrade the database
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openldap
Version: 14
Hardware: i386
OS: Linux
low
medium
Target Milestone: ---
Assignee: Jan Vcelak
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On: 661682
Blocks: 685119
TreeView+ depends on / blocked
 
Reported: 2010-12-20 12:13 UTC by Jan Vcelak
Modified: 2013-03-04 01:28 UTC (History)
6 users (show)

(edit)
Clone Of: 661682
: 685119 (view as bug list)
(edit)
Last Closed: 2011-04-21 05:33:15 UTC


Attachments (Terms of Use)

Description Jan Vcelak 2010-12-20 12:13:00 UTC
Cloning - I would like to investigate this problem more closely.

+++ This bug was initially created as a clone of Bug #661682 +++

Created attachment 467729 [details]
Anaconda logs

Description of problem:

Uprade from f13->f14 (i386) got stuck at middle of installing packages. I could switch to console, but not back to GUI, since GUI was stuck.

Version-Release number of selected component (if applicable):

I don't know the runtime version of anaconda, I did preupgrade from f13, whichever was the version it happened to use at the time of yesterday.

How reproducible:

I don't know.

Steps to Reproduce:
1. I updated F13
2. I run preupgrade to download the packages and rebooted
3. At boot I had to tell anaconda to use proxies while it asked for it
4. Just let run until got stuck
    
Actual results:

Upgrade GUI got stuck with the latest error message:

15:13:35,562 WARN anaconda: /usr/lib/python2.7/site-packages/pyanaconda/iw/progress_gui.py:67: GtkWarning: Failed to set text from markup due to error parsing markup: Error on line 2 char 32: Element 'markup' was closed, but the currently open element is 'NAME'
  self.infolabel.set_markup(txt)



Expected results:

Upgrade to finnish the package installs.

Additional info:

See attachments for more logs

--- Additional comment from clumens@redhat.com on 2010-12-09 15:16:02 CET ---

You appear to be hung up on the package scriptlet for this package:

Upgrading openldap-servers-2.4.23-4.fc14.i686
bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
Expect poor performance for suffix "dc=my-domain,dc=com".

How long did you let it sit?  There's really nothing anaconda can do here (except redo the UI to actually refresh, but ugh).

--- Additional comment from ilkka.tengvall@nsn.com on 2010-12-10 14:43:39 CET ---

It was sitting there over night. I haven't even configured openldap-servers in my system, too bad the default settings fail the upgrade.

Comment 1 Jan Vcelak 2011-01-06 19:19:57 UTC
Ilkka,

was your F13 up-to-date before the upgrade? Do you know which version of openldap you had?

I haven't encountered any problem when upgrading from  openldap-2.4.21-11.fc13 to openldap-servers-2.4.23-4.fc14. I tried upgrading with database filled with quite a lot of data, with empty database and even with not-created database. Everything seems to be fine.

Jan

Comment 2 Ilkka Tengvall 2011-01-07 07:45:50 UTC
I still have the /var lvm snapshot still available. It seems the last version of updated ldap was:

Aug 24 15:49:29 Updated: openldap-2.4.21-10.fc13.x86_64
Aug 24 15:50:21 Updated: openldap-clients-2.4.21-10.fc13.x86_64
Aug 24 15:50:45 Updated: openldap-2.4.21-10.fc13.i686
Aug 24 15:52:38 Updated: openldap-devel-2.4.21-10.fc13.x86_64

Hmmm, that's weird, I seem to have both 64/32 bit versions installed. I have Wind River environment on this PC, and it requires certain packages for 32 bit also. I don't see another reason for that.

and yes, the F13 was up-to-date.

And I had nothing in ldap database, it was unused. I installed ldap stuff to debug something, but I was not using the server for anything real, never did anything with the database.

Comment 3 Jan Vcelak 2011-01-07 12:14:20 UTC
I managed to reproduce this by damaging the database a little bit before the upgrade. (You might not even know, if you weren't using openldap-servers. Unclean server shutdown is also possible.) In fact it is the same problem as here (bug 667768). It seems that slapd tools (slaptest, slapcat, slapadd, ...) get stuck if the database is inconsistent.

Unfortunately it can not be fixed as easily as the previous bug. There is no way to check that database is OK before doing the database export (which is done during the upgrade).

This is definitely a bug, not very serious one. I suppose that nobody with damaged ldap database will run package upgrade knowingly. I will investigate this more and try to find some solution.

Ilkka, Thanks for reporting. :)

Comment 4 Ilkka Tengvall 2011-01-07 12:35:17 UTC
heh, thanks, good to know it wasn't only in between my ears after all :D !

Comment 5 Jan Vcelak 2011-01-20 19:02:42 UTC
Can be fixed by adding 'db_recover' before database migration.

Comment 6 Jan Vcelak 2011-01-25 13:32:49 UTC
Fixed in openldap-2.4.23-7.fc14

Comment 7 Fedora Update System 2011-01-25 13:35:06 UTC
openldap-2.4.23-7.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/openldap-2.4.23-7.fc14

Comment 8 Fedora Update System 2011-01-25 21:02:01 UTC
openldap-2.4.23-7.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update openldap'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/openldap-2.4.23-7.fc14

Comment 9 Fedora Update System 2011-02-02 12:43:15 UTC
openldap-2.4.23-8.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/openldap-2.4.23-8.fc14

Comment 10 Fedora Update System 2011-03-01 12:41:58 UTC
Package openldap-2.4.21-12.fc13:
* should fix your issue,
* was pushed to the Fedora 13 updates-testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openldap-2.4.21-12.fc13'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/openldap-2.4.21-12.fc13
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2011-03-01 12:42:45 UTC
Package openldap-2.4.23-9.fc14:
* should fix your issue,
* was pushed to the Fedora 14 updates-testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/openldap-2.4.23-9.fc14
then log in and leave karma (feedback).

Comment 12 Ilkka Tengvall 2011-03-02 09:52:36 UTC
thanks for solving this. I don't though know what to put into karma, since I don't want to test it by re-installing old f13 over my existing f14 and upgrade it back to f14. I can test it though if you tell me what you did to break the db in purpose to get it stuck, and a way to repeat the error. Only if you want me to, I trust your tests though.

Comment 13 Ilkka Tengvall 2011-03-02 10:10:52 UTC
Sorry, I was too hasty... The upgrade got stuck:

--&<-----------------------------------------------------------
[itengval@pikkud ~]$ sudo yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14
Loaded plugins: presto, refresh-packagekit
updates-testing/metalink                                 |  17 kB     00:00     
updates-testing                                          | 4.7 kB     00:00     
updates-testing/primary_db                               | 790 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-devel-2.4.23-4.fc14.i686
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-clients-2.4.23-4.fc14.i686
--> Processing Dependency: openldap = 2.4.23-4.fc14 for package: openldap-servers-2.4.23-4.fc14.i686
---> Package openldap.i686 0:2.4.23-9.fc14 set to be updated
--> Running transaction check
---> Package openldap-clients.i686 0:2.4.23-9.fc14 set to be updated
---> Package openldap-devel.i686 0:2.4.23-9.fc14 set to be updated
---> Package openldap-servers.i686 0:2.4.23-9.fc14 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch      Version             Repository            Size
================================================================================
Updating:
 openldap              i686      2.4.23-9.fc14       updates-testing      256 k
Updating for dependencies:
 openldap-clients      i686      2.4.23-9.fc14       updates-testing      157 k
 openldap-devel        i686      2.4.23-9.fc14       updates-testing      1.1 M
 openldap-servers      i686      2.4.23-9.fc14       updates-testing      2.0 M

Transaction Summary
================================================================================
Upgrade       4 Package(s)

Total download size: 3.5 M
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
updates-testing/prestodelta                              |  22 kB     00:00     
Processing delta metadata
Package(s) data still to download: 3.5 M
(1/4): openldap-2.4.23-9.fc14.i686.rpm                   | 256 kB     00:00     
(2/4): openldap-clients-2.4.23-9.fc14.i686.rpm           | 157 kB     00:00     
(3/4): openldap-devel-2.4.23-9.fc14.i686.rpm             | 1.1 MB     00:00     
(4/4): openldap-servers-2.4.23-9.fc14.i686.rpm           | 2.0 MB     00:00     
--------------------------------------------------------------------------------
Total                                           5.5 MB/s | 3.5 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : openldap-2.4.23-9.fc14.i686                              1/8 
  Updating       : openldap-clients-2.4.23-9.fc14.i686                      2/8 
  Updating       : openldap-servers-2.4.23-9.fc14.i686                      3/8 
--&<-----------------------------------------------------------

Looking simultaneously at the yum log

--&<-----------------------------------------------------------
Mar 01 14:41:05 Updated: ruby-libs-1.8.7.334-1.fc14.i686
Mar 01 14:41:05 Installed: nss-softokn-freebl-devel-3.12.9-5.fc14.i686
Mar 01 14:41:05 Updated: nss-softokn-devel-3.12.9-5.fc14.i686
Mar 01 14:41:05 Updated: nss-devel-3.12.9-8.fc14.i686
Mar 02 11:43:09 Updated: gpxe-roms-qemu-1.0.1-3.fc14.noarch
Mar 02 11:43:09 Updated: 1:NetworkManager-glib-0.8.3.996-1.fc14.i686
Mar 02 11:43:13 Updated: 1:NetworkManager-0.8.3.996-1.fc14.i686
Mar 02 11:43:24 Updated: 1:NetworkManager-gnome-0.8.3.996-1.fc14.i686
Mar 02 11:44:13 Updated: openldap-2.4.23-9.fc14.i686
Mar 02 11:44:13 Updated: openldap-clients-2.4.23-9.fc14.i686
--&<-----------------------------------------------------------

So there is no log of openldap-servers. It's been 20 minutes now stuck there. And DB should be rather empty, I don't use it for anything.

You can here see from ps awfux what it's doing:

--&<-----------------------------------------------------------
root      4191  0.0  0.0   8332  1596 pts/0    S+   11:43   0:00  |   \_ sudo yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14
root      4192  0.9  3.2 124196 100104 pts/0   S+   11:43   0:11  |       \_ /usr/bin/python /usr/bin/yum update --enablerepo=updates-testing openldap-2.4.23
root      4248  0.0  0.0   5084  1092 pts/0    S+   11:44   0:00  |           \_ /bin/sh /home/itengval/rpmbuild/rpm/tmp/rpm-tmp.5uZ0N8 2
root      4252  0.0  0.0   5748  1136 pts/0    S+   11:44   0:00  |               \_ runuser -m -s /usr/sbin/slapadd -- ldap -q -l /var/lib/ldap/upgrade.ldif
ldap      4253  0.0  0.1  13832  3300 pts/0    Sl+  11:44   0:00  |                   \_ slapadd -q -l /var/lib/ldap/upgrade.ldif
--&<-----------------------------------------------------------


and here is what the ldap is having open (lsof -p 4253):

--&<-----------------------------------------------------------
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
slapadd 4253 ldap  cwd    DIR  253,1     4096      2 /
slapadd 4253 ldap  rtd    DIR  253,1     4096      2 /
slapadd 4253 ldap  txt    REG  253,1  2241560  26958 /usr/sbin/slapd
slapadd 4253 ldap  mem    REG  253,1  1577412  26181 /lib/libdb-4.8.so
slapadd 4253 ldap  mem    REG  253,1   105200  18360 /lib/libresolv-2.13.so
slapadd 4253 ldap  mem    REG  253,1  1274636  14000 /usr/lib/libnss3.so
slapadd 4253 ldap  mem    REG  253,1   107528  70518 /usr/lib/libnssutil3.so
slapadd 4253 ldap  mem    REG  253,1    12164  70427 /lib/libplds4.so
slapadd 4253 ldap  mem    REG  253,1    16676  69396 /lib/libplc4.so
slapadd 4253 ldap  mem    REG  253,1   240412  17322 /lib/libnspr4.so
slapadd 4253 ldap  mem    REG  253,1   133344  18349 /lib/libpthread-2.13.so
slapadd 4253 ldap  mem    REG  253,1    19776  14990 /lib/libdl-2.13.so
slapadd 4253 ldap  mem    REG  253,1    84848  17743 /lib/libz.so.1.2.5
slapadd 4253 ldap  mem    REG  253,1   169060  14538 /usr/lib/libsmime3.so
slapadd 4253 ldap  mem    REG  253,1   216536  14705 /usr/lib/libssl3.so
slapadd 4253 ldap  mem    REG  253,1   298084  12682 /lib/libfreebl3.so
slapadd 4253 ldap  mem    REG  253,1    36132  19914 /lib/libcrypt-2.13.so
slapadd 4253 ldap  mem    REG  253,1   137160  14144 /lib/ld-2.13.so
slapadd 4253 ldap  mem    REG  253,1  1847224  16957 /lib/libc-2.13.so
slapadd 4253 ldap  mem    REG  253,1   102740  20866 /usr/lib/libsasl2.so.2.0.23
slapadd 4253 ldap  mem    REG  253,1    36852  61664 /usr/lib/libltdl.so.7.2.2
slapadd 4253 ldap    0r  FIFO    0,8      0t0 164117 pipe
slapadd 4253 ldap    1w   CHR    1,3      0t0   4032 /dev/null
slapadd 4253 ldap    2u   REG  253,1      138  17497 /tmp/tmpcAgZ4K
slapadd 4253 ldap    3r   REG  253,1        0 218516 /var/lib/ldap/upgrade.ldif
--&<-----------------------------------------------------------


this is the temp file content:

--&<-----------------------------------------------------------
$ sudo cat /tmp/tmpcAgZ4K
bdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
Expect poor performance for suffix "dc=my-domain,dc=com".
--&<-----------------------------------------------------------

and notice the file upgrade.ldif is empty

Any additional info you would like to get from the environment (my work laptop)?

br,

 it

Comment 14 Jan Vcelak 2011-03-18 20:12:49 UTC
If upgrade.ldif is empty, then the recovery tool was unable to fix the broken database (1). Or: the database is in the old format which cannot be opened by current version of db4 tools (2).

The first possibility is very unlikely. But the second one probably happened. I found another nasty bug in the upgrade script - the database is not migrated if we the upgrade is done from OpenLDAP with embedded DB4 to some newer version.

If you were not using openldap-servers, you probably didn't notice it. And the problem appeared now (after some improvements in the upgrade script).

I rewrote the script to be able to handle this situation without damaging the data and without a freeze during the package upgrade. I will push the update soon.

Comment 15 Jan Vcelak 2011-03-18 22:06:55 UTC
Hm... I was not entirely right. The problem didn't disappear entirely.

I found this issue:
http://www.openldap.org/its/index.cgi?findid=6853

And upstream fix seems to work. :-)

Comment 16 Jan Vcelak 2011-03-19 00:14:49 UTC
Fixed in openldap-2.4.23-10.fc14

Comment 17 Fedora Update System 2011-03-19 00:20:52 UTC
openldap-2.4.23-10.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/openldap-2.4.23-10.fc14

Comment 18 Fedora Update System 2011-03-19 00:23:47 UTC
openldap-2.4.24-2.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/openldap-2.4.24-2.fc15

Comment 19 Fedora Update System 2011-04-21 05:32:49 UTC
openldap-2.4.24-2.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2011-09-25 03:46:23 UTC
openldap-2.4.23-10.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.