Bug 1529442

Summary: Upgrade of DS database from Fedora 26 version to Fedora 27 version.
Product: [Fedora] Fedora Reporter: stephen.hocking
Component: 389-ds-baseAssignee: mreynolds
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 27CC: chris, dgilmore, mreynolds, rmeggins, vashirov
Target Milestone: ---   
Target Release: ---   
Hardware: armv7hl   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-10 16:09:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description stephen.hocking 2017-12-28 06:08:21 UTC
Description of problem:389 DS database upgrade fails with undefined symbol.


Version-Release number of selected component (if applicable):1.3.7.8


How reproducible: Straightforward, but tedious.


Steps to Reproduce:
1. Set up FreeIPA under Fedora 26
2. Upgrade machine in place to Fedora 27
3. Profit!

Actual results:
DS database fails to upgrade, preventing directory service from starting.


Expected results:
DS database properly upgraded

Additional info:
Dec 28 16:41:29 ldap02.craftncomp.com systemd[1]: Starting 389 Directory Server CRAFTNCOMP-COM....
Dec 28 16:41:32 ldap02.craftncomp.com ns-slapd[6073]: [28/Dec/2017:16:41:32.652510725 +1100] - NOTICE - config_set_port - Non-Secure Port Disabled
Dec 28 16:41:32 ldap02.craftncomp.com ns-slapd[6073]: [28/Dec/2017:16:41:32.957764725 +1100] - ERR - symload_report_error - Netscape Portable Runtime error -5975: /usr/lib/dirsrv/plugins/libreplication-plugin.so: undefined symbol: replication_legacy_plugin_init
Dec 28 16:41:32 ldap02.craftncomp.com ns-slapd[6073]: [28/Dec/2017:16:41:32.962107391 +1100] - ERR - symload_report_error - Could not load symbol "replication_legacy_plugin_init" from "/usr/lib/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy Replication Plugin
Dec 28 16:41:32 ldap02.craftncomp.com ns-slapd[6073]: [28/Dec/2017:16:41:32.966819058 +1100] - ERR - load_plugin_entry - Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"

Issue does not exist on other architecture that I have access to (x86_64)

Comment 1 mreynolds 2018-02-13 22:11:37 UTC
It appears the 389-ds-base upgrade scripts were not run because the upgrade itself failed:

Updating instance (slapd-AUSIL-US)...
Error: could not parse nsstate 00f38b11d21db201d28880b54b3f89010100000000000000 - tsdiff is 6882719651.34442 seconds or 79661.1070757456 days
[18/02/13:22:07:33] - [Setup] Info Error: could not parse nsState from cn=uniqueid generator,cn=config.  Value: 00f38b11d21db201d28880b54b3f89010100000000000000
[18/02/13:22:07:33] - [Setup] Fatal Error: could not update the directory server.
[18/02/13:22:07:33] - [Setup] Fatal Exiting . . .
Log file is '/tmp/setup_Axodi.log'

Comment 3 mreynolds 2018-02-13 22:28:11 UTC
Can I please get a copy of the dse.ldif ?  Thanks!

Comment 5 Dennis Gilmore 2018-02-13 22:58:37 UTC
What is the best way to sanitise the dse.ldif file for public consumption?

Comment 6 mreynolds 2018-02-13 23:38:51 UTC
I just want to see the nsstate attributes from the config file:

For example:

dn: cn=uniqueid generator,cn=config
...
nsState:: AOTX9AgR6AH8RL9OzdYuUQEAAAAAAAAA


dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
...
nsState:: AQAAAAAAAACGX4NaAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAA==


I would also be interested to see what is in the DS errors log from the time of the upgrade.

Also, if you install a new instance does it succeed and start up?

Thanks,
Mark

Comment 8 Dennis Gilmore 2018-02-14 18:47:28 UTC
[14/Feb/2018:14:17:56.205106134 +0000] - NOTICE - config_set_port - Non-Secure Port Disabled
[14/Feb/2018:14:17:56.708422560 +0000] - ERR - symload_report_error - Netscape Portable Runtime error -5975: /usr/lib/dirsrv/plugins/libreplication-plugin.so: undefined symbol: replication_legacy_plugin_init
[14/Feb/2018:14:17:56.736406686 +0000] - ERR - symload_report_error - Could not load symbol "replication_legacy_plugin_init" from "/usr/lib/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy Replication Plugin
[14/Feb/2018:14:17:56.762026971 +0000] - ERR - load_plugin_entry - Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
[14/Feb/2018:18:39:48.410643462 +0000] - NOTICE - config_set_port - Non-Secure Port Disabled
[14/Feb/2018:18:39:48.645801750 +0000] - ERR - symload_report_error - Netscape Portable Runtime error -5975: /usr/lib/dirsrv/plugins/libreplication-plugin.so: undefined symbol: replication_legacy_plugin_init
[14/Feb/2018:18:39:48.650509431 +0000] - ERR - symload_report_error - Could not load symbol "replication_legacy_plugin_init" from "/usr/lib/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy Replication Plugin
[14/Feb/2018:18:39:48.654962903 +0000] - ERR - load_plugin_entry - Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"


that is the output in the errors log file. I can try setup a new ipa replica and see if it works

Comment 9 mreynolds 2018-02-14 19:10:25 UTC
Unfortunately it looks like these logs are not from the upgrade attempt, but from the server just starting up.  However, I finally got a arm lab machine, and I was able to reproduce the issue by just running "setup.pl -u".


# setup.pl -u
...
...
Full DN of administrative user [cn=Directory Manager]: 
Password for this user: 
Updating instance (slapd-host)...
Error: could not parse nsstate 002623afd11db2019284abee1150f11a0000000000000000 - tsdiff is 12664395670.0362 seconds or 146578.653588382 days
Error: could not parse nsState from cn=uniqueid generator,cn=config.  Value: 002623afd11db2019284abee1150f11a0000000000000000
Error: could not update the directory server.
Exiting . . .
Log file is '/tmp/setupAkZWCf.log'



Continuing investigation...

Comment 10 mreynolds 2018-02-14 23:42:50 UTC
Problem identified (integer overflow), and patch is out for review...

Comment 11 mreynolds 2018-05-10 16:09:26 UTC
This has been fixed

Comment 12 chris 2019-08-02 20:17:35 UTC
So I know this bug is supposedly 'fixed', but I'm experiencing this exact same problem when upgrading an existing IPA server from 4.4.4 to 4.7.

I have an existing Fedora 26 system running freeipa 4.4.4.  When I upgrade it to any newer edition however, freeipa breaks during the upgrade process every time and at every newer revision, up to Fedora 30 and freeipa 4.7 with the following error condition:
=====
ns-slapd[1293]: [01/Aug/2019:17:26:22.138679916 -0700] - ERR - symload_report_error - Could not load symbol "replication_legacy_plugin_init" from "/usr/lib64/dirsrv/plugins/libreplication-plugin.so"
=====

Makes no difference if I'm upgrading to Fedora 27, 28, 29, or 30 (or the related freeipa revision that's available for each distro version).

Since this issue is stated to be closed referring to the 'currentrelease', I am in search of some assistance in understanding and remedying a freeipa upgrade to 'currentrelease' since every rev up to and including 4.7 is still exhibiting this same behavior.

Any insights or assistance is greatly appreciated.

Thanks,
-Chris