Bug 1130252

Also, if I try to start manually after upgrade, it fails with:

[root@rhel6-1 ~]# service dirsrv start
Starting dirsrv: 
    PKI-IPA.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    TESTRELM-TEST.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
  *** Error: 2 instance(s) failed to start


I think this may be a duplicate of bug#1112581

going to modify the upgrade and see if this is resolved.

No, I'm still seeing the same results after adding pki-ca to the upgrade:

[root@rhel6-2 ~]# rpm -q ipa-server pki-ca 389-ds-base
ipa-server-3.0.0-42.el6.x86_64
pki-ca-9.0.3-37.el6.noarch
389-ds-base-1.2.11.15-39.el6.x86_64

From messages I see the same results:

Aug 14 12:52:24 rhel6-2 yum[4303]: Updated: 389-ds-base-libs-1.2.11.15-39.el6.x86_64
Aug 14 12:52:32 rhel6-2 named[4051]: LDAP error: Can't contact LDAP server
Aug 14 12:52:32 rhel6-2 named[4051]: connection to the LDAP server was lost
Aug 14 12:52:32 rhel6-2 named[4051]: bind to LDAP server failed: Can't contact LDAP server
Aug 14 12:52:32 rhel6-2 named[4051]: ldap_psearch_watcher failed to handle LDAP connection error. Recon
nection in 60s
Aug 14 12:52:48 rhel6-2 yum[4303]: Updated: 389-ds-base-1.2.11.15-39.el6.x86_64

If I upgrade openldap, I now get this:

[root@rhel6-2 slapd-TEST-QE]# service dirsrv restart
Shutting down dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TEST-QE...                                             [  OK  ]
Starting dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TEST-QE...[14/Aug/2014:17:17:50 -0500] - Information: Non-Secure Port Disabled
                                                           [  OK  ]

And I can't see the 389 port open now:

[root@rhel6-2 dirsrv]# netstat -taupne |grep 389
tcp        0      0 :::7389                     :::*                        LISTEN      0          56704      17958/ns-slapd      


I still can't start ipa.  How can I track down why slapd isn't starting on port 389?  Is there a logging level I should use?

I'm changing component here to 389-ds-base since that seems to be the main piece here.

(In reply to Scott Poore from comment #4)
> If I upgrade openldap, I now get this:

What do you mean by 'openldap'?  Do you mean you upgraded IPA?

>     PKI-IPA...                                             [  OK  ]
>     TEST-QE...[14/Aug/2014:17:17:50 -0500] - Information: Non-Secure Port
> Disabled

This message is output when nsslapd-port is set to 0, which IPA does during upgrade.  There is likely some failure that IPA is encountering during upgrade, which caused it to bail out before resetting nsslapd-port to 389.

Moving this back to ipa.

(In reply to Nathan Kinder from comment #5)
> This message is output when nsslapd-port is set to 0, which IPA does during
> upgrade.  There is likely some failure that IPA is encountering during
> upgrade, which caused it to bail out before resetting nsslapd-port to 389.

I just read your earlier comments, and I believe that the upgrade failure is likely fallout from the earlier errors encountered by the openldap/389-ds-base mismatch.  You're now dealing with the old port setting that was left behind.

OpenLDAP changed things such that they dropped a library IIRC.  There may be a missing dependency here in 389-ds-base now that I think about it more.  If you repeat your upgrade test from 6.5 -> 6.6, but update openldap manually first, does everything work?

Nathan, 

You are correct.  If I update openldap before or with ipa, it's working cleanly now.

So, for my case, the fix is either:

yum update -y openldap; yum update 'ipa*' sssd

or

yum update -y 'ipa*' sssd openldap

Is this something that can/should be fixed with an updated rpm dependency?

Thanks,
Scott

(In reply to Scott Poore from comment #7)
> Nathan, 
> 
> You are correct.  If I update openldap before or with ipa, it's working
> cleanly now.
> 
> So, for my case, the fix is either:
> 
> yum update -y openldap; yum update 'ipa*' sssd
> 
> or
> 
> yum update -y 'ipa*' sssd openldap
> 
> Is this something that can/should be fixed with an updated rpm dependency?
> 

Quite possibly.  What were the exact old and new versions of openldap in your test?

old:
openldap-2.4.23-32.el6_4.1.x86_64

new:
openldap-2.4.39-8.el6.x86_64

This commit 1270ca4b759cc489bfd25be8d500b6cbce63ab92 introduced @ldap_lib_ldif@, but the substitute was not added to configure.ac.
+AC_SUBST(ldap_lib_ldif)

Since this change was made, libldif was not linked to the directory server.  So, the openldap versions which do not have this change make the directory server fail to start.
    Newer versions of openldap have moved the ldif api into the main libldap
    At configure time we need to see if libldap has the api, or if we need
    to link with a separate libldif

Adding the above line and regenerating configure fixes this problem.  This affects all the supported versions.

Upstream ticket:
https://fedorahosted.org/389/ticket/47875

Steps to verify:

1. downgrade openldap to this version
   openldap-2.4.23-32.el6_4.1.x86_64
2. install rhel-6.6 389-ds-base.
   If it installs fine, the fix is verified.

Note: without the fix, the installation fails with this error.
/usr/bin/pwdhash-bin: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
Could not import LDIF file '/tmp/ldifHKngyV.ldif'.  Error: 32512.  Output: importing data ...
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline

I wonder if IPA should have a clone of this bug. There should be a failsafe in the IPA updater such that it ALWAYS resets the listeners back to their initial values (so 389 and security on)

Actually, this is a very good idea. We want to make upgrade process smoother. I will clone the Bugzilla.

Since it requires specific version of openldap to be downgraded, its not possible to automate in TET. Hence, marking qe_test_coverage- flag.

[root@iceman ~]# setup-ds.pl -u

==============================================================================
This program will update the 389 Directory Server.

It is recommended that you have "root" privilege to perform the update.
Tips for using this  program:
  - Press "Enter" to choose the default and go to the next screen
  - Type "Control-B" or the word "back" then "Enter" to go back to the previous screen
  - Type "Control-C" to cancel the update

Would you like to continue with update? [yes]: 

==============================================================================

The update process can work in one of two modes:

  - Online: The changes are made to the running directory servers using LDAP.
            The operations must be performed as an administrative user.
            You must provide the name and password, for each instance
            if there is more than one instance of directory server.
            Some operations may require a directory server restart to take
            effect.  The update script will notify you if you need to restart
            the server.

  - Offline: The changes are made to the server configuration files.  The
             servers MUST FIRST BE SHUTDOWN BY YOU.  The script will not
             shutdown the servers for you.  You MUST shutdown the
             servers in order to use this mode.  A username and password
             are not required to use Offline mode.  If the servers are not
             shutdown, CHANGES WILL BE LOST.

To summarize:
  Online - servers remain running - you must provide admin name and password
           for each server - servers may need to be restarted
  Offline - servers must be shutdown - no username or password required

Which update mode do you want to use? [quit]: Offline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
Finished successful update of directory server.
Please restart your directory servers.
Exiting . . .
Log file is '/tmp/setupsjKF9J.log'

[root@iceman ~]# service dirsrv restart
Shutting down dirsrv: 
    C2... server already stopped                           [FAILED]
    M1... server already stopped                           [FAILED]
    M2... server already stopped                           [FAILED]
    TestInst1... server already stopped                    [FAILED]
  *** Error: 4 instance(s) unsuccessfully stopped          [FAILED]
Starting dirsrv: 
    C2.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    M1.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    M2.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    TestInst1.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
  *** Error: 4 instance(s) failed to start
[root@iceman ~]# rpm -qa |grep -i 389-ds-base
389-ds-base-1.2.11.15-40.el6.x86_64
389-ds-base-devel-1.2.11.15-40.el6.x86_64
389-ds-base-libs-1.2.11.15-40.el6.x86_64
[root@iceman ~]# rpm -qa |grep -i openldap
openldap-debuginfo-2.4.23-26.el6.x86_64
openldap-devel-2.4.23-32.el6_4.1.x86_64
openldap-clients-2.4.23-32.el6_4.1.x86_64
compat-openldap-2.3.43-2.el6.x86_64
openldap-2.4.23-32.el6_4.1.x86_64

A per comment #18, marking the bug as Assigned

Additional info:

The same works if the openldap packages updated to...
openldap-devel-2.4.39-8.el6.x86_64
openldap-2.4.39-8.el6.x86_64
openldap-clients-2.4.39-8.el6.x86_64

With the latest build of 389-ds-base, I could successfully upgrade an existing instance and create new instances. Restarting of directory server instances also work. Hence, marking the bug as Verified.

Build tested - 389-ds-base-1.2.11.15-41

  - Offline: The changes are made to the server configuration files.  The
             servers MUST FIRST BE SHUTDOWN BY YOU.  The script will not
             shutdown the servers for you.  You MUST shutdown the
             servers in order to use this mode.  A username and password
             are not required to use Offline mode.  If the servers are not
             shutdown, CHANGES WILL BE LOST.

To summarize:
  Online - servers remain running - you must provide admin name and password
           for each server - servers may need to be restarted
  Offline - servers must be shutdown - no username or password required

Which update mode do you want to use? [quit]: Offline
[21/Aug/2014:09:11:38 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52141
[21/Aug/2014:09:11:38 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:38 -0400] Upgrade DN Format - passsync1489: Start upgrade dn format.
[21/Aug/2014:09:11:38 -0400] Upgrade DN Format - Instance passsync1489 in /var/lib/dirsrv/slapd-C2/db/passsync1489 is up-to-date
[21/Aug/2014:09:11:39 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52140
[21/Aug/2014:09:11:39 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:39 -0400] Upgrade DN Format - userRoot: Start upgrade dn format.
[21/Aug/2014:09:11:39 -0400] Upgrade DN Format - Instance userRoot in /var/lib/dirsrv/slapd-C2/db/userRoot is up-to-date
[21/Aug/2014:09:11:56 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52299
[21/Aug/2014:09:11:56 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:56 -0400] Upgrade DN Format - NetscapeRoot: Start upgrade dn format.
[21/Aug/2014:09:11:56 -0400] Upgrade DN Format - Instance NetscapeRoot in /var/lib/dirsrv/slapd-M1/db/NetscapeRoot is up-to-date
[21/Aug/2014:09:11:57 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52298
[21/Aug/2014:09:11:57 -0400] - Import allocates 3223756KB import cache.

FYI, I also tested an IPA upgrade and works great.

Thanks.

[root@rhel6-1 ~]# yum update 'ipa*' sssd
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Update Process
Resolving Dependencies
...cut...
Installed:
  sssd-common.x86_64 0:1.11.6-23.el6                                                                   

Dependency Installed:
  libbasicobjects.x86_64 0:0.1.1-11.el6             python-sssdconfig.noarch 0:1.11.6-23.el6          
  sssd-ad.x86_64 0:1.11.6-23.el6                    sssd-common-pac.x86_64 0:1.11.6-23.el6            
  sssd-ipa.x86_64 0:1.11.6-23.el6                   sssd-krb5.x86_64 0:1.11.6-23.el6                  
  sssd-krb5-common.x86_64 0:1.11.6-23.el6           sssd-ldap.x86_64 0:1.11.6-23.el6                  
  sssd-proxy.x86_64 0:1.11.6-23.el6                

Updated:
  ipa-admintools.x86_64 0:3.0.0-42.el6                   ipa-client.x86_64 0:3.0.0-42.el6              
  ipa-python.x86_64 0:3.0.0-42.el6                       ipa-server.x86_64 0:3.0.0-42.el6              
  ipa-server-selinux.x86_64 0:3.0.0-42.el6               sssd.x86_64 0:1.11.6-23.el6                   

Dependency Updated:
  389-ds-base.x86_64 0:1.2.11.15-41.el6             389-ds-base-libs.x86_64 0:1.2.11.15-41.el6        
  libcollection.x86_64 0:0.6.2-11.el6               libdhash.x86_64 0:0.4.3-11.el6                    
  libini_config.x86_64 0:1.1.0-11.el6               libipa_hbac.x86_64 0:1.11.6-23.el6                
  libipa_hbac-python.x86_64 0:1.11.6-23.el6         libpath_utils.x86_64 0:0.2.1-11.el6               
  libref_array.x86_64 0:0.1.4-11.el6                libsss_idmap.x86_64 0:1.11.6-23.el6               
  sssd-client.x86_64 0:1.11.6-23.el6               

Replaced:
  libsss_autofs.x86_64 0:1.9.2-129.el6                                                                 

Complete!


[root@rhel6-1 ~]# ipactl status
Directory Service: RUNNING
KDC Service: RUNNING
KPASSWD Service: RUNNING
DNS Service: RUNNING
MEMCACHE Service: RUNNING
HTTP Service: RUNNING
CA Service: RUNNING

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1385.html