Bug 1130252 - RHEL6.6 dirsrv not running after ipa upgrade
Summary: RHEL6.6 dirsrv not running after ipa upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base
Version: 6.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Noriko Hosoi
QA Contact: Sankar Ramalingam
URL:
Whiteboard:
Depends On:
Blocks: 1131187
TreeView+ depends on / blocked
 
Reported: 2014-08-14 16:32 UTC by Scott Poore
Modified: 2014-10-14 07:53 UTC (History)
7 users (show)

Fixed In Version: 389-ds-base-1.2.11.15-41.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1131187 (view as bug list)
Environment:
Last Closed: 2014-10-14 07:53:57 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1385 normal SHIPPED_LIVE 389-ds-base bug fix and enhancement update 2014-10-14 01:27:42 UTC

Description Scott Poore 2014-08-14 16:32:23 UTC
Description of problem:

Upgrading a RHEL6.5 server to 6.6 version of IPA (and components) results in IPA not running.  I also see named failure during upgrade output:

  Updating   : ipa-server-3.0.0-42.el6.x86_64                                                    26/45 
Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7
  Updating   : ipa-server-selinux-3.0.0-42.el6.x86_64                                            27/45 

Some digging through ipaupgrade.log shows failures to stop dirsrv as if it's already stopped:

2014-08-14T16:14:13Z DEBUG Upgrading IPA:
2014-08-14T16:14:13Z DEBUG   [1/8]: stopping directory server
2014-08-14T16:14:13Z DEBUG args=/sbin/service dirsrv stop TESTRELM-TEST
2014-08-14T16:14:13Z DEBUG stdout=Shutting down dirsrv: 
    TESTRELM-TEST... server already stopped[FAILED]
  *** Error: 1 instance(s) unsuccessfully stopped[FAILED]

Then looking at messages for yum update and named messages shows:

Aug 14 11:12:06 rhel6-1 yum[14053]: Updated: 389-ds-base-libs-1.2.11.15-39.el6.x86_64
Aug 14 11:12:12 rhel6-1 named[4089]: LDAP error: Can't contact LDAP server
Aug 14 11:12:12 rhel6-1 named[4089]: connection to the LDAP server was lost
Aug 14 11:12:12 rhel6-1 named[4089]: bind to LDAP server failed: Can't contact LDAP server
Aug 14 11:12:12 rhel6-1 named[4089]: ldap_psearch_watcher failed to handle LDAP connection error. Recon
nection in 60s
Aug 14 11:12:25 rhel6-1 yum[14053]: Updated: 389-ds-base-1.2.11.15-39.el6.x86_64

And in dirsrv errors log I can see that it was stopped but I don't see anything about it being started:

[14/Aug/2014:11:12:09 -0500] - slapd shutting down - signaling operation threads
[14/Aug/2014:11:12:09 -0500] - slapd shutting down - closing down internal subsystems and plugins
[14/Aug/2014:11:12:09 -0500] - Waiting for 4 database threads to stop
[14/Aug/2014:11:12:09 -0500] - All database threads now stopped
[14/Aug/2014:11:12:09 -0500] - slapd stopped.


Version-Release number of selected component (if applicable):
389-ds-base-1.2.11.15-39.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1.  on RHEL6.5 host, ipa-server-install # with dns configured
2.  setup RHEL6.6 yum repo configs
3.  yum update 'ipa*' sssd -y

Actual results:
dirsrv (and thus ipa) not running after upgrade.

Expected results:
everything running.

Additional info:

Comment 2 Scott Poore 2014-08-14 17:20:26 UTC
Also, if I try to start manually after upgrade, it fails with:

[root@rhel6-1 ~]# service dirsrv start
Starting dirsrv: 
    PKI-IPA.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    TESTRELM-TEST.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
  *** Error: 2 instance(s) failed to start


I think this may be a duplicate of bug#1112581

going to modify the upgrade and see if this is resolved.

Comment 3 Scott Poore 2014-08-14 18:15:25 UTC
No, I'm still seeing the same results after adding pki-ca to the upgrade:

[root@rhel6-2 ~]# rpm -q ipa-server pki-ca 389-ds-base
ipa-server-3.0.0-42.el6.x86_64
pki-ca-9.0.3-37.el6.noarch
389-ds-base-1.2.11.15-39.el6.x86_64

From messages I see the same results:

Aug 14 12:52:24 rhel6-2 yum[4303]: Updated: 389-ds-base-libs-1.2.11.15-39.el6.x86_64
Aug 14 12:52:32 rhel6-2 named[4051]: LDAP error: Can't contact LDAP server
Aug 14 12:52:32 rhel6-2 named[4051]: connection to the LDAP server was lost
Aug 14 12:52:32 rhel6-2 named[4051]: bind to LDAP server failed: Can't contact LDAP server
Aug 14 12:52:32 rhel6-2 named[4051]: ldap_psearch_watcher failed to handle LDAP connection error. Recon
nection in 60s
Aug 14 12:52:48 rhel6-2 yum[4303]: Updated: 389-ds-base-1.2.11.15-39.el6.x86_64

Comment 4 Scott Poore 2014-08-15 02:04:44 UTC
If I upgrade openldap, I now get this:

[root@rhel6-2 slapd-TEST-QE]# service dirsrv restart
Shutting down dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TEST-QE...                                             [  OK  ]
Starting dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TEST-QE...[14/Aug/2014:17:17:50 -0500] - Information: Non-Secure Port Disabled
                                                           [  OK  ]

And I can't see the 389 port open now:

[root@rhel6-2 dirsrv]# netstat -taupne |grep 389
tcp        0      0 :::7389                     :::*                        LISTEN      0          56704      17958/ns-slapd      


I still can't start ipa.  How can I track down why slapd isn't starting on port 389?  Is there a logging level I should use?

I'm changing component here to 389-ds-base since that seems to be the main piece here.

Comment 5 Nathan Kinder 2014-08-15 03:10:57 UTC
(In reply to Scott Poore from comment #4)
> If I upgrade openldap, I now get this:

What do you mean by 'openldap'?  Do you mean you upgraded IPA?

>     PKI-IPA...                                             [  OK  ]
>     TEST-QE...[14/Aug/2014:17:17:50 -0500] - Information: Non-Secure Port
> Disabled

This message is output when nsslapd-port is set to 0, which IPA does during upgrade.  There is likely some failure that IPA is encountering during upgrade, which caused it to bail out before resetting nsslapd-port to 389.

Moving this back to ipa.

Comment 6 Nathan Kinder 2014-08-15 04:17:29 UTC
(In reply to Nathan Kinder from comment #5)
> This message is output when nsslapd-port is set to 0, which IPA does during
> upgrade.  There is likely some failure that IPA is encountering during
> upgrade, which caused it to bail out before resetting nsslapd-port to 389.

I just read your earlier comments, and I believe that the upgrade failure is likely fallout from the earlier errors encountered by the openldap/389-ds-base mismatch.  You're now dealing with the old port setting that was left behind.

OpenLDAP changed things such that they dropped a library IIRC.  There may be a missing dependency here in 389-ds-base now that I think about it more.  If you repeat your upgrade test from 6.5 -> 6.6, but update openldap manually first, does everything work?

Comment 7 Scott Poore 2014-08-15 14:31:08 UTC
Nathan, 

You are correct.  If I update openldap before or with ipa, it's working cleanly now.

So, for my case, the fix is either:

yum update -y openldap; yum update 'ipa*' sssd

or

yum update -y 'ipa*' sssd openldap

Is this something that can/should be fixed with an updated rpm dependency?

Thanks,
Scott

Comment 8 Nathan Kinder 2014-08-15 16:24:50 UTC
(In reply to Scott Poore from comment #7)
> Nathan, 
> 
> You are correct.  If I update openldap before or with ipa, it's working
> cleanly now.
> 
> So, for my case, the fix is either:
> 
> yum update -y openldap; yum update 'ipa*' sssd
> 
> or
> 
> yum update -y 'ipa*' sssd openldap
> 
> Is this something that can/should be fixed with an updated rpm dependency?
> 

Quite possibly.  What were the exact old and new versions of openldap in your test?

Comment 9 Scott Poore 2014-08-15 17:41:56 UTC
old:
openldap-2.4.23-32.el6_4.1.x86_64

new:
openldap-2.4.39-8.el6.x86_64

Comment 10 Noriko Hosoi 2014-08-15 22:51:28 UTC
This commit 1270ca4b759cc489bfd25be8d500b6cbce63ab92 introduced @ldap_lib_ldif@, but the substitute was not added to configure.ac.
+AC_SUBST(ldap_lib_ldif)

Since this change was made, libldif was not linked to the directory server.  So, the openldap versions which do not have this change make the directory server fail to start.
    Newer versions of openldap have moved the ldif api into the main libldap
    At configure time we need to see if libldap has the api, or if we need
    to link with a separate libldif

Adding the above line and regenerating configure fixes this problem.  This affects all the supported versions.

Comment 11 Noriko Hosoi 2014-08-15 23:07:30 UTC
Upstream ticket:
https://fedorahosted.org/389/ticket/47875

Comment 13 Noriko Hosoi 2014-08-17 03:56:01 UTC
Steps to verify:

1. downgrade openldap to this version
   openldap-2.4.23-32.el6_4.1.x86_64
2. install rhel-6.6 389-ds-base.
   If it installs fine, the fix is verified.

Note: without the fix, the installation fails with this error.
/usr/bin/pwdhash-bin: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
Could not import LDIF file '/tmp/ldifHKngyV.ldif'.  Error: 32512.  Output: importing data ...
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline

Comment 14 Rob Crittenden 2014-08-18 13:43:08 UTC
I wonder if IPA should have a clone of this bug. There should be a failsafe in the IPA updater such that it ALWAYS resets the listeners back to their initial values (so 389 and security on)

Comment 15 Martin Kosek 2014-08-18 15:51:26 UTC
Actually, this is a very good idea. We want to make upgrade process smoother. I will clone the Bugzilla.

Comment 17 Sankar Ramalingam 2014-08-19 12:12:36 UTC
Since it requires specific version of openldap to be downgraded, its not possible to automate in TET. Hence, marking qe_test_coverage- flag.

Comment 18 Sankar Ramalingam 2014-08-19 12:14:47 UTC
[root@iceman ~]# setup-ds.pl -u

==============================================================================
This program will update the 389 Directory Server.

It is recommended that you have "root" privilege to perform the update.
Tips for using this  program:
  - Press "Enter" to choose the default and go to the next screen
  - Type "Control-B" or the word "back" then "Enter" to go back to the previous screen
  - Type "Control-C" to cancel the update

Would you like to continue with update? [yes]: 

==============================================================================

The update process can work in one of two modes:

  - Online: The changes are made to the running directory servers using LDAP.
            The operations must be performed as an administrative user.
            You must provide the name and password, for each instance
            if there is more than one instance of directory server.
            Some operations may require a directory server restart to take
            effect.  The update script will notify you if you need to restart
            the server.

  - Offline: The changes are made to the server configuration files.  The
             servers MUST FIRST BE SHUTDOWN BY YOU.  The script will not
             shutdown the servers for you.  You MUST shutdown the
             servers in order to use this mode.  A username and password
             are not required to use Offline mode.  If the servers are not
             shutdown, CHANGES WILL BE LOST.

To summarize:
  Online - servers remain running - you must provide admin name and password
           for each server - servers may need to be restarted
  Offline - servers must be shutdown - no username or password required

Which update mode do you want to use? [quit]: Offline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
./ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
Finished successful update of directory server.
Please restart your directory servers.
Exiting . . .
Log file is '/tmp/setupsjKF9J.log'

[root@iceman ~]# service dirsrv restart
Shutting down dirsrv: 
    C2... server already stopped                           [FAILED]
    M1... server already stopped                           [FAILED]
    M2... server already stopped                           [FAILED]
    TestInst1... server already stopped                    [FAILED]
  *** Error: 4 instance(s) unsuccessfully stopped          [FAILED]
Starting dirsrv: 
    C2.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    M1.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    M2.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
    TestInst1.../usr/sbin/ns-slapd: symbol lookup error: /usr/lib64/dirsrv/libslapd.so.0: undefined symbol: ldif_getline
                                                           [FAILED]
  *** Error: 4 instance(s) failed to start
[root@iceman ~]# rpm -qa |grep -i 389-ds-base
389-ds-base-1.2.11.15-40.el6.x86_64
389-ds-base-devel-1.2.11.15-40.el6.x86_64
389-ds-base-libs-1.2.11.15-40.el6.x86_64
[root@iceman ~]# rpm -qa |grep -i openldap
openldap-debuginfo-2.4.23-26.el6.x86_64
openldap-devel-2.4.23-32.el6_4.1.x86_64
openldap-clients-2.4.23-32.el6_4.1.x86_64
compat-openldap-2.3.43-2.el6.x86_64
openldap-2.4.23-32.el6_4.1.x86_64

Comment 19 Sankar Ramalingam 2014-08-19 12:15:35 UTC
A per comment #18, marking the bug as Assigned

Comment 20 Sankar Ramalingam 2014-08-19 12:19:45 UTC
Additional info:

The same works if the openldap packages updated to...
openldap-devel-2.4.39-8.el6.x86_64
openldap-2.4.39-8.el6.x86_64
openldap-clients-2.4.39-8.el6.x86_64

Comment 21 Sankar Ramalingam 2014-08-21 13:21:11 UTC
With the latest build of 389-ds-base, I could successfully upgrade an existing instance and create new instances. Restarting of directory server instances also work. Hence, marking the bug as Verified.

Build tested - 389-ds-base-1.2.11.15-41

  - Offline: The changes are made to the server configuration files.  The
             servers MUST FIRST BE SHUTDOWN BY YOU.  The script will not
             shutdown the servers for you.  You MUST shutdown the
             servers in order to use this mode.  A username and password
             are not required to use Offline mode.  If the servers are not
             shutdown, CHANGES WILL BE LOST.

To summarize:
  Online - servers remain running - you must provide admin name and password
           for each server - servers may need to be restarted
  Offline - servers must be shutdown - no username or password required

Which update mode do you want to use? [quit]: Offline
[21/Aug/2014:09:11:38 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52141
[21/Aug/2014:09:11:38 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:38 -0400] Upgrade DN Format - passsync1489: Start upgrade dn format.
[21/Aug/2014:09:11:38 -0400] Upgrade DN Format - Instance passsync1489 in /var/lib/dirsrv/slapd-C2/db/passsync1489 is up-to-date
[21/Aug/2014:09:11:39 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52140
[21/Aug/2014:09:11:39 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:39 -0400] Upgrade DN Format - userRoot: Start upgrade dn format.
[21/Aug/2014:09:11:39 -0400] Upgrade DN Format - Instance userRoot in /var/lib/dirsrv/slapd-C2/db/userRoot is up-to-date
[21/Aug/2014:09:11:56 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52299
[21/Aug/2014:09:11:56 -0400] - Import allocates 3223756KB import cache.
[21/Aug/2014:09:11:56 -0400] Upgrade DN Format - NetscapeRoot: Start upgrade dn format.
[21/Aug/2014:09:11:56 -0400] Upgrade DN Format - Instance NetscapeRoot in /var/lib/dirsrv/slapd-M1/db/NetscapeRoot is up-to-date
[21/Aug/2014:09:11:57 -0400] - check_and_set_import_cache: pagesize: 4096, pages: 2014848, procpages: 52298
[21/Aug/2014:09:11:57 -0400] - Import allocates 3223756KB import cache.

Comment 22 Scott Poore 2014-08-21 15:13:53 UTC
FYI, I also tested an IPA upgrade and works great.

Thanks.

[root@rhel6-1 ~]# yum update 'ipa*' sssd
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Setting up Update Process
Resolving Dependencies
...cut...
Installed:
  sssd-common.x86_64 0:1.11.6-23.el6                                                                   

Dependency Installed:
  libbasicobjects.x86_64 0:0.1.1-11.el6             python-sssdconfig.noarch 0:1.11.6-23.el6          
  sssd-ad.x86_64 0:1.11.6-23.el6                    sssd-common-pac.x86_64 0:1.11.6-23.el6            
  sssd-ipa.x86_64 0:1.11.6-23.el6                   sssd-krb5.x86_64 0:1.11.6-23.el6                  
  sssd-krb5-common.x86_64 0:1.11.6-23.el6           sssd-ldap.x86_64 0:1.11.6-23.el6                  
  sssd-proxy.x86_64 0:1.11.6-23.el6                

Updated:
  ipa-admintools.x86_64 0:3.0.0-42.el6                   ipa-client.x86_64 0:3.0.0-42.el6              
  ipa-python.x86_64 0:3.0.0-42.el6                       ipa-server.x86_64 0:3.0.0-42.el6              
  ipa-server-selinux.x86_64 0:3.0.0-42.el6               sssd.x86_64 0:1.11.6-23.el6                   

Dependency Updated:
  389-ds-base.x86_64 0:1.2.11.15-41.el6             389-ds-base-libs.x86_64 0:1.2.11.15-41.el6        
  libcollection.x86_64 0:0.6.2-11.el6               libdhash.x86_64 0:0.4.3-11.el6                    
  libini_config.x86_64 0:1.1.0-11.el6               libipa_hbac.x86_64 0:1.11.6-23.el6                
  libipa_hbac-python.x86_64 0:1.11.6-23.el6         libpath_utils.x86_64 0:0.2.1-11.el6               
  libref_array.x86_64 0:0.1.4-11.el6                libsss_idmap.x86_64 0:1.11.6-23.el6               
  sssd-client.x86_64 0:1.11.6-23.el6               

Replaced:
  libsss_autofs.x86_64 0:1.9.2-129.el6                                                                 

Complete!


[root@rhel6-1 ~]# ipactl status
Directory Service: RUNNING
KDC Service: RUNNING
KPASSWD Service: RUNNING
DNS Service: RUNNING
MEMCACHE Service: RUNNING
HTTP Service: RUNNING
CA Service: RUNNING

Comment 23 errata-xmlrpc 2014-10-14 07:53:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1385.html


Note You need to log in before you can comment on or make changes to this bug.