Bug 895298 - IPA upgrade error restarting named when dirsrv off before upgrade
IPA upgrade error restarting named when dirsrv off before upgrade
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ipa (Show other bugs)
6.4
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Rob Crittenden
Namita Soman
: Regression
Depends On:
Blocks: 902691
  Show dependency treegraph
 
Reported: 2013-01-14 20:44 EST by Scott Poore
Modified: 2013-10-07 14:59 EDT (History)
4 users (show)

See Also:
Fixed In Version: ipa-3.0.0-22.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-21 04:32:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Scott Poore 2013-01-14 20:44:54 EST
Description of problem:

If dirsrv is stopped before an upgrade, named not restarted cleanly during the IPA upgrade.

This is what I see during yum update 'ipa*':

  Updating   : ipa-server-3.0.0-21.el6.x86_64                                                    34/72 
Unexpected error
CalledProcessError: Command '/sbin/service named restart ' returned non-zero exit status 7
  Updating   : ipa-server-selinux-3.0.0-21.el6.x86_64                                            35/72 


Version-Release number of selected component (if applicable):
2.2.0 -> 3.0.0 update

How reproducible:
always

Steps to Reproduce:
1.  Install RHEL6.3 IPA server
2.  Point yum repos for RHEL6.4
3.  service dirsrv stop
4.  yum update 'ipa*'

  
Actual results:
fails to restart named and things don't work afterwards


Expected results:
restarts 

Additional info:

Looks like ipaupgrade.log shows that dirsrv restarted after attempted named restart...not sure if it matters.

/var/log/ipaupgrade.log:

2013-01-11T22:35:14Z INFO Changes to named.conf have been made, restart named
2013-01-11T22:35:14Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2013-01-11T22:35:17Z DEBUG args=/sbin/service named restart
2013-01-11T22:35:17Z DEBUG stdout=Stopping named: .[  OK  ]^M
Starting named: [FAILED]^M

2013-01-11T22:35:17Z DEBUG stderr=
2013-01-11T22:35:17Z INFO   File "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line 614, in run_script
    return_value = main_function()

  File "/usr/sbin/ipa-upgradeconfig", line 660, in main
    bindinstance.BindInstance(fstore).restart()

  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 268, in restart
    self.service.restart(instance_name, capture_output=capture_output, wait=wait)

  File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py", line 76, in restart
    ipautil.run(["/sbin/service", self.service_name, "restart", instance_name], capture_output=capture_output)

  File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 316, in run
    raise CalledProcessError(p.returncode, args)

2013-01-11T22:35:17Z INFO The ipa-upgradeconfig command failed, exception: CalledProcessError: Command '/sbin/service named restart ' returned non-zero exit status 7

Attempts to start named alone show this failure in /var/log/messages:

Jan 14 19:42:15 rhel6-1 named[32388]: sizing zone task pool based on 6 zones
Jan 14 19:42:15 rhel6-1 named[32388]: set up managed keys zone for view _default, file 'dynamic/managed-keys.bind'
Jan 14 19:42:15 rhel6-1 named[32388]: Failed to init credentials (Cannot contact any KDC for realm 'TESTRELM.COM')
Jan 14 19:42:15 rhel6-1 named[32388]: loading configuration: failure
Jan 14 19:42:15 rhel6-1 named[32388]: exiting (due to fatal error)


Now, I can work around this with ipactl stop/start:

[root@rhel6-1 ipa-upgrade]# ipactl stop
Stopping CA Service
Stopping pki-ca:                                           [  OK  ]
Stopping HTTP Service
Stopping httpd:                                            [  OK  ]
Stopping MEMCACHE Service
Stopping ipa_memcached:                                    [  OK  ]
Stopping DNS Service
Stopping named:                                            [  OK  ]
Stopping KPASSWD Service
Stopping Kerberos 5 Admin Server:                          [FAILED]
Stopping KDC Service
Stopping Kerberos 5 KDC:                                   [FAILED]
Stopping Directory Service
Shutting down dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TESTRELM-COM...                                        [  OK  ]

[root@rhel6-1 ipa-upgrade]# ipactl start
Starting Directory Service
Starting dirsrv: 
    PKI-IPA...                                             [  OK  ]
    TESTRELM-COM...                                        [  OK  ]
Starting KDC Service
Starting Kerberos 5 KDC:                                   [  OK  ]
Starting KPASSWD Service
Starting Kerberos 5 Admin Server:                          [  OK  ]
Starting DNS Service
Starting named:                                            [  OK  ]
Starting MEMCACHE Service
Starting ipa_memcached:                                    [  OK  ]
Starting HTTP Service
Starting httpd:                                            [  OK  ]
Starting CA Service
Starting pki-ca:                                           [  OK  ]

[root@rhel6-1 ipa-upgrade]# kinit admin
Password for admin@TESTRELM.COM: 

[root@rhel6-1 ipa-upgrade]#
Comment 2 Rob Crittenden 2013-01-14 21:38:22 EST
I assume this would happen if one also did ipactl stop before upgrading.
Comment 3 Rob Crittenden 2013-01-14 21:46:55 EST
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3350
Comment 6 Scott Poore 2013-01-17 23:11:26 EST
hmm....I'm still seeing this (and more now):

  Updating   : ipa-server-3.0.0-22.el6.x86_64                                                                    52/95 
Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7
certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n ocspSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n subsystemCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /etc/httpd/alias -n ipaCert -c dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ra_cert -p /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 1
certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n Server-Cert cert-pki-ca -c dogtag-ipa-renew-agent -P XXXXXXXX' returned non-zero exit status 1
Unable to find certmonger request ID for auditSigning Cert
  Updating   : ipa-server-selinux-3.0.0-22.el6.x86_64                                                            53/95 


This make any sense?  Do you need me to send logs?
Comment 7 Rob Crittenden 2013-01-18 09:20:06 EST
The failure to restart named will be seen, and is fine. We just don't want to blow up the rest of the upgrade.

The other errors should be fixed by selinux-policy 3.7.19-193. Can you see if you have any AVCs?
Comment 8 Scott Poore 2013-01-18 16:28:37 EST

No AVC and I've got that version of selinux-policy (after the ugprade at least):

[root@rhel6-1 log]# ausearch -m avc
<no matches>

[root@rhel6-1 log]# rpm -q selinux-policy
selinux-policy-3.7.19-193.el6.noarch


I do see this in the /var/log/ipaupgrade.log file though:

2013-01-18T04:07:06Z DEBUG args=/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX
2013-01-18T04:07:06Z DEBUG stdout=No CA with name "dogtag-ipa-renew-agent" found.

2013-01-18T04:07:06Z DEBUG stderr=
2013-01-18T04:07:06Z ERROR certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1
Comment 9 Rob Crittenden 2013-01-18 16:30:09 EST
Is this the first master installed or a replica of the first master?
Comment 10 Scott Poore 2013-01-18 19:37:16 EST
First master.  There's also a second and a client in the env.
Comment 11 Rob Crittenden 2013-01-21 11:20:27 EST
My current guess is this is an rpm ordering issue. I think that the IPA upgrade is happening before the updated certmonger is installed, and certmonger provides this CA.
Comment 12 Scott Poore 2013-01-21 17:39:40 EST
Ok, we moved certmonger upgrade issue to bug #902474
Comment 13 Scott Poore 2013-01-24 21:30:27 EST
Verified.

Version ::

ipa-server-3.0.0-23.el6.x86_64

Automated Test Results ::

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: upgrade_bz_895298_check_master: IPA upgrade error restarting named when dirsrv off before upgrade
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [17:03:38] ::  Machine in recipe is MASTER
:: [17:03:38] ::  Backing up and submitting /var/log/ipaupgrade.log
:: [   PASS   ] :: File '/var/log/ipaupgrade.log' should not contain 'The ipa-upgradeconfig command failed.*named restart'
:: [   PASS   ] :: BZ 895298 not found
2013-01-23T21:58:17Z ERROR Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7
:: [   PASS   ] :: Running 'grep 'ERROR Failed to restart named' /var/log/ipaupgrade.log'
:: [17:03:42] ::  workaround: restarting everything
Shutting down dirsrv: 
    PKI-IPA...[  OK  ]
    TESTRELM-COM...
MARK-LWD-LOOP -- 2013-01-23 17:03:45 --
[  OK  ]
Starting dirsrv: 
    PKI-IPA...[  OK  ]
    TESTRELM-COM...[  OK  ]
Stopping Kerberos 5 KDC: [FAILED]
Starting Kerberos 5 KDC: [  OK  ]
Stopping Kerberos 5 Admin Server: [FAILED]
Starting Kerberos 5 Admin Server: [  OK  ]
Stopping named: [  OK  ]
Starting named: [  OK  ]
Stopping ipa_memcached: [  OK  ]
Starting ipa_memcached: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Stopping pki-ca: [  OK  ]
Starting pki-ca: [  OK  ]
Restarting Directory Service
Restarting KDC Service
Restarting KPASSWD Service
Restarting DNS Service
Restarting MEMCACHE Service
Restarting HTTP Service
Restarting CA Service
:: [   PASS   ] :: Running 'ipactl restart'
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[  OK  ]
:: [   PASS   ] :: Running 'service sssd restart'
:: [17:04:27] ::  Backing up and submitting /var/log/ipaupgrade.log
:: [   PASS   ] :: Running 'rhts-sync-set -s 'upgrade_bz_895298_check_master.83' -m 10.16.76.37'
Comment 15 errata-xmlrpc 2013-02-21 04:32:25 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0528.html

Note You need to log in before you can comment on or make changes to this bug.