Bug 895298
| Summary: | IPA upgrade error restarting named when dirsrv off before upgrade | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Scott Poore <spoore> |
| Component: | ipa | Assignee: | Rob Crittenden <rcritten> |
| Status: | CLOSED ERRATA | QA Contact: | Namita Soman <nsoman> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.4 | CC: | cpelland, dpal, jgalipea, mkosek |
| Target Milestone: | rc | Keywords: | Regression |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ipa-3.0.0-22.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-02-21 09:32:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 902691 | ||
I assume this would happen if one also did ipactl stop before upgrading. Upstream ticket: https://fedorahosted.org/freeipa/ticket/3350 Fixed upstream: master: https://fedorahosted.org/freeipa/changeset/476aacd69963dd94de3af3d640fca783d77b4eb8 ipa-3-1: https://fedorahosted.org/freeipa/changeset/cab85b7c9a7e8b7c82915512f7d4718c4cbcbef9 ipa-3-0: https://fedorahosted.org/freeipa/changeset/a89d96fa473c70bb2c89d7a120713fcfb9463263 hmm....I'm still seeing this (and more now): Updating : ipa-server-3.0.0-22.el6.x86_64 52/95 Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n ocspSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n subsystemCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /etc/httpd/alias -n ipaCert -c dogtag-ipa-renew-agent -C /usr/lib64/ipa/certmonger/renew_ra_cert -p /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 1 certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n Server-Cert cert-pki-ca -c dogtag-ipa-renew-agent -P XXXXXXXX' returned non-zero exit status 1 Unable to find certmonger request ID for auditSigning Cert Updating : ipa-server-selinux-3.0.0-22.el6.x86_64 53/95 This make any sense? Do you need me to send logs? The failure to restart named will be seen, and is fine. We just don't want to blow up the rest of the upgrade. The other errors should be fixed by selinux-policy 3.7.19-193. Can you see if you have any AVCs? No AVC and I've got that version of selinux-policy (after the ugprade at least): [root@rhel6-1 log]# ausearch -m avc <no matches> [root@rhel6-1 log]# rpm -q selinux-policy selinux-policy-3.7.19-193.el6.noarch I do see this in the /var/log/ipaupgrade.log file though: 2013-01-18T04:07:06Z DEBUG args=/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX 2013-01-18T04:07:06Z DEBUG stdout=No CA with name "dogtag-ipa-renew-agent" found. 2013-01-18T04:07:06Z DEBUG stderr= 2013-01-18T04:07:06Z ERROR certmonger failed to start tracking certificate: Command '/usr/bin/getcert start-tracking -d /var/lib/pki-ca/alias -n auditSigningCert cert-pki-ca -c dogtag-ipa-renew-agent -B /usr/lib64/ipa/certmonger/stop_pkicad -C /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" -P XXXXXXXX' returned non-zero exit status 1 Is this the first master installed or a replica of the first master? First master. There's also a second and a client in the env. My current guess is this is an rpm ordering issue. I think that the IPA upgrade is happening before the updated certmonger is installed, and certmonger provides this CA. Ok, we moved certmonger upgrade issue to bug #902474 Verified. Version :: ipa-server-3.0.0-23.el6.x86_64 Automated Test Results :: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ LOG ] :: upgrade_bz_895298_check_master: IPA upgrade error restarting named when dirsrv off before upgrade :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [17:03:38] :: Machine in recipe is MASTER :: [17:03:38] :: Backing up and submitting /var/log/ipaupgrade.log :: [ PASS ] :: File '/var/log/ipaupgrade.log' should not contain 'The ipa-upgradeconfig command failed.*named restart' :: [ PASS ] :: BZ 895298 not found 2013-01-23T21:58:17Z ERROR Failed to restart named: Command '/sbin/service named restart ' returned non-zero exit status 7 :: [ PASS ] :: Running 'grep 'ERROR Failed to restart named' /var/log/ipaupgrade.log' :: [17:03:42] :: workaround: restarting everything Shutting down dirsrv: PKI-IPA...[ OK ] TESTRELM-COM... MARK-LWD-LOOP -- 2013-01-23 17:03:45 -- [ OK ] Starting dirsrv: PKI-IPA...[ OK ] TESTRELM-COM...[ OK ] Stopping Kerberos 5 KDC: [FAILED] Starting Kerberos 5 KDC: [ OK ] Stopping Kerberos 5 Admin Server: [FAILED] Starting Kerberos 5 Admin Server: [ OK ] Stopping named: [ OK ] Starting named: [ OK ] Stopping ipa_memcached: [ OK ] Starting ipa_memcached: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Stopping pki-ca: [ OK ] Starting pki-ca: [ OK ] Restarting Directory Service Restarting KDC Service Restarting KPASSWD Service Restarting DNS Service Restarting MEMCACHE Service Restarting HTTP Service Restarting CA Service :: [ PASS ] :: Running 'ipactl restart' Stopping sssd: [ OK ] Starting sssd: [ OK ] [ OK ] :: [ PASS ] :: Running 'service sssd restart' :: [17:04:27] :: Backing up and submitting /var/log/ipaupgrade.log :: [ PASS ] :: Running 'rhts-sync-set -s 'upgrade_bz_895298_check_master.83' -m 10.16.76.37' Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2013-0528.html |
Description of problem: If dirsrv is stopped before an upgrade, named not restarted cleanly during the IPA upgrade. This is what I see during yum update 'ipa*': Updating : ipa-server-3.0.0-21.el6.x86_64 34/72 Unexpected error CalledProcessError: Command '/sbin/service named restart ' returned non-zero exit status 7 Updating : ipa-server-selinux-3.0.0-21.el6.x86_64 35/72 Version-Release number of selected component (if applicable): 2.2.0 -> 3.0.0 update How reproducible: always Steps to Reproduce: 1. Install RHEL6.3 IPA server 2. Point yum repos for RHEL6.4 3. service dirsrv stop 4. yum update 'ipa*' Actual results: fails to restart named and things don't work afterwards Expected results: restarts Additional info: Looks like ipaupgrade.log shows that dirsrv restarted after attempted named restart...not sure if it matters. /var/log/ipaupgrade.log: 2013-01-11T22:35:14Z INFO Changes to named.conf have been made, restart named 2013-01-11T22:35:14Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2013-01-11T22:35:17Z DEBUG args=/sbin/service named restart 2013-01-11T22:35:17Z DEBUG stdout=Stopping named: .[ OK ]^M Starting named: [FAILED]^M 2013-01-11T22:35:17Z DEBUG stderr= 2013-01-11T22:35:17Z INFO File "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line 614, in run_script return_value = main_function() File "/usr/sbin/ipa-upgradeconfig", line 660, in main bindinstance.BindInstance(fstore).restart() File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 268, in restart self.service.restart(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py", line 76, in restart ipautil.run(["/sbin/service", self.service_name, "restart", instance_name], capture_output=capture_output) File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 316, in run raise CalledProcessError(p.returncode, args) 2013-01-11T22:35:17Z INFO The ipa-upgradeconfig command failed, exception: CalledProcessError: Command '/sbin/service named restart ' returned non-zero exit status 7 Attempts to start named alone show this failure in /var/log/messages: Jan 14 19:42:15 rhel6-1 named[32388]: sizing zone task pool based on 6 zones Jan 14 19:42:15 rhel6-1 named[32388]: set up managed keys zone for view _default, file 'dynamic/managed-keys.bind' Jan 14 19:42:15 rhel6-1 named[32388]: Failed to init credentials (Cannot contact any KDC for realm 'TESTRELM.COM') Jan 14 19:42:15 rhel6-1 named[32388]: loading configuration: failure Jan 14 19:42:15 rhel6-1 named[32388]: exiting (due to fatal error) Now, I can work around this with ipactl stop/start: [root@rhel6-1 ipa-upgrade]# ipactl stop Stopping CA Service Stopping pki-ca: [ OK ] Stopping HTTP Service Stopping httpd: [ OK ] Stopping MEMCACHE Service Stopping ipa_memcached: [ OK ] Stopping DNS Service Stopping named: [ OK ] Stopping KPASSWD Service Stopping Kerberos 5 Admin Server: [FAILED] Stopping KDC Service Stopping Kerberos 5 KDC: [FAILED] Stopping Directory Service Shutting down dirsrv: PKI-IPA... [ OK ] TESTRELM-COM... [ OK ] [root@rhel6-1 ipa-upgrade]# ipactl start Starting Directory Service Starting dirsrv: PKI-IPA... [ OK ] TESTRELM-COM... [ OK ] Starting KDC Service Starting Kerberos 5 KDC: [ OK ] Starting KPASSWD Service Starting Kerberos 5 Admin Server: [ OK ] Starting DNS Service Starting named: [ OK ] Starting MEMCACHE Service Starting ipa_memcached: [ OK ] Starting HTTP Service Starting httpd: [ OK ] Starting CA Service Starting pki-ca: [ OK ] [root@rhel6-1 ipa-upgrade]# kinit admin Password for admin: [root@rhel6-1 ipa-upgrade]#