Red Hat Bugzilla – Bug 1401088
IPA upgrade of replica without DNS fails during restart of named-pkcs11
Last modified: 2017-08-01 05:44:33 EDT
This bug is created as a clone of upstream ticket: https://fedorahosted.org/freeipa/ticket/6503 When upgrading a pre-4.4 replica which has no IPA DNS configured (there is IPA DNS server in topology) to 4.4.2 (or ipa-4.4.0-12.el7), the upgrade erroneously tries to restart named-pkcs11 and fails: {{{ 2016-11-22T14:12:02Z DEBUG args=/bin/systemctl start named-pkcs11.service 2016-11-22T14:12:02Z DEBUG Process finished, return code=1 2016-11-22T14:12:02Z DEBUG stdout= 2016-11-22T14:12:02Z DEBUG stderr=Job for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details. 2016-11-22T14:12:02Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2016-11-22T14:12:02Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1867, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1725, in upgrade_configuration bind.start() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 345, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 285, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 515, in run raise CalledProcessError(p.returncode, arg_string, str(output)) 2016-11-22T14:12:02Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1 2016-11-22T14:12:02Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1 2016-11-22T14:12:02Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information }}} Upon further inspection the failure was tracked into the following piece of code which checks whether bind is configured and running: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipaserver/install/server/upgrade.py?h=ipa-4-4#n1722 The issue is that there is the following content in sysrestore.state dumped even if DNS is not configured on replica: {{{ [named] dns_record_0_0 = SRV _ldap._tcp 0 100 389 replica1 dns_record_0_1 = SRV _kerberos._tcp 0 100 88 replica1 dns_zone_0 = ipa.test dns_record_0_3 = SRV _kerberos-master._tcp 0 100 88 replica1 dns_record_0_2 = SRV _kerberos._udp 0 100 88 replica1 dns_record_0_5 = SRV _kpasswd._tcp 0 100 464 replica1 dns_record_0_4 = SRV _kerberos-master._udp 0 100 88 replica1 dns_record_0_7 = SRV _ntp._udp 0 100 123 replica1 dns_record_0_6 = SRV _kpasswd._udp 0 100 464 replica1 }}} Since the `StateFile.has_state` logic just checks for the presence of the section in the state file, it erroneously thinks that named is configured, hence the error. We need to either fix `StateFile` to actually search for enabled=True to decide whether the service was enabled, or fix `BindInstance` to not dump zone info into state file. Steps to reproduce: 1,) install 4.2 or 4.3 master w/ DNS 2.) create a replica from the master w/o DNS 3.) upgrade master to IPA v4.4 4.) now try to update the replica to the same version Expected results: The replica upgrades successfully Actual results: The upgrade fails with the following error: {{{ Cleanup : freeipa-common-4.3.2-2.fc24.noarch 19/19 IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1 The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information }}}
Fixed upstream master: https://fedorahosted.org/freeipa/changeset/f0e09c42b76f229486e5dea097cd2b6602999943 ipa-4-4: https://fedorahosted.org/freeipa/changeset/bf28d79afeff4575adc9ba0618b5acbf0cf51009
ipa-server version: ipa-server-4.5.0-13.el7.x86_64 Tested the bug with following setup: 1) install 4.2 master w/ DNS (In my case 4.4.0-14, 4.2.0-15) 2) create a replica from the master w/o DNS 3) upgrade master to IPA (In my case 4.5.0-13) 4) now try to update the replica to the same version Observations: 1. Verified that on upgrading the replica server (when setup without DNS) to latest version it is upgraded successfully. 2. Upon upgrade the "ipactl status" and "ipactl restart" command are run successfully both on IPA-Master and Replica. 3. No failures are observed under /var/log/ipaupgrade.log on REPLICA server. 4. Verified the same for following upgrade paths: - RHEL 7.1.z > RHEL 7.4 (Replica upgrade fails Logged separate bz1456774) - RHEL 7.2.z > RHEL 7.4 - RHEL 7.3.z > RHEL 7.4 Thus on the basis of above observation marking status of bug to "VERIFIED".
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2304