Bug 1401088

Summary: IPA upgrade of replica without DNS fails during restart of named-pkcs11
Product: Red Hat Enterprise Linux 7 Reporter: Petr Vobornik <pvoborni>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Nikhil Dehadrai <ndehadra>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: mbasti, ndehadra, pvoborni, rcritten, tscherf
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.4.0-14.el7.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1404169 (view as bug list) Environment:
Last Closed: 2017-08-01 09:44:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1404169    

Description Petr Vobornik 2016-12-02 18:42:45 UTC
This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/6503

When upgrading a pre-4.4 replica which has no IPA DNS configured (there is IPA DNS server in topology) to 4.4.2 (or ipa-4.4.0-12.el7), the upgrade erroneously tries to restart named-pkcs11 and fails:

{{{
2016-11-22T14:12:02Z DEBUG args=/bin/systemctl start named-pkcs11.service
2016-11-22T14:12:02Z DEBUG Process finished, return code=1
2016-11-22T14:12:02Z DEBUG stdout=
2016-11-22T14:12:02Z DEBUG stderr=Job for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details.

2016-11-22T14:12:02Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2016-11-22T14:12:02Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1867, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1725, in upgrade_configuration
    bind.start()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 345, in start
    self.service.start(instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 285, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 515, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))

2016-11-22T14:12:02Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1
2016-11-22T14:12:02Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1
2016-11-22T14:12:02Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
}}}

Upon further inspection the failure was tracked into the following piece of code which checks whether bind is configured and running: https://git.fedorahosted.org/cgit/freeipa.git/tree/ipaserver/install/server/upgrade.py?h=ipa-4-4#n1722

The issue is that there is the following content in sysrestore.state dumped even if DNS is not configured on replica:

{{{
[named]
dns_record_0_0 = SRV _ldap._tcp 0 100 389 replica1
dns_record_0_1 = SRV _kerberos._tcp 0 100 88 replica1
dns_zone_0 = ipa.test
dns_record_0_3 = SRV _kerberos-master._tcp 0 100 88 replica1
dns_record_0_2 = SRV _kerberos._udp 0 100 88 replica1
dns_record_0_5 = SRV _kpasswd._tcp 0 100 464 replica1
dns_record_0_4 = SRV _kerberos-master._udp 0 100 88 replica1
dns_record_0_7 = SRV _ntp._udp 0 100 123 replica1
dns_record_0_6 = SRV _kpasswd._udp 0 100 464 replica1
}}}

Since the `StateFile.has_state` logic just checks for the presence of the section in the state file, it erroneously thinks that named is configured, hence the error.

We need to either fix `StateFile` to actually search for enabled=True to decide whether the service was enabled, or fix `BindInstance` to not dump zone info into state file.

Steps to reproduce:

1,) install 4.2 or 4.3 master w/ DNS

2.) create a replica from the master w/o DNS

3.) upgrade master to IPA v4.4

4.) now try to update the replica to the same version

Expected results:

The replica upgrades successfully

Actual results:

The upgrade fails with the following error:

{{{
 Cleanup     : freeipa-common-4.3.2-2.fc24.noarch                                                            19/19 
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start named-pkcs11.service' returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
}}}

Comment 6 Nikhil Dehadrai 2017-05-31 12:35:55 UTC
ipa-server version: ipa-server-4.5.0-13.el7.x86_64

Tested the bug with following setup:
1) install 4.2 master w/ DNS (In my case 4.4.0-14, 4.2.0-15)
2) create a replica from the master w/o DNS 
3) upgrade master to IPA (In my case 4.5.0-13)
4) now try to update the replica to the same version

Observations:
1. Verified that on upgrading the replica server (when setup without DNS) to latest version it is upgraded successfully.
2. Upon upgrade the "ipactl status" and "ipactl restart" command are run successfully both on IPA-Master and Replica.
3. No failures are observed under /var/log/ipaupgrade.log on REPLICA server.
4. Verified the same for following upgrade paths:
- RHEL 7.1.z > RHEL 7.4 (Replica upgrade fails Logged separate bz1456774)
- RHEL 7.2.z > RHEL 7.4
- RHEL 7.3.z > RHEL 7.4

Thus on the basis of above observation marking status of bug to "VERIFIED".

Comment 8 errata-xmlrpc 2017-08-01 09:44:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304