Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionMartin Babinsky
2017-05-23 11:46:13 UTC
Description of problem:
When testing upgrade from RHEL 7.3 ipa-server image (version 4.4.0-41 from official registry) to a development version of RHEL 7.4 image, the upgrade code fails due to attempting to unconfigure ipa_memcache service which is not present in FreeIPA 4.5.x:
```console
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/8]: saving configuration
[2/8]: disabling listeners
[3/8]: enabling DS global lock
[4/8]: starting directory server
[5/8]: updating schema
[6/8]: upgrading server
[7/8]: stopping directory server
[8/8]: restoring configuration
Done.
Update complete
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
Publish directory already set to new location
Upgraded /etc/httpd/conf.d/ipa.conf to version 26
Upgraded /etc/httpd/conf.d/ipa-pki-proxy.conf to version 10
/etc/dirsrv/slapd-EXAMPLE-TEST/certmap.conf is now managed by IPA. It will be overwritten. A backup of the original will be made.
Upgraded /etc/dirsrv/slapd-EXAMPLE-TEST/certmap.conf to version 3
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
Upgrading IPA services
Unconfiguring ipa_memcached
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl stop ipa_memcached.service' returned non-zero exit status 5
```
ipaupgrade.log shows:
```console
Starting external process
2017-05-23T08:05:01Z DEBUG args=/bin/systemctl stop ipa_memcached.service
2017-05-23T08:05:02Z DEBUG Process finished, return code=5
2017-05-23T08:05:02Z DEBUG stdout=
2017-05-23T08:05:02Z DEBUG stderr=Failed to stop ipa_memcached.service: Unit ipa_memcached.service not loaded.
2017-05-23T08:05:02Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-05-23T08:05:02Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1905, in upgrade
upgrade_configuration()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1655, in upgrade_configuration
uninstall_ipa_memcached()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 87, in uninstall_ipa_memcached
ipa_memcached.uninstall()
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 681, in uninstall
self.stop()
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 398, in stop
self.service.stop(instance_name, capture_output=capture_output)
File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 283, in stop
ipautil.run(args, skip_output=not capture_output)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 495, in run
raise CalledProcessError(p.returncode, arg_string, str(output))
```
The root cause of this error is in the fact that the new image bind-mounts the old image data volume whose var/lib/ipa/sysrestore/sysrestore.state holds configuration status for ipa_memcached.
This leads the upgrader to attempt to unconfigure the service, however there is no ipa_memcached unit file present anymore, so an attempt to stop it inevitably fails. Additionally, even if the state file lists the service as not running and not enabled, the uninstaller still attempts to unconditionally stop and disable the service.
We should either honor the settings in the state file or make it possible to recover gracefully from the mismatch between the reported state and actual state in the environment.
Version-Release number of selected component (if applicable):
ipa-version in the RHEL 7.4 image: ipa-server-common.noarch 0:4.5.0-13.el7
How reproducible:
Always
Steps to Reproduce:
1. install ipa-server container from registry.access.redhat.com/rhel7/ipa-server:latest
2. upgrade to a devel version of ipa-server RHEL 7.4 image
Actual results:
Upgrade fails on failed attempt to stop ipa_memcached
Expected results:
Upgrade is successful and the new container can continue to work with old data
Verified the bug on the basis of following observations:
Version-Release number of selected component (if applicable):
ipa-docker image: 4.5.0.8
BIND:
bind-dyndb-ldap-11.1-4.el7.x86_64
bind-9.9.4-51.el7.x86_64
IPA-VERSION:
ipa-server-4.5.0-21.el7.x86_64
Atomic host version:
-bash-4.2# atomic host status
State: idle
Deployments:
● atomic-host:rhel-atomic-host/7/x86_64/standard
Version: 7.4.0 (2017-07-28 00:26:01)
Commit: 846fb0e18e65bd9a62fc9d952627413c6467c33c2d726449a1d7ad7690bbb93a
Steps
1. Setup IPA server and REPLICA server using ipa-docker image(4.4.0.45 i.e RHEL 7.3.z).
2. Upgrade the ipa-server and replica using latest image.(4.5.0.8)
Actual results:
1. After step2, Upgrade is successful and the issue mentioned is not observed.
2. But after upgrade the user is unable to login to server UI for IPA/Replica both and same error message "Login failed due to an unknown reason." is noticed. (In my case from rhel 7.3.z to rhel 7.4.z). Thus we have logged a separate bug BZ1476782 for it.
Thus on the basis of above observations , marking this bug as "VERIFIED"
Description of problem: When testing upgrade from RHEL 7.3 ipa-server image (version 4.4.0-41 from official registry) to a development version of RHEL 7.4 image, the upgrade code fails due to attempting to unconfigure ipa_memcache service which is not present in FreeIPA 4.5.x: ```console Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/8]: saving configuration [2/8]: disabling listeners [3/8]: enabling DS global lock [4/8]: starting directory server [5/8]: updating schema [6/8]: upgrading server [7/8]: stopping directory server [8/8]: restoring configuration Done. Update complete Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] Publish directory already set to new location Upgraded /etc/httpd/conf.d/ipa.conf to version 26 Upgraded /etc/httpd/conf.d/ipa-pki-proxy.conf to version 10 /etc/dirsrv/slapd-EXAMPLE-TEST/certmap.conf is now managed by IPA. It will be overwritten. A backup of the original will be made. Upgraded /etc/dirsrv/slapd-EXAMPLE-TEST/certmap.conf to version 3 [Verifying that CA proxy configuration is correct] [Verifying that KDC configuration is using ipa-kdb backend] Upgrading IPA services Unconfiguring ipa_memcached IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl stop ipa_memcached.service' returned non-zero exit status 5 ``` ipaupgrade.log shows: ```console Starting external process 2017-05-23T08:05:01Z DEBUG args=/bin/systemctl stop ipa_memcached.service 2017-05-23T08:05:02Z DEBUG Process finished, return code=5 2017-05-23T08:05:02Z DEBUG stdout= 2017-05-23T08:05:02Z DEBUG stderr=Failed to stop ipa_memcached.service: Unit ipa_memcached.service not loaded. 2017-05-23T08:05:02Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-05-23T08:05:02Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1905, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1655, in upgrade_configuration uninstall_ipa_memcached() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 87, in uninstall_ipa_memcached ipa_memcached.uninstall() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 681, in uninstall self.stop() File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 398, in stop self.service.stop(instance_name, capture_output=capture_output) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 283, in stop ipautil.run(args, skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 495, in run raise CalledProcessError(p.returncode, arg_string, str(output)) ``` The root cause of this error is in the fact that the new image bind-mounts the old image data volume whose var/lib/ipa/sysrestore/sysrestore.state holds configuration status for ipa_memcached. This leads the upgrader to attempt to unconfigure the service, however there is no ipa_memcached unit file present anymore, so an attempt to stop it inevitably fails. Additionally, even if the state file lists the service as not running and not enabled, the uninstaller still attempts to unconditionally stop and disable the service. We should either honor the settings in the state file or make it possible to recover gracefully from the mismatch between the reported state and actual state in the environment. Version-Release number of selected component (if applicable): ipa-version in the RHEL 7.4 image: ipa-server-common.noarch 0:4.5.0-13.el7 How reproducible: Always Steps to Reproduce: 1. install ipa-server container from registry.access.redhat.com/rhel7/ipa-server:latest 2. upgrade to a devel version of ipa-server RHEL 7.4 image Actual results: Upgrade fails on failed attempt to stop ipa_memcached Expected results: Upgrade is successful and the new container can continue to work with old data