With recent Fedora 28 nightly composes - after the freeipa and systemd updates that make new deployments of FreeIPA work - the upgrade test fails. The upgrade test runs a server deployment on the previous release (so, Fedora 27 in this case), then upgrades it to the new release, and checks it still works. Which...it doesn't. On the first boot after running the upgrade, ipa.service fails to start. The journal shows this: Mar 19 10:26:59 ipa001.domain.local ipactl[839]: IPA version error: data needs to be upgraded (expected version '4.6.90.pre1-1.fc28', current version '4.6.3-2.fc27') Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Automatically running upgrade, for details see /var/log/ipaupgrade.log Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Be patient, this may take a few minutes. Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Automatic upgrade failed: Update complete Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Upgrading the configuration of the IPA services Mar 19 10:26:59 ipa001.domain.local ipactl[839]: [Verifying that root certificate is published] Mar 19 10:26:59 ipa001.domain.local ipactl[839]: [Migrate CRL publish directory] Mar 19 10:26:59 ipa001.domain.local ipactl[839]: CRL tree already moved Mar 19 10:26:59 ipa001.domain.local ipactl[839]: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Unexpected error - see /var/log/ipaupgrade.log for details: Mar 19 10:26:59 ipa001.domain.local ipactl[839]: KeyError: 'WSGI_PREFIX_DIR' Mar 19 10:26:59 ipa001.domain.local ipactl[839]: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information Mar 19 10:26:59 ipa001.domain.local ipactl[839]: See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again Mar 19 10:26:59 ipa001.domain.local ipactl[839]: Aborting ipactl Mar 19 10:26:59 ipa001.domain.local systemd[1]: ipa.service: Main process exited, code=exited, status=1/FAILURE Mar 19 10:26:59 ipa001.domain.local systemd[1]: ipa.service: Failed with result 'exit-code'. Mar 19 10:26:59 ipa001.domain.local systemd[1]: Failed to start Identity, Policy, Audit. /var/log/ipaupgrade.log doesn't show much more, just this: 2018-03-19T17:26:58Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2018-03-19T17:26:58Z DEBUG File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute return_value = self.run() File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run server.upgrade() File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1953, in upgrade upgrade_configuration() File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1656, in upgrade_configuration "ipa.conf.template")) File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 191, in upgrade_file update_conf(sub_dict, filename, template) File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 117, in update_conf template = ipautil.template_file(template_filename, sub_dict) File "/usr/lib/python3.6/site-packages/ipapython/ipautil.py", line 313, in template_file return template_str(f.read(), vars) File "/usr/lib/python3.6/site-packages/ipapython/ipautil.py", line 300, in template_str val = string.Template(txt).substitute(vars) File "/usr/lib64/python3.6/string.py", line 130, in substitute return self.pattern.sub(convert, self.template) File "/usr/lib64/python3.6/string.py", line 123, in convert return str(mapping[named]) 2018-03-19T17:26:58Z DEBUG The ipa-server-upgrade command failed, exception: KeyError: 'WSGI_PREFIX_DIR' 2018-03-19T17:26:58Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: KeyError: 'WSGI_PREFIX_DIR' 2018-03-19T17:26:58Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information I will attach the full /var/log tarball for inspection. Proposing as a Beta blocker per the upgrade criterion, and also the discussion in https://bugzilla.redhat.com/show_bug.cgi?id=1503321 , where we stated: "The decision to classify this bug as an AcceptedBlocker was made, even though there is no current criteria to warrant blocking on this. We plan to make a near-future change to the criteria so that a bug like this from this point on will be blocker-qualifying." I think we never actually got around to doing that, but the intent is still on record :/
Created attachment 1410226 [details] tarball of all of /var/log from a failure
I have a fix for this, tracked as https://pagure.io/freeipa/issue/7454 upstream.
freeipa-4.6.90.pre1-3.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
freeipa-4.6.90.pre1-3.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
Fails with a similar error for a different variable, now: 2018-03-20T22:41:54Z DEBUG The ipa-server-upgrade command failed, exception: KeyError: 'GSSAPI_SESSION_KEY' I'm gonna call this basically the same issue, as it looks like just another value missing from the same template dictionary. We can call the issue fixed when *all* the values that ought to be in this dict, are in it. :P
Looking at this through a Logical Eye, it looks an awful lot like this was caused by: https://pagure.io/freeipa/c/e6c707b168067ebb3705c21efc377acd29b23fff thus, we need to add *all* the bits that commit added to httpinstance.py , to the sub_dict used here as well. I'm adjusting (by hand, cos that's just how I roll) ab's patch in the Fedora Rawhide and F28 package builds to do this, and sending out a -4 build. I'll comment on the upstream bug to make sure it gets done upstream as well.
freeipa-4.6.90.pre1-4.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
Testing indicates that -4 fixes this, but we run into two other issues: https://bugzilla.redhat.com/show_bug.cgi?id=1558817 https://bugzilla.redhat.com/show_bug.cgi?id=1558818
freeipa-4.6.90.pre1-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
freeipa-4.6.90.pre1-6.1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
+1 blocker
+1 Blocker
That's +3 (counting me), setting accepted.
freeipa-4.6.90.pre1-6.1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.