Bug 1558818

Summary: FreeIPA server upgrade from Fedora 27 to Fedora 28 fails during upgrade boot with "kadmin.local: No KCM server found while opening default credentials cache"
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: freeipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 28CC: abokovoy, ipa-maint, jcholast, jhrozek, pvoborni, rcritten, robatino, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: freeipa-4.6.90.pre1-6.1.fc28 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-21 20:31:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1469205    
Attachments:
Description Flags
tarball of all of /var/log from a failure none

Description Adam Williamson 2018-03-21 05:01:45 UTC 
After https://bugzilla.redhat.com/show_bug.cgi?id=1558354 is fully fixed (i.e. with freeipa-4.6.90.pre1-4.fc28), upgrade of a FreeIPA server from Fedora 27 to Fedora 28 still fails. There seem to be two different failures, one during the actual upgrade boot, one during the first boot after upgrade: the IPA upgrade process runs both times, and fails differently each time.

This bug is for the *first* failure, the one that occurs during the system upgrade boot itself. https://bugzilla.redhat.com/show_bug.cgi?id=1558817 is the bug for the *second* failure, the one that occurs during the first boot after upgrade.

In this case, we see this in ipaupgrade.log:

2018-03-21T03:00:48Z DEBUG args=['/usr/sbin/kadmin.local', '-q', 'addprinc -randkey dogtag/ipa001.domain.local@DOMAIN.
LOCAL', '-x', 'ipa-setup-override-restrictions']
2018-03-21T03:00:48Z DEBUG Process finished, return code=1
2018-03-21T03:00:48Z DEBUG stdout=
2018-03-21T03:00:48Z DEBUG stderr=kadmin.local: No KCM server found while opening default credentials cache

Attaching a tarball of the whole /var/log archive, including logs for *both* failures. The logs for this failure are in the second-to-last boot in the logs, around 20:00 (system local time - journal timestamps), 03:00 (UTC time - IPA log timestamps).

Proposing as a Beta blocker. We should probably establish whether fixing either, both, or one specific one of the two bugs is necessary to make the upgrade work successfully.
Comment 1 Adam Williamson 2018-03-21 05:02:05 UTC 
Created attachment 1410941 [details]
tarball of all of /var/log from a failure
Comment 2 Alexander Bokovoy 2018-03-21 05:35:10 UTC 
If we are in F28 already, our krb5.conf points to use KCM as a default credentials cache, so kadmin.local tries to use it. However, SSSD might be offline at this point and thus KCM is not available.

For kadmin.local it does not matter what we are using because there is no real authentication involved, so perhaps for the purpose of the IPA upgrade process we could use some empty FILE-based credentials cache to avoid such issues.
Comment 3 Fedora Update System 2018-03-21 10:38:41 UTC 
freeipa-4.6.90.pre1-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
Comment 4 Fedora Update System 2018-03-21 14:13:06 UTC 
freeipa-4.6.90.pre1-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
Comment 5 Adam Williamson 2018-03-21 16:10:02 UTC 
I'm going to change the proposal here to Beta FE, as it turns out the other bug - https://bugzilla.redhat.com/show_bug.cgi?id=1558817 - is really more critical here. This one is kind of a consequence of that one (sssd failing). It'd still be nice to have this fixed, though, to give the upgrade a higher chance of working if there *is* some issue with sssd.
Comment 6 Fedora Update System 2018-03-21 16:25:13 UTC 
freeipa-4.6.90.pre1-6.1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-8cff0f34f6
Comment 7 Fedora Update System 2018-03-21 20:31:47 UTC 
freeipa-4.6.90.pre1-6.1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.