Description of problem: Following update to sssd-1.16.0-8.fc27.x86_64, unable to login as Kerberos user. sss_cache -E appears to resolve the issue for one user. Not sure what ultimately fixed my issue, time perhaps. Version-Release number of selected component (if applicable): Fedora 27 sssd-1.16.0-8.fc27.x86_64 How reproducible: Not sure, happened to at least two users on tech-list. Steps to Reproduce: 1. Apply patch. 2. Reboot. 3. Try to log in as a user from Kerberos. Actual results: Couldn't authenticate / log in. Expected results: Login successful. Additional info: Will attach some logs and dnf history info.
Created attachment 1413791 [details] dnf history info
Having this problem this morning as well, similarly after receiving sssd updates and rebooting to new kernel. sss_cache -E does not seem to have solved the issue. Would be happy to provide whatever debug material might be useful. # systemctl status sssd-kcm.socket Unit sssd-kcm.socket could not be found. I don't know much about debugging PAM or sssd, but as best I can tell authn is not consulting anything except localuser auth.
I should probably add that I do not know what fixed the issue for me. I found some errors in the sssd logs that indicated a couple options I had set weren't valid anymore. One of them I moved to a different section , the other I went without. I also commented the following line from /etc/krb5.conf: # dns_canonicalize_hostname = false Rebooted, still not working. Did a kinit successfully, still wasn't working, then I decided to get serious and tailed a bunch of log files to try to debug and tried it again, and it just worked. :-/
(In reply to Luke Meyer from comment #3) > Having this problem this morning as well, similarly after receiving sssd > updates and rebooting to new kernel. sss_cache -E does not seem to have > solved the issue. Would be happy to provide whatever debug material might be > useful. > > # systemctl status sssd-kcm.socket > Unit sssd-kcm.socket could not be found. > > I don't know much about debugging PAM or sssd, but as best I can tell authn > is not consulting anything except localuser auth. I would suggest to start by adding debug_level=10 into all the sssd.conf sections, restarting the sssd service and submitting the logs.
Thanks for filing this Jeffrey. sss_cache -E definitely resolved this for me and it definitely appeared after installing sssd-*-1.16.1-1.fc27.x86_64. I've been a little eager in vacuuming system journal but if there be logs elsewhere, I'll happily post them.
FWIW, I do not have the simple_allow_users config set; mine required cache clearing to resolve. I have not yet tried to duplicate the issue.
I hit this issue as well. My symptoms are exactly as Chris described in an email thread (upgraded and could no longer log in). I didn't know about the 'sss_cache -E' option. Instead I followed the steps below: 1. Logged in as a local user 2. Switched to root (su root) 3. Switched to my network user (su <username>) 4. kinit <username> 5. Logged out 6. Logged in as <username> I suspect this is the long winded way of just clearing the cache. I'm convinced that something changed in the latest version of sssd that is causing this issue. Its a little nerve racking because you can't even log in! Happy to provide any logs or debugging info. sssd-1.16.1-1.fc27.x86_64
Hi Siddharth, if these are indeed the only steps you undertook then you did not touch SSSD at all. However you did create a new ccache. I wonder if there is a SELinux labeling issue that is causing SSSD to fail to create/access the ccache the first time which is fixed by an unconfined user call to kinit ?
This occurred yesterday after an update to F-28 Same symptoms, same resolution. Before clearing the cache, I set log level to 9 and restarted sssd. Not sure there is anything useful in there but I'm attaching the timely output from `journalctl -u sssd` just in case.
Current versions: sssd-1.16.3-1.fc28.x86_64 sssd-krb5-1.16.3-1.fc28.x86_64 sssd-krb5-common-1.16.3-1.fc28.x86_64
Created attachment 1477530 [details] cbredesen's sssd log This log file was captured using log level 9 BUT I didn't put that config param in every section so I'm not sure it has all you need. Anyway here it is.
Michal is preparing an update that should fix this
I was impacted by this again today. Applied updates last night, picked up sssd-1.16.3-1. This morning when I booted up couldn't get in. I tried the following: 1. Login as local user. 2. Get on VPN. 3. sss_cache -E. 4. kinit #successfully 5. Still unable to log in. 6. Poke around for quite some time trying to figure it out. 7. Reboot. 8. Repeat 1-4 again, still unable to log in. 9. Change debug_level from 4 to 10 and restart sssd to debug. 10. It works. So, like last time, it's working again, but I don't really know why it takes so long to start working again.
From 1.16.3 announce[0]: .... IMPORTANT: The 1.16.2 release was storing the cached passwords without a salt prefix string. This bug was fixed in this release, but any password hashes generated by 1.16.2 are incompatible with the hashes generated by 1.16.3. The effect is that upgrade from 1.16.2 to 1.16.3 should be done when the authentication server is reachable so that the first authentication after the upgrade fix the cached password. .... [0] https://docs.pagure.org/SSSD.sssd/users/relnotes/notes_1_16_3.html
Comment 19 is very useful, thank you. A log line, visible at the default log level, would be great I think. When I peruse my sssd logs I see nothing about it blowing away cached passwords (did I miss it?). I'm not sure how much this is used in the wild but I can imagine it preventing a support case or several.
sssd-1.16.3-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-344f454318
sssd-1.16.3-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-71139c3aae
sssd-1.16.3-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-344f454318
sssd-1.16.3-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-71139c3aae
(In reply to Chris Bredesen from comment #20) > I'm not sure how > much this is used in the wild but I can imagine it preventing a support case > or several. Support cases are a RHEL thing and luckily this bug never made it to RHEL. We broke the hashes in 1.16.2 and fixed them again in 1.16.3, but we were able to backport the fix to RHEL-7.6 already.
I'm really sorry about the breakage. I really didn't connect the dots between this earlier report (in fact, I totally forget about it) and the bug and I thought that a release note upstream would be enough because I was under the impression that nobody hit the bug when we introduced it. The full story is that the bug was introduced in 1.16.2, in 1.16.3 we corrected the hashes, but as I said above, I didn't think many people are using this feature. Please, give the update that mzidek built (comments #23 and #24 have a link) a try and karma so that we can get the update to stable (especially F-27 never had the 1.16.3 version hit stable) so that fewer people are hit during an upgrade.
No worries Jakub, I appreciate the additional context. I can update my F28 station but how would I reliably duplicate this? Downgrade to .2, clear cache, create cache and then update to current testing version?
(In reply to Chris Bredesen from comment #27) > No worries Jakub, I appreciate the additional context. I can update my F28 > station but how would I reliably duplicate this? Downgrade to .2, clear > cache, create cache and then update to current testing version? I think the following should work: - downgrade to 1.16.2 - log in while online to put the 'wrong' hash into the cache - go offline (either disconnect from the VPN, shut down wifi or such) - upgrade to 1.16.3-1 while offline - log in With the broken version 1.16.3-1 you won't be able to log in. You should be able to log in with 1.16.3-2.
sssd-1.16.3-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
sssd-1.16.3-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.