I've been running with my current Kerberos setup for over a year with no issues, used for authentication via GSSAPI and nfs4, among other things. In the last day, It's stopped working without any config changes on my part. What seems to have changed is that krb5 is looking for host principals using the fqdn auto-assigned by my ISP, rather than the simple hostnames I've been using previously. I tried to add host principals for those hostnames to the client keytabs but can't, as they're not there already. I already use rdns = false, and dns_canonicalize_hostname = fallback and false didn't help.
To clarify it IS granting tickets, but not accepting them.
It sounds like what you mean is that the KDC provides tickets to the client, but the server is not handling them.
So please provide:
- What service stopped working
- Logs of said service if you have any
- What client is being used
- What tickets do you find in the cache after attempt
- What ticket id you expect to see there
Provide versions as needed if a different library/program version leads to different results for any of the above
ssh connections with GSSAPI, mounting/listing an nfs4 share that uses krb5:krb5p:krb5i.
For example I'm using BAMBOO as my realm, and ssh to any host on this realm with -vvv gives:
debug1: Unspecified GSS failure. Minor code may provide more information
Server krbtgt/HSD1.IL.COMCAST.NET@BAMBOO not found in Kerberos database
It should be looking for krbtgt/BAMBOO@BAMBOO.
Sounds like this may be a regression in SSH actually.
Can you please run the ssh client after setting KRB5_TRACE=/tmp/trace.txt and after a kdestroy and clean kinit ?
Then attach the trace.txt file that gets generated.
Created attachment 1699195 [details]
Here you are. Though an ssh regression wouldn't explain the NFS4 issue. Share is automounted with a systemd unit file.
It would explain it if ssh does not forward credentials ...
The trace seem to indicate SSH is trying to connect to: bamboo.hsd1.il.comcast.net
What command line command have you used exactly ?
Did you ssh to a FQDN or are you using a shorthand ?
KRB5_TRACE=/tmp/trace.txt ssh bamboo.
bamboo.hsd1.il.comcast.net doesn't resolve.
right, so can you tell what command line did you use to produce that log ?
Debug log from ssh would help too.
KRB5_TRACE=/tmp/trace.txt ssh bamboo, I'll attach that.
Created attachment 1699197 [details]
what does "host bamboo" return you on the command line ?
looks like a host resolution issue to me
gwyn@gwythsefyll]$ host bamboo
Host bamboo not found: 3(NXDOMAIN)
[gwyn@gwythsefyll]$ ping bamboo
PING bamboo (192.168.0.13) 56(84) bytes of data.
64 bytes from bamboo (192.168.0.13): icmp_seq=1 ttl=64 time=87.7 ms
It's not in DNS, it's in /etc/hosts. But it should still resolve.
What's your host resolution stack in nsswitch.conf ?
hosts: files mdns4_minimal [NOTFOUND=return] dns myhostname
Last thing that we modified around here was, which indeed could lead similar behavior if GSSAPITrustDns is set to yes. Possible workaround (or debugging steps) could be using GSSAPIServerIdentity configuration option in ssh_config.
But this change landed 3 months ago, so it might actually be some other change elsewhere. I would recommend you to try besict what version was last working for you (of openssh, krb5, glibc, not sure what else could affect it).
That's what I was thinking, but only krb5 was updated in the last 24 hours, of that list. openssh June 16th, glibc April 24th. But I believe krb5 1.18.2-7 worked.
Would it be possible to downgrade libkrb5 to confirm ?
It looks like it was actually 1.18-1 that was the old version, and it didn't help. Should I try downgrading the server as well?
The issue is clearly client side, and looks related to host resolution.
If downgrading krb5 doesn't help it just means krb5 is not the change that triggered the issue, but some other package (or network configuration) change.
the fact you cannot resolve the hostname via the "host" command seem to indicate some major issue there, so I would focus on that.
Forgot that "host" never look at files.
What do you expect your entry named "bamboo" to resolve to as a server name ?
"bamboo.bamboo" ? if so do you have that in your etc/hosts file ?
getent hosts bamboo
Which matches what's in /etc/hosts.
We have another similar bug #1852320, where reporter claims downgrade of krb5-libs solved the issue so it really looks like krb5 regression.
Created attachment 1699304 [details]
New KRB5 ssh trace
This morning, with krb5-libs 1.18-1, NFS4 works, I can ssh with GSSAPI to a debian machine running krb5-user 1.17-3, and the KRB5 trace for failing ssh has changed.
what principal name is your server expecting?
Is it literally host/bamboo@BAMBOO ?
Or something else ?
are you aware of a recent change in libkrb5 that could trigger a different behavior in this case ?
I reupgraded, and nfs still works, and ssh still doesn't.
The kerberos and ssh traces are still the same or is there anything new?
Created attachment 1699568 [details]
krb5 ssh trace july 1
I think so, but here it is in case I missed something.
Well, the most obvious change would be the setting of `dns_canonicalize_hostname = fallback`. Can you check if this is the culprit by retrying with `dns_canonicalize_hostname = true` and `dns_canonicalize_hostname = false`?
I can confirm that none of those three options work.
...and now NFS doesn't work again.
Downgrading client and server to 1.8-1 fixes everything.
By 1.8-1 do you mean 1.18-1? (And by extension: are you talking about krb5 versions here, or something else?)
With 1.18.2, do you see any change when you set `qualify_shortname = ""` with `dns_canonicalize_hostname = false` or with `dns_canonicalize_hostname = fallback`?
I do, yes. And it appears that 1.18.2 works with:
qualify_shortname = ""
dns_canonicalize_hostname = false
FEDORA-2020-1be0542fdd has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-1be0542fdd
FEDORA-2020-1be0542fdd has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-1be0542fdd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-1be0542fdd
See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-1be0542fdd has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.