Bug 1852041 - Settings to ignore rdns seem not to work
Summary: Settings to ignore rdns seem not to work
Alias: None
Product: Fedora
Classification: Fedora
Component: krb5
Version: 32
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Robbie Harwood
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2020-06-29 16:06 UTC by Gwyn Ciesla
Modified: 2020-07-15 01:12 UTC (History)
10 users (show)

Fixed In Version: krb5-1.18.2-10.fc33 krb5-1.18.2-10.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-07-15 01:12:18 UTC
Type: Bug

Attachments (Terms of Use)
KRB5 trace (15.95 KB, text/plain)
2020-06-29 16:51 UTC, Gwyn Ciesla
no flags Details
ssh log (19.29 KB, text/plain)
2020-06-29 17:16 UTC, Gwyn Ciesla
no flags Details
New KRB5 ssh trace (11.67 KB, text/plain)
2020-06-30 13:18 UTC, Gwyn Ciesla
no flags Details
krb5 ssh trace july 1 (27.77 KB, text/plain)
2020-07-01 19:49 UTC, Gwyn Ciesla
no flags Details

Description Gwyn Ciesla 2020-06-29 16:06:48 UTC
I've been running with my current Kerberos setup for over a year with no issues, used for authentication via GSSAPI and nfs4, among other things. In the last day, It's stopped working without any config changes on my part.  What seems to have changed is that krb5 is looking for host principals using the fqdn auto-assigned by my ISP, rather than the simple hostnames I've been using previously. I tried to add host principals for those hostnames to the client keytabs but can't, as they're not there already. I already use rdns = false, and dns_canonicalize_hostname = fallback and false didn't help.

Comment 1 Gwyn Ciesla 2020-06-29 16:13:52 UTC
To clarify it IS granting tickets, but not accepting them.

Comment 2 Simo Sorce 2020-06-29 16:28:18 UTC
It sounds like what you mean is that the KDC provides tickets to the client, but the server is not handling them.
So please provide:

- What service stopped working
- Logs of said service if you have any
- What client is being used
- What tickets do you find in the cache after attempt
- What ticket id you expect to see there

Provide versions as needed if a different library/program version leads to different results for any of the above

Comment 3 Gwyn Ciesla 2020-06-29 16:40:15 UTC
ssh connections with GSSAPI, mounting/listing an nfs4 share that uses krb5:krb5p:krb5i.

For example I'm using BAMBOO as my realm, and ssh to any host on this realm with -vvv gives:

debug1: Unspecified GSS failure.  Minor code may provide more information
Server krbtgt/HSD1.IL.COMCAST.NET@BAMBOO not found in Kerberos database
 It should be looking for krbtgt/BAMBOO@BAMBOO.

Comment 4 Simo Sorce 2020-06-29 16:47:40 UTC
Sounds like this may be a regression in SSH actually.

Can you please run the ssh client after setting KRB5_TRACE=/tmp/trace.txt and after a kdestroy and clean kinit ?

Then attach the trace.txt file that gets generated.

Comment 5 Gwyn Ciesla 2020-06-29 16:51:08 UTC
Created attachment 1699195 [details]
KRB5 trace

Here you are. Though an ssh regression wouldn't explain the NFS4 issue. Share is automounted with a systemd unit file.

Comment 6 Simo Sorce 2020-06-29 17:00:52 UTC
It would explain it if ssh does not forward credentials ...

The trace seem to indicate SSH is trying to connect to: bamboo.hsd1.il.comcast.net

What command line command have you used exactly ?
Did you ssh to a FQDN or are you using a shorthand ?

Comment 7 Gwyn Ciesla 2020-06-29 17:05:49 UTC
KRB5_TRACE=/tmp/trace.txt ssh bamboo.

bamboo.hsd1.il.comcast.net doesn't resolve.

Comment 8 Simo Sorce 2020-06-29 17:10:32 UTC
right, so can you tell what command line did you use to produce that log ?

Comment 9 Jakub Jelen 2020-06-29 17:12:48 UTC
Debug log from ssh would help too.

Comment 10 Gwyn Ciesla 2020-06-29 17:13:47 UTC
KRB5_TRACE=/tmp/trace.txt ssh bamboo, I'll attach that.

Comment 11 Gwyn Ciesla 2020-06-29 17:16:48 UTC
Created attachment 1699197 [details]
ssh log

Comment 12 Simo Sorce 2020-06-29 17:33:13 UTC
what does "host bamboo" return you on the command line ?
looks like a host resolution issue to me

Comment 13 Gwyn Ciesla 2020-06-29 17:35:27 UTC
How odd.

gwyn@gwythsefyll]$ host bamboo
Host bamboo not found: 3(NXDOMAIN)
[gwyn@gwythsefyll]$ ping bamboo
PING bamboo ( 56(84) bytes of data.
64 bytes from bamboo ( icmp_seq=1 ttl=64 time=87.7 ms

It's not in DNS, it's in /etc/hosts.  But it should still resolve.

Comment 14 Simo Sorce 2020-06-29 18:25:19 UTC
What's your host resolution stack in nsswitch.conf ?

Comment 15 Gwyn Ciesla 2020-06-29 18:26:54 UTC
hosts:      files mdns4_minimal [NOTFOUND=return] dns myhostname

Comment 16 Jakub Jelen 2020-06-29 18:43:16 UTC
Last thing that we modified around here was, which indeed could lead similar behavior if GSSAPITrustDns is set to yes. Possible workaround (or debugging steps) could be using GSSAPIServerIdentity configuration option in ssh_config.


But this change landed 3 months ago, so it might actually be some other change elsewhere. I would recommend you to try besict what version was last working for you  (of openssh, krb5, glibc, not sure what else could affect it).

Comment 17 Gwyn Ciesla 2020-06-29 18:52:21 UTC
That's what I was thinking, but only krb5 was updated in the last 24 hours, of that list. openssh June 16th, glibc April 24th. But I believe krb5 1.18.2-7 worked.

Comment 18 Simo Sorce 2020-06-29 19:25:27 UTC
Would it be possible to downgrade libkrb5 to confirm ?

Comment 19 Gwyn Ciesla 2020-06-29 19:44:50 UTC
It looks like it was actually 1.18-1 that was the old version, and it didn't help. Should I try downgrading the server as well?

Comment 20 Simo Sorce 2020-06-29 20:23:26 UTC
The issue is clearly client side, and looks related to host resolution.
If downgrading krb5 doesn't help it just means krb5 is not the change that triggered the issue, but some other package (or network configuration) change.

the fact you cannot resolve the hostname via the "host" command seem to indicate some major issue there, so I would focus on that.

Comment 21 Simo Sorce 2020-06-29 20:25:50 UTC
Forgot that "host" never look at files.

What do you expect your entry named "bamboo" to resolve to as a server name ?
"bamboo.bamboo" ? if so do you have that in your etc/hosts file ?

Comment 22 Gwyn Ciesla 2020-06-29 20:39:23 UTC
getent hosts bamboo
 returns    bamboo

Which matches what's in /etc/hosts.

Comment 23 Jakub Jelen 2020-06-30 09:07:06 UTC
We have another similar bug #1852320, where reporter claims downgrade of krb5-libs solved the issue so it really looks like krb5 regression.

Comment 24 Gwyn Ciesla 2020-06-30 13:18:56 UTC
Created attachment 1699304 [details]
New KRB5 ssh trace

This morning, with krb5-libs 1.18-1, NFS4 works, I can ssh with GSSAPI to a debian machine running krb5-user 1.17-3, and the KRB5 trace for failing ssh has changed.

Comment 25 Simo Sorce 2020-06-30 14:45:44 UTC
what principal name is your server expecting?

Is it literally host/bamboo@BAMBOO ?

Or something else ?

Comment 26 Simo Sorce 2020-06-30 14:46:24 UTC
are you aware of a recent change in libkrb5 that could trigger a different behavior in this case ?

Comment 27 Gwyn Ciesla 2020-06-30 14:52:01 UTC
Yes, host/bamboo@BAMBOO.

Comment 28 Gwyn Ciesla 2020-07-01 17:26:57 UTC
I reupgraded, and nfs still works, and ssh still doesn't.

Comment 29 Jakub Jelen 2020-07-01 19:45:27 UTC
The kerberos and ssh traces are still the same or is there anything new?

Comment 30 Gwyn Ciesla 2020-07-01 19:49:37 UTC
Created attachment 1699568 [details]
krb5 ssh trace july 1

I think so, but here it is in case I missed something.

Comment 31 Robbie Harwood 2020-07-01 20:33:00 UTC
Well, the most obvious change would be the setting of `dns_canonicalize_hostname = fallback`.  Can you check if this is the culprit by retrying with `dns_canonicalize_hostname = true` and `dns_canonicalize_hostname = false`?

Comment 32 Gwyn Ciesla 2020-07-01 20:37:55 UTC
I can confirm that none of those three options work.

Comment 33 Gwyn Ciesla 2020-07-02 13:06:33 UTC
...and now NFS doesn't work again.

Comment 34 Gwyn Ciesla 2020-07-06 18:42:00 UTC
Downgrading client and server to 1.8-1 fixes everything.

Comment 35 Robbie Harwood 2020-07-08 19:25:32 UTC
By 1.8-1 do you mean 1.18-1?  (And by extension: are you talking about krb5 versions here, or something else?)

With 1.18.2, do you see any change when you set `qualify_shortname = ""` with `dns_canonicalize_hostname = false` or with `dns_canonicalize_hostname = fallback`?

Comment 36 Gwyn Ciesla 2020-07-08 19:44:25 UTC
I do, yes. And it appears that 1.18.2 works with:

    qualify_shortname = ""
    dns_canonicalize_hostname = false

Comment 37 Fedora Update System 2020-07-08 20:49:37 UTC
FEDORA-2020-1be0542fdd has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-1be0542fdd

Comment 38 Fedora Update System 2020-07-09 19:58:32 UTC
FEDORA-2020-1be0542fdd has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-1be0542fdd`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-1be0542fdd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 39 Fedora Update System 2020-07-15 01:12:18 UTC
FEDORA-2020-1be0542fdd has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.