Bug 1463665 - Failure to obtain ticket when default credential in a collection is not available
Failure to obtain ticket when default credential in a collection is not avail...
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: krb5 (Show other bugs)
rawhide
Unspecified Unspecified
medium Severity unspecified
: ---
: ---
Assigned To: Robbie Harwood
Fedora Extras Quality Assurance
https://github.com/krb5/krb5/pull/687
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-21 08:50 EDT by Bastien Nocera
Modified: 2017-09-05 15:05 EDT (History)
11 users (show)

See Also:
Fixed In Version: krb5-1.15.1-27
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-05 15:05:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libsoup-based reproducer (1.02 KB, text/plain)
2017-06-30 11:25 EDT, Debarshi Ray
no flags Details

  None (edit)
Description Bastien Nocera 2017-06-21 08:50:39 EDT
krb5-libs-1.15.1-8.fc26.x86_64
webkitgtk4-2.16.4-1.fc26.x86_64
gnome-online-accounts-3.24.1-1.fc26.x86_64

1. In Online Accounts, create Kerberos logins for FEDORAPROJECT.ORG and REDHAT.COM
2. Run klist:
$ klist -A
Ticket cache: KEYRING:persistent:1000:krb_ccache_WBWOu0s
Default principal: foo@REDHAT.COM

Valid starting     Expires            Service principal
21/06/17 12:12:13  21/06/17 22:12:09  HTTP/errata-web.redhat.com@REDHAT.COM
21/06/17 12:12:13  21/06/17 22:12:09  HTTP/errata-web.redhat.com@
21/06/17 12:12:09  21/06/17 22:12:09  krbtgt/REDHAT.COM@REDHAT.COM

Ticket cache: KEYRING:persistent:1000:1000
Default principal: foo@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
21/06/17 11:25:09  22/06/17 11:25:08  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
	renew until 26/06/17 23:25:00

2. Wait until the REDHAT.COM ticket expires (or run "kinit -l 15m foo@REDHAT.COM" to make the timeout shorter)
3. Run klist again:
$ klist
klist: Credentials cache keyring 'persistent:1000:krb_ccache_WBWOu0s' not found
$ klist -A
Ticket cache: KEYRING:persistent:1000:krb_ccache_KFw8AGB
Default principal: foo@REDHAT.COM

Valid starting     Expires            Service principal
21/06/17 14:06:58  22/06/17 00:06:58  krbtgt/REDHAT.COM@REDHAT.COM

Ticket cache: KEYRING:persistent:1000:1000
Default principal: foo@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
21/06/17 11:25:09  22/06/17 11:25:08  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
	renew until 26/06/17 23:25:00

Running "kswitch -p foo@REDHAT.COM" allows accessing an SSO enabled website again through epiphany/webkit (the internal RH errata tool)
Comment 1 Robbie Harwood 2017-06-26 14:26:38 EDT
I believe epiphany is using bare krb5, not GSSAPI.  As such, it will not be able to take advantage of the dynamic realm matching logic that would make this work.
Comment 2 Bastien Nocera 2017-06-27 06:02:20 EDT
(In reply to Robbie Harwood from comment #1)
> I believe epiphany is using bare krb5, not GSSAPI.

Reference? Because I'm pretty certain it does have GSSAPI.
Comment 3 Robbie Harwood 2017-06-27 11:22:15 EDT
(In reply to Bastien Nocera from comment #2)
> (In reply to Robbie Harwood from comment #1)
> > I believe epiphany is using bare krb5, not GSSAPI.
> 
> Reference? Because I'm pretty certain it does have GSSAPI.

I don't know.  I honestly cannot find the project source after fifteen minutes of looking.  Since you seem confident that it does, perhaps you have a link you could share?
Comment 4 Bastien Nocera 2017-06-27 11:50:55 EDT
(In reply to Robbie Harwood from comment #3)
> (In reply to Bastien Nocera from comment #2)
> > (In reply to Robbie Harwood from comment #1)
> > > I believe epiphany is using bare krb5, not GSSAPI.
> > 
> > Reference? Because I'm pretty certain it does have GSSAPI.
> 
> I don't know.  I honestly cannot find the project source after fifteen
> minutes of looking.

epiphany lives in GNOME git:
http://git.gnome.org/browse/epiphany/

But that's not going to be much help, as the GSSAPI is implemented neither in WebKitGTK+ not epiphany, but in libsoup.

>  Since you seem confident that it does, perhaps you have
> a link you could share?

libsoup implementation, in GNOME git as well:
https://git.gnome.org/browse/libsoup/tree/libsoup/soup-auth-negotiate.c

See also:
https://bugzilla.redhat.com/show_bug.cgi?id=1329368

Dan would know if there are particular quirks to the libsoup implementation of GSSAPI.
Comment 5 Dan Winship 2017-06-27 12:15:46 EDT
(In reply to Bastien Nocera from comment #4)
> Dan would know if there are particular quirks to the libsoup implementation
> of GSSAPI.

I really only understand that code as "http auth method that does mysterious things". Tomas Popela probably understands the GSSAPI-specific bits better than I do.

But I would imagine it's just a matter of passing some flag to the gss_init_sec_context() call in soup_gss_client_step() or something like that...
Comment 6 Alexander Bokovoy 2017-06-27 14:52:48 EDT
GSSAPI code in libsoup should be able to choose proper ticket indeed. However, the logic might break if there is no way to find out which realm is the target service mapped into.

So I would start with looking into what realm that particular host is mapped into.  Can you show your /etc/krb5.conf?
Comment 7 Bastien Nocera 2017-06-28 05:42:28 EDT
(In reply to Alexander Bokovoy from comment #6)
> GSSAPI code in libsoup should be able to choose proper ticket indeed.
> However, the logic might break if there is no way to find out which realm is
> the target service mapped into.
> 
> So I would start with looking into what realm that particular host is mapped
> into.  Can you show your /etc/krb5.conf?

It's the same one that's in the krb5-libs package. See comment 0.
Comment 8 Alexander Bokovoy 2017-06-28 05:51:43 EDT
For the sake of this discussion, I'll paste krb5-libs package's krb5.conf here:

-----------------
# To opt out of the system crypto-policies configuration of krb5, remove the
# symlink at /etc/krb5.conf.d/crypto-policies which will not be recreated.
includedir /etc/krb5.conf.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
# default_realm = EXAMPLE.COM

[realms]
# EXAMPLE.COM = {
#  kdc = kerberos.example.com
#  admin_server = kerberos.example.com
# }

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM
-----------------

Several comments:

1. there are no explicitly defined realms and mappings between DNS domains and realms in the default configuration. That's fine for defaults.

2. There is no way to look up realm information from DNS (dns_lookup_realm = false). It means when realm is unknown, libkrb5 will not be doing search of a TXT record in the DNS domain to know what realm is that.

3. SRV records in the DNS domain would be used to discover KDCs because this is enabled by default.

As result, if your realm provides SRV records in the DNS to find out a proper KDC for the realm, this will work automatically.

For FEDORAPROJECT.ORG realm we ship special configuration that defines it in fedora-packager-0.6.0.1-2.fc26.noarch (/etc/krb5.conf.d/fedoraproject_org file).

As for REDHAT.COM, there is nothing in redhat.com's DNS and nothing in Fedora packages that would provide such information. Thus, libkrb5 needs explicit configuration for REDHAT.COM to deal with it. Do you have it explicitly configured in one of snippets in /etc/krb5.conf.d/? If not, that's the problem you have here.
Comment 9 Bastien Nocera 2017-06-28 06:19:09 EDT
(In reply to Alexander Bokovoy from comment #8)
> For the sake of this discussion, I'll paste krb5-libs package's krb5.conf
> here:
<snip>
> For FEDORAPROJECT.ORG realm we ship special configuration that defines it in
> fedora-packager-0.6.0.1-2.fc26.noarch (/etc/krb5.conf.d/fedoraproject_org
> file).

Which I don't have installed, because fedora-packager isn't installed, koji and fedpkg are (and I'm not sure what I'd do with YubiKey support packages either).

Why would the krb5 snippet be in fedora-packager instead of in the package that would consumne it?
$ grep -i realm /etc/rpkg/fedpkg.conf 
kerberos_realms = FEDORAPROJECT.ORG
$ rpm -qf /etc/rpkg/fedpkg.conf
fedpkg-1.28-1.fc26.noarch

> As for REDHAT.COM, there is nothing in redhat.com's DNS and nothing in
> Fedora packages that would provide such information. Thus, libkrb5 needs
> explicit configuration for REDHAT.COM to deal with it. Do you have it
> explicitly configured in one of snippets in /etc/krb5.conf.d/? If not,
> that's the problem you have here.

$ ls -l /etc/krb5.conf.d/
total 0
0 lrwxrwxrwx. 1 root root 42 Apr 28 22:36 crypto-policies -> /etc/crypto-policies/back-ends/krb5.config

Seems like the defaults are broken, no?
Comment 10 Alexander Bokovoy 2017-06-28 06:39:39 EDT
no, defaults aren't broken. Remember that defaults are to look up KDC for a known realm in DNS.

The difference between FEDORAPROJECT.ORG is REDHAT.COM realms is that the former one is publishing its configuration in DNS while the latter one is not, at least for the public DNS view:

$ dig -t SRV +short _kerberos._tcp.fedoraproject.org
20 1 1088 id.fedoraproject.org.
$

$ dig -t SRV +short _kerberos._tcp.redhat.com
$

If you are on VPN, things might look different though and there REDHAT.COM might be discoverable. However, you didn't say what was your state when the ticket disappeared.

More to that, a default krb5 in F26 would first try to use KDC proxy configuration if it is available:

$ dig -t URI +short _kerberos.fedoraproject.org
10 1 "krb5srv:m:kkdcp:https://id.fedoraproject.org/KdcProxy/"
$

The discovery I talk about above happens only if there is no explicit configuration for the realm in question already.

Can you try to define an explicit configuration for REDHAT.COM realm by providing a drop-in file /etc/krb5.conf.d/redhat_com? Something like

[realms]
 REDHAT.COM = {
        kdc = kdc-for-redhat-com
 }
[domain_realm]
 .redhat.com = REDHAT.COM
 redhat.com = REDHAT.COM

where 'kdc-for-redhat-com' is a first one you get in response for "dig -t SRV +short _kerberos._tcp.redhat.com" while on VPN. You can specify all servers from the answer too, each on own 'kdc = ...' line.

My current suspicion is that there might be some bug in GSSAPI implementation in libkrb5 that does not trigger a proper look up for the case when a TGT ticket has expired and there is no explicit configuration for that realm.
Comment 12 Nathaniel McCallum 2017-06-28 08:12:30 EDT
libsoup-2.58.1-1.fc26.x86_64
krb5-libs-1.15.1-8.fc26.x86_64
gnome-online-accounts-3.24.1-1.fc26.x86_64
webkitgtk4-2.16.4-1.fc26.x86_64

I'm not able to reproduce the problem. I kinit to two realms. When I run klist I get the default realm and when I do klist -A I get both realms. Then, I let the default realm expire. Now a klist produces the error shown above ("ccache not found"). However, klist -A shows only the non-expired ccache and shows it listed as default.

Attempting to go to a SSO enabled site using the expired realm fails (as expected). Attempting to go to a SSO enabled site using the non-expired realm works (as expected).

I have tested this with both krb5.conf realm snippets and with SRV record location.
Comment 13 Debarshi Ray 2017-06-29 05:43:52 EDT
(In reply to Bastien Nocera from comment #9)
> (In reply to Alexander Bokovoy from comment #8)
> > For FEDORAPROJECT.ORG realm we ship special configuration that defines it in
> > fedora-packager-0.6.0.1-2.fc26.noarch (/etc/krb5.conf.d/fedoraproject_org
> > file).
> 
> Which I don't have installed, because fedora-packager isn't installed, koji
> and fedpkg are (and I'm not sure what I'd do with YubiKey support packages
> either).

At the risk of going off-topic, installing fedora-packager is certainly helpful and recommended. It provides fedora-packager-setup that helps set up the certificates and pulls in fedpkg, koji, etc..

See https://fedoraproject.org/wiki/Package_maintenance_guide, although it could be improved by not suggesting that fedpkg be installed separately.
Comment 14 Debarshi Ray 2017-06-29 06:28:45 EDT
(In reply to Nathaniel McCallum from comment #12)
> libsoup-2.58.1-1.fc26.x86_64
> krb5-libs-1.15.1-8.fc26.x86_64
> gnome-online-accounts-3.24.1-1.fc26.x86_64
> webkitgtk4-2.16.4-1.fc26.x86_64
> 
> I'm not able to reproduce the problem. I kinit to two realms. When I run
> klist I get the default realm and when I do klist -A I get both realms.
> Then, I let the default realm expire. Now a klist produces the error shown
> above ("ccache not found"). However, klist -A shows only the non-expired
> ccache and shows it listed as default.

That sounds like you managed to reproduce the problem.

Just so we are on the same page, I think Bastien started off with valid REDHAT.COM and FEDORAPROJECT.ORG credentials in (1), with REDHAT.COM being the default. He waited for the REDHAT.COM tgt to expire in (2). Then in (3) he had a new REDHAT.COM tgt in klist -A, but nothing in klist because the new REDHAT.COM tgt didn't get set as the default.

A few data points that might be relevant:

(a) Bastien is using GNOME's Settings -> Online Accounts (ie. gnome-online-accounts, which is basically another krb5-libs frontend) to acquire the tickets.

(b) Bastien's REDHAT.COM principal is not allowed to renew tickets. So he always gets a new one on expiry. Mine is different because my principal was recently allowed to  renewal tickets.

(c) All FEDORAPROJECT.ORG tgts can be renewed.

I think it might be GNOME's fault that the new REDHAT.COM credentials were not set as default. However, I am not completely sure. For a very long time, before FEDORAPROJECT.ORG became a thing, my REDHAT.COM principal couldn't do renewals either, just like Bastien's. And I never had a report where the newly acquired ticket didn't get set as default. Maybe the fact that there is a second Kerberos realm plays a role here?

I think the absence of a default principal might not be relevant here. Right now, on my machine, I have valid FEDORAPROJECT.ORG (default) and REDHAT.COM tickets, but https://errata.devel.redhat.com/ still prompts me for a password in Firefox. Epiphany reports 401 Unauthorized. It works once I use kswitch to switch the default to REDHAT.COM. Wasn't https://github.com/krb5/krb5/pull/606 meant to address this? Or is this a matter of the errata tool using raw Kerberos instead of GSSAPI? Although did Simo say that it uses GSSAPI and it does respond with a 'WWW-Authenticate: Negotiate':

$ curl -v https://errata.devel.redhat.com/
...
...
...
< HTTP/1.1 401 Unauthorized
< Date: Thu, 29 Jun 2017 10:24:05 GMT
< Server: Apache/2.4.18 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< WWW-Authenticate: Negotiate
< WWW-Authenticate: Basic realm="Kerberos 5 Login"
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
Comment 15 Debarshi Ray 2017-06-29 06:30:43 EDT
(In reply to Alexander Bokovoy from comment #10)
> Can you try to define an explicit configuration for REDHAT.COM realm by
> providing a drop-in file /etc/krb5.conf.d/redhat_com? Something like
> 
> [realms]
>  REDHAT.COM = {
>         kdc = kdc-for-redhat-com
>  }
> [domain_realm]
>  .redhat.com = REDHAT.COM
>  redhat.com = REDHAT.COM

There is bug 1409819 against rhpkg for this.
Comment 16 Debarshi Ray 2017-06-29 06:39:31 EDT
(In reply to Debarshi Ray from comment #14)
> I think the absence of a default principal might not be relevant here. Right
> now, on my machine, I have valid FEDORAPROJECT.ORG (default) and REDHAT.COM
> tickets, but https://errata.devel.redhat.com/ still prompts me for a
> password in Firefox. Epiphany reports 401 Unauthorized. It works once I use
> kswitch to switch the default to REDHAT.COM. Wasn't
> https://github.com/krb5/krb5/pull/606 meant to address this?

Something's up with https://errata.devel.redhat.com/.

I can access https://redhat.orangehrm.com/ just fine in both Epiphany and Firefox with a FEDORAPROJECT.ORG tgt set as default, and a REDHAT.COM one which isn't.
Comment 17 Alexander Bokovoy 2017-06-29 06:42:55 EDT
To be clear, when we talk about 'an app uses GSSAPI  or raw Kerberos API', we talk about the client side. This is where a selection of a ticket from a ccache collection differs.

GSSAPI krb5 implementation in MIT Kerberos is capable of finding a ccache from the ccache collection that contains a ticket from the same realm as a target principal. Raw Kerberos API only works with currently selected ("default") ccache from the ccache collection.

Thus, for GSSAPI-based application, there should be no difference where the actual ticket is located within the ccache -- as long as that ticket exists. If GNOME Online Accounts are able to re-acquire a TGT to REDHAT.COM's principal, it should just work.

Does GNOME Online Accounts use raw Kerberos API itself? May be it needs to switch to a correct ccache first before renewing the ticket/reacquiring a new one?
Comment 18 Alexander Bokovoy 2017-06-29 06:46:36 EDT
I have no problems with the errata tool. I just tried it with Firefox and looked into the network exchanges -- errata tool sends 401, asking for an authorization. My client sends Negotiate request, then receives 200 with Negotiate response and then complete the sequence. All this while REDHAT.COM's ccache is not a default one in the collection.
Comment 19 Simo Sorce 2017-06-29 06:49:32 EDT
Setting explicit domain_realm mapping is not necessary anymore and is a red herring.

The problem here is in GSSAPI failing to autoselect when the default credential is missing. It works fine when defualt creds are set and everything else is otherwise the same.

So I think this is a bug in GSSAPI and it is why I asked Bastien to open this bug.

Robbie can you investigate why, when here are 2 valid credentials and one is set as default everything works, but when the default is missing suddenly GSSAPI breaks ?

This can be easily reproduced by acquiring 2 creds in a KEYRING ccache and then manually "breaking" the key that holds the default creds reference and point it to an invented and unexisting subcred name.
Comment 20 Debarshi Ray 2017-06-29 10:34:54 EDT
(In reply to Alexander Bokovoy from comment #17)
> To be clear, when we talk about 'an app uses GSSAPI  or raw Kerberos API',
> we talk about the client side. This is where a selection of a ticket from a
> ccache collection differs.

Umm... I recall Nathaniel mentioning that parts of the Red Hat infrastructure are antiquated in the sense that they still use "raw Kerberos" instead of GSSAPI. That's how I understood it as a server-side issue. Maybe I am totally confused?

> Does GNOME Online Accounts use raw Kerberos API itself? May be it needs to
> switch to a correct ccache first before renewing the ticket/reacquiring a
> new one?

In case the TGT cannot be renewed, GOA uses krb5_get_init_creds_password to get a new ticket:
https://git.gnome.org/browse/gnome-online-accounts/tree/src/goaidentity/goakerberosidentity.c#n1290

In case it can be renewed, GOA uses krb5_get_renewed_creds:
https://git.gnome.org/browse/gnome-online-accounts/tree/src/goaidentity/goakerberosidentity.c#n1526

These credentials are stored with krb5_cc_store_cred:
https://git.gnome.org/browse/gnome-online-accounts/tree/src/goaidentity/goakerberosidentity.c#n1202
Comment 21 Debarshi Ray 2017-06-29 10:38:16 EDT
(In reply to Alexander Bokovoy from comment #18)
> I have no problems with the errata tool. I just tried it with Firefox and
> looked into the network exchanges -- errata tool sends 401, asking for an
> authorization. My client sends Negotiate request, then receives 200 with
> Negotiate response and then complete the sequence. All this while
> REDHAT.COM's ccache is not a default one in the collection.

Well, right now Firefox isn't even throwing the password dialog at me. It just keeps spinning. Epiphany continues to throw 401 Unauthorized as before. This is with a default FEDORAPROJECT.ORG, and a REDHAT.COM principals.

I'll write a simple libsoup program to demonstrate this.
Comment 22 Dan Winship 2017-06-29 10:42:59 EDT
That's probably bug 1439798? I'm not sure what the current state of gssapi bugfixes in fedora libsoup is.
Comment 23 Tomas Popela 2017-06-29 10:52:52 EDT
(In reply to Dan Winship from comment #22)
> That's probably bug 1439798? I'm not sure what the current state of gssapi
> bugfixes in fedora libsoup is.

Ah yeah, this needs to be backported to Fedora as well (unless you will do a new libsoup release (at least for f26))
Comment 24 Tomas Popela 2017-06-30 03:13:42 EDT
(In reply to Tomas Popela from comment #23)
> (In reply to Dan Winship from comment #22)
> > That's probably bug 1439798? I'm not sure what the current state of gssapi
> > bugfixes in fedora libsoup is.
> 
> Ah yeah, this needs to be backported to Fedora as well (unless you will do a
> new libsoup release (at least for f26))

https://bodhi.fedoraproject.org/updates/FEDORA-2017-4305176f2a
Comment 25 Debarshi Ray 2017-06-30 08:25:30 EDT
That doesn't seem to fix it for me. Both Epiphany and Firefox (it doesn't use libsoup, but still) are behaving the same way as in comment 14 and comment 21.

A while ago I had filed bug 1406072 against Firefox. Do you think there is work that needs to happen in Firefox?
Comment 26 Nathaniel McCallum 2017-06-30 08:57:36 EDT
If there is a bug in GSSAPI/Kerberos, it would be manifested in both wget and curl. Could you work up a simple shell script reproducer? That would at least help us see the problem (which I still don't currently understand).
Comment 27 Debarshi Ray 2017-06-30 11:25 EDT
Created attachment 1293284 [details]
libsoup-based reproducer

Run it as:
$ ./a.out https://errata.devel.redhat.com/
...
...
$ ./a.out https://redhat.orangehrm.com/
...
...
(and so on)
Comment 28 Nathaniel McCallum 2017-06-30 11:28:56 EDT
Can you put this in a shell script which does the ccache management to trigger a failure?
Comment 29 Debarshi Ray 2017-07-03 09:46:29 EDT
For me the ccache management to trigger the failure is:
(a) Add dray@REDHAT.COM through Settings -> Online Accounts
(b) Add rishi@FEDORAPROJECT.ORG through Settings -> Online Accounts
(c) $ kswitch -p rishi@FEDORAPROJECT.ORG
(so that dray@REDHAT.COM is no longer the default)
(d) ./a.out https://errata.devel.redhat.com/ -> doesn't work, but
    ./a.out https://redhat.orangehrm.com/ does.

I haven't checked whether kinit vs the GUI matters or not. I will setup a fresh new VM and check.
Comment 30 Robbie Harwood 2017-07-10 18:13:12 EDT
Thanks for your patience.  The sequence of commands (using bare kinit instead of GOA) in #29 does not seem to trigger this for me (rawhide).

Further, if I have my fp.o credential active on the top, both continue to work.  This is true after the fp.o has expired (at which point it is implicitly removed from the ccache and no longer shows in `klist -A`).

So I don't have a reproducer right now.
Comment 31 Debarshi Ray 2017-07-13 11:51:35 EDT
So far I was using my workhorse Fedora 25 host.

To get a clean and more recent environment, I installed a fresh new Fedora Workstation 26 VM from the live image in GNOME Boxes. Then:
  $ sudo dnf update
  ...
  $ sudo dnf install krb5-workstation
  ...

Then I used kinit to setup my Kerberos principals:

[rishi@kolache-f26 ~]$ klist
klist: Credentials cache keyring 'persistent:1000:1000' not found
[rishi@kolache-f26 ~]$ kinit rishi@FEDORAPROJECT.ORG
Password for rishi@FEDORAPROJECT.ORG: 
[rishi@kolache-f26 ~]$ kinit dray@REDHAT.COM
Password for dray@REDHAT.COM: 
[rishi@kolache-f26 ~]$ klist
Ticket cache: KEYRING:persistent:1000:krb_ccache_UfxLoyL
Default principal: dray@REDHAT.COM

Valid starting       Expires              Service principal
2017-07-13T17:23:13  2017-07-14T03:23:13  krbtgt/REDHAT.COM@REDHAT.COM
	renew until 2017-07-20T17:23:07
[rishi@kolache-f26 ~]$ kswitch -p rishi@FEDORAPROJECT.ORG
[rishi@kolache-f26 ~]$ klist
Ticket cache: KEYRING:persistent:1000:1000
Default principal: rishi@FEDORAPROJECT.ORG

Valid starting       Expires              Service principal
2017-07-13T17:23:00  2017-07-14T17:22:53  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
	renew until 2017-07-20T17:22:53
[rishi@kolache-f26 ~]$ klist -A
Ticket cache: KEYRING:persistent:1000:1000
Default principal: rishi@FEDORAPROJECT.ORG

Valid starting       Expires              Service principal
2017-07-13T17:23:00  2017-07-14T17:22:53  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
	renew until 2017-07-20T17:22:53

Ticket cache: KEYRING:persistent:1000:krb_ccache_UfxLoyL
Default principal: dray@REDHAT.COM

Valid starting       Expires              Service principal
2017-07-13T17:23:13  2017-07-14T03:23:13  krbtgt/REDHAT.COM@REDHAT.COM
	renew until 2017-07-20T17:23:07
[rishi@kolache-f26 ~]$ 

Then I set up the Red Hat Corporate IT CA roots, so that our test cases involving the REDHAT.COM principal don't have to deal with certificate problems:
$ cd /etc/pki/ca-trust/source/anchors
$ sudo wget https://password.corp.redhat.com/RH-IT-Root-CA.crt
$ sudo update-ca-trust extract
$ sudo wget https://password.corp.redhat.com/legacy.crt
$ sudo update-ca-trust extract 

Then I built the attached test case:
$ sudo dnf install libsoup-devel glib2-devel
$ gcc soup-negotiate.c $(pkg-config --cflags --libs libsoup-2.4 glib-2.0)

This ends in a 401:
$ ./a.out https://errata.devel.redhat.com/

But this ends in a 200:
$ ./a.out https://redhat.orangehrm.com/

I get similar results in Firefox, which uses a different HTTP implementation than the test case. https://errata.devel.redhat.com/ prompts me for my Kerberos credentials, while https://redhat.orangehrm.com/ just works.
Comment 32 Debarshi Ray 2017-07-19 06:00:43 EDT
I was debugging this with Tomas today, and he thinks that the following WARNING could be indicative of a problem on the server:

$ ./a.out https://errata.devel.redhat.com/
> GET / HTTP/1.1
> Soup-Debug-Timestamp: 1500458207
> Soup-Debug: SoupSession 1 (0xcc9150), SoupMessage 1 (0xcd20d0), SoupSocket 1 (0xd06c60)
> Host: errata.devel.redhat.com
> Accept-Encoding: gzip, deflate
> Connection: Keep-Alive
  

(process:13835): libsoup-WARNING **: gssapi step failed: Unspecified GSS failure.  Minor code may provide more information: Server krbtgt/HOST.PROD.ENG.BOS.REDHAT.COM@FEDORAPROJECT.ORG not found in Kerberos database
< HTTP/1.1 401 Unauthorized
< Soup-Debug-Timestamp: 1500458209
< Soup-Debug: SoupMessage 1 (0xcd20d0)
< Date: Wed, 19 Jul 2017 09:56:47 GMT
< Server: Apache/2.4.25 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< WWW-Authenticate: Negotiate
< WWW-Authenticate: Basic realm="Kerberos 5 Login"
< Content-Length: 381
< Keep-Alive: timeout=5, max=100
< Connection: Keep-Alive
< Content-Type: text/html; charset=iso-8859-1
< 
< <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
< <html><head>
< <title>401 Unauthorized</title>
< </head><body>
< <h1>Unauthorized</h1>
< <p>This server could not verify that you
< are authorized to access the document
< requested.  Either you supplied the wrong
< credentials (e.g., bad password), or your
< browser doesn't understand how to supply
< the credentials required.</p>
< </body></html>
Comment 33 Matt Rogers 2017-07-19 10:58:34 EDT
(In reply to Debarshi Ray from comment #14)
> Wasn't https://github.com/krb5/krb5/pull/606 meant to address this? 

That patch helps with ccache selection for non-default credentials only when the target principal is host-based with no explicit realm component (i.e., "server@redhat.com") and no domain_realm mapping for redhat.com -> REDHAT.COM. If it's just a service principal then the patch would not help in this case.
Comment 34 Robbie Harwood 2017-07-19 16:44:31 EDT
Thank you Debarshi.  I have reproduced the issue.
Comment 35 Robbie Harwood 2017-07-19 17:56:44 EDT
So on further testing: with the KDC explicitly configured for the REDHAT.COM realm, everything works.  However, if the KDC is /not/ configured and instead pulled out of DNS, orange works while errata does not.
Comment 36 Robbie Harwood 2017-07-27 11:32:29 EDT
Okay, so this is working as intended, unfortunately.

The best thing to do would be to get the [domain_realm] mapping in krb5.conf for REDHAT.COM set up.

The next best thing to do is, in krb5.conf, set `dns_lookup_realm = True` and set `realm_try_domains` (probably to 5 or so).

I'll look into whether we can do something more intuitive in this use case.
Comment 37 Nathaniel McCallum 2017-07-27 11:39:57 EDT
We need to do something more intuitive and automatic - unless there are specific security concerns. Can you describe what is happening?
Comment 38 Robbie Harwood 2017-07-27 12:36:40 EDT
(In reply to Nathaniel McCallum from comment #37)
> We need to do something more intuitive and automatic - unless there are
> specific security concerns. Can you describe what is happening?

The errata.redhat.com tool redirects to machines with hostnames like errata-01.host.prod.eng.bos.redhat.com.  When we have the [domain_realm] section present, anything that ends in ".redhat.com" gets mapped into the REDHAT.COM realm.  When we don't have this section present, it works differently: up to realm_try_domains additional subdomains are allowed to be present in a hostname for it to match a realm.

So, for instance, if realm_try_domains=0 (default), saml.redhat.com will match into the REDHAT.COM domain, but foo.bar.redhat.com will not; if realm_try_domains=1, then both will match, and so on.
Comment 39 Nathaniel McCallum 2017-07-28 13:49:00 EDT
Let's bring this up to upstream. This seems to me like a solvable problem.
Comment 40 Debarshi Ray 2017-08-01 06:19:30 EDT
Thank you for pursuing this!
Comment 41 Jan Kurik 2017-08-15 03:04:04 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Note You need to log in before you can comment on or make changes to this bug.