Bug 961969 - Cannot log into IPA with browsers on OS X with kerberos credentials
Cannot log into IPA with browsers on OS X with kerberos credentials
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ipa (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Rob Crittenden
Namita Soman
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-10 16:59 EDT by Vincent Danen
Modified: 2013-06-06 05:53 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-06 05:53:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
error log file (24.90 KB, text/plain)
2013-05-13 17:45 EDT, Vincent Danen
no flags Details

  None (edit)
Description Vincent Danen 2013-05-10 16:59:38 EDT
On the mac, I am unable to log into IPA's web ui without a password (and a valid kerberos ticket).

The output of klist shows I have a valid ticket, and I can log into the web UI if I type in my authentication information, but I am unable to do so automatically.

This is true for Safari, Firefox, or Chrome.  On Firefox, I followed the automatic instructions for Firefox and was still unable to login.

I do have the CA certificate installed on the system.

I have a mediawiki instance running on another server that uses kerberos for auth, and it works.

If I do the same with Firefox on Fedora 18, it works fine.

Looks like the problem is here:

[Fri May 10 11:52:11 2013] [error] ipa: ERROR: 500 Internal Server Error: login_kerberos: KRB5CCNAME not defined in HTTP request environment

This environment is not set on OS X, although I'm not sure where or why this is required on IPA, but not on the other box?
Comment 1 Rob Crittenden 2013-05-10 17:01:54 EDT
This suggests that negotiation did not succeed.

Can you set LogLevel debug in /etc/httpd/conf.d/nss.conf and restart Apache? This will give us a clearer picture of what is going on in mod_auth_kerb.
Comment 3 Vincent Danen 2013-05-13 17:45:48 EDT
Created attachment 747454 [details]
error log file

Slightly scrubbed logfile with debug enabled in nss.conf; connection was from Safari on OS X 10.8.3 with a valid kerberos ticket.
Comment 4 Rob Crittenden 2013-05-13 18:10:06 EDT
Can you share the equivalent time period for the krb5kdc log?

The way this is supposed to work is this the web server uses s4u2proxy to make the request on behalf of the user.

I'd expect to see a request and DELEGATION in the KDC log.
Comment 5 Vincent Danen 2013-05-13 18:20:36 EDT
Sorry, should have thought of that.  Pasting it below since it's not too big.

May 13 15:41:07 ipa.mydomain.com krb5kdc[11249](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.0.15: NEEDED_PREAUTH: host/ipa.mydomain.com@MYDOMAIN.COM for krbtgt/MYDOMAIN.COM@MYDOMAIN.COM, Additional pre-authentication required
May 13 15:41:07 ipa.mydomain.com krb5kdc[11249](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.0.15: ISSUE: authtime 1368481267, etypes {rep=18 tkt=18 ses=18}, host/ipa.mydomain.com@MYDOMAIN.COM for krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
May 13 15:41:08 ipa.mydomain.com krb5kdc[11249](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.0.15: ISSUE: authtime 1368481267, etypes {rep=18 tkt=18 ses=18}, host/ipa.mydomain.com@MYDOMAIN.COM for ldap/ipa.mydomain.com@MYDOMAIN.COM
May 13 15:41:28 ipa.mydomain.com krb5kdc[11249](info): TGS_REQ (7 etypes {18 17 16 23 3 2 1}) 192.168.0.50: ISSUE: authtime 1368208590, etypes {rep=18 tkt=18 ses=18}, vdanen@MYDOMAIN.COM for HTTP/ipa.mydomain.com@MYDOMAIN.COM
May 13 15:41:28 ipa.mydomain.com krb5kdc[11249](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.0.15: NEEDED_PREAUTH: HTTP/ipa.mydomain.com@MYDOMAIN.COM for krbtgt/MYDOMAIN.COM@MYDOMAIN.COM, Additional pre-authentication required
May 13 15:41:28 ipa.mydomain.com krb5kdc[11249](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.0.15: ISSUE: authtime 1368481288, etypes {rep=18 tkt=18 ses=18}, HTTP/ipa.mydomain.com@MYDOMAIN.COM for krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
Comment 6 Martin Kosek 2013-05-15 09:01:14 EDT
Hm, this looks suspicious:

[Mon May 13 15:41:28 2013] [debug] src/mod_auth_kerb.c(1939): [client 192.168.0.50] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos, referer: https://ipa.mydomain.com/ipa/ui/index.html
[Mon May 13 15:41:28 2013] [debug] src/mod_auth_kerb.c(1278): [client 192.168.0.50] Acquiring creds for HTTP@ipa.mydomain.com, referer: https://ipa.mydomain.com/ipa/ui/index.html
[Mon May 13 15:41:28 2013] [debug] src/mod_auth_kerb.c(1372): [client 192.168.0.50] Using principal HTTP/ipa.mydomain.com@ANNVIX.CA for s4u2proxy, referer: https://ipa.mydomain.com/ipa/ui/index.html
[Mon May 13 15:41:28 2013] [error] [client 192.168.0.50] Credentials for HTTP/ipa.mydomain.com@ANNVIX.CA have expired or will soon expire - now 1368481288 endtime 1368456467, referer: https://ipa.mydomain.com/ipa/ui/index.html
[Mon May 13 15:41:28 2013] [debug] src/mod_auth_kerb.c(1372): [client 192.168.0.50] Using principal HTTP/ipa.mydomain.com@ANNVIX.CA for s4u2proxy, referer: https://ipa.mydomain.com/ipa/ui/index.html
[Mon May 13 15:41:28 2013] [error] [client 192.168.0.50] Credentials for HTTP/ipa.mydomain.com@ANNVIX.CA have expired or will soon expire - now 1368481288 endtime 1368456467, referer: https://ipa.mydomain.com/ipa/ui/index.html

Isn't the http keytab expired? You can try following:

# klist -kt /etc/httpd/conf/ipa.keytab 
Keytab name: FILE:/etc/httpd/conf/ipa.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 05/15/13 05:52:27 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/15/13 05:52:27 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/15/13 05:52:27 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/15/13 05:52:27 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM

# kinit -kt /etc/httpd/conf/ipa.keytab HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM

Valid starting     Expires            Service principal
05/15/13 08:43:02  05/16/13 08:43:02  krbtgt/IDM.LAB.BOS.REDHAT.COM@IDM.LAB.BOS.REDHAT.COM

If the kinit dies we have the root cause. But I assume this is not the case as you said it works for you on Fedora 18.

Is there any difference in how these browsers do negotiation? I see this header in my F18 Chrome:

Authorization:Negotiate YIIFiwYGK...4YMA==
Comment 7 Vincent Danen 2013-05-15 13:19:06 EDT
Ah yes, that is definitely an issue:

# klist -kt /etc/httpd/conf/ipa.keytab
Keytab name: FILE:/etc/httpd/conf/ipa.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
# klist -kt /etc/httpd/conf/ipa.keytab
Keytab name: FILE:/etc/httpd/conf/ipa.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
   2 12/10/12 11:37:27 HTTP/ipa.mydomain.com@MYDOMAIN.COM
[root@ipa ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/ipa.mydomain.com@MYDOMAIN.COM

Valid starting     Expires            Service principal
05/15/13 11:00:36  05/16/13 11:00:36  krbtgt/mydomain.com@MYDOMAIN.COM

So why isn't that keytab being updated?

Actually, when I look at the server that has mediawiki installed on it, it looks quite similar (these dates are from when I first installed IPA I think):

# klist -kt /etc/httpd/conf/http.keytab
Keytab name: FILE:/etc/httpd/conf/http.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/12 15:14:25 HTTP/wiki.mydomain.com@MYDOMAIN.COM
   1 12/11/12 15:14:25 HTTP/wiki.mydomain.com@MYDOMAIN.COM
   1 12/11/12 15:14:26 HTTP/wiki.mydomain.com@MYDOMAIN.COM
   1 12/11/12 15:14:26 HTTP/wiki.mydomain.com@MYDOMAIN.COM

Looking at response headers with the mediawiki I see:

WWW-Authenticate	Negotiate oRQw...CAg==, Negotiate oRQw...CAg==

I do not see anything similar when I go to the IPA UI with Chrome.
Comment 8 Martin Kosek 2013-05-17 02:57:30 EDT
This is a very good question. Rob, is there any means of automatic keytab renewal in IPA? I know that SSSD has plans and test implementation for a deamon for keytab rotation but it is not in yet.

Anyway, until the automatic keytab rotation is resolved, one could rotate the keys manually:

# cd /etc/httpd/conf/
# kinit admin
Password for admin@IDM.LAB.BOS.REDHAT.COM: 
# klist -kt ipa.keytab            
Keytab name: FILE:ipa.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 05/16/13 02:34:54 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/16/13 02:34:54 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/16/13 02:34:54 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   2 05/16/13 02:34:54 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
# mv /etc/httpd/conf/ipa.keytab /etc/httpd/conf/ipa.keytab.bkp
# ipa-getkeytab -s $IPA_SERVER -p HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM -k /etc/httpd/conf/ipa.keytab
# klist -kt ipa.keytab
Keytab name: FILE:ipa.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   3 05/17/13 02:48:02 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   3 05/17/13 02:48:02 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   3 05/17/13 02:48:02 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM
   3 05/17/13 02:48:02 HTTP/vm-037.idm.lab.bos.redhat.com@IDM.LAB.BOS.REDHAT.COM



As for the inability to SSO from Mac, this may be deficiency of mod_auth_kerb. I may be wrong, but these lines in mod_auth_kerb.c looked suspicious:

static int
kerb_authenticate_user(request_rec *r)
{
...
   log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
	      "kerb_authenticate_user entered with user %s and auth_type %s",
	      (MK_USER)?MK_USER:"(NULL)",type?type:"(NULL)");

...

   /* get what the user sent us in the HTTP header */
   auth_line = (char *)MK_TABLE_GET(r->headers_in, (r->proxyreq == PROXYREQ_PROXY)
	                                    ? "Proxy-Authorization"
					    : "Authorization");
   if (!auth_line) {
      set_kerb_auth_headers(r, conf, use_krb4, use_krb5, 
	                    (use_krb5) ? "\0" : NULL);
      return HTTP_UNAUTHORIZED;
   }

If I read the code right, if Mac browser does not send Authorization/Proxy-Authorization header, the SSO may not work.
Comment 9 Rob Crittenden 2013-05-17 09:38:31 EDT
I think you're confusing the keytab with the ccache. The keytab holds the identity that of the web server. Its ccache (in various locations on different distros) holds its client tickets. The former doesn't normally change without admin intervention, the latter is managed by mod_auth_kerb.

Kerberos has to integrate into the HTTP authentication mechanism. This involves the server sending the client a WWW-Authenticate header including the type, the realm and any type-specific inforation (e.g. nonce with Digest auth). The client then responds with an Authorization header.

So you'll note that the first a web browser hits IPA mod_auth_kerb logs something like kerb_authenticate_user entered with user (NULL) and auth_type Kerberos. This is because the user sent no credentials because the browser didn't know they were required. The server responds with a WWW-Authenticate header.

The subsequent client response will include a base64-encoded GSSAPI request in the Authorization header with negotiate as the auth type (and this is why we require SSL on Kerberos HTTP connections).

So yeah, if the browser isn't sending an Authorization header then SSO won't work. The reason for this may be that the browser isn't authorized to negotiate. In Firefox this is configured using the negotiate options in about:config. In Chrome I believe you need to pass command-line options when it starts, --auth-server-whitelist="*.example.com"

Here is an oldish wiki page on it, http://freeipa.org/page/Fedora_Chrome

Note that you shouldn't (or need to) set a delegate whitelist.
Comment 10 Vincent Danen 2013-05-17 17:37:54 EDT
(In reply to comment #9)
> So yeah, if the browser isn't sending an Authorization header then SSO won't
> work. The reason for this may be that the browser isn't authorized to
> negotiate. In Firefox this is configured using the negotiate options in
> about:config. In Chrome I believe you need to pass command-line options when
> it starts, --auth-server-whitelist="*.example.com"

Ok, but I think you missed the part where it works fine with mediawiki on rhel6, but not the IPA UI (same domain, same local network, etc.).

Why does it work on my rhel6 box with mediawiki but not the IPA ui?  All browsers work with mediawiki.

For my wiki virtualhost I have:

    <Location />
        AuthType Kerberos
        AuthName "Kerberos Login"
        KrbMethodNegotiate On
        KrbMethodK5Passwd Off
	KrbServiceName HTTP
        KrbAuthRealms MYDOMAIN.COM
        Krb5Keytab /etc/httpd/conf/http.keytab
	KrbSaveCredentials off
        #KrbServiceName HTTP
        #KrbVerifyKDC on
        #require user vdanen@MYDOMAIN.COM adanen@MYDOMAIN.COM
        require valid-user
    </Location>

This works -- I've not had to "log in" to mediawiki for two years, and I'm on that wiki pretty much every day.  This is true for Safari, Chrome, and Firefox.  They all work.

The only difference between the two that I can spot is that KrbSaveCredentials is on for the IPA host and _not_ for the mediawiki host.  I toggled that and restarted httpd on the IPA server and it didn't make a difference (for the browser; it's possible that is a red herring).
Comment 11 Rob Crittenden 2013-05-17 18:15:30 EDT
We need KrbSaveCredentials set in order to authenticate as you to LDAP. For the case where KrbConstrainedDelegation is enabled we don't actually require the TGT be forwarded.

If you want to try to set network.negotiate-auth.delegation-uris in Firefox to your domain that may fix things, we just prefer using S4U2Proxy.

Lets try Firefox since it has some client-side logging capabilities. This is what one would do on a Linux box, I'm hoping the same will work in a MacOS shell:

export NSPR_LOG_MODULES=negotiateauth:5
export NSPR_LOG_FILE=/tmp/moz.log
firefox
Comment 12 Dmitri Pal 2013-05-21 20:32:20 EDT
Any luck?
Comment 14 Vincent Danen 2013-05-24 11:13:29 EDT
Sorry, just got around to trying this now.  A tab was restored to the wiki, so it shows up as well (first); I guess it's good to see the difference:

2024264064[10055f110]:   service = wiki.mydomain.com
2024264064[10055f110]:   using negotiate-gss
2024264064[10055f110]: entering nsAuthGSSAPI::nsAuthGSSAPI()
2024264064[10055f110]: Attempting to load gss functions
2024264064[10055f110]: entering nsAuthGSSAPI::Init()
2024264064[10055f110]: nsHttpNegotiateAuth::GenerateCredentials() [challenge=Negotiate]
2024264064[10055f110]: entering nsAuthGSSAPI::GetNextToken()
2024264064[10055f110]:   leaving nsAuthGSSAPI::GetNextToken [rv=0]
2024264064[10055f110]:   Sending a token of length 711
2024264064[10055f110]:   service = ipa.mydomain.com
2024264064[10055f110]:   using negotiate-gss
2024264064[10055f110]: entering nsAuthGSSAPI::nsAuthGSSAPI()
2024264064[10055f110]: entering nsAuthGSSAPI::Init()
2024264064[10055f110]: nsHttpNegotiateAuth::GenerateCredentials() [challenge=Negotiate]
2024264064[10055f110]: entering nsAuthGSSAPI::GetNextToken()
2024264064[10055f110]:   leaving nsAuthGSSAPI::GetNextToken [rv=0]
2024264064[10055f110]:   Sending a token of length 710
Comment 16 Rob Crittenden 2013-05-28 11:44:18 EDT
This looks normal from the client side.

As a test, can you configure Firefox on the Mac to delegate to your domain and try again?
Comment 17 Vincent Danen 2013-05-30 12:26:41 EDT
Aha!

I only had network.negotiate-auth.trusted-uris set in about:config.  I've also set network.negotiate-auth.delegation-uris to my domain and also set network.negotiate-auth.using-native-gsslib to "true".  It seems that the "configure" (can't recall the link name now) when you go to the IPA server doesn't set these? 

Ok, based on the above I then poked at Chrome a little bit.  Seems it requires passing another commandline argument: --auth-negotiate-delegate-whitelist="*mydomain.com" (I previously was only passing --auth-server-whitelist).

So now it works.

Perhaps some fodder for documentation.  In light of this, I'd say this bug can be closed and maybe some documentation can/should be updated (I'll be updating my wiki documentation right away here).
Comment 18 Vincent Danen 2013-05-30 12:28:14 EDT
As an aside, the ipa/config/ssbrowser.html page, under Firefox configuration, does _not_ mention delegation at all.
Comment 19 Rob Crittenden 2013-05-30 14:08:32 EDT
That is because delegation should no longer be required. I had you do that as a test.

Since you tweaked two options, delegation and native gssapi, it may be worth to try again un-setting delegation.

With delegation you send the full TGT to IPA which then uses it to perform ldap operations on your behalf. We now use S4U2Proxy where you use your TGT to authenticate, then the web server uses its credentials to perform ldap operations on your behalf.

I suppose it is possible that this is an issue with the Mac GSSAPI implementation.
Comment 20 Martin Kosek 2013-06-06 05:53:52 EDT
Setting

network.negotiate-auth.using-native-gsslib true

is already covered in chapter 4.3.3 in documentation:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Identity_Management_Guide/index.html

I am closing this Bug as NOTABUG.

Note You need to log in before you can comment on or make changes to this bug.