Bug 1290831 - gssntlmssp breaks firefox access to some web sites
gssntlmssp breaks firefox access to some web sites
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: gssntlmssp (Show other bugs)
22
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Simo Sorce
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-11 10:11 EST by Martin Wilck
Modified: 2016-07-19 15:23 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-19 15:23:33 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ff-ntlm.log: log with gssntlmssp enabled, ff-nontlm.log: log with gssntlmssp disabled (26.73 KB, application/x-xz)
2015-12-11 10:11 EST, Martin Wilck
no flags Details

  None (edit)
Description Martin Wilck 2015-12-11 10:11:10 EST
Created attachment 1104723 [details]
ff-ntlm.log: log with gssntlmssp enabled, ff-nontlm.log: log with gssntlmssp disabled

Description of problem:

If the gssntlmssp package is installed and "negotiate-auth" is enabled for a domain, but Kerberos authentification fails for some server in that domain, firefox fails to access the page (returns HTTP error 401 - authentication failed without prompting for a password).

The reason for this failure is totally in the dark even for experienced users. It took me 2 days of debugging to understand what was going on. 
Severity: high because of that.

I saw this on F21, but my analysis and the packages used make me believe very strongly that the problem also exists on newer Fedora releases.

This problem won't occur in properly administered sites where fallback to ntlmssp wouln't be necessary because kerberos works. However, in such sites the  gssntlmssp package wouldn't be necessary at all.

The biggest impact of this problem could be worked around by documenting properly that users are expected to either configure winbind or use NTLM_USER_FILE if this package is installed (see below).


Version-Release number of selected component (if applicable):
firefox-42.0-2.fc21.x86_64
krb5-libs-1.12.2-19.fc21.x86_64
gssntlmssp-0.6.0-1.fc21.x86_64

How reproducible:
Access a web service in a domain that is trusted for negotiate-auth (member of network.negotiate-auth.trusted-uris" config variable in firefox but isn't properly registered in the kerberos / AD domain.

Actual results:
Error 401 returned, without giving the user the possibility to enter user ID and password.

Expected results:
Pop-up window is displayed, letting user enter the credentials

Additional info:

WORKAROUND: 

a) deinstall and/or disable gssntlmssp
   (but on my system gssntlmssp is pulled in by purple-sipe)

c) proper winbind configuration (complex, not in line with Fedora SSSD policy https://www.fedorahosted.org/sssd/wiki/SSSD-vs-Winbind, thus doesn't apply to my system, NOT TESTED).

c) set an environment variable NTLM_USER_FILE with the name of a file containing DOMAIN:USER:PASSWORD, if this is acceptable by security policy (PASSWORD is in plain text in this file, thus readable by anyone with superuser permissions)

c) works, but it is hardly documented at all. The only hint to it (apart from reading the source code is on the home page of the project https://fedorahosted.org/gss-ntlmssp/; there's nothing the package documentation or elsewhere in Fedora docs. Also I am not aware of any tool that would generate the file for the user; this would be a reasonable thing to do for pam_krb5, for example.

But anyway, storing plain passwords on disk in this manner is really not good security practice.


Further analysis
----------------

I am attaching logs generated with 

NSPR_LOG_MODULES=negotiateauth:5,timestamp,nsHttp:5,NTLM:5
NSPR_LOG_FILE=/dev/stderr
KRB5_TRACE=/dev/stderr

Both with and without gssntlmssp (and without any of the workarounds mentioned above).

Here is what happens:

1) client tries to access the page and gets back a 401 error with

  WWW-Authenticate: negotiate
  WWW-Authenticate: ntlm

Firefox is now supposed to try these mechanisms in turn.

2) firefox tries "negotiate" auth and enters GSSAPI with SPNEGO mech.

3) SPNEGO tries predefined mechanisms in turn: KRB5, KRB5_OLD, and KRB5_WRONG, IAKERB (krb5: lib/gssapi/krb5/gssapiP_krb5.h). The first three will try to get a ticket via kerberos directly. The GSSAPI/SPNEGO/IAKERB mechanism returns control to the browser and has the browser send an "Authorization: Negotiate" token with a wrapped kerberos ticket request to the server. The token also contains the list of supported SPNEGO mechanisms (at this point, IAKERB and NTLMSSP). 

4) The server returns with another "WWW-Authenticate: Negotiate" tag, containing a challenge for NTLMSSP. Control is passed back to GSSAPI. GSSAPI detects that IAKERB is not supported and NTLMSSP advertised instead by the server. 

5) GSSAPI starts NTLMSSP mechanism through gssntlmssp. But because firefox never passes user creds to GSSAPI, gssntlmssp doesn't have any usable credentials and the authentication fails. 

[ A comment in firefox' source file nsHttpNegotiateAuth.cpp says

// Always set *identityInvalid == FALSE here.  This 
// will prevent the browser from popping up the authentication
// prompt window.  Because GSSAPI does not have an API
// for fetching initial credentials (ex: A Kerberos TGT),
// there is no correct way to get the users credentials.

I am not sure if this is correct but gssntlmssp's implementation seems to support the notion ]

6) the GSSAPI call ultimately fails, and thus the nsAuthGSSAPI authetication method in firefox fails, too.

7) Because the server hasn't sent "WWW-Authenticate: NTLM" any more after the first response in 3), firefox is now out of luck and can't continue. Therefore it simply displays a HTTP 401 error page.


If gssntlmssp is disabled, the sequence of events is subtly different:

3a) GSSAPI will generate a slightly different SPNEGO/IAKERB token to send to the server, which doesn't advertize NTLMSSP support. 

4a) The server returns an empty "WWW-Authenticate: Negotiate" tag and a "WWW-Authenticate: NTLM" tag.

5a) nsAuthGSSAPI::GetNextToken fails immediately without calling into GSSAPI again, because no input token was received from the server.

6a) nsAuthGSSAPI fails.

7a) Because the server has returned "WWW-Authenticate: NTLM" in the last HTTP response, firefox falls back to generic NTLM and is displays a username/password prompt, allowing the user to access the site.
Comment 1 Simo Sorce 2016-02-23 10:32:20 EST
Hi Martin,
thank you for the detailed report, and sorry for the late reply.

Your analysis is spot on, my preferred solution would be to fix Firefox, however I understand that it may take  while s Mozilla moves slowly on these matters.

One option you have is to disable the mechanism by commenting the mechanism id in /etc/gss/mech.d/ntlmssp.conf, assumin you do not need to use it in SIPE.

If you need it in SIPE, I am not entirely sure what we can do here, as GSSAPI has only a system wide configuration file for mechanisms and it can't be overridden with environment variables or other options on a per application basis.
Comment 2 Martin Wilck 2016-02-23 10:54:26 EST
(In reply to Simo Sorce from comment #1)
> Hi Martin,
> thank you for the detailed report, and sorry for the late reply.
> 
> Your analysis is spot on, my preferred solution would be to fix Firefox,
> however I understand that it may take  while s Mozilla moves slowly on these
> matters.

My personal workaround was to use NTLM_USER_FILE, so I don't need a quick fix :-)

An upstream mozilla bug report might be worthwhile. But the crucial point seems to be 5) above - the mozilla guys see no way to pass the credentials to GSSAPI. If we open a bug there, they're likely to reject it for that reason.

Would it make sense to try to talk to the GSSAPI folks? Someone would need to work out what has to be done to enable a proper password callback for this situation. It can't be impossible, can it? The most obvious solution in a modern Linux architecture would be using dbus and a password agent of some sort, such as gnome-keyring...
Comment 3 Simo Sorce 2016-02-24 17:37:50 EST
(In reply to Martin Wilck from comment #2)
> (In reply to Simo Sorce from comment #1)
> > Hi Martin,
> > thank you for the detailed report, and sorry for the late reply.
> > 
> > Your analysis is spot on, my preferred solution would be to fix Firefox,
> > however I understand that it may take  while s Mozilla moves slowly on these
> > matters.
> 
> My personal workaround was to use NTLM_USER_FILE, so I don't need a quick
> fix :-)
> 
> An upstream mozilla bug report might be worthwhile. But the crucial point
> seems to be 5) above - the mozilla guys see no way to pass the credentials
> to GSSAPI. If we open a bug there, they're likely to reject it for that
> reason.

They probably didn't look carefully at the API, they can use gss_acquire_cred_with_password() if they want to try to acquire password credentials, and use them to establish a context.

> Would it make sense to try to talk to the GSSAPI folks? Someone would need
> to work out what has to be done to enable a proper password callback for
> this situation. It can't be impossible, can it? The most obvious solution in
> a modern Linux architecture would be using dbus and a password agent of some
> sort, such as gnome-keyring...

Which is not needed, strictly, see above, however I am torn on whether we should allow the broser to ask you for credentials. That said we will have mechanisms to provide the browser with some credentials behind the scene eventually, so all we need is to make sure Firefox catches the right error condition and asks for a password *only* if context establishment fails due to lack of credentials, if it fails for other reasons Firefox shouldn't prompt (perhaps we can get a tunable that can be set to not prompting by default).

One of the reasons asking for a password may be wrong is that when using krb5 a user may not have a password at all, but may be using other means to obtain credentials, like pkinit or a 2FA based pre-auth method that gss_acquire_cred_with_password() wouldn't be able to fulfill at all.
Comment 4 Martin Wilck 2016-02-25 04:42:19 EST
Hmm. Someone needs to figure out how to do this right. It doesn't make sense to point the mozillal guys to gss_acquire_cred_with_password() if it's ultimately not the right way to do it.

My co-workers using Windows never have to bother about things like this. They enter their password at logon, and Windows takes care of the rest. It uses KRB5 tickets for services that support it, and for services that don't (as the one that caused my pain here) it will pull out the plain text password from some secret place where stores it under the hood (at least I'm guessing this is what happens - I have no other explanation why Windows is able to do an NTMLSSP login without asking for the password. Maybe I'm missing something?).

This is the sort of user experience that Linux should aim at, too. Experienced users may want to control whether or not the plaintext password is cached somewhere, but ordinary users will simply want the Windows "single-sign-on" behavior.

I think this functionality belongs in something like systemd-logind. It should support as many algorithms and authentication schemes as possible, ask for passwords only if absolutely necessary, and provide a secure and GUI-agnostic API for applications to retrieve credentials for whatever purpose. Only after having verified that the entity asking for the credentials has the right to do so, of course.

"You may say I'm a dreamer, but I'm not the only one" :-)
Comment 5 Simo Sorce 2016-02-25 14:12:13 EST
(In reply to Martin Wilck from comment #4)
> Hmm. Someone needs to figure out how to do this right. It doesn't make sense
> to point the mozillal guys to gss_acquire_cred_with_password() if it's
> ultimately not the right way to do it.
> 
> My co-workers using Windows never have to bother about things like this.
> They enter their password at logon, and Windows takes care of the rest. It
> uses KRB5 tickets for services that support it, and for services that don't
> (as the one that caused my pain here) it will pull out the plain text
> password from some secret place where stores it under the hood (at least I'm
> guessing this is what happens - I have no other explanation why Windows is
> able to do an NTMLSSP login without asking for the password. Maybe I'm
> missing something?).
> 
> This is the sort of user experience that Linux should aim at, too.
> Experienced users may want to control whether or not the plaintext password
> is cached somewhere, but ordinary users will simply want the Windows
> "single-sign-on" behavior.

This is the user experience I aplannig to achieve eventually (we can already do that with kerberos), but it requires a lot of plumbing and changes so it takes time.

> I think this functionality belongs in something like systemd-logind. It
> should support as many algorithms and authentication schemes as possible,
> ask for passwords only if absolutely necessary, and provide a secure and
> GUI-agnostic API for applications to retrieve credentials for whatever
> purpose. Only after having verified that the entity asking for the
> credentials has the right to do so, of course.

I plan to suupport this caching in SSSD+GSS-Proxy, we can't do it in system-logind because you need a spearate daemon to keep the hash out of rach of normal user processes, as that is a clear-text password equivalent.

> "You may say I'm a dreamer, but I'm not the only one" :-)

I dream too, it's just that dream=making takes time, but hey patches are welcome to speed it up :-)
Comment 6 David Woodhouse 2016-05-18 03:34:58 EDT
Hm, why is the client even trying to negotiate NTLMSSP when it has no credentials? Sure, fixing Firefox would be great... but Firefox is massively resistant even to simple and obvious fixes (like spawning ntlm_auth from the right bloody place).
Comment 7 Martin Wilck 2016-05-19 07:51:41 EDT
(In reply to David Woodhouse from comment #6)
> Hm, why is the client even trying to negotiate NTLMSSP when it has no
> credentials?

My understanding is that firefox treats GSSAPI basically as a "black box" with no interaction. A callback through which GSSAPI could ask the user for a passwort is unsupported. GSSAPI walks through the list of SPNEGO mechanisms - all of them except NTLMSSP function without user callback.

See also the comment quoted from the Mozilla sources under 5) above.
Comment 8 David Woodhouse 2016-05-19 09:07:09 EDT
Right. Given that we didn't use gssapi_acquire_creds_with_password(), and given that there are no 'default' creds to be found in $NTLM_USER_FILE or winbind, why in $DEITY's name is SPNEGO attempting to negotiate the use of NTLMSSP? Or why is NTLMSSP allowing itself to be used at all? If it doesn't have any creds, surely that should be the same as if it isn't install at all?

(Let's assume, for the moment, that nobody in the real world would ever want to mess around with NTLMSSP 'anonymous' authentication, and can explicitly enable that crap if they ever did.)
Comment 9 Simo Sorce 2016-05-20 13:42:55 EDT
gssntlmssp will return an error at the first invocation of gss_init_sec_context (either direct or via spnego) if it is not provided or cannot acquire credentials.

The al problen here is no gssntlmssp, but IAKERB, which causes the whole SPNEGO processing to start and the server seeing that the client support NTLMSSP only sends back Negotiate headers and not the NTLM headers.

IIUC When Firefox ends up in a failure it sees no NTLM headers and decideds not to try the builtin NTLM auth.

It is not easy to workaround this because SPNEGO does not want to try to acquire credentials for all mechanisms when the context is being established and rather simply calls into the following mechanisms ony if the previous one fail, this is because acquiring creds may be an expensive operation (AS requests and such for krb5).

Now this means the spnego mech advertizes all supported mechanisms regardless of whether it has crdentials unless you acquire_cred before hand (which would cause spnego to *know* what creds are available).

If firefox were smart enough it would acquire_cred and/or acquire_cred_with_password as a fallback, but Firefox isn't.

It's not ideal, ideas welcome
Comment 10 Simo Sorce 2016-05-20 14:07:01 EDT
Let me take my comment back, Greg Hudson pointed out that SPNEGO does in fact enumerate available credenials (at least in 1.14/master it does), and so NTLMSSP should not be advertied from the client if no credentials are available (NTLM user file or winbind).
Comment 11 David Woodhouse 2016-05-20 14:21:20 EDT
So this is fixed in Fedora 23 because we have 1.14.1 there?
Comment 12 David Woodhouse 2016-05-20 14:24:29 EDT
(And actually, for at least some servers my recollection is that they don't *care* whether NTLMSSP was one of the mechanisms offered in SPNEGO; they don't offer WWW-Authenticate: NTLM at all after WWW-Authenticate: Negotiate has been attempted. So even before gssntlmssp, the attempt to do IAKERB was screwing us over and preventing the NTLM authentication. There's a separate request somewhere for "make it possible to disable IAKERB... and do so by default" IIRC.)
Comment 13 Simo Sorce 2016-05-20 15:31:23 EDT
Ok after a fine gdb session I figured out the problem, gssntlmssp was incorrectly returning a credential structure when no credentials were available.

This was causing SPNEGO to advertize NTLMSSP as a viable mech when it shouldn't have.

Proposed fix here:
https://github.com/simo5/gss-ntlmssp/commit/26597c648d494f8757deec7da85bd4f0c340f06e

Will make packages as soon as I get another pair of eyes on it for review.
Comment 14 Fedora Update System 2016-05-20 16:38:41 EDT
gssntlmssp-0.6.0-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-014408dded
Comment 15 Fedora Update System 2016-05-20 16:38:48 EDT
gssntlmssp-0.6.0-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d6f6098254
Comment 16 Fedora Update System 2016-05-21 23:50:24 EDT
gssntlmssp-0.6.0-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-d6f6098254
Comment 17 Fedora Update System 2016-05-21 23:56:08 EDT
gssntlmssp-0.6.0-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-014408dded
Comment 18 Fedora Update System 2016-06-03 12:28:56 EDT
gssntlmssp-0.7.0-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-06e192287b
Comment 19 Fedora Update System 2016-06-03 12:29:05 EDT
gssntlmssp-0.7.0-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b2c360ff35
Comment 20 Fedora Update System 2016-06-04 14:25:58 EDT
gssntlmssp-0.7.0-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-06e192287b
Comment 21 Fedora Update System 2016-06-04 23:58:15 EDT
gssntlmssp-0.7.0-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b2c360ff35
Comment 22 Fedora Update System 2016-06-18 14:45:42 EDT
gssntlmssp-0.7.0-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
Comment 23 Fedora Update System 2016-06-30 15:53:32 EDT
gssntlmssp-0.7.0-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 24 Fedora End Of Life 2016-07-19 15:23:33 EDT
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.