Created attachment 1104723 [details] ff-ntlm.log: log with gssntlmssp enabled, ff-nontlm.log: log with gssntlmssp disabled Description of problem: If the gssntlmssp package is installed and "negotiate-auth" is enabled for a domain, but Kerberos authentification fails for some server in that domain, firefox fails to access the page (returns HTTP error 401 - authentication failed without prompting for a password). The reason for this failure is totally in the dark even for experienced users. It took me 2 days of debugging to understand what was going on. Severity: high because of that. I saw this on F21, but my analysis and the packages used make me believe very strongly that the problem also exists on newer Fedora releases. This problem won't occur in properly administered sites where fallback to ntlmssp wouln't be necessary because kerberos works. However, in such sites the gssntlmssp package wouldn't be necessary at all. The biggest impact of this problem could be worked around by documenting properly that users are expected to either configure winbind or use NTLM_USER_FILE if this package is installed (see below). Version-Release number of selected component (if applicable): firefox-42.0-2.fc21.x86_64 krb5-libs-1.12.2-19.fc21.x86_64 gssntlmssp-0.6.0-1.fc21.x86_64 How reproducible: Access a web service in a domain that is trusted for negotiate-auth (member of network.negotiate-auth.trusted-uris" config variable in firefox but isn't properly registered in the kerberos / AD domain. Actual results: Error 401 returned, without giving the user the possibility to enter user ID and password. Expected results: Pop-up window is displayed, letting user enter the credentials Additional info: WORKAROUND: a) deinstall and/or disable gssntlmssp (but on my system gssntlmssp is pulled in by purple-sipe) c) proper winbind configuration (complex, not in line with Fedora SSSD policy https://www.fedorahosted.org/sssd/wiki/SSSD-vs-Winbind, thus doesn't apply to my system, NOT TESTED). c) set an environment variable NTLM_USER_FILE with the name of a file containing DOMAIN:USER:PASSWORD, if this is acceptable by security policy (PASSWORD is in plain text in this file, thus readable by anyone with superuser permissions) c) works, but it is hardly documented at all. The only hint to it (apart from reading the source code is on the home page of the project https://fedorahosted.org/gss-ntlmssp/; there's nothing the package documentation or elsewhere in Fedora docs. Also I am not aware of any tool that would generate the file for the user; this would be a reasonable thing to do for pam_krb5, for example. But anyway, storing plain passwords on disk in this manner is really not good security practice. Further analysis ---------------- I am attaching logs generated with NSPR_LOG_MODULES=negotiateauth:5,timestamp,nsHttp:5,NTLM:5 NSPR_LOG_FILE=/dev/stderr KRB5_TRACE=/dev/stderr Both with and without gssntlmssp (and without any of the workarounds mentioned above). Here is what happens: 1) client tries to access the page and gets back a 401 error with WWW-Authenticate: negotiate WWW-Authenticate: ntlm Firefox is now supposed to try these mechanisms in turn. 2) firefox tries "negotiate" auth and enters GSSAPI with SPNEGO mech. 3) SPNEGO tries predefined mechanisms in turn: KRB5, KRB5_OLD, and KRB5_WRONG, IAKERB (krb5: lib/gssapi/krb5/gssapiP_krb5.h). The first three will try to get a ticket via kerberos directly. The GSSAPI/SPNEGO/IAKERB mechanism returns control to the browser and has the browser send an "Authorization: Negotiate" token with a wrapped kerberos ticket request to the server. The token also contains the list of supported SPNEGO mechanisms (at this point, IAKERB and NTLMSSP). 4) The server returns with another "WWW-Authenticate: Negotiate" tag, containing a challenge for NTLMSSP. Control is passed back to GSSAPI. GSSAPI detects that IAKERB is not supported and NTLMSSP advertised instead by the server. 5) GSSAPI starts NTLMSSP mechanism through gssntlmssp. But because firefox never passes user creds to GSSAPI, gssntlmssp doesn't have any usable credentials and the authentication fails. [ A comment in firefox' source file nsHttpNegotiateAuth.cpp says // Always set *identityInvalid == FALSE here. This // will prevent the browser from popping up the authentication // prompt window. Because GSSAPI does not have an API // for fetching initial credentials (ex: A Kerberos TGT), // there is no correct way to get the users credentials. I am not sure if this is correct but gssntlmssp's implementation seems to support the notion ] 6) the GSSAPI call ultimately fails, and thus the nsAuthGSSAPI authetication method in firefox fails, too. 7) Because the server hasn't sent "WWW-Authenticate: NTLM" any more after the first response in 3), firefox is now out of luck and can't continue. Therefore it simply displays a HTTP 401 error page. If gssntlmssp is disabled, the sequence of events is subtly different: 3a) GSSAPI will generate a slightly different SPNEGO/IAKERB token to send to the server, which doesn't advertize NTLMSSP support. 4a) The server returns an empty "WWW-Authenticate: Negotiate" tag and a "WWW-Authenticate: NTLM" tag. 5a) nsAuthGSSAPI::GetNextToken fails immediately without calling into GSSAPI again, because no input token was received from the server. 6a) nsAuthGSSAPI fails. 7a) Because the server has returned "WWW-Authenticate: NTLM" in the last HTTP response, firefox falls back to generic NTLM and is displays a username/password prompt, allowing the user to access the site.
Hi Martin, thank you for the detailed report, and sorry for the late reply. Your analysis is spot on, my preferred solution would be to fix Firefox, however I understand that it may take while s Mozilla moves slowly on these matters. One option you have is to disable the mechanism by commenting the mechanism id in /etc/gss/mech.d/ntlmssp.conf, assumin you do not need to use it in SIPE. If you need it in SIPE, I am not entirely sure what we can do here, as GSSAPI has only a system wide configuration file for mechanisms and it can't be overridden with environment variables or other options on a per application basis.
(In reply to Simo Sorce from comment #1) > Hi Martin, > thank you for the detailed report, and sorry for the late reply. > > Your analysis is spot on, my preferred solution would be to fix Firefox, > however I understand that it may take while s Mozilla moves slowly on these > matters. My personal workaround was to use NTLM_USER_FILE, so I don't need a quick fix :-) An upstream mozilla bug report might be worthwhile. But the crucial point seems to be 5) above - the mozilla guys see no way to pass the credentials to GSSAPI. If we open a bug there, they're likely to reject it for that reason. Would it make sense to try to talk to the GSSAPI folks? Someone would need to work out what has to be done to enable a proper password callback for this situation. It can't be impossible, can it? The most obvious solution in a modern Linux architecture would be using dbus and a password agent of some sort, such as gnome-keyring...
(In reply to Martin Wilck from comment #2) > (In reply to Simo Sorce from comment #1) > > Hi Martin, > > thank you for the detailed report, and sorry for the late reply. > > > > Your analysis is spot on, my preferred solution would be to fix Firefox, > > however I understand that it may take while s Mozilla moves slowly on these > > matters. > > My personal workaround was to use NTLM_USER_FILE, so I don't need a quick > fix :-) > > An upstream mozilla bug report might be worthwhile. But the crucial point > seems to be 5) above - the mozilla guys see no way to pass the credentials > to GSSAPI. If we open a bug there, they're likely to reject it for that > reason. They probably didn't look carefully at the API, they can use gss_acquire_cred_with_password() if they want to try to acquire password credentials, and use them to establish a context. > Would it make sense to try to talk to the GSSAPI folks? Someone would need > to work out what has to be done to enable a proper password callback for > this situation. It can't be impossible, can it? The most obvious solution in > a modern Linux architecture would be using dbus and a password agent of some > sort, such as gnome-keyring... Which is not needed, strictly, see above, however I am torn on whether we should allow the broser to ask you for credentials. That said we will have mechanisms to provide the browser with some credentials behind the scene eventually, so all we need is to make sure Firefox catches the right error condition and asks for a password *only* if context establishment fails due to lack of credentials, if it fails for other reasons Firefox shouldn't prompt (perhaps we can get a tunable that can be set to not prompting by default). One of the reasons asking for a password may be wrong is that when using krb5 a user may not have a password at all, but may be using other means to obtain credentials, like pkinit or a 2FA based pre-auth method that gss_acquire_cred_with_password() wouldn't be able to fulfill at all.
Hmm. Someone needs to figure out how to do this right. It doesn't make sense to point the mozillal guys to gss_acquire_cred_with_password() if it's ultimately not the right way to do it. My co-workers using Windows never have to bother about things like this. They enter their password at logon, and Windows takes care of the rest. It uses KRB5 tickets for services that support it, and for services that don't (as the one that caused my pain here) it will pull out the plain text password from some secret place where stores it under the hood (at least I'm guessing this is what happens - I have no other explanation why Windows is able to do an NTMLSSP login without asking for the password. Maybe I'm missing something?). This is the sort of user experience that Linux should aim at, too. Experienced users may want to control whether or not the plaintext password is cached somewhere, but ordinary users will simply want the Windows "single-sign-on" behavior. I think this functionality belongs in something like systemd-logind. It should support as many algorithms and authentication schemes as possible, ask for passwords only if absolutely necessary, and provide a secure and GUI-agnostic API for applications to retrieve credentials for whatever purpose. Only after having verified that the entity asking for the credentials has the right to do so, of course. "You may say I'm a dreamer, but I'm not the only one" :-)
(In reply to Martin Wilck from comment #4) > Hmm. Someone needs to figure out how to do this right. It doesn't make sense > to point the mozillal guys to gss_acquire_cred_with_password() if it's > ultimately not the right way to do it. > > My co-workers using Windows never have to bother about things like this. > They enter their password at logon, and Windows takes care of the rest. It > uses KRB5 tickets for services that support it, and for services that don't > (as the one that caused my pain here) it will pull out the plain text > password from some secret place where stores it under the hood (at least I'm > guessing this is what happens - I have no other explanation why Windows is > able to do an NTMLSSP login without asking for the password. Maybe I'm > missing something?). > > This is the sort of user experience that Linux should aim at, too. > Experienced users may want to control whether or not the plaintext password > is cached somewhere, but ordinary users will simply want the Windows > "single-sign-on" behavior. This is the user experience I aplannig to achieve eventually (we can already do that with kerberos), but it requires a lot of plumbing and changes so it takes time. > I think this functionality belongs in something like systemd-logind. It > should support as many algorithms and authentication schemes as possible, > ask for passwords only if absolutely necessary, and provide a secure and > GUI-agnostic API for applications to retrieve credentials for whatever > purpose. Only after having verified that the entity asking for the > credentials has the right to do so, of course. I plan to suupport this caching in SSSD+GSS-Proxy, we can't do it in system-logind because you need a spearate daemon to keep the hash out of rach of normal user processes, as that is a clear-text password equivalent. > "You may say I'm a dreamer, but I'm not the only one" :-) I dream too, it's just that dream=making takes time, but hey patches are welcome to speed it up :-)
Hm, why is the client even trying to negotiate NTLMSSP when it has no credentials? Sure, fixing Firefox would be great... but Firefox is massively resistant even to simple and obvious fixes (like spawning ntlm_auth from the right bloody place).
(In reply to David Woodhouse from comment #6) > Hm, why is the client even trying to negotiate NTLMSSP when it has no > credentials? My understanding is that firefox treats GSSAPI basically as a "black box" with no interaction. A callback through which GSSAPI could ask the user for a passwort is unsupported. GSSAPI walks through the list of SPNEGO mechanisms - all of them except NTLMSSP function without user callback. See also the comment quoted from the Mozilla sources under 5) above.
Right. Given that we didn't use gssapi_acquire_creds_with_password(), and given that there are no 'default' creds to be found in $NTLM_USER_FILE or winbind, why in $DEITY's name is SPNEGO attempting to negotiate the use of NTLMSSP? Or why is NTLMSSP allowing itself to be used at all? If it doesn't have any creds, surely that should be the same as if it isn't install at all? (Let's assume, for the moment, that nobody in the real world would ever want to mess around with NTLMSSP 'anonymous' authentication, and can explicitly enable that crap if they ever did.)
gssntlmssp will return an error at the first invocation of gss_init_sec_context (either direct or via spnego) if it is not provided or cannot acquire credentials. The al problen here is no gssntlmssp, but IAKERB, which causes the whole SPNEGO processing to start and the server seeing that the client support NTLMSSP only sends back Negotiate headers and not the NTLM headers. IIUC When Firefox ends up in a failure it sees no NTLM headers and decideds not to try the builtin NTLM auth. It is not easy to workaround this because SPNEGO does not want to try to acquire credentials for all mechanisms when the context is being established and rather simply calls into the following mechanisms ony if the previous one fail, this is because acquiring creds may be an expensive operation (AS requests and such for krb5). Now this means the spnego mech advertizes all supported mechanisms regardless of whether it has crdentials unless you acquire_cred before hand (which would cause spnego to *know* what creds are available). If firefox were smart enough it would acquire_cred and/or acquire_cred_with_password as a fallback, but Firefox isn't. It's not ideal, ideas welcome
Let me take my comment back, Greg Hudson pointed out that SPNEGO does in fact enumerate available credenials (at least in 1.14/master it does), and so NTLMSSP should not be advertied from the client if no credentials are available (NTLM user file or winbind).
So this is fixed in Fedora 23 because we have 1.14.1 there?
(And actually, for at least some servers my recollection is that they don't *care* whether NTLMSSP was one of the mechanisms offered in SPNEGO; they don't offer WWW-Authenticate: NTLM at all after WWW-Authenticate: Negotiate has been attempted. So even before gssntlmssp, the attempt to do IAKERB was screwing us over and preventing the NTLM authentication. There's a separate request somewhere for "make it possible to disable IAKERB... and do so by default" IIRC.)
Ok after a fine gdb session I figured out the problem, gssntlmssp was incorrectly returning a credential structure when no credentials were available. This was causing SPNEGO to advertize NTLMSSP as a viable mech when it shouldn't have. Proposed fix here: https://github.com/simo5/gss-ntlmssp/commit/26597c648d494f8757deec7da85bd4f0c340f06e Will make packages as soon as I get another pair of eyes on it for review.
gssntlmssp-0.6.0-4.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-014408dded
gssntlmssp-0.6.0-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d6f6098254
gssntlmssp-0.6.0-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-d6f6098254
gssntlmssp-0.6.0-4.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-014408dded
gssntlmssp-0.7.0-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-06e192287b
gssntlmssp-0.7.0-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-b2c360ff35
gssntlmssp-0.7.0-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-06e192287b
gssntlmssp-0.7.0-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-b2c360ff35
gssntlmssp-0.7.0-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.
gssntlmssp-0.7.0-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.