Bug 1047776

Summary: GSS-Proxy no longer authenticates (for httpd with GSS_USE_PROXY=1): ISE and Unspecified GSS failure. Minor code may provide more information (, Permission denied)
Product: Red Hat Enterprise Linux 7 Reporter: Jan Pazdziora <jpazdziora>
Component: gssproxyAssignee: Simo Sorce <ssorce>
Status: CLOSED WORKSFORME QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: ejnersan, gdeschner, jpazdziora, rmainz
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-06 17:58:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Pazdziora 2014-01-02 08:35:01 UTC
Description of problem:

When using the GSS-Proxy with mod_auth_kerb based on

   https://fedorahosted.org/gss-proxy/wiki/Apache

starting with gssproxy 0.3.0 I always get Internal Server Error.

Version-Release number of selected component (if applicable):

gssproxy-0.3.0-0.el7.x86_64
gssproxy-0.3.0-5.el7.x86_64
gssproxy-0.3.0-6.el7.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Configure httpd with mod_auth_kerb to use GSS-Proxy, based on https://fedorahosted.org/gss-proxy/wiki/Apache
2. Run curl --negotiate -u : http://<the-httpd-server>/private

Actual results:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
[...]

and the error_log on the server says

[auth_kerb:error] [pid 10422] [client 10.16.96.136:60242] gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, Permission denied)

Expected results:

OK

Additional info:

This is a spin off of bug 1008777.

Things work with gssproxy-0.2.3-6.el7.x86_64, with the same configuration.

Comment 1 Jan Pazdziora 2014-01-02 08:35:32 UTC
Problem is the same on Fedora 20 with gssproxy-0.3.0-0.fc20.x86_64.

Comment 2 Jan Pazdziora 2014-01-02 08:40:37 UTC
The bug is there with setenforce 0 as well, so it's not SELinux related.

Comment 3 Jan Pazdziora 2014-01-02 08:43:13 UTC
Note -- the rebase to 0.3.0 went in via bug 1023369.

Comment 4 Simo Sorce 2014-01-02 14:05:32 UTC
(In reply to Jan Pazdziora from comment #1)
> Problem is the same on Fedora 20 with gssproxy-0.3.0-0.fc20.x86_64.

I am confused I just fixed a few bugs in this code and with master it all just works.
But when it was failing, it was failing with different errors.
Either no error (SPNEGO does not relay errors, until a patch I contributed over the break is backported), or replay cache errors.
Secondarily there was a user validation error.

I have never seen a permission denied error, fcan you strace the process and tell me what it fails to open ?

Comment 5 Jan Pazdziora 2014-01-03 08:52:08 UTC
(In reply to Simo Sorce from comment #4)
> (In reply to Jan Pazdziora from comment #1)
> > Problem is the same on Fedora 20 with gssproxy-0.3.0-0.fc20.x86_64.
> 
> I am confused I just fixed a few bugs in this code and with master it all
> just works.

So you say that for you and the latest RHEL 7 compose, following https://fedorahosted.org/gss-proxy/wiki/Apache gives you a working setup where the HTTP request passes?

> But when it was failing, it was failing with different errors.
> Either no error (SPNEGO does not relay errors, until a patch I contributed
> over the break is backported), or replay cache errors.
> Secondarily there was a user validation error.
> 
> I have never seen a permission denied error, fcan you strace the process and
> tell me what it fails to open ?

It comes from httpd trying to open /etc/krb5.keytab.

The problem seems to be that GSS-Proxy does not authenticate and Apache (mod_auth_kerb, I assume) tries to use the default keytab path to do the authentication.

So the fact that it says Permission denied is not really anything coming from GSS-Proxy, it's a fallback operation from Apache after GSS-Proxy silently did not authenticate.

Comment 6 Simo Sorce 2014-01-03 13:57:52 UTC
(In reply to Jan Pazdziora from comment #5)
> It comes from httpd trying to open /etc/krb5.keytab.
> 
> The problem seems to be that GSS-Proxy does not authenticate and Apache
> (mod_auth_kerb, I assume) tries to use the default keytab path to do the
> authentication.
> 
> So the fact that it says Permission denied is not really anything coming
> from GSS-Proxy, it's a fallback operation from Apache after GSS-Proxy
> silently did not authenticate.

Ok, so gss-proxy's proxymech tried with the proxy, failed hard, and retried locally. To get the real error returned from gss-proxy please set the environment variable: GSSPROXY_BEHAVIOR = REMOTE_ONLY

This mode is not very well tested but should work for your environment and report if gss-proxy is encountering a specific error.

Comment 7 Jan Pazdziora 2014-01-06 01:03:29 UTC
(In reply to Simo Sorce from comment #6)
> 
> Ok, so gss-proxy's proxymech tried with the proxy, failed hard, and retried
> locally. To get the real error returned from gss-proxy please set the
> environment variable: GSSPROXY_BEHAVIOR = REMOTE_ONLY

With

GSSPROXY_BEHAVIOR=REMOTE_ONLY

added to /etc/sysconfig/httpd I get

[Sun Jan 05 20:02:15.189109 2014] [auth_kerb:error] [pid 10468] [client 10.16.40.243:46492] gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information ()

in my /var/log/httpd/error_log.

Comment 9 Simo Sorce 2014-01-06 17:58:58 UTC
It was a configuration error.

You left in the nfs/client stuff which has "allow_any_uid = yes".
This means it will match any request unles something matched first.
It was thus matchin gyour request, and then failing because nfs/client specifies "cred_usage = initiate" but we are doing an accept. But it would have failed anyway due to the wrong keytab.

The solution was to simply move the http service on top of the config file so that euid = 48 would match first.

Another option is to specify a different socket for the http service and set the GSSPROXY_SOCKET evn variable.

Another option is to add selinuc_context so that service/HTTP matches only on 
the apache label and nfs/client only on the nfs label.

However it is obvious the problem here is that gssproxy is not logging well enough to make the problem easily understandable, maybe we should open an RFE upstream to log operations, what service they matched and what was the outcome, so that you can find what is going on w/o gdb ...

I will close this bug as working for me, now.

Comment 10 Jan Pazdziora 2014-01-07 01:03:20 UTC
(In reply to Simo Sorce from comment #9)
> It was a configuration error.
> 
> You left in the nfs/client stuff which has "allow_any_uid = yes".
> This means it will match any request unles something matched first.
> It was thus matchin gyour request, and then failing because nfs/client
> specifies "cred_usage = initiate" but we are doing an accept. But it would
> have failed anyway due to the wrong keytab.

It looks like the

  cred_usage = initiate
  allow_any_uid = yes

lines are new lines in gssproxy-0.3.0. Moving the service/HTTP section up helps.

I've amended https://fedorahosted.org/gss-proxy/wiki/Apache now.

Comment 11 Ejner Fergo 2014-11-27 13:40:24 UTC
I still get this error on Fedora 21:

==> /var/log/httpd/error_log <==
[Thu Nov 27 14:31:15.967483 2014] [auth_kerb:error] [pid 1001] [client 10.0.3.1:53508] gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, Permission denied)

Followed the guide in the link by Jan Pazdziora to the letter, exept I deleted all the nfs parts in gssproxy.conf, leaving only service/HTTP (this part was at the top in any case)

gssproxy-0.3.1-4.fc21.x86_64
mod_auth_kerb-5.4-30.fc21.x86_64
httpd-2.4.10-9.fc21.x86_64

Comment 12 Jan Pazdziora 2014-12-04 11:14:13 UTC
(In reply to Ejner Fergo from comment #11)
> I still get this error on Fedora 21:
> 
> ==> /var/log/httpd/error_log <==
> [Thu Nov 27 14:31:15.967483 2014] [auth_kerb:error] [pid 1001] [client
> 10.0.3.1:53508] gss_acquire_cred() failed: Unspecified GSS failure.  Minor
> code may provide more information (, Permission denied)
> 
> Followed the guide in the link by Jan Pazdziora to the letter, exept I
> deleted all the nfs parts in gssproxy.conf, leaving only service/HTTP (this
> part was at the top in any case)
> 
> gssproxy-0.3.1-4.fc21.x86_64
> mod_auth_kerb-5.4-30.fc21.x86_64
> httpd-2.4.10-9.fc21.x86_64

Thank you for the report. I've filed Fedora bug 1170576 now.