Bug 1168962

Summary: gssproxy is not working with httpd on ppc64 and s390x
Product: Red Hat Enterprise Linux 7 Reporter: Patrik Kis <pkis>
Component: gssproxyAssignee: Simo Sorce <ssorce>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: dpal, eguan, jpazdziora, ksiddiqu, pkis
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gssproxy-0.4.1-2.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 09:30:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch that fixes a cast error that break S390 and alike
none
fixed patch, for reference
none
console output with verification steps none

Description Patrik Kis 2014-11-28 14:37:56 UTC
Description of problem:
When the same test is executed as in bug 1008777 (https://fedorahosted.org/gss-proxy/wiki/Apache) and selinux is switched to permissive because the mentioned bug, the scenario is working on x86_64 and ppc64le architectures but not on ppc64 and s390x.
Not sure if the problem is in gssprxy (but it looks like); I was not able to get closer to the root cause.
The test page is accessible directly with mod_auth_kerb on all architectures, problems starts when gssproxy is configured.

Version-Release number of selected component (if applicable):
gssproxy-0.3.0-9.el7.s390x
krb5-libs-1.12.2-8.el7.s390x
httpd-2.4.6-29.el7.s390x
mod_auth_kerb-5.4-28.el7.s390x
selinux-policy-3.13.1-9.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. KDC set up, keytab created ...

# setenforce 0
# cat /var/www/html/private 
Test page to test GSSAPI through gssproxy
# cat /etc/httpd/conf.d/gssapi.conf 
<Location /private>
  AuthType Kerberos
  AuthName "Kerberos Login"
  KrbMethodNegotiate On
  require valid-user
</Location>
# cat /etc/sysconfig/httpd
LANG=C
GSS_USE_PROXY=1
# cat /etc/gssproxy/gssproxy.conf
[service/HTTP]
  mechs = krb5
  cred_store = keytab:/var/lib/gssproxy/clients/http.keytab
  cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
  euid = 48
# klist -kt /var/lib/gssproxy/clients/http.keytab
Keytab name: FILE:/var/lib/gssproxy/clients/http.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
   2 11/28/2014 08:10:04 HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
# echo aaa | kinit alice
Password for alice: 
# klist 
Ticket cache: KEYRING:persistent:0:krb_ccache_PKowI4K
Default principal: alice

Valid starting       Expires              Service principal
11/28/2014 09:12:55  11/29/2014 09:12:55  krbtgt/ZMRAZ.COM
# curl --negotiate -u : -i http://`hostname`/private
HTTP/1.1 401 Unauthorized
Date: Fri, 28 Nov 2014 14:13:02 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_kerb/5.4
WWW-Authenticate: Negotiate
WWW-Authenticate: Basic realm="Kerberos Login"
Content-Length: 381
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 500 Internal Server Error
Date: Fri, 28 Nov 2014 14:13:02 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) mod_auth_kerb/5.4
Content-Length: 527
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 root@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

# klist 
Ticket cache: KEYRING:persistent:0:krb_ccache_PKowI4K
Default principal: alice

Valid starting       Expires              Service principal
11/28/2014 09:13:02  11/29/2014 09:12:55  HTTP/ibm-z10-35.rhts.eng.bos.redhat.com
11/28/2014 09:12:55  11/29/2014 09:12:55  krbtgt/ZMRAZ.COM


Actual results:

# gssproxy -i -d
Debug Enabled
Client connected (fd = 10) (pid = 31897) (uid = 48) (gid = 48) (context = system_u:system_r:httpd_t:s0)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48, socket: (null)

==> /var/log/httpd/error_log <==
[Fri Nov 28 09:13:02.255605 2014] [auth_kerb:error] [pid 31901] [client 10.16.66.226:47582] gss_acquire_cred() failed: Unspecified GSS failure.  Minor code may provide more information (, Permission denied)

Comment 1 Simo Sorce 2014-12-10 13:41:44 UTC
Can you please check the audit log ?
I think this is probably a duplicate for bz1008777

Comment 2 Patrik Kis 2014-12-11 08:21:41 UTC
(In reply to Simo Sorce from comment #1)
> Can you please check the audit log ?
> I think this is probably a duplicate for bz1008777

I don't think is is duplicate of bz1008777. Note that in this case the test were executed with selinux in permissive mode and it worked only on x86_64 and ppc64le.
On s390x, for example, in permissive mode the web server still responded with "500 Internal Server Error". Audit log shows the well known denial:

----
time->Thu Dec 11 03:14:16 2014
type=SYSCALL msg=audit(1418285656.865:982579): arch=80000016 syscall=5 success=yes exit=11 a0=3fff000abc0 a1=2c1 a2=180 a3=3fff4b98a18 items=0 ppid=1 pid=56509 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="gssproxy" exe="/usr/sbin/gssproxy" subj=system_u:system_r:gssproxy_t:s0 key=(null)
type=AVC msg=audit(1418285656.865:982579): avc:  denied  { create } for  pid=56509 comm="gssproxy" name="HTTP_0" scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file


but it was only logged this time (note: success=yes), so there must be a problem somewhere else too which seems to be ppc64 and s390x related.

Comment 3 Simo Sorce 2014-12-12 18:20:03 UTC
Ok then I need a s390x or ppc64 machine with development tool (debuginfo/gdb) installed to log in and see what is going on. There may be some subtle endianess bug somewhere I guess.

Comment 5 Simo Sorce 2014-12-15 16:41:47 UTC
Created attachment 969156 [details]
Patch that fixes a cast error that break S390 and alike

Can you check if this patch fixes the issue for you ?

Comment 6 Simo Sorce 2014-12-15 16:42:39 UTC
Ah btw I added this to the HTTP service definition:
cred_store = rcache:none:none

This avoids selinux denials.

Comment 7 Jan Pazdziora (Red Hat) 2014-12-16 08:58:17 UTC
(In reply to Simo Sorce from comment #5)
> Created attachment 969156 [details]
> Patch that fixes a cast error that break S390 and alike
> 
> Can you check if this patch fixes the issue for you ?

You want the line

+    size_t cypherlen;

to read

+    size_t cipherlen;

or the code won't compile.

I confirm that with the fixed patch things start to work on a s390x machine.

Comment 8 Simo Sorce 2014-12-16 17:42:15 UTC
arghh, how did it compile for me ? ...
updating patch, but thanks fo confirming.

Comment 9 Simo Sorce 2014-12-16 17:44:04 UTC
Created attachment 969676 [details]
fixed patch, for reference

Comment 10 Simo Sorce 2015-01-05 21:57:35 UTC
Patrik,
do you need a scratch build to test this ?

Comment 11 Patrik Kis 2015-01-06 10:09:08 UTC
(In reply to Simo Sorce from comment #10)
> Patrik,
> do you need a scratch build to test this ?

No thanks, the issue is not that urgent for me. Jan confirmed that the patch is working. But if you want to test it once more, create the build and I can run my auto test on it.

Comment 12 Dmitri Pal 2015-04-15 16:49:36 UTC
Upstream ticket:
https://fedorahosted.org/gss-proxy/ticket/146

Comment 13 Roland Mainz 2015-06-12 00:13:04 UTC
Taking bug myself...

Comment 14 Roland Mainz 2015-07-10 01:03:14 UTC
Fixed in gssproxy-0.4.1-2.el7 ...

... marking bug as MODIFIED.

Comment 16 Kaleem 2015-08-13 11:31:05 UTC
Verified.

[root@ibm-p730-06-lp1 ~]# rpm -q gssproxy mod_auth_kerb ipa-client
gssproxy-0.4.1-3.el7.ppc64
mod_auth_kerb-5.4-28.el7.ppc64
ipa-client-4.2.0-3.el7.ppc64
[root@ibm-p730-06-lp1 ~]#

and 

[root@ibm-z10-05 ~]# rpm -q gssproxy mod_auth_kerb ipa-client
gssproxy-0.4.1-3.el7.s390x
mod_auth_kerb-5.4-28.el7.s390x
ipa-client-4.2.0-3.el7.s390x
[root@ibm-z10-05 ~]#

Please find the attached console output of verification steps taken.

Comment 17 Kaleem 2015-08-13 11:33:03 UTC
Created attachment 1062477 [details]
console output with verification steps

Comment 20 errata-xmlrpc 2015-11-19 09:30:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2298.html