Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1487350 - SELinux sometimes prevents sec=krb5 mounts
Summary: SELinux sometimes prevents sec=krb5 mounts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.4
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Lukas Vrabec
QA Contact: Milos Malik
Mirek Jahoda
URL:
Whiteboard:
Depends On:
Blocks: 1665815
TreeView+ depends on / blocked
 
Reported: 2017-08-31 17:29 UTC by Orion Poplawski
Modified: 2019-08-06 12:52 UTC (History)
18 users (show)

Fixed In Version: selinux-policy-3.13.1-233.el7
Doc Type: Bug Fix
Doc Text:
.SELinux now allow `gssd_t` processes to access kernel keyrings of other processes Previously, an allow rule for the `gssd_t` type was missing in the SELinux policy. As a consequence, SELinux in enforcing mode occasionally prevented processes running as `gssd_t` from accessing kernel keyrings of other processes and could block for example `sec=krb5` mounts. The rule has been added to the policy, and processes running as `gssd_t` are now able to access keyrings of other processes.
Clone Of:
: 1665815 (view as bug list)
Environment:
Last Closed: 2019-08-06 12:51:45 UTC
Target Upstream Version:


Attachments (Terms of Use)
unconfned processes (2.31 KB, text/plain)
2017-10-30 15:09 UTC, Orion Poplawski
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1532243 0 medium CLOSED SELinux is preventing /usr/sbin/rpc.gssd from read access on the key Unknown. 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2019:2127 0 None None None 2019-08-06 12:52:06 UTC

Internal Links: 1532243

Description Orion Poplawski 2017-08-31 17:29:03 UTC
Description of problem:

On some systems I'm seeing sec=krb5 mounts fail sometimes.  AVC logs show:

type=AVC msg=audit(1504200164.309:3727): avc:  denied  { read } for  pid=657 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key
type=AVC msg=audit(1504200164.309:3728): avc:  denied  { write } for  pid=657 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key

Allowing via custom selinux policy seems to clear things up.

Version-Release number of selected component (if applicable):
nfs-utils-1.3.0-0.48.el7.x86_64
selinux-policy-3.13.1-166.el7.noarch


How reproducible:
Sporadic - some boots okay, some fail.

Comment 2 R P Herrold 2017-08-31 21:13:01 UTC
Orion,

conceptually this seems like a race condition which mounting and umonting remote filesystems need intervention to handle well.  see:

https://www.freedesktop.org/software/systemd/man/systemd.mount.html
https://bugzilla.redhat.com/show_bug.cgi?id=881481#c1

Perhaps there is a need of an RFE to add a 'hint' to the systemd automatic dependency generation when krb5 is present?

I know that with plain old NFS mounts, simply adding _netdev in the fstab has not been enough on boxes I administer to reliably avoid network shutdown before NFS umounts are done.  

Just a thought

Comment 3 Orion Poplawski 2017-08-31 21:37:31 UTC
These mounts are all handled by autofs and so are only mounted much later.  Near as I can tell though a machine can either boot up into a good state or a bad state.

Comment 4 R P Herrold 2017-08-31 22:50:16 UTC
yeah ... but similar -- there is a autofs systemd target out there 

/usr/lib/systemd/system/autofs.service

but that file makes no mention of krb5 

Checking the available packages, I don't see a systemd one as to krb5

# yum list \*krb5\*


I see a Debian bug which implies that such a krb5.service has been written

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790262


I have to think perhaps adding the target and an 'After' clause would help?

The example below is for non-Kerberized NFS in a protected environment, but has an example of the 'After='

[root@centos-7 ~]# grep -v "^#" /etc/systemd/system/home.mount
[Unit]
Description=home devices  
Wants=network-online.target
After=network-online.target
Conflicts=umount.target

[Mount]
What=nas1.first.owlriver.net:/NFS_export/home
Where=/home
Type=nfs

[Install]
WantedBy=machines.target 

[root@centos-7 ~]#

Comment 5 Lukas Vrabec 2017-10-30 15:05:00 UTC
Could you attach output of: 

# ps -efZ | grep unconfined_service_t 

THanks,
Lukas.

Comment 6 Orion Poplawski 2017-10-30 15:09:38 UTC
Created attachment 1345506 [details]
unconfned processes

Here's an example from one system.

Comment 8 Orion Poplawski 2018-01-29 18:54:46 UTC
New one:

type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for  pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=key

Comment 9 Brian J. Murrell 2018-05-10 17:48:20 UTC
(In reply to Orion Poplawski from comment #8)
> New one:
> 
> type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for 
> pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0
> tcontext=system_u:system_r:httpd_t:s0 tclass=key

I get tons of these:

type=AVC msg=audit(1525973209.105:154645): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.297:154646): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.353:154647): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.371:154648): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.376:154649): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.377:154650): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.567:154651): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.569:154652): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.601:154653): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.602:154654): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.603:154655): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973785.870:154692): avc:  denied  { getpgid } for  pid=18840 comm="fail2ban-server" scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:system_r:fail2ban_t:s0 tclass=process

I suspect these are corresponding with rpc.gssd's:

[pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) = 161860634
[pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb", KEY_SPEC_PROCESS_KEYRING) = 451990255
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary", 0) = 731119501
[pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
[pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1 EINVAL (Invalid argument)
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES (Permission denied)

Comment 11 Lukas Vrabec 2018-12-04 14:34:21 UTC
Hi All, 

Is this regression? I'm not sure what happened in rpc.gssd code. Gssd is trying to manupulate with manipulate the kernel's key management facility.

Moving this to nfs-utils component. Feel free to move it back after some coments related to our SELinux issue. 

Thanks,
Lukas.

Comment 12 Lukas Vrabec 2018-12-05 08:31:28 UTC
Adding temporary fix to make it working with SELinux. 

$ cat local_gssd_key.te 
policy_module(local_gssd_key,1.0)

require {
	attribute domain;
	type gssd_t;
}

allow gssd_t domain:key { read write };

$ make -f /usr/share/selinux/devel/Makefile local_gssd_key.pp
# semodule -i local_gssd_key.pp

Comment 13 Steve Dickson 2018-12-06 17:56:47 UTC
(In reply to Brian J. Murrell from comment #9)
> (In reply to Orion Poplawski from comment #8)
> 
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

I have no idea who is manipulating these keys... It is not
in the rpc.gssd code... so It must be in one the libraries 
that its linked with
$ ldd /usr/sbin/rpc.gssd
	linux-vdso.so.1 =>  (0x00007ffe81726000)
	libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fcc5ce9d000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fcc5cc50000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fcc5c968000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fcc5c735000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fcc5c531000)
	libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fcc5c306000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcc5c0ea000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fcc5bd1d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fcc5bb0f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fcc5b90b000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fcc5b707000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fcc5b4ee000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcc5d2fa000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fcc5b2d4000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fcc5b0ad000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fcc5ae4b000)

rpc.gssd also used the gssproxy code... but I took a look
int that code and did not see any key calls.

Would it be possible to get a backtrace to see what is making those calls?

Comment 16 Steve Dickson 2018-12-20 16:21:44 UTC
(In reply to Brian J. Murrell from comment #9)
>
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

Hmm... since this is happening during krb5 mounts which means
name@domain need to be mapped in to real uids and gids.

This is done by the kernel doing an upcall to nfsidmap(5)
which does indeed used key rings to store the id mappings.

So this mostly likely nfsidmap not rpc.gssd.

But I have no idea how to tell SELinux it is ok fir
nfsidmap to create and manipulate key rings... 
SELinux changes were not needed in the past.

So I'm going to throw this back to the selinux-policy team
to see what their opinion is... If something needs
to change in nfs-utils please throw it back

Comment 35 errata-xmlrpc 2019-08-06 12:51:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2127


Note You need to log in before you can comment on or make changes to this bug.