1487350 – SELinux sometimes prevents sec=krb5 mounts

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1487350 - SELinux sometimes prevents sec=krb5 mounts

Summary: SELinux sometimes prevents sec=krb5 mounts

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	selinux-policy
Sub Component:
Version:	7.4
Hardware:	All
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	Lukas Vrabec
QA Contact:	Milos Malik
Docs Contact:	Mirek Jahoda
URL:
Whiteboard:
Depends On:
Blocks:	1665815
TreeView+	depends on / blocked

Reported:	2017-08-31 17:29 UTC by Orion Poplawski
Modified:	2019-08-06 12:52 UTC (History)
CC List:	18 users (show)
Fixed In Version:	selinux-policy-3.13.1-233.el7
Doc Type:	Bug Fix
Doc Text:	.SELinux now allow `gssd_t` processes to access kernel keyrings of other processes Previously, an allow rule for the `gssd_t` type was missing in the SELinux policy. As a consequence, SELinux in enforcing mode occasionally prevented processes running as `gssd_t` from accessing kernel keyrings of other processes and could block for example `sec=krb5` mounts. The rule has been added to the policy, and processes running as `gssd_t` are now able to access keyrings of other processes.
Clone Of:
Clones:	1665815 (view as bug list)
Environment:
Last Closed:	2019-08-06 12:51:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
unconfned processes (2.31 KB, text/plain) 2017-10-30 15:09 UTC, Orion Poplawski	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1532243	0	medium	CLOSED	SELinux is preventing /usr/sbin/rpc.gssd from read access on the key Unknown.	2021-02-22 00:41:40 UTC
Red Hat Product Errata	RHBA-2019:2127	0	None	None	None	2019-08-06 12:52:06 UTC

Internal Links: 1532243

Description Orion Poplawski 2017-08-31 17:29:03 UTC

Description of problem:

On some systems I'm seeing sec=krb5 mounts fail sometimes.  AVC logs show:

type=AVC msg=audit(1504200164.309:3727): avc:  denied  { read } for  pid=657 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key
type=AVC msg=audit(1504200164.309:3728): avc:  denied  { write } for  pid=657 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=key

Allowing via custom selinux policy seems to clear things up.

Version-Release number of selected component (if applicable):
nfs-utils-1.3.0-0.48.el7.x86_64
selinux-policy-3.13.1-166.el7.noarch


How reproducible:
Sporadic - some boots okay, some fail.

Comment 2 R P Herrold 2017-08-31 21:13:01 UTC

Orion,

conceptually this seems like a race condition which mounting and umonting remote filesystems need intervention to handle well.  see:

https://www.freedesktop.org/software/systemd/man/systemd.mount.html
https://bugzilla.redhat.com/show_bug.cgi?id=881481#c1

Perhaps there is a need of an RFE to add a 'hint' to the systemd automatic dependency generation when krb5 is present?

I know that with plain old NFS mounts, simply adding _netdev in the fstab has not been enough on boxes I administer to reliably avoid network shutdown before NFS umounts are done.  

Just a thought

Comment 3 Orion Poplawski 2017-08-31 21:37:31 UTC

These mounts are all handled by autofs and so are only mounted much later.  Near as I can tell though a machine can either boot up into a good state or a bad state.

Comment 4 R P Herrold 2017-08-31 22:50:16 UTC

yeah ... but similar -- there is a autofs systemd target out there 

/usr/lib/systemd/system/autofs.service

but that file makes no mention of krb5 

Checking the available packages, I don't see a systemd one as to krb5

# yum list \*krb5\*


I see a Debian bug which implies that such a krb5.service has been written

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790262


I have to think perhaps adding the target and an 'After' clause would help?

The example below is for non-Kerberized NFS in a protected environment, but has an example of the 'After='

[root@centos-7 ~]# grep -v "^#" /etc/systemd/system/home.mount
[Unit]
Description=home devices  
Wants=network-online.target
After=network-online.target
Conflicts=umount.target

[Mount]
What=nas1.first.owlriver.net:/NFS_export/home
Where=/home
Type=nfs

[Install]
WantedBy=machines.target 

[root@centos-7 ~]#

Comment 5 Lukas Vrabec 2017-10-30 15:05:00 UTC

Could you attach output of: 

# ps -efZ | grep unconfined_service_t 

THanks,
Lukas.

Comment 6 Orion Poplawski 2017-10-30 15:09:38 UTC

Created attachment 1345506 [details]
unconfned processes

Here's an example from one system.

Comment 8 Orion Poplawski 2018-01-29 18:54:46 UTC

New one:

type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for  pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=key

Comment 9 Brian J. Murrell 2018-05-10 17:48:20 UTC

(In reply to Orion Poplawski from comment #8)
> New one:
> 
> type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for 
> pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0
> tcontext=system_u:system_r:httpd_t:s0 tclass=key

I get tons of these:

type=AVC msg=audit(1525973209.105:154645): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.297:154646): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.353:154647): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.371:154648): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.376:154649): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.377:154650): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.567:154651): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.569:154652): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.601:154653): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.602:154654): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.603:154655): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973785.870:154692): avc:  denied  { getpgid } for  pid=18840 comm="fail2ban-server" scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:system_r:fail2ban_t:s0 tclass=process

I suspect these are corresponding with rpc.gssd's:

[pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) = 161860634
[pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb", KEY_SPEC_PROCESS_KEYRING) = 451990255
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary", 0) = 731119501
[pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
[pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1 EINVAL (Invalid argument)
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES (Permission denied)

Comment 11 Lukas Vrabec 2018-12-04 14:34:21 UTC

Hi All, 

Is this regression? I'm not sure what happened in rpc.gssd code. Gssd is trying to manupulate with manipulate the kernel's key management facility.

Moving this to nfs-utils component. Feel free to move it back after some coments related to our SELinux issue. 

Thanks,
Lukas.

Comment 12 Lukas Vrabec 2018-12-05 08:31:28 UTC

Adding temporary fix to make it working with SELinux. 

$ cat local_gssd_key.te 
policy_module(local_gssd_key,1.0)

require {
	attribute domain;
	type gssd_t;
}

allow gssd_t domain:key { read write };

$ make -f /usr/share/selinux/devel/Makefile local_gssd_key.pp
# semodule -i local_gssd_key.pp

Comment 13 Steve Dickson 2018-12-06 17:56:47 UTC

(In reply to Brian J. Murrell from comment #9)
> (In reply to Orion Poplawski from comment #8)
> 
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

I have no idea who is manipulating these keys... It is not
in the rpc.gssd code... so It must be in one the libraries 
that its linked with
$ ldd /usr/sbin/rpc.gssd
	linux-vdso.so.1 =>  (0x00007ffe81726000)
	libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fcc5ce9d000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fcc5cc50000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fcc5c968000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fcc5c735000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fcc5c531000)
	libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fcc5c306000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcc5c0ea000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fcc5bd1d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fcc5bb0f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fcc5b90b000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fcc5b707000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fcc5b4ee000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcc5d2fa000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fcc5b2d4000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fcc5b0ad000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fcc5ae4b000)

rpc.gssd also used the gssproxy code... but I took a look
int that code and did not see any key calls.

Would it be possible to get a backtrace to see what is making those calls?

Comment 16 Steve Dickson 2018-12-20 16:21:44 UTC

(In reply to Brian J. Murrell from comment #9)
>
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

Hmm... since this is happening during krb5 mounts which means
name@domain need to be mapped in to real uids and gids.

This is done by the kernel doing an upcall to nfsidmap(5)
which does indeed used key rings to store the id mappings.

So this mostly likely nfsidmap not rpc.gssd.

But I have no idea how to tell SELinux it is ok fir
nfsidmap to create and manipulate key rings... 
SELinux changes were not needed in the past.

So I'm going to throw this back to the selinux-policy team
to see what their opinion is... If something needs
to change in nfs-utils please throw it back

Comment 35 errata-xmlrpc 2019-08-06 12:51:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2127

Note You need to log in before you can comment on or make changes to this bug.