Bug 1487350

Orion,

conceptually this seems like a race condition which mounting and umonting remote filesystems need intervention to handle well.  see:

https://www.freedesktop.org/software/systemd/man/systemd.mount.html
https://bugzilla.redhat.com/show_bug.cgi?id=881481#c1

Perhaps there is a need of an RFE to add a 'hint' to the systemd automatic dependency generation when krb5 is present?

I know that with plain old NFS mounts, simply adding _netdev in the fstab has not been enough on boxes I administer to reliably avoid network shutdown before NFS umounts are done.  

Just a thought

These mounts are all handled by autofs and so are only mounted much later.  Near as I can tell though a machine can either boot up into a good state or a bad state.

yeah ... but similar -- there is a autofs systemd target out there 

/usr/lib/systemd/system/autofs.service

but that file makes no mention of krb5 

Checking the available packages, I don't see a systemd one as to krb5

# yum list \*krb5\*


I see a Debian bug which implies that such a krb5.service has been written

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790262


I have to think perhaps adding the target and an 'After' clause would help?

The example below is for non-Kerberized NFS in a protected environment, but has an example of the 'After='

[root@centos-7 ~]# grep -v "^#" /etc/systemd/system/home.mount
[Unit]
Description=home devices  
Wants=network-online.target
After=network-online.target
Conflicts=umount.target

[Mount]
What=nas1.first.owlriver.net:/NFS_export/home
Where=/home
Type=nfs

[Install]
WantedBy=machines.target 

[root@centos-7 ~]#

Could you attach output of: 

# ps -efZ | grep unconfined_service_t 

THanks,
Lukas.

Created attachment 1345506 [details]
unconfned processes

Here's an example from one system.

New one:

type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for  pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=key

(In reply to Orion Poplawski from comment #8)
> New one:
> 
> type=AVC msg=audit(1517251818.976:341657): avc:  denied  { read } for 
> pid=1848 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0
> tcontext=system_u:system_r:httpd_t:s0 tclass=key

I get tons of these:

type=AVC msg=audit(1525973209.105:154645): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.297:154646): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.353:154647): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.371:154648): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.376:154649): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.377:154650): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.567:154651): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.569:154652): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.601:154653): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.602:154654): avc:  denied  { read } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973209.603:154655): avc:  denied  { write } for  pid=1024 comm="rpc.gssd" scontext=system_u:system_r:gssd_t:s0 tcontext=system_u:system_r:system_cronjob_t:s0-s0:c0.c1023 tclass=key
type=AVC msg=audit(1525973785.870:154692): avc:  denied  { getpgid } for  pid=18840 comm="fail2ban-server" scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:system_r:fail2ban_t:s0 tclass=process

I suspect these are corresponding with rpc.gssd's:

[pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) = 161860634
[pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb", KEY_SPEC_PROCESS_KEYRING) = 451990255
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary", 0) = 731119501
[pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
[pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1 EINVAL (Invalid argument)
[pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY (Required key not available)
[pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES (Permission denied)

Hi All, 

Is this regression? I'm not sure what happened in rpc.gssd code. Gssd is trying to manupulate with manipulate the kernel's key management facility.

Moving this to nfs-utils component. Feel free to move it back after some coments related to our SELinux issue. 

Thanks,
Lukas.

Adding temporary fix to make it working with SELinux. 

$ cat local_gssd_key.te 
policy_module(local_gssd_key,1.0)

require {
	attribute domain;
	type gssd_t;
}

allow gssd_t domain:key { read write };

$ make -f /usr/share/selinux/devel/Makefile local_gssd_key.pp
# semodule -i local_gssd_key.pp

(In reply to Brian J. Murrell from comment #9)
> (In reply to Orion Poplawski from comment #8)
> 
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

I have no idea who is manipulating these keys... It is not
in the rpc.gssd code... so It must be in one the libraries 
that its linked with
$ ldd /usr/sbin/rpc.gssd
	linux-vdso.so.1 =>  (0x00007ffe81726000)
	libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fcc5ce9d000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fcc5cc50000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fcc5c968000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fcc5c735000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fcc5c531000)
	libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fcc5c306000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcc5c0ea000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fcc5bd1d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fcc5bb0f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fcc5b90b000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fcc5b707000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fcc5b4ee000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcc5d2fa000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fcc5b2d4000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fcc5b0ad000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fcc5ae4b000)

rpc.gssd also used the gssproxy code... but I took a look
int that code and did not see any key calls.

Would it be possible to get a backtrace to see what is making those calls?

(In reply to Brian J. Murrell from comment #9)
>
> I suspect these are corresponding with rpc.gssd's:
> 
> [pid 26466] keyctl(KEYCTL_GET_PERSISTENT, 0, KEY_SPEC_PROCESS_KEYRING) =
> 161860634
> [pid 26466] keyctl(KEYCTL_SEARCH, 161860634, "keyring", "_krb",
> KEY_SPEC_PROCESS_KEYRING) = 451990255
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "user", "krb_ccache:primary",
> 0) = 731119501
> [pid 26466] keyctl(KEYCTL_READ, 731119501, NULL, 0) = 9
> [pid 26466] keyctl(KEYCTL_READ, 731119501, "", 9) = 9
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] keyctl(KEYCTL_SEARCH, 0, "user", "__krb5_princ__", 0) = -1
> EINVAL (Invalid argument)
> [pid 26466] keyctl(KEYCTL_SEARCH, 451990255, "keyring", "0", 0) = -1 ENOKEY
> (Required key not available)
> [pid 26466] add_key("keyring", "0", NULL, 0, 451990255) = -1 EACCES
> (Permission denied)

Hmm... since this is happening during krb5 mounts which means
name@domain need to be mapped in to real uids and gids.

This is done by the kernel doing an upcall to nfsidmap(5)
which does indeed used key rings to store the id mappings.

So this mostly likely nfsidmap not rpc.gssd.

But I have no idea how to tell SELinux it is ok fir
nfsidmap to create and manipulate key rings... 
SELinux changes were not needed in the past.

So I'm going to throw this back to the selinux-policy team
to see what their opinion is... If something needs
to change in nfs-utils please throw it back

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2127