Bug 1985000

Summary: selinux is preventing the execution of lvmlockctl during sanlock device failure testing
Product: Red Hat Enterprise Linux 8 Reporter: Corey Marthaler <cmarthal>
Component: selinux-policyAssignee: Zdenek Pytela <zpytela>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.5CC: lvrabec, mmalik, plautrba, ssekidde, teigland
Target Milestone: betaKeywords: AutoVerified, Triaged
Target Release: 8.6Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: selinux-policy-3.14.3-83.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-10 15:14:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2021-07-22 15:48:16 UTC
Description of problem:
We expected messages from lvmlockctl while testing feature bug 1932586 and learned they weren't occurring due to selinux.


[root@host-162 ~]# grep avc /var/log/audit/audit.log
type=AVC msg=audit(1626470874.195:1018): avc:  denied  { execute } for  pid=21538 comm="sanlock-helper" name="lvmlockctl" dev="vda3" ino=8897230 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0
type=AVC msg=audit(1626470913.524:1019): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626470914.527:1020): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626470915.530:1021): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626470916.533:1022): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626470917.536:1023): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626470918.547:1024): avc:  denied  { signal } for  pid=17048 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904837.991:7071): avc:  denied  { execute } for  pid=27373 comm="sanlock-helper" name="lvmlockctl" dev="vda3" ino=9010775 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0
type=AVC msg=audit(1626904877.720:7072): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904878.722:7073): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904879.724:7074): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904880.727:7075): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904881.729:7076): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0
type=AVC msg=audit(1626904882.731:7077): avc:  denied  { signal } for  pid=1421 comm="sanlock-helper" scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0


Version-Release number of selected component (if applicable):
selinux-policy-3.14.3-68.el8.noarch


How reproducible:
Everytime

Comment 1 Zdenek Pytela 2021-07-30 18:08:21 UTC
Please gather all AVC denials with full auditing enabled:

1) Open the /etc/audit/rules.d/audit.rules file in an editor.
2) Remove the following line if it exists:
-a task,never
3) Add the following line to the end of the file:
-w /etc/shadow -p w
4) Restart the audit daemon:
  # service auditd restart
5) Re-run the scenario.
6) Collect AVC denials:
  # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

I see bz#1932586 has not been resolved yet, is the updated package available somewhere so that we can reproduce the problem first?

Comment 3 Zdenek Pytela 2021-08-06 13:43:42 UTC
Thank you for the command output, these permissions were requested:
allow sanlock_t bin_t:file execute;
allow sanlock_t lvm_t:process signal;

type=PROCTITLE msg=audit(08/05/2021 12:32:33.296:108) : proctitle=/usr/sbin/sanlock daemon 
type=PATH msg=audit(08/05/2021 12:32:33.296:108) : item=0 name=/usr/sbin/lvmlockctl inode=9277210 dev=fd:03 mode=file,555 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(08/05/2021 12:32:33.296:108) : cwd=/ 
type=SYSCALL msg=audit(08/05/2021 12:32:33.296:108) : arch=x86_64 syscall=execve success=no exit=EACCES(Permission denied) a0=0x558fdeda2ef0 a1=0x7fff9cacc0f0 a2=0x7fff9cacc8c0 a3=0x7f2f59968780 items=1 ppid=1524 pid=1663 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sanlock-helper exe=/usr/sbin/sanlock subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(08/05/2021 12:32:33.296:108) : avc:  denied  { execute } for  pid=1663 comm=sanlock-helper name=lvmlockctl dev="vda3" ino=9277210 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0 
----
type=PROCTITLE msg=audit(08/05/2021 12:33:13.197:109) : proctitle=/usr/sbin/sanlock daemon 
type=OBJ_PID msg=audit(08/05/2021 12:33:13.197:109) : opid=1541 oauid=unset ouid=root oses=-1 obj=system_u:system_r:lvm_t:s0 ocomm=lvmlockd 
type=SYSCALL msg=audit(08/05/2021 12:33:13.197:109) : arch=x86_64 syscall=kill success=no exit=EACCES(Permission denied) a0=0x605 a1=SIGTERM a2=0x200 a3=0x323ca items=0 ppid=1523 pid=1524 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sanlock-helper exe=/usr/sbin/sanlock subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(08/05/2021 12:33:13.197:109) : avc:  denied  { signal } for  pid=1524 comm=sanlock-helper scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0 

The first one will be probably resolved with lvmlockctl labeling:

-r-xr-xr-x. 1 root root system_u:object_r:lvm_exec_t:s0 2797920 Feb 22 10:53 /usr/sbin/lvm
-r-xr-xr-x. 1 root root system_u:object_r:bin_t:s0       180744 Feb 22 10:53 /usr/sbin/lvmlockctl

I see also reference to a sanlock-helper command - is it an executable, a process name, something else?

Comment 4 David Teigland 2021-08-10 15:19:43 UTC
sanlock-helper is a helper process created by the sanlock daemon process.  The helper process is created before sanlock does setgid/setuid because the helper process should remain root so that it can do SIGKILL on other pids.

Comment 5 Zdenek Pytela 2021-10-20 14:56:08 UTC
I've submitted a Fedora PR to address the issue:
https://github.com/fedora-selinux/selinux-policy/pull/925

David,

Will you be able to test some custom scenarios with SELinux enabled once we have the build (probably today or tomorrow) with these changes in selinux-policy?

Particularly note these changes only address the denials we saw in the audit logs, and there was only the "signal" permission which is any signal except signull/sigkill/sigstop/sigchld which have distinct permission name, so this is currently not covered by the changes:

(In reply to David Teigland from comment #4)
> sanlock-helper is a helper process created by the sanlock daemon process. 
> The helper process is created before sanlock does setgid/setuid because the
> helper process should remain root so that it can do SIGKILL on other pids.

Comment 6 David Teigland 2021-10-20 15:59:27 UTC
yes, I can try it.

Comment 7 Zdenek Pytela 2021-10-20 16:29:31 UTC
To backport:
commit 361550e925da7d661e99dbfa93bad2743dbbc0d2
Author: Zdenek Pytela <zpytela>
Date:   Wed Oct 20 16:06:09 2021 +0200

    Support sanlock VG automated recovery on storage access loss

Comment 12 David Teigland 2021-10-21 19:59:15 UTC
Installed the scratch rpm and ran the steps from bug 1932586 which worked.

# rpm -q selinux-policy
selinux-policy-3.14.3-82.el8.noarch


# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: lvmlockd lost access to locks in VG ab.
Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: Successful VG ab kill command /usr/sbin/my_vg_kill_script.sh ab

Comment 15 Zdenek Pytela 2021-10-27 13:11:08 UTC
(In reply to David Teigland from comment #12)
> Installed the scratch rpm and ran the steps from bug 1932586 which worked.
> 
> # rpm -q selinux-policy
> selinux-policy-3.14.3-82.el8.noarch
> 
> 
> # cat /etc/selinux/config 
> 
> # This file controls the state of SELinux on the system.
> # SELINUX= can take one of these three values:
> #     enforcing - SELinux security policy is enforced.
> #     permissive - SELinux prints warnings instead of enforcing.
> #     disabled - No SELinux policy is loaded.
> SELINUX=enforcing
> # SELINUXTYPE= can take one of these three values:
> #     targeted - Targeted processes are protected,
> #     minimum - Modification of targeted policy. Only selected processes are
> protected. 
> #     mls - Multi Level Security protection.
> SELINUXTYPE=targeted
> 
> 
> Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: lvmlockd lost access
> to locks in VG ab.
> Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: Successful VG ab
> kill command /usr/sbin/my_vg_kill_script.sh ab

David,

Thank you for the confirmation the service works. Still, I have a question: which signals sanlock-helper need to be allowed to send to lvm? 

#c0 says "signal" which is a generic signal, in SELinux different to kill/stop/child/null
#c4 and #c12 says "sigkill"
or even both? I need to rewrite the commit so I'd like to see all necessary permissions are allowed.

Comment 16 David Teigland 2021-10-27 15:30:19 UTC
sanlock-helper sends both SIGTERM and SIGKILL.

Comment 19 Zdenek Pytela 2021-11-03 20:02:38 UTC
I've decided to revert the previous commit and submit new ones:
https://github.com/fedora-selinux/selinux-policy/pull/933

Comment 20 Zdenek Pytela 2021-11-04 16:06:51 UTC
To backport:
commit d9d3fac0ffa9aac27dc87887137516d458104f71 (HEAD -> rawhide, upstream/rawhide)
Author: Zdenek Pytela <zpytela>
Date:   Wed Nov 3 20:59:53 2021 +0100

    Support sanlock VG automated recovery on storage access loss 2/2

commit 1e6936f6e2ce0fcfb26866d5f5d99b6e363d6113
Author: Zdenek Pytela <zpytela>
Date:   Wed Nov 3 20:56:33 2021 +0100

    Support sanlock VG automated recovery on storage access loss 1/2

and revert the previous 2 commits:
d55456395921719026dd65ab95647bdfecd56769
a0414e5e14f6dba91dff8e4583869e7842070f13

Comment 30 errata-xmlrpc 2022-05-10 15:14:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (selinux-policy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1995