Please gather all AVC denials with full auditing enabled:

1) Open the /etc/audit/rules.d/audit.rules file in an editor.
2) Remove the following line if it exists:
-a task,never
3) Add the following line to the end of the file:
-w /etc/shadow -p w
4) Restart the audit daemon:
  # service auditd restart
5) Re-run the scenario.
6) Collect AVC denials:
  # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

I see bz#1932586 has not been resolved yet, is the updated package available somewhere so that we can reproduce the problem first?

Thank you for the command output, these permissions were requested:
allow sanlock_t bin_t:file execute;
allow sanlock_t lvm_t:process signal;

type=PROCTITLE msg=audit(08/05/2021 12:32:33.296:108) : proctitle=/usr/sbin/sanlock daemon 
type=PATH msg=audit(08/05/2021 12:32:33.296:108) : item=0 name=/usr/sbin/lvmlockctl inode=9277210 dev=fd:03 mode=file,555 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(08/05/2021 12:32:33.296:108) : cwd=/ 
type=SYSCALL msg=audit(08/05/2021 12:32:33.296:108) : arch=x86_64 syscall=execve success=no exit=EACCES(Permission denied) a0=0x558fdeda2ef0 a1=0x7fff9cacc0f0 a2=0x7fff9cacc8c0 a3=0x7f2f59968780 items=1 ppid=1524 pid=1663 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sanlock-helper exe=/usr/sbin/sanlock subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(08/05/2021 12:32:33.296:108) : avc:  denied  { execute } for  pid=1663 comm=sanlock-helper name=lvmlockctl dev="vda3" ino=9277210 scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0 
----
type=PROCTITLE msg=audit(08/05/2021 12:33:13.197:109) : proctitle=/usr/sbin/sanlock daemon 
type=OBJ_PID msg=audit(08/05/2021 12:33:13.197:109) : opid=1541 oauid=unset ouid=root oses=-1 obj=system_u:system_r:lvm_t:s0 ocomm=lvmlockd 
type=SYSCALL msg=audit(08/05/2021 12:33:13.197:109) : arch=x86_64 syscall=kill success=no exit=EACCES(Permission denied) a0=0x605 a1=SIGTERM a2=0x200 a3=0x323ca items=0 ppid=1523 pid=1524 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sanlock-helper exe=/usr/sbin/sanlock subj=system_u:system_r:sanlock_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(08/05/2021 12:33:13.197:109) : avc:  denied  { signal } for  pid=1524 comm=sanlock-helper scontext=system_u:system_r:sanlock_t:s0-s0:c0.c1023 tcontext=system_u:system_r:lvm_t:s0 tclass=process permissive=0 

The first one will be probably resolved with lvmlockctl labeling:

-r-xr-xr-x. 1 root root system_u:object_r:lvm_exec_t:s0 2797920 Feb 22 10:53 /usr/sbin/lvm
-r-xr-xr-x. 1 root root system_u:object_r:bin_t:s0       180744 Feb 22 10:53 /usr/sbin/lvmlockctl

I see also reference to a sanlock-helper command - is it an executable, a process name, something else?

sanlock-helper is a helper process created by the sanlock daemon process.  The helper process is created before sanlock does setgid/setuid because the helper process should remain root so that it can do SIGKILL on other pids.

I've submitted a Fedora PR to address the issue:
https://github.com/fedora-selinux/selinux-policy/pull/925

David,

Will you be able to test some custom scenarios with SELinux enabled once we have the build (probably today or tomorrow) with these changes in selinux-policy?

Particularly note these changes only address the denials we saw in the audit logs, and there was only the "signal" permission which is any signal except signull/sigkill/sigstop/sigchld which have distinct permission name, so this is currently not covered by the changes:

(In reply to David Teigland from comment #4)
> sanlock-helper is a helper process created by the sanlock daemon process. 
> The helper process is created before sanlock does setgid/setuid because the
> helper process should remain root so that it can do SIGKILL on other pids.

yes, I can try it.

To backport:
commit 361550e925da7d661e99dbfa93bad2743dbbc0d2
Author: Zdenek Pytela <zpytela>
Date:   Wed Oct 20 16:06:09 2021 +0200

    Support sanlock VG automated recovery on storage access loss

Installed the scratch rpm and ran the steps from bug 1932586 which worked.

# rpm -q selinux-policy
selinux-policy-3.14.3-82.el8.noarch


# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: lvmlockd lost access to locks in VG ab.
Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: Successful VG ab kill command /usr/sbin/my_vg_kill_script.sh ab

(In reply to David Teigland from comment #12)
> Installed the scratch rpm and ran the steps from bug 1932586 which worked.
> 
> # rpm -q selinux-policy
> selinux-policy-3.14.3-82.el8.noarch
> 
> 
> # cat /etc/selinux/config 
> 
> # This file controls the state of SELinux on the system.
> # SELINUX= can take one of these three values:
> #     enforcing - SELinux security policy is enforced.
> #     permissive - SELinux prints warnings instead of enforcing.
> #     disabled - No SELinux policy is loaded.
> SELINUX=enforcing
> # SELINUXTYPE= can take one of these three values:
> #     targeted - Targeted processes are protected,
> #     minimum - Modification of targeted policy. Only selected processes are
> protected. 
> #     mls - Multi Level Security protection.
> SELINUXTYPE=targeted
> 
> 
> Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: lvmlockd lost access
> to locks in VG ab.
> Oct 21 14:50:36 dct-rhel8-cluster-n1 lvmlockctl[11444]: Successful VG ab
> kill command /usr/sbin/my_vg_kill_script.sh ab

David,

Thank you for the confirmation the service works. Still, I have a question: which signals sanlock-helper need to be allowed to send to lvm? 

#c0 says "signal" which is a generic signal, in SELinux different to kill/stop/child/null
#c4 and #c12 says "sigkill"
or even both? I need to rewrite the commit so I'd like to see all necessary permissions are allowed.

sanlock-helper sends both SIGTERM and SIGKILL.

I've decided to revert the previous commit and submit new ones:
https://github.com/fedora-selinux/selinux-policy/pull/933

To backport:
commit d9d3fac0ffa9aac27dc87887137516d458104f71 (HEAD -> rawhide, upstream/rawhide)
Author: Zdenek Pytela <zpytela>
Date:   Wed Nov 3 20:59:53 2021 +0100

    Support sanlock VG automated recovery on storage access loss 2/2

commit 1e6936f6e2ce0fcfb26866d5f5d99b6e363d6113
Author: Zdenek Pytela <zpytela>
Date:   Wed Nov 3 20:56:33 2021 +0100

    Support sanlock VG automated recovery on storage access loss 1/2

and revert the previous 2 commits:
d55456395921719026dd65ab95647bdfecd56769
a0414e5e14f6dba91dff8e4583869e7842070f13

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (selinux-policy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1995