This is the denial interpreted:

----
type=PROCTITLE msg=audit(09/01/2022 12:02:11.629:726) : proctitle=/usr/sbin/sbd query-watchdog
type=AVC msg=audit(09/01/2022 12:02:11.629:726) : avc:  denied  { sys_ptrace } for  pid=52395 comm=sbd capability=sys_ptrace  scontext=system_u:system_r:sbd_t:s0 tcontext=system_u:system_r:sbd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(09/01/2022 12:02:11.629:726) : arch=x86_64 syscall=readlink success=no exit=EACCES(Permission denied) a0=0x7ffe7787fa40 a1=0x7ffe7787f830 a2=0xff a3=0x0 items=0 ppid=52391 pid=52395 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sbd exe=/usr/sbin/sbd subj=system_u:system_r:sbd_t:s0 key=(null)

It still misses some data which would appear when full auditing was enabled:

1) Open the /etc/audit/rules.d/audit.rules file in an editor.
2) Remove the following line if it exists:
-a task,never
3) Add the following line to the end of the file:
-w /etc/shadow -p w
4) Restart the audit daemon:
  # service auditd restart
5) Re-run your scenario.
6) Collect AVC denials:
  # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

Thank you, it is clearer now:
----
type=PROCTITLE msg=audit(09/12/2022 15:00:59.857:4015) : proctitle=/usr/sbin/sbd query-watchdog
type=PATH msg=audit(09/12/2022 15:00:59.857:4015) : item=0 name=/proc/851/fd/0 inode=21560 dev=00:05 mode=link,500 ouid=root ogid=root rdev=00:00 obj=system_u:system_r:rpcbind_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(09/12/2022 15:00:59.857:4015) : cwd=/var/lib/pcsd
type=SYSCALL msg=audit(09/12/2022 15:00:59.857:4015) : arch=x86_64 syscall=readlink success=no exit=EACCES(Permission denied) a0=0x7ffd9edd46c0 a1=0x7ffd9edd44b0 a2=0xff a3=0x0 items=1 ppid=538646 pid=538650 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=sbd exe=/usr/sbin/sbd subj=system_u:system_r:sbd_t:s0 key=(null)
type=AVC msg=audit(09/12/2022 15:00:59.857:4015) : avc:  denied  { sys_ptrace } for  pid=538650 comm=sbd capability=sys_ptrace  scontext=system_u:system_r:sbd_t:s0 tcontext=system_u:system_r:sbd_t:s0 tclass=capability permissive=0
----

Do you know which process has pid 851 or why does sbd want to access its file descriptors?

Can you check if the following permission is sufficient?

  # cat local_sbd_ptrace.cil
(allow sbd_t sbd_t (capability (sys_ptrace)))
  # semodule -i local_sbd_ptrace.cil

(In reply to Zdenek Pytela from comment #4)
> Thank you, it is clearer now:
> ----
> type=PROCTITLE msg=audit(09/12/2022 15:00:59.857:4015) :
> proctitle=/usr/sbin/sbd query-watchdog
> type=PATH msg=audit(09/12/2022 15:00:59.857:4015) : item=0
> name=/proc/851/fd/0 inode=21560 dev=00:05 mode=link,500 ouid=root ogid=root
> rdev=00:00 obj=system_u:system_r:rpcbind_t:s0 nametype=NORMAL cap_fp=none
> cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
> type=CWD msg=audit(09/12/2022 15:00:59.857:4015) : cwd=/var/lib/pcsd
> type=SYSCALL msg=audit(09/12/2022 15:00:59.857:4015) : arch=x86_64
> syscall=readlink success=no exit=EACCES(Permission denied) a0=0x7ffd9edd46c0
> a1=0x7ffd9edd44b0 a2=0xff a3=0x0 items=1 ppid=538646 pid=538650 auid=unset
> uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root
> fsgid=root tty=(none) ses=unset comm=sbd exe=/usr/sbin/sbd
> subj=system_u:system_r:sbd_t:s0 key=(null)
> type=AVC msg=audit(09/12/2022 15:00:59.857:4015) : avc:  denied  {
> sys_ptrace } for  pid=538650 comm=sbd capability=sys_ptrace 
> scontext=system_u:system_r:sbd_t:s0 tcontext=system_u:system_r:sbd_t:s0
> tclass=capability permissive=0
> ----
> 
> Do you know which process has pid 851 or why does sbd want to access its
> file descriptors?

The process with pid 851:
rpc          851  0.0  0.1  67240  5528 ?        Ss   Sep06   0:00 /usr/bin/rpcbind -w -f

I'm not sure about the sbd access to its file descriptors, Klaus will probably know more about it.


> Can you check if the following permission is sufficient?
> 
>   # cat local_sbd_ptrace.cil
> (allow sbd_t sbd_t (capability (sys_ptrace)))
>   # semodule -i local_sbd_ptrace.cil

The proposed permission seems to be sufficient, there were no new sbd avc denials after manual re-run run of the scenario.

hmm ... thought I had tested that but not with the privileges of pcsd - which I guess is
happening here - but dropping the shell privileges ... maybe did something wrong there.

Reason for scanning over the fds is to check if the device was opened by
sbdy else - respectively who - similar as lsof would do.

Guess it does actually not matter which process has pid 851 in the test above.
Good news should be that apart from the AVC denials it should fall back to previous
behavior. And on the node the pcs command had been invoked it should work without
issues as it would usually be done in an interactive root shell.

And I'm afraid there is not much I could do differently in sbd to achieve the
desired behavior. I mean besides taking back the feature.
Getting the identity of the watchdog-devices discovered would have failed
before that busy check. So in cases none of those had failed I could
skip the busy-check and thus prevent the AVC denials. In the cases the
implementation had been made for (sbdy is hogging the device) the
AVC denials would of course still be there.

(In reply to Klaus Wenninger from comment #7)
> And I'm afraid there is not much I could do differently in sbd to achieve the
> desired behavior. I mean besides taking back the feature.
> Getting the identity of the watchdog-devices discovered would have failed
> before that busy check. So in cases none of those had failed I could
> skip the busy-check and thus prevent the AVC denials. In the cases the
> implementation had been made for (sbdy is hogging the device) the
> AVC denials would of course still be there.

But iirc there was a reason why I hadn't gone that route in the first
place. But I don't remember ...

To backport:
commit 533de74a9a344542ab504915938b636698fd9838
Author: Zdenek Pytela <zpytela>
Date:   Wed Oct 5 14:12:22 2022 +0200

    Allow sbd the sys_ptrace capability

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (selinux-policy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2965