Caught in enforcing mode:
----
type=PROCTITLE msg=audit(09/16/2021 04:53:03.632:549) : proctitle=/usr/sbin/libvirtd --timeout 120 
type=PATH msg=audit(09/16/2021 04:53:03.632:549) : item=0 name=/run/libvirt/virtlockd-sock inode=1375 dev=00:19 mode=socket,600 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:virt_var_run_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(09/16/2021 04:53:03.632:549) : cwd=/ 
type=SOCKADDR msg=audit(09/16/2021 04:53:03.632:549) : saddr={ saddr_fam=local path=/run/libvirt/virtlockd-sock } 
type=SYSCALL msg=audit(09/16/2021 04:53:03.632:549) : arch=x86_64 syscall=connect success=no exit=EACCES(Permission denied) a0=0x3 a1=0x7fc28ece3ef0 a2=0x6e a3=0x1 items=1 ppid=1 pid=6987 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc-worker exe=/usr/sbin/libvirtd subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(09/16/2021 04:53:03.632:549) : avc:  denied  { connectto } for  pid=6987 comm=rpc-worker path=/run/libvirt/virtlockd-sock scontext=system_u:system_r:svirt_t:s0:c274,c531 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0 
----

# rpm -qa selinux\* \*libvirt\* \*qemu\* | sort
ipxe-roms-qemu-20200823-7.git4bd064de.el9.noarch
libvirt-client-7.6.0-2.el9.x86_64
libvirt-daemon-7.6.0-2.el9.x86_64
libvirt-daemon-driver-qemu-7.6.0-2.el9.x86_64
libvirt-libs-7.6.0-2.el9.x86_64
qemu-guest-agent-6.0.0-12.el9.x86_64
qemu-img-6.0.0-12.el9.x86_64
qemu-kvm-common-6.0.0-12.el9.x86_64
qemu-kvm-core-6.0.0-12.el9.x86_64
selinux-policy-34.1.16-1.el9_b.noarch
selinux-policy-targeted-34.1.16-1.el9_b.noarch
#

Caught in permissive mode:
----
type=PROCTITLE msg=audit(09/16/2021 04:55:44.866:559) : proctitle=/usr/sbin/libvirtd --timeout 120 
type=PATH msg=audit(09/16/2021 04:55:44.866:559) : item=0 name=/run/libvirt/virtlockd-sock inode=1375 dev=00:19 mode=socket,600 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:virt_var_run_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(09/16/2021 04:55:44.866:559) : cwd=/ 
type=SOCKADDR msg=audit(09/16/2021 04:55:44.866:559) : saddr={ saddr_fam=local path=/run/libvirt/virtlockd-sock } 
type=SYSCALL msg=audit(09/16/2021 04:55:44.866:559) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7f8020834ef0 a2=0x6e a3=0x1 items=1 ppid=1 pid=7035 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc-worker exe=/usr/sbin/libvirtd subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(09/16/2021 04:55:44.866:559) : avc:  denied  { connectto } for  pid=7035 comm=rpc-worker path=/run/libvirt/virtlockd-sock scontext=system_u:system_r:svirt_t:s0:c745,c891 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=1 
----
type=PROCTITLE msg=audit(09/16/2021 04:55:45.513:571) : proctitle=/usr/sbin/virtlockd 
type=PATH msg=audit(09/16/2021 04:55:45.513:571) : item=0 name=/var/lib/libvirt/images/vmguest.img inode=33554641 dev=fd:01 mode=file,644 ouid=qemu ogid=qemu rdev=00:00 obj=system_u:object_r:svirt_image_t:s0:c745,c891 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(09/16/2021 04:55:45.513:571) : cwd=/ 
type=SYSCALL msg=audit(09/16/2021 04:55:45.513:571) : arch=x86_64 syscall=openat success=yes exit=12 a0=0xffffff9c a1=0x561c7344dfd0 a2=O_RDWR a3=0x0 items=1 ppid=1 pid=7036 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlockd exe=/usr/sbin/virtlockd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(09/16/2021 04:55:45.513:571) : avc:  denied  { dac_override } for  pid=7036 comm=virtlockd capability=dac_override  scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability permissive=1 
type=AVC msg=audit(09/16/2021 04:55:45.513:571) : avc:  denied  { search } for  pid=7036 comm=virtlockd name=images dev="vda1" ino=33554640 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:virt_image_t:s0 tclass=dir permissive=1 
----

The vmguest machine starts successfully, even if the vmguest.img is full of zeros.

# ls -ldZ /var/lib/libvirt/images/
drwx--x--x. 2 root root system_u:object_r:virt_image_t:s0 25 Sep 16 04:53 /var/lib/libvirt/images/
# ls -lZ /var/lib/libvirt/images/
total 4
-rw-r--r--. 1 qemu qemu system_u:object_r:svirt_image_t:s0:c605,c764 0 Sep 16 04:53 vmguest.img
#

Not sure where the dac_override AVC comes from.
Ownership change from root:root to qemu:qemu did not help.

After allowing the search and connectto permissions, dac_override is still there:
----
type=PROCTITLE msg=audit(10/13/2021 06:16:19.108:948) : proctitle=/usr/sbin/virtlockd 
type=PATH msg=audit(10/13/2021 06:16:19.108:948) : item=0 name=/var/lib/libvirt/images/vmguest.img inode=6891369 dev=fd:01 mode=file,644 ouid=qemu ogid=qemu rdev=00:00 obj=system_u:object_r:svirt_image_t:s0:c38,c203 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(10/13/2021 06:16:19.108:948) : cwd=/ 
type=SYSCALL msg=audit(10/13/2021 06:16:19.108:948) : arch=x86_64 syscall=openat success=no exit=EACCES(Permission denied) a0=0xffffff9c a1=0x55d7027c6000 a2=O_RDWR a3=0x0 items=1 ppid=1 pid=9845 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlockd exe=/usr/sbin/virtlockd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(10/13/2021 06:16:19.108:948) : avc:  denied  { dac_override } for  pid=9845 comm=virtlockd capability=dac_override  scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability permissive=0

It is not clear why:
# ls -lZa /var/lib/libvirt/images/
total 1024
drwx--x--x. 2 root root system_u:object_r:virt_image_t:s0          25 Oct 13 06:12 .
drwxr-xr-x. 9 root root system_u:object_r:virt_var_lib_t:s0       106 Oct 13 06:08 ..
-rw-r--r--. 1 root root unconfined_u:object_r:virt_image_t:s0 1048576 Oct 13 06:12 vmguest.img

# ps -eo pid,ppid,euid,command,context | grep -e CONTEXT -e virtlockd
    PID    PPID  EUID COMMAND                     CONTEXT
   9845       1     0 /usr/sbin/virtlockd         system_u:system_r:virtlogd_t:s0-s0:c0.c1023

No other denial appears with dontaudit rules disabled.

capabilities(7) reads:

       CAP_DAC_OVERRIDE
              Bypass  file  read, write, and execute permission checks.  (DAC is an abbreviation
              of "discretionary access control".)

       CAP_DAC_READ_SEARCH
              * Bypass file read permission checks and directory  read  and  execute  permission
                checks;
              * invoke open_by_handle_at(2);
              * use the linkat(2) AT_EMPTY_PATH flag to create a link to a file referred to by a
                file descriptor.

Ondrej,

Would you have a hint how to understand it? virtlockd runs as root, wants to O_RDWR access a file owned by root in a directory owned by root.

The code path taken suggests that the vmguest.img file's owner uid&gid doesn't match the process' uid&gid (root:root). Also in the PATH record there is "ouid=qemu ogid=qemu". Something must have changed the owner of the file to root in between the access attempt and you listing the directory contents (or it's an entirely different file, as the timestamps suggest). In my case it's indeed owned by qemu:qemu right after `virsh create`:

drwx--x--x. 2 root root system_u:object_r:virt_image_t:s0                 25 Oct 13 09:25 .
drwxr-xr-x. 9 root root system_u:object_r:virt_var_lib_t:s0              106 Oct 13 08:30 ..
-rw-r--r--. 1 qemu qemu system_u:object_r:svirt_image_t:s0:c693,c824 1048576 Oct 13 09:25 vmguest.img

Thank you, I overlooked the ouid=qemu entries. That something surely is libvirtd:

----
type=PROCTITLE msg=audit(10/13/2021 10:21:04.548:1200) : proctitle=/usr/sbin/libvirtd --timeout 120
type=PATH msg=audit(10/13/2021 10:21:04.548:1200) : item=0 name=/var/lib/libvirt/images/vmguest2.img inode=6973371 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:svirt_image_t:s0:c311,c968 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(10/13/2021 10:21:04.548:1200) : cwd=/
type=SYSCALL msg=audit(10/13/2021 10:21:04.548:1200) : arch=x86_64 syscall=chown success=yes exit=0 a0=0x7ff904045510 a1=qemu a2=qemu a3=0x0 items=1 ppid=10381 pid=10473 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc-worker exe=/usr/sbin/libvirtd subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 key=chown
----
type=PROCTITLE msg=audit(10/13/2021 10:21:05.185:1203) : proctitle=/usr/sbin/libvirtd --timeout 120
type=PATH msg=audit(10/13/2021 10:21:05.185:1203) : item=0 name=/var/lib/libvirt/images/vmguest2.img inode=6973371 dev=fd:01 mode=file,644 ouid=qemu ogid=qemu rdev=00:00 obj=unconfined_u:object_r:virt_image_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(10/13/2021 10:21:05.185:1203) : cwd=/
type=SYSCALL msg=audit(10/13/2021 10:21:05.185:1203) : arch=x86_64 syscall=chown success=yes exit=0 a0=0x7ff90404bc90 a1=root a2=root a3=0x0 items=1 ppid=10381 pid=10488 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc-worker exe=/usr/sbin/libvirtd subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 key=chown

The question is why the ownership needs to be changed twice.

Jiri,

We came across a problem that libvirtd requires the dac_override capability when a new vm is created.
According to the audit records, it seems ownership of the image file changes twice: first from root:root to qemu:qemu and then back.
Can you confirm this observation is correct? Is this behaviour expected?

With dynamic_ownership turned on (which is the default), libvirt chowns disk
images to qemu:qemu when starting a domain and resets the ownership back to
root:root when the domain stops. The question is whether all the messages were
generated while starting a domain or while starting and stopping it. Seeing
multiple ownership changes when a domain gets started and remains running
would seem strange to me. Anyway, when a domain is not running, its disk
images should be "root:root system_u:object_r:virt_image_t:s0" while disk
images of a running domain should be owned by qemu:qemu with
system_u:object_r:svirt_image_t:s0:c*,c* label.

What I don't understand, though, is how this is related to the original report
(and a possible associated fix) of rpc-worker not being able to access
/run/libvirt/virtlockd-sock. It looks similar to bug 1792713 (which got
auto-closed in the meantime).

Michal could you please check what is going on here as you are more familiar
with the ownership handling code and virtlockd?

So virtlockd is libvirt's reimplementation of sanlock, well the good bits of it. The way it works is that when libvirt wants to start a guest, it forks off and just before exec()-ing QEMU binary the child connects to virtlockd. The connection is then intentionally leaked to QEMU. This way, the virtlockd knows when the QEMU dies and can automatically unlock all the disks QEMU had.

The chown()/setfilecon_raw() duplicity is only apparent. Libvirt needs to set seclabels before starting QEMU so that QEMU can open it. And if QEMU (or connecting to the virtlockd) fails for whatever reason, it has failed after seclabels were set. Therefore, libvirt restores the original ones (recored in XATTRs, but that's another story). So the duplicity is expected really.

What is unexpected is that the child which connects to virtlockd should run as root:root and with libvirtd's SELinux context, because setexeccon_raw(), UID/GID change, capabilities drop happens only after the child connected to virtlockd. One thing that I did not mention yet is that libvirt calls setsockcreatecon_raw() with something like "system_u:system_r:svirt_t:s0:c282,c802" (where the range changes dynamically to be unique per each guest), but honestly - I don't understand why that would be an issue - the socket already exists at that point (either because systemd created it (when using socket activation), or because it was created by virtlockd itself).

It's unclear to me what has changed, but this part of libvirt wasn't changed in ages. Maybe previously policy was allowing svirt_t to connect and now it isn't anymore? At any rate, bug 1792713 (which Jirka references above) is the same issue.

This commit:

commit 2d69f99ffb42aeebdc09f1fcda46153fbd688a68 (HEAD -> rawhide, upstream/rawhide)
Author: Nikola Knazekova <nknazeko>
Date:   Thu May 5 12:33:31 2022 +0200

    Allow svirt connectto virtlogd

    Fix bz#2000881

addresses all denials but dac_override.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (selinux-policy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8283