Hide Forgot
Description of problem: On a freshly deployed osp-16.1, creating a VM on a compute node leads to denials in the audit.log: type=AVC msg=audit(1594984849.260:10946): avc: denied { read write } for pid=44259 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=15431 scontext=system_u:system_r:svirt_t:s0:c470,c554 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 This is non-blocking though, not really sure what svirt is trying to do in there. Note the overcloud has been built using the standard OC images, meaning there isn't any LVM on the system This also happens when we migrate a VM from a compute to another. Version-Release number of selected component (if applicable): openstack-selinux-0.8.20-0.20200428133425.3300746.el8ost.noarch rhosp-director-images-x86_64-16.1-20200714.3.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. Deploy 16.1 with latest puddle/compose 2. Create a VM on a compute 3. Check the audit.log Actual results: We see a denial (but the VM is created) Expected results: We shouldn't see any denials Additional info: The following policy is generated using audit2allow, not sure if it's a good thing or not: module svirt-lvm 1.0; require { type svirt_t; type lvm_control_t; class chr_file { read write }; } #============= svirt_t ============== allow svirt_t lvm_control_t:chr_file { read write };
A few things (but no real root cause yet): - Given the AVC, DanPB says on IRC that "allowing write access to /dev/mapper/control sounds like a very bad idea" - So we enabled libvirt debug filters that will also print out debugging for SELinux, but there's nothing in there. I.e. this yields nothing: ()[root@oc0-compute1 libvirt]# egrep -i 'lvm|mapper' libvirtd.log ()[root@oc0-compute1 libvirt]# - Nothing related to LVM or /dev/mapper/control in `journalctl` on the relevant Compute node - But audit.log reports the same AVC: --- type=AVC msg=audit(1594987242.026:11795): avc: denied { read write } for pid=52726 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=1217 scontext=system_u:system_r:svirt_t:s0:c170,c695 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 ---
For the record, that's the full guest QEMU command-line, including libvirt and QEMU versions in use: --------------------------------------------------------------------------------- 2020-07-17 11:20:49.233+0000: starting up libvirt version: 6.0.0, package: 17.2.module+el8.2.0+6629+3fc0f2c2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-05-14-11:03:02, ), qemu version: 4.2.0qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950, kernel: 4.18.0-193.6.3.el8_2.x86_64, hostname: oc0-compute0.mydomain.tld LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-instance-00000001/.config \ QEMU_AUDIO_DRV=none \ /usr/libexec/qemu-kvm \ -name guest=instance-00000001,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes \ -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \ -cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,perfctr-core=on,amd-ssbd=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,monitor=off,svm=off \ -m 512 \ -overcommit mem-lock=off \ -smp 1,sockets=1,dies=1,cores=1,threads=1 \ -uuid 8a234ced-b36c-461d-acaa-2658992b24e5 \ -smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=20.3.1-0.20200626213433.38ee1f3.el8ost,serial=8a234ced-b36c-461d-acaa-2658992b24e5,uuid=8a234ced-b36c-461d-acaa-2658992b24e5,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=33,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/fbd4b81c3120e48c414d41e26399f983592f0f56","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/8a234ced-b36c-461d-acaa-2658992b24e5/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \ -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \ -device virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:07:a6:89,bus=pci.0,addr=0x3 \ -add-fd set=3,fd=38 \ -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \ -device isa-serial,chardev=charserial0,id=serial0 \ -device usb-tablet,id=input0,bus=usb.0,port=1 \ -vnc 172.16.13.102:0 \ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/0 (label charserial0) 2020-07-17T11:20:49.311581Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead ---------------------------------------------------------------------------------
Some more logs for the context: type=VIRT_MACHINE_ID msg=audit(1594987241.930:11779): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 vm-ctx=system_u:system_r:svirt_t:s0:c170,c695 img-ctx=system_u:object_r:svirt_image_t:s0:c170,c695 model=selinux exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_MACHINE_ID msg=audit(1594987241.930:11780): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 vm-ctx=+107:+107 img-ctx=+107:+107 model=dac exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=ANOM_PROMISCUOUS msg=audit(1594987241.980:11781): dev=tap5fda37a3-f0 prom=256 old_prom=0 auid=4294967295 uid=993 gid=1000 ses=4294967295AUID="unset" UID="openvswitch" GID="hugetlbfs" type=VIRT_RESOURCE msg=audit(1594987241.984:11782): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=net reason=open vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 net=fa:16:3e:30:64:70 path="/dev/net/tun" rdev=0A:C8 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987241.999:11783): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=net reason=open vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 net=fa:16:3e:30:64:70 path="/dev/vhost-net" rdev=0A:EE exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11784): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=deny vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=all exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11785): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=major category=pty maj=88 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11786): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/null" rdev=01:03 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11787): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/full" rdev=01:07 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11788): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/zero" rdev=01:05 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11789): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/random" rdev=01:08 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11790): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/urandom" rdev=01:09 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11791): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/ptmx" rdev=05:02 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11792): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/kvm" rdev=0A:E8 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11793): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/rtc" rdev=FB:00 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.018:11794): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=cgroup reason=allow vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 cgroup="/sys/fs/cgroup/devices/machine/qemu-1-instance-00000002.libvirt-qemu/" class=path path="/dev/hpet" rdev=0A:E4 acl=rw exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=AVC msg=audit(1594987242.026:11795): avc: denied { read write } for pid=52726 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=1217 scontext=system_u:system_r:svirt_t:s0:c170,c695 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 type=USER_ACCT msg=audit(1594987242.063:11796): pid=52735 uid=42435 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='op=PAM:accounting grantors=pam_unix acct="neutron" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="unknown(42435)" AUID="unset" type=USER_CMD msg=audit(1594987242.063:11797): pid=52735 uid=42435 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='cwd="/" cmd=6E657574726F6E2D726F6F7477726170202F6574632F6E657574726F6E2F726F6F74777261702E636F6E66206970206E65746E732065786563206F766E6D6574612D36336438326433612D636262312D343534302D623964632D6530623863653433336235332073797363746C202D77206E65742E697076342E636F6E662E616C6C2E70726F6D6F74655F7365636F6E6461726965733D31 exe="/usr/bin/sudo" terminal=? res=success'UID="unknown(42435)" AUID="unset" type=CRED_REFR msg=audit(1594987242.063:11798): pid=52735 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=USER_START msg=audit(1594987242.063:11799): pid=52735 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=USER_END msg=audit(1594987242.320:11800): pid=52735 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=CRED_DISP msg=audit(1594987242.320:11801): pid=52735 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.348:11802): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=disk reason=start vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 old-disk="?" new-disk="/var/lib/nova/instances/350dc719-a54b-4b58-a24b-462a20c17127/disk" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.348:11803): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=net reason=start vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 old-net="?" new-net="fa:16:3e:30:64:70" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.348:11804): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=chardev reason=start vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 old-chardev="?" new-chardev="/dev/pts/1" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.348:11805): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=mem reason=start vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 old-mem=0 new-mem=524288 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_RESOURCE msg=audit(1594987242.348:11806): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm resrc=vcpu reason=start vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 old-vcpu=0 new-vcpu=1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset" type=VIRT_CONTROL msg=audit(1594987242.348:11807): pid=52329 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:spc_t:s0 msg='virt=kvm op=start reason=booted vm="instance-00000002" uuid=350dc719-a54b-4b58-a24b-462a20c17127 vm-pid=52726 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
Created attachment 1701742 [details] Generated XML for the instance
strace of the QEMU startup shows this sequence: openat(AT_FDCWD, "/proc/thread-self/attr/sockcreate", O_RDWR|O_CLOEXEC) = 3 write(3, "system_u:system_r:svirt_t:s0:c380,c858\0", 39) = 39 close(3) = 0 close(3) = 0 openat(AT_FDCWD, "/proc/thread-self/attr/sockcreate", O_RDWR|O_CLOEXEC) = 3 close(3) = 0 close(3) = 0 stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 236), ...}) = 0 openat(AT_FDCWD, "/dev/mapper/control", O_RDWR) = 3 close(4) = 0 stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 236), ...}) = 0 close(4) = 0 stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 236), ...}) = 0 close(4) = 0 openat(AT_FDCWD, "/proc/thread-self/attr/exec", O_RDWR|O_CLOEXEC) = 4 write(4, "system_u:system_r:svirt_t:s0:c380,c858\0", 39) = 39 close(4) = 0 close(4) = 0 close(4) = 0 close(40) = 0 close(41) = 0 execve("/usr/libexec/qemu-kvm", ["/usr/libexec/qemu-kvm", "-name", "guest=instance-00000003,debug-threads=on", "-S", "-object", "secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-instance-00000003/master-key.aes", "-machine", "pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off", "-cpu", "EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,perfctr-core=on,amd-ssbd=on,virt-ssbd=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,monitor=off,svm=off", "-m", "512", "-overcommit", "mem-lock=off", "-smp", "1,sockets=1,dies=1,cores=1,threads=1", "-uuid", "a97feed7-1d75-4193-86e6-b4c1e657e03a", "-smbios", "type=1,manufacturer=Red Hat,product=OpenStack Compute,version=20.3.1-0.20200626213433.38ee1f3.el8ost,serial=a97feed7-1d75-4193-86e6-b4c1e657e03a,uuid=a97feed7-1d75-4193-86e6-b4c1e657e03a,family=Virtual Machine", "-no-user-config", "-nodefaults", "-chardev", "socket,id=charmonitor,fd=34,server,nowait", "-mon", "chardev=charmonitor,id=monitor,mode=control", "-rtc", "base=utc,driftfix=slew", "-global", "kvm-pit.lost_tick_policy=delay", "-no-hpet", "-no-shutdown", "-boot", "strict=on", "-device", "piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2", "-blockdev", "{\"driver\":\"file\",\"filename\":\"/var/lib/nova/instances/_base/40b95d117af8543436a404bf09f6690aefaebf6b\",\"node-name\":\"libvirt-2-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}", "-blockdev", "{\"node-name\":\"libvirt-2-format\",\"read-only\":true,\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"raw\",\"file\":\"libvirt-2-storage\"}", "-blockdev", "{\"driver\":\"file\",\"filename\":\"/var/lib/nova/instances/a97feed7-1d75-4193-86e6-b4c1e657e03a/disk\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}", "-blockdev", "{\"node-name\":\"libvirt-1-format\",\"read-only\":false,\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":\"libvirt-2-format\"}", "-device", "virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on", "-netdev", "tap,fd=36,id=hostnet0,vhost=on,vhostfd=37", "-device", "virtio-net-pci,rx_queue_size=512,host_mtu=1442,netdev=hostnet0,id=net0,mac=fa:16:3e:7f:db:a3,bus=pci.0,addr=0x3", "-add-fd", "set=3,fd=39", "-chardev", "pty,id=charserial0,logfile=/dev/fdset/3,logappend=on", "-device", "isa-serial,chardev=charserial0,id=serial0", "-device", "usb-tablet,id=input0,bus=usb.0,port=1", "-vnc", "172.16.13.62:1", "-device", "cirrus-vga,id=video0,bus=pci.0,addr=0x2", "-device", "virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5", "-sandbox", "on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny", "-msg", "timestamp=on"], 0x7fb30c037e20 /* 7 vars */) = 0 We can see in this trace that libvirt is opening /dev/mapper/control, and then nothing ever closes FD=3. IOW, we're leaking FD=3 into the QEMU process, and then at time of execve SELinux is reporting the AVC because we've left a read-write FD open and svirt, rightly, isn't allowing it. I can't immediately see what code in libvirt will be opening this file though
Created attachment 1701743 [details] strace log of startup
Created attachment 1701749 [details] Libvirt debug log for an instance creation
Shortly before /dev/mapper/control is opened, strace logs show this: unlink("/run/libvirt/qemu/2-instance-00000003.dev/hpet") = -1 ENOENT (No such file or directory) mknod("/run/libvirt/qemu/2-instance-00000003.dev/hpet", S_IFCHR|0600, makedev(10, 228)) = 0 lchown("/run/libvirt/qemu/2-instance-00000003.dev/hpet", 0, 0) = 0 chmod("/run/libvirt/qemu/2-instance-00000003.dev/hpet", 020600) = 0 Shortly after /dev/mapper/control is opened, strace logs show this: stat("/dev/hugepages", {st_mode=S_IFDIR|0775, st_size=0, ...}) = 0 stat("/run/libvirt/qemu/2-instance-00000003.hugepages", 0x7fb3191a9ea0) = -1 ENOENT (No such file or directory) stat("/run/libvirt/qemu", {st_mode=S_IFDIR|0755, st_size=200, ...}) = 0 mkdir("/run/libvirt/qemu/2-instance-00000003.hugepages", 0777) = 0 mount("/dev/hugepages", "/run/libvirt/qemu/2-instance-00000003.hugepages", 0x7fb323fd6d63, MS_MOVE, NULL) = 0 We can match this up with the libvirt debug logs for the instance: 2020-07-20 12:32:27.189+0000: 90277: debug : virFileMakeParentPath:3056 : path=/run/libvirt/qemu/3-instance-00000005.dev/hpet 2020-07-20 12:32:27.189+0000: 90277: debug : virFileMakePathHelper:2993 : path=/run/libvirt/qemu/3-instance-00000005.dev mode=0777 2020-07-20 12:32:27.189+0000: 90277: debug : qemuDomainCreateDeviceRecursive:14642 : Creating dev /dev/hpet 2020-07-20 12:32:27.189+0000: 90277: debug : qemuDomainSetupAllDisks:14933 : Setting up disks 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllDisks:14942 : Setup all disks 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllHostdevs:14975 : Setting up hostdevs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllHostdevs:14982 : Setup all hostdevs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllMemories:15006 : Setting up memories 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllMemories:15013 : Setup all memories 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllChardevs:15043 : Setting up chardevs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllChardevs:15051 : Setup all chardevs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllGraphics:15107 : Setting up graphics 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllGraphics:15115 : Setup all graphics 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllInputs:15141 : Setting up inputs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllInputs:15148 : Setup all inputs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllRNGs:15180 : Setting up RNGs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupAllRNGs:15188 : Setup all RNGs 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupLoader:15200 : Setting up loader 2020-07-20 12:32:27.190+0000: 90277: debug : qemuDomainSetupLoader:15224 : Setup loader 2020-07-20 12:32:27.190+0000: 90277: debug : virFileMakePathHelper:2993 : path=/run/libvirt/qemu/3-instance-00000005.hugepages mode=0777 2020-07-20 12:32:27.190+0000: 90277: debug : virFileMakePathHelper:2993 : path=/run/libvirt/qemu mode=0777 This points towards qemuDomainSetupAllDisks as being a likely cause of opening /dev/mapper/control. This method in turn calls qemuDomainSetupDisk, which calls virDevMapperGetTargets() This in turn calls into dm_task_create in libdevmapper.so In the LVM2 package source we see code in the source file device_mapper/ioctl/libdm-iface.c : #ifdef DM_IOCTLS static int _control_fd = -1; static int _hold_control_fd_open = 0; ...snip... static int _open_and_assign_control_fd(const char *control) { if ((_control_fd = open(control, O_RDWR)) < 0) { log_sys_error("open", control); return 0; } return 1; } #endif this matches our strace log. I think there are multiple bugs here: - device-mapper should be setting O_CLOEXEC here to prevent its global FD being left open in child process. This is a potentially very serious flaw as a FD for /dev/mapper/control gives significant privileges over the host and children should never get this - libvirt is not calling dm_lib_release() so it doesn't clean up its global state. This would workaround the lack of use of O_CLOEXEC, but device-mapper should be fixed regardless. It simply isn't acceptable to open FDs without O_CLOEXEC in a multi-threaded application in the modern world. - libvirt should actually not be using libdevice-mapper.so at all in this context. We're running in between fork & exec, which means we're only permitted to use async signal safe functions. Libvirt uses malloc because we know our platforms make malloc safe, but anything else not declared async signal safe is still 100% off limits. I see zero evidence that libdevice-mapper is written to be async signal safe. Purely by luck we're not triggering a function that will cause a deadlock, or maybe we are but havent hit the deadlock race yet.
Hi Daniel, It seems the same issue with Bug 1822522 - avc: denied { read write } for pid=3449 comm="qemu-kvm" path="/dev/mapper/control". And there is patch for the issue: https://bugzilla.redhat.com/show_bug.cgi?id=1822522#c6
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2020-July/msg01500.html
I've merged patches upstream: e450ebb4c6 virDevMapperGetTargets: Don't ignore EBADF 2249455654 virdevmapper: Don't use libdevmapper to obtain dependencies b8ebbe0545 virDevMapperGetTargetsImpl: Use VIR_AUTOSTRINGLIST ae5752aabc virdevmapper.c: Join two WITH_DEVMAPPER sections together v6.6.0-rc1-4-ge450ebb4c6
*** Bug 1835909 has been marked as a duplicate of this bug. ***
*** Bug 1822522 has been marked as a duplicate of this bug. ***
Reproduced with libvirt-daemon-6.0.0-17.3.el8.x86_64. 1.Start a guest without multipath disk: #virsh start rhel8.3 Domain rhel8.3 started # virsh domblklist rhel8.3 Target Source ------------------------------------------------------------------ vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2.1 2.Check the audit log: # ausearch -m avc | grep -i control | grep -i mapper type=AVC msg=audit(1600045610.463:8082): avc: denied { read write } for pid=14416 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=24622 scontext=system_u:system_r:svirt_t:s0:c380,c661 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 Verified with libvirt-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64. 1.Start a guest without multipath disk: #virsh start rhel8.3 Domain rhel8.3 started # virsh domblklist rhel8.3 Target Source ------------------------------------------------------------------ vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2.1 2.Check the audit log: #ausearch -m avc | grep -i control | grep -i mapper no output
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137