Description of problem: Libvirt fails to restore a VM the state of which was previously saved to a block device when namespaces are enabled (which is the default) producing the following SELinux AVC for the QEMU process trying to read from the block device: type=AVC msg=audit(1573810056.889:155): avc: denied { read } for pid=2291 comm="qemu-kvm" path="/dev/vdb" dev="devtmpfs" ino=12586 scontext=system_u:system_r:svirt_t:s0:c238,c315 tcontext=system_u:object_r:svirt_image_t:s0:c315,c352 tclass=blk_file permissive=0 Version-Release number of selected component (if applicable): libvirt.x86_64 5.6.0-5.module+el8.1.0+4229+2e4e348c qemu-kvm.x86_64 15:4.1.0-10.module+el8.1.0+4234+33aa4f57 How reproducible: 100% Steps to Reproduce: 1. Add the block device to the cgroup_device_acl list in qemu.conf Note: See the BZ this bug depends on to understand why the block device needs to go into the cgroup ACL list until the former is addressed. 2. Save a running domain $ virsh save <domain> /dev/<disk> Domain alpine saved to /dev/<disk> $ ls -lZ /dev/<disk> brw-rw----. 1 root disk system_u:object_r:svirt_image_t:s0:c315,c352 253 3. Try to restore the domain from the block device $ virsh restore /dev/<disk> error: Failed to restore domain from /dev/<disk> error: internal error: qemu unexpectedly closed the monitor: 2019-11-15T09:27:25.621428 Z qemu-kvm: Not a migration stream 2019-11-15T09:27:25.621446Z qemu-kvm: load of migration failed: Invalid argument Note: The error QEMU produces is confusing in this case, but to give you heads up, it's produced in <qemu.git>/migration/savevm.v:qemu_loadvm_state() when QEMU tries to read the header from the block device to check for the magic it expects to find there but is unable to read it. 4. Check the audit.log, an AVC should have been produced (see above for the AVC content) 5. Put selinux into permissive mode $ setenforce 0 6. Re-try the restore $ virsh restore /dev/<disk> Domain restored from /dev/vdb $ ls -lZ /dev/<disk> brw-rw----. 1 root root system_u:object_r:svirt_image_t:s0:c315,c352 253 Actual results: Expected results: Libvirt succeeds in restoring a VM from a block device Additional info: If the same sequence of operations is performed with namespaces disabled (put namespaces = [] in qemu.conf), everything works as expected, no AVC is produced, so this is related only to namespaces.
Ah, so I know what the problem is. When resuming a domain, libvirt opens the file (/dev/vdb in this case) and keeps the FD so that it can be passed later to qemu. Then, when namespaces are enabled, private /dev is set up (currently withouth /dev/vdb - bug 1772439 but even if I patch it this bug still demonstrates itself), and since the domain uses namespaces the /dev/vdb inside the private namespace is relabeled. But the FD from the parent namespace is passed to qemu and the /dev/vdb there is not relabeled. Therefore, the moment qemu tries to read from the FD it gets an error. Let me see if I can come up with a clever way to fix this.
Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-April/msg00185.html
*** Bug 1772439 has been marked as a duplicate of this bug. ***
I've just pushed the fix upstream: 28fdfd20f2 qemu: Label restore path outside of secdriver transactions 55cbb94e2e security: Introduce virSecurityManagerDomainSetPathLabelRO v6.2.0-172-g28fdfd20f2
Hi, Michal Could you help with 2 questions below please? Thank you in advance! 1. When I try to reproduce this bug, I encountered different behaviours. Could you help check whether they can be regarded as same issue with comment0? If not, any further steps can help reproduce comment0? (1).On rhel8.2: libvirt-daemon-6.0.0-24.module+el8.2.1+6997+c666f621.x86_64 qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514.x86_64 # virsh save avocado-vt-vm1 /dev/sdc Domain avocado-vt-vm1 saved to /dev/sdc # ls -lZ /dev/sdc brw-rw----. 1 root disk system_u:object_r:svirt_image_t:s0:c345,c797 8, 32 Jun 4 06:43 /dev/sdc # virsh restore /dev/sdc error: Failed to restore domain from /dev/sdc error: internal error: child reported (status=125): unable to set security context 'system_u:object_r:virt_content_t:s0' on '/dev/sdc': No such file or directory # grep avc -i /var/log/audit/audit.log (no output) # setenforce 0 # virsh restore /dev/sdc Domain restored from /dev/sdc Disable namespaces in qemu.conf and set enforcing can also restore it successfully. (2). On rhel8.1: libvirt-daemon-5.6.0-5.module+el8.1.0+4229+2e4e348c.x86_64 qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a.x86_64 # virsh restore /dev/sdb error: Failed to restore domain from /dev/sdb error: internal error: child reported (status=125): unable to get SELinux context of /dev/sdb: No such file or directory # grep avc -i /var/log/audit/audit.log (no output) # setenforce 0 # virsh restore /dev/sdb error: Failed to restore domain from /dev/sdb error: internal error: child reported (status=125): unable to get SELinux context of /dev/sdb: No such file or directory # vim /etc/libvirt/qemu.conf namespaces = [] # systemctl restart libvirtd # getenforce Enforcing # virsh restore /dev/sdb Domain restored from /dev/sdb 2. When verify on rhel8.3, an extra avc denied occurs when namespaces is enabled even though restore succeeds. Is it an issue? libvirt-daemon-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64 qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64 # grep namespace /etc/libvirt/qemu.conf ...#namespaces = [ "mount" ] # getenforce Enforcing # virsh save avocado-vt-vm1 /dev/sdb Domain avocado-vt-vm1 saved to /dev/sdb # ls -lZ /dev/sdb brw-rw----. 1 root disk system_u:object_r:svirt_image_t:s0:c206,c683 8, 16 Jun 16 23:27 /dev/sdb # virsh restore /dev/sdb Domain restored from /dev/sdb # grep avc -i /var/log/audit/audit.log type=AVC msg=audit(1592364498.484:6305): avc: denied { read write } for pid=805908 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=14717 scontext=system_u:system_r:svirt_t:s0:c646,c806 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 # setenforce 0 # virsh restore /dev/sdb Domain restored from /dev/sdb # grep avc -i /var/log/audit/audit.log type=AVC msg=audit(1592365656.113:6338): avc: denied { read write } for pid=806210 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" ino=14717 scontext=system_u:system_r:svirt_t:s0:c515,c907 tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=1 # vim /etc/libvirt/qemu.conf namespaces = [] # systemctl restart libvirtd # getenforce Enforcing # virsh restore /dev/sdb Domain restored from /dev/sdb # grep avc -i /var/log/audit/audit.log (no output)
(In reply to yanqzhan from comment #8) > Hi, Michal > Could you help with 2 questions below please? Thank you in advance! > 1. When I try to reproduce this bug, I encountered different behaviours. > Could you help check whether they can be regarded as same issue with > comment0? If not, any further steps can help reproduce comment0? > > (1).On rhel8.2: > libvirt-daemon-6.0.0-24.module+el8.2.1+6997+c666f621.x86_64 > qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514.x86_64 > > # virsh save avocado-vt-vm1 /dev/sdc > Domain avocado-vt-vm1 saved to /dev/sdc > > # ls -lZ /dev/sdc > brw-rw----. 1 root disk system_u:object_r:svirt_image_t:s0:c345,c797 8, 32 > Jun 4 06:43 /dev/sdc > > # virsh restore /dev/sdc > error: Failed to restore domain from /dev/sdc > error: internal error: child reported (status=125): unable to set security > context 'system_u:object_r:virt_content_t:s0' on '/dev/sdc': No such file or > directory > > # grep avc -i /var/log/audit/audit.log > (no output) > > # setenforce 0 > # virsh restore /dev/sdc > Domain restored from /dev/sdc > > Disable namespaces in qemu.conf and set enforcing can also restore it > successfully. Right. This is the bug that we are fixing here, even though it may not look like it. The thing is, when setting a seclabel libvirt has two ways ot doing that - if namespaces are enabled then it enters the namespace and sets the seclables. If the namespaces are disabled then it sets the seclabels directly. What you are seeing is just another way this bug demonstrates itself. The fix from comment 4 makes libvirt ignore namespaces on 'virsh restore'. This is because the block device the domain is restoring from (e.g. /dev/sdc in this case) is opened before domain is started. The domain XML is read from the device (the save file has some structure), and the corresponding FD is then seeked to the start of qemu migration data. The FD is then passed to QEMU to restore guest from. Anyway, the point is that the device is opened out of namespaces (there is no namespace anyway - the domain is not running yet). And therefore, when giving the FD to QEMU the seclabel must be set on the device that is outside of the namespace - that is the file that QEMU reads data from. > > (2). On rhel8.1: > libvirt-daemon-5.6.0-5.module+el8.1.0+4229+2e4e348c.x86_64 > qemu-kvm-4.1.0-9.module+el8.1.0+4210+23b2046a.x86_64 > > # virsh restore /dev/sdb > error: Failed to restore domain from /dev/sdb > error: internal error: child reported (status=125): unable to get SELinux > context of /dev/sdb: No such file or directory > > # grep avc -i /var/log/audit/audit.log > (no output) > > # setenforce 0 > # virsh restore /dev/sdb > error: Failed to restore domain from /dev/sdb > error: internal error: child reported (status=125): unable to get SELinux > context of /dev/sdb: No such file or directory > > # vim /etc/libvirt/qemu.conf > namespaces = [] > # systemctl restart libvirtd > # getenforce > Enforcing > # virsh restore /dev/sdb > Domain restored from /dev/sdb And this is the same bug. > > > 2. When verify on rhel8.3, an extra avc denied occurs when namespaces is > enabled even though restore succeeds. Is it an issue? > libvirt-daemon-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64 > qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64 > > # grep namespace /etc/libvirt/qemu.conf > ...#namespaces = [ "mount" ] > # getenforce > Enforcing > # virsh save avocado-vt-vm1 /dev/sdb > Domain avocado-vt-vm1 saved to /dev/sdb > > # ls -lZ /dev/sdb > brw-rw----. 1 root disk system_u:object_r:svirt_image_t:s0:c206,c683 8, 16 > Jun 16 23:27 /dev/sdb > > # virsh restore /dev/sdb > Domain restored from /dev/sdb > > # grep avc -i /var/log/audit/audit.log > type=AVC msg=audit(1592364498.484:6305): avc: denied { read write } for > pid=805908 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" > ino=14717 scontext=system_u:system_r:svirt_t:s0:c646,c806 > tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=0 > > # setenforce 0 > # virsh restore /dev/sdb > Domain restored from /dev/sdb > > # grep avc -i /var/log/audit/audit.log > type=AVC msg=audit(1592365656.113:6338): avc: denied { read write } for > pid=806210 comm="qemu-kvm" path="/dev/mapper/control" dev="devtmpfs" > ino=14717 scontext=system_u:system_r:svirt_t:s0:c515,c907 > tcontext=system_u:object_r:lvm_control_t:s0 tclass=chr_file permissive=1 > > # vim /etc/libvirt/qemu.conf > namespaces = [] > # systemctl restart libvirtd > > # getenforce > Enforcing > > # virsh restore /dev/sdb > Domain restored from /dev/sdb > > # grep avc -i /var/log/audit/audit.log > (no output) I've met this issue too, but unfortunately I have no idea what might be causing it. It is tracked in bug 1822522.
Thanks Michal's detailed reply. Then mark this bug as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137