Bug 1771500 - Refcount should not accumulate when the second vm using same disk fails to start [rhel-av-8.1.0.z]
Summary: Refcount should not accumulate when the second vm using same disk fails to st...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Michal Privoznik
QA Contact: yafu
URL:
Whiteboard:
Depends On: 1740024
Blocks: 1652078 1780154
TreeView+ depends on / blocked
 
Reported: 2019-11-12 13:41 UTC by Oneata Mircea Teodor
Modified: 2020-04-29 09:35 UTC (History)
14 users (show)

Fixed In Version: libvirt-5.6.0-6.1.el8
Doc Type: Bug Fix
Doc Text:
Cause: When starting up a guest libvirt needs to prepare all the files that qemu is going to touch. This involves setting up security labels, i.e. DAC labels (aka user:group owners), and SELinux labels so that qemu can access desired files. At the same time, when 'remember_owner' is set in qemu.conf (or if commented out it's on by default), libvirt uses extended attributes (aka XATTRs) to record original labels, so that they can be restored back when qemu shuts down. This comes handy when the file is in user's $HOME (e.g. installation ISO) - because once qemu no longer uses it, the file has original owner/SELinux label and thus ordinary user doesn't lose access to it. However, the XATTRs are a bit tricky - each security driver (= piece of code that handles only DAC, or only SELinux) uses its own namespace (so that they are independent) and also has their own reference counter (the reference counter reflects how many times given file is in use). This means that each file has two reference counters - one for DAC and one for SELinux. Ideally, these are in sync - the file has both labels after all. Anyway, libvirt does perform a rollback, that is - a list of [path; desired DAC label] pairs is gathered and then set at once. The same applies for SELinux labels (this means we have two independent lists). If something goes wrong while processing the list, a rollback is performed => the original owner is restored. However, libvirt did not perform rollback on DAC list when applying SELinux list failed. Another way to look at this (leaving out 'remember_owner' feature completely) is: when setting up labels DAC labels are set first (i.e. chown) and only after that SELinux labels are set (i.e. chcon). But, if setting up DAC labels succeeded and setting up SELinux failed, the files were left with DAC labels changed which creates a vulnerability. Consequence: If setting up SELinux labels failed at some point, the XATTR refcounter for DAC would increase (if remember_owner is enabled, which is default), and/or all files configured for the domain (disks, sockets, ...) are left with the owner that domain was configured to run under (if remember_owner is disabled). Fix: The fix consists of performing a rollback on DAC driver when SELinux driver fails. Result: The XATTR refcounter for DAC no longer increases and files keep their original owner.
Clone Of: 1740024
: 1780154 (view as bug list)
Environment:
Last Closed: 2020-02-17 05:25:10 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0510 0 None None None 2020-02-17 05:25:39 UTC

Comment 7 yafu 2019-11-29 07:01:22 UTC
Verified with libvirt-5.6.0-6.1.module+el8.1.0+4754+8d38b36b.x86_64.

Test steps:
# virsh start vm1
Domain vm1 started

# getfattr -m trusted.libvirt.security -d /var/lib/libvirt/images/78.qcow2 
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/images/78.qcow2
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.ref_selinux="1"
trusted.libvirt.security.selinux="unconfined_u:object_r:virt_image_t:s0"
trusted.libvirt.security.timestamp_dac="1573809643"
trusted.libvirt.security.timestamp_selinux="1573809643"

# virsh start vm2
error: Failed to start domain vm2
error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /var/lib/libvirt/images/78.qcow2 which is already in use

#getfattr -m trusted.libvirt.security -d /var/lib/libvirt/images/78.qcow2 
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/images/78.qcow2
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.ref_selinux="1"
trusted.libvirt.security.selinux="unconfined_u:object_r:virt_image_t:s0"
trusted.libvirt.security.timestamp_dac="1573809643"
trusted.libvirt.security.timestamp_selinux="1573809643"

# virsh destroy vm1
Domain vm1 destroyed

#getfattr -m trusted.libvirt.security -d /var/lib/libvirt/images/78.qcow2
no output

#virsh start vm1
Domain vm1 started

Also test disk hotplug, and other devices, such as: memory, console/serial device, input device, etc.
For channel device, there is an existing bug:
Bug 1754392 - Xattr for “guestfwd channel” device will not be cleaned after VM was destroyed or device was detached; thus Other VM can not use i

Comment 9 yafu 2019-12-04 03:22:04 UTC
Hi  Michal,

The xattr was cleared when trying to start the second vm using same disk with libvirt-5.6.0-6.2.x86_64. And it works well with libvirt-5.6.0-6.1.x86_64.
Could you help to check the issue please?
Thanks.

Test steps:
1.# virsh start vm1
Domain vm1 started

2.# getfattr -m trusted.libvirt.security -d /var/lib/libvirt/images/test.qcow2
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/images/test.qcow2
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.ref_selinux="1"
trusted.libvirt.security.selinux="system_u:object_r:virt_image_t:s0"
trusted.libvirt.security.timestamp_dac="1574640796"
trusted.libvirt.security.timestamp_selinux="1574640796"

3.Start another guest using /var/lib/libvirt/images/test.qcow2:
# virsh start vm2
error: Failed to start domain vm2
error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /var/lib/libvirt/images/test.qcow2 which is already in use

4.The xattr was cleared after step3:
# getfattr -m trusted.libvirt.security -d /var/lib/libvirt/images/test.qcow2
no output

Comment 10 Michal Privoznik 2019-12-04 16:42:14 UTC
Yes, this is a bug in the patch I've pushed. Proposing the fix here:

https://www.redhat.com/archives/libvir-list/2019-December/msg00246.html

Comment 11 Michal Privoznik 2019-12-05 14:03:33 UTC
Actually, I've talked to lmen and we agreed that the fix from comment 10 will be in a separate bug. So I'm moving this bug over to QA and will clone this one shortly.

Comment 12 yafu 2019-12-06 01:35:03 UTC
According to comment9-11, move the bug to verified.

Comment 14 errata-xmlrpc 2020-02-17 05:25:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0510


Note You need to log in before you can comment on or make changes to this bug.