Bug 1803551

Summary: Active layer commit broken on block storage (internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /path... which is already in use
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Nir Soffer <nsoffer>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: yisun
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.2CC: jdenemar, jiyan, jsuchane, lmen, pkrempa, virt-maint, vjuranek, xuzhang, yafu
Target Milestone: rcKeywords: Regression, Upstream
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-6.0.0-10.el8 Doc Type: Bug Fix
Doc Text:
Cause: When starting a virtual machine or changing security labels on files in general, libvirt records the original owner of the file in XATTRs. When restoring the label back (e.g. on domain shutdown), the XATTRs are used to recall the original owner (instead of defaulting to root:root). This also means that when a file has XATTRs set, it is in use by some domain. To avoid changing the seclabel (and cutting the qemu off), libvirt mandates the same label to be used. For instance, when two domains want to use the same disk at once (e.g. an installation medium), they have to use the same label to avoid disabling access to each other. But, when it comes to backing chains, libvirt remembers the owner only for the topmost parent of the chain. The rest has no XATTRs set. However, when doing block commit, our security drivers ignored this fact and set XATTRs on all images they changed seclabels on. Worse, when finishing the job and restoring the labels, XATTRs were not cleared out. Consequence: XATTRs were set even if they shouldn't have been - resulting in libvirt thinking an image is already in use, even when it clearly was not. Fix: Libvirt tells security code what images are the topmost images and thus effectively suppresses remembering for images below. Result: Blockcommit works again. Yay!
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-05 09:57:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1788185    
Attachments:
Description Flags
logs
none
pure libvirt reproducer script none

Description Nir Soffer 2020-02-16 21:37:08 UTC 
Created attachment 1663375 [details]
logs

Description of problem:

Active layer commit is broken after doing internal commit on block storage.
Libvirt seems to try to access RHV block devices under /dev/vgname/lvname
when it should use block devices in private namespace.

Live merge of top volume in RHV fails with:

2020-02-15 18:15:51,592-0500 ERROR (jsonrpc/4) [virt.vm] (vmId='54b2e4ab-1f9b-440d-8bb3-9896635ac388') Live merge failed (job: bdbfbdb1-1fce-421a-8f6a-e1e68a8b0bb4) (vm:5344
)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5342, in merge
    bandwidth, flags)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 823, in blockCommit
    if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
libvirt.libvirtError: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164 which is already in use


Version-Release number of selected component (if applicable):
# rpm -q libvirt-daemon
libvirt-daemon-6.0.0-4.module+el8.2.0+5642+838f3513.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create VM with disk on block storage
2. Create snapshot 1
3. Create snapshot 2
4. Delete snapshot 1 (internal merge, COMMIT block job) - works
5. Delete snapshot 2 (leaf merge, ACTIVE_COMMIT job) - fails

Actual results:
Live merge fails with error about modifying selinux label on /rhev/...
which is a symlink to /dev/vgname/lvname.

Expected results:
Libvirt should should not access block devices under /rhev/... since it
creates nodes in private namespaces for the vm.


Additional info:

This is the image chain before step 4 (internal merge):

# qemu-img info --backing-chain /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/2e0606c4-6174-4c7a-97b8-470002398983
image: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/2e0606c4-6174-4c7a-97b8-470002398983
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
backing file: cbd57db8-fb61-4317-b3d0-87dc5111e6a3 (actual path: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/cbd57db8-fb61-4317-b3d0-87dc5111e6a3)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/cbd57db8-fb61-4317-b3d0-87dc5111e6a3
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
backing file: 4093e21a-73f7-451a-90d1-2b8d41685164 (actual path: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164)
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false


These are the actual block devices:


# ls -lhZ `realpath /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/2e0606c4-6174-4c7a-97b8-470002398983`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 30 Feb 15 18:04 /dev/dm-30

# ls -lhZ `realpath /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/cbd57db8-fb61-4317-b3d0-87dc5111e6a3`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 29 Feb 15 18:04 /dev/dm-29

# ls -lhZ `realpath /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 15 18:04 /dev/dm-28


After deleting snap1 (commit cbd57db8-fb61-4317-b3d0-87dc5111e6a3 into 4093e21a-73f7-451a-90d1-2b8d41685164):

# ls -lhZ `realpath /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/2e0606c4-6174-4c7a-97b8-470002398983`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 30 Feb 15 18:04 /dev/dm-30

# ls -lhZ `realpath /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 15 18:12 /dev/dm-28


This is the VM XML before the step 3:


    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/2e0606c4-6174-4c7a-97b8-470002398983' index='6'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore type='block' index='5'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/cbd57db8-fb61-4317-b3d0-87dc5111e6a3'>
          <seclabel model='dac' relabel='no'/>
        </source>
        <backingStore type='block' index='1'>
          <format type='qcow2'/>
          <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164'>
            <seclabel model='dac' relabel='no'/>
          </source>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>


In libvirtd log we see:


2020-02-15 23:15:51.579+0000: 18431: debug : virDomainBlockCommit:10517 : dom=0x7f6ab4055e40, (VM: name=lsm-block, uuid=54b2e4ab-1f9b-440d-8bb3-9896635ac388), disk=vda, base=vda[1], top=<null>, bandwidth=0, flags=0xc
2020-02-15 23:15:51.579+0000: 18431: debug : qemuDomainObjBeginJobInternal:9460 : Starting job: job=modify agentJob=none asyncJob=none (vm=0x7f6ab4004a80 name=lsm-block, current job=none agentJob=none async=none)
2020-02-15 23:15:51.579+0000: 18431: debug : qemuDomainObjBeginJobInternal:9509 : Started job: modify (async=none vm=0x7f6ab4004a80 name=lsm-block)
2020-02-15 23:15:51.579+0000: 18431: debug : virStorageFileChainLookup:1680 : 1: /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164
2020-02-15 23:15:51.579+0000: 18431: debug : qemuDomainStorageSourceAccessModify:11765 : src='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164' readonly=1 force_ro=0 force_rw=1 revoke=0 chain=0
2020-02-15 23:15:51.581+0000: 18431: debug : virFileClose:110 : Closed fd 39
2020-02-15 23:15:51.584+0000: 18431: error : virProcessRunInFork:1161 : internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164 which is already in use


Note - this works:

1. Create VM with disk on block storage
2. Create snapshot 1
3. Delete snapshot 1 (ACTIVE_COMMIT)


This is a regression this flow used to work in RHEL 7.7.
Comment 1 Nir Soffer 2020-02-16 21:54:39 UTC 
Workaround in RHV side:

1. Stop the VM
2. Start the VM
3. Delete the snapshot (ACTIVE_COMMIT block job)

So the affect on RHV customers is only short downtime when restarting the VM.
Comment 2 Nir Soffer 2020-02-17 12:48:46 UTC 
Adding more info on this flow using nsenter to inspect the vm mount namespace.

I don't see anything that explaining the issue but the nodes created during
snapshots look suspicious.


Starting the VM
---------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164' index='1'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>

Host namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 06:52 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164`
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 06:52 /dev/dm-28

VM mount namspace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 06:52 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164`
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 28 Feb 17 06:59 /dev/dm-28


Creating snapshot 1
-------------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790' index='3'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164'>
          <seclabel model='dac' relabel='no'/>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>

Host namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:02 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:02 /dev/91630622-c645-4397-a9fe-9ddf26690500/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790 -> ../dm-29

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 07:02 /dev/dm-28
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 29 Feb 17 07:02 /dev/dm-29

VM mount namspace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0  8 Feb 17 06:52 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 50 Feb 17 07:02 /dev/91630622-c645-4397-a9fe-9ddf26690500/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790 -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-29


# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 28 Feb 17 07:01 /dev/dm-28
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 29 Feb 17 07:02 /dev/dm-29

Issues:

The new node looks wrong:

    -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-29

Why not:

    -> ../dm-29

selinux lable looks wrong:

    system_u:object_r:svirt_image_t:s0:c839,c863

Shouldn't it be:

    system_u:object_r:virt_content_t:s0


Creating snapshot 2
-------------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/ecc9776b-baee-4815-b5ba-ebb31796a01d' index='4'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore type='block' index='3'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790'>
          <seclabel model='dac' relabel='no'/>
        </source>
        <backingStore type='block' index='1'>
          <format type='qcow2'/>
          <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164'>
            <seclabel model='dac' relabel='no'/>
          </source>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>

Host namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790 -> ../dm-29
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> ../dm-30

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 07:09 /dev/dm-28
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 29 Feb 17 07:09 /dev/dm-29
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 30 Feb 17 07:09 /dev/dm-30

VM mount namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0  8 Feb 17 06:52 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 50 Feb 17 07:02 /dev/91630622-c645-4397-a9fe-9ddf26690500/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790 -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-29
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 50 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-30

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 28 Feb 17 07:01 /dev/dm-28
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 29 Feb 17 07:09 /dev/dm-29
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 30 Feb 17 07:12 /dev/dm-30

Issues:

Same as for snapshot 1.


Deleting snapshot 1
-------------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/ecc9776b-baee-4815-b5ba-ebb31796a01d' index='4'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164'>
          <seclabel model='dac' relabel='no'/>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>


Host namspace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:14 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> ../dm-30

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 07:14 /dev/dm-28
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 30 Feb 17 07:09 /dev/dm-30

VM mount namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0  8 Feb 17 06:52 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 50 Feb 17 07:02 /dev/91630622-c645-4397-a9fe-9ddf26690500/d49e84b0-0925-4f9b-b8c9-c3fc7d0b9790 -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-29
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 50 Feb 17 07:09 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> /dev/91630622-c645-4397-a9fe-9ddf26690500/../dm-30

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu system_u:object_r:virt_content_t:s0          253, 28 Feb 17 07:13 /dev/dm-28
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 29 Feb 17 07:09 /dev/dm-29
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c839,c863 253, 30 Feb 17 07:17 /dev/dm-30


Deleting snapshot 2
-------------------

Operation failed with:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5342, in merge
    bandwidth, flags)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 823, in blockCommit
    if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
libvirt.libvirtError: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164 which is already in use


After stopping and starting the VM
----------------------------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/ecc9776b-baee-4815-b5ba-ebb31796a01d' index='1'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore type='block' index='3'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164'>
          <seclabel model='dac' relabel='no'/>
        </source>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>


Host namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> ../dm-29

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 07:25 /dev/dm-28
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 29 Feb 17 07:25 /dev/dm-29

VM mount namspace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> ../dm-29

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu system_u:object_r:virt_content_t:s0          253, 28 Feb 17 07:25 /dev/dm-28
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c314,c638 253, 29 Feb 17 07:26 /dev/dm-29


Deleting snapshot 2
-------------------

VM XML:

    <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/91630622-c645-4397-a9fe-9ddf26690500/images/9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6/4093e21a-73f7-451a-90d1-2b8d41685164' index='3'>
        <seclabel model='dac' relabel='no'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6</serial>
      <boot order='1'/>
      <alias name='ua-9f36c5ff-2ed1-4d1a-a7ad-365e5e1fb7b6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>


Host namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:29 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu    system_u:object_r:fixed_disk_device_t:s0 253, 28 Feb 17 07:29 /dev/dm-28

VM mount namespace:

# ls -lhZ /dev/91630622-c645-4397-a9fe-9ddf26690500/*
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/4093e21a-73f7-451a-90d1-2b8d41685164 -> ../dm-28
lrwxrwxrwx. 1 root root system_u:object_r:device_t:s0 8 Feb 17 07:25 /dev/91630622-c645-4397-a9fe-9ddf26690500/ecc9776b-baee-4815-b5ba-ebb31796a01d -> ../dm-29

# ls -lhZ `realpath /dev/91630622-c645-4397-a9fe-9ddf26690500/*`
brw-rw----. 1 vdsm qemu system_u:object_r:svirt_image_t:s0:c314,c638 253, 28 Feb 17 07:34 /dev/dm-28
brw-rw----. 1 vdsm qemu system_u:object_r:fixed_disk_device_t:s0     253, 29 Feb 17 07:29 /dev/dm-29
Comment 4 Peter Krempa 2020-02-26 14:58:48 UTC 
Created attachment 1665991 [details]
pure libvirt reproducer script

I've attached a reproducer script with libvirt-only interactions.

Don't forget to fix the 'VG' variable to an existing LVM volume group and the rhv.xml file to set the <emulator> element to qemu-4.2 to use blockdev.

Note that the script will not clean up the logical volumes created.

Tested with permissive selinux only.
Comment 5 Michal Privoznik 2020-02-27 12:10:30 UTC 
Thanks Peter for the script! It helped really. I've posted patches upstream:

https://www.redhat.com/archives/libvir-list/2020-February/msg01113.html
Comment 7 Michal Privoznik 2020-03-09 08:31:48 UTC 
v2:

https://www.redhat.com/archives/libvir-list/2020-March/msg00218.html
Comment 8 Michal Privoznik 2020-03-09 13:18:46 UTC 
And I've pushed the fix upstream:

13eb6c1468 (HEAD, origin/master, origin/HEAD, qemu_domain_storage_source_access_allow_v2, master) qemu: Tell secdrivers which images are top parent
62f3d8adbc security: Introduce VIR_SECURITY_DOMAIN_IMAGE_PARENT_CHAIN_TOP flag

v6.1.0-60-g13eb6c1468
Comment 13 yisun 2020-03-31 09:29:31 UTC 
Tested on libvirt-6.0.0-14.module+el8.2.0+6069+78a1cb09.x86_64
And result is PASS
Thx Peter for the reproduce script. Add a line "virsh dumpxml $VM | awk '/<disk/,/<\/disk/'" to print disk xml before blockcommit, so the test result as follow:

1. prepare a vg
[root@dell-per740xd-11 bugs]# vgdisplay
  --- Volume group ---
  VG Name               vg_test
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  22
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               92.00 MiB
  PE Size               4.00 MiB
  Total PE              23
  Alloc PE / Size       15 / 60.00 MiB
  Free  PE / Size       8 / 32.00 MiB
  VG UUID               88kmRI-ZvTY-O1E1-GjUX-ACY3-y3Fg-Zvb4Zb

2. prepare the vm xml "rhv.xml" and test script "test.sh"
3. run the test script
[root@dell-per740xd-11 bugs]# sh test.sh
+ VG=vg_test
+ VM=rhv
+ virsh destroy rhv
error: failed to get domain 'rhv'

+ lvcreate -n test-base -L 20M vg_test
  Logical volume "test-base" created.
+ lvcreate -n test-snap1 -L 20M vg_test
  Logical volume "test-snap1" created.
+ lvcreate -n test-snap2 -L 20M vg_test
  Logical volume "test-snap2" created.
+ qemu-img create -f qcow2 /dev/vg_test/test-base 10M
Formatting '/dev/vg_test/test-base', fmt=qcow2 size=10485760 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ qemu-img create -f qcow2 -F qcow2 -b /dev/vg_test/test-base /dev/vg_test/test-snap1
Formatting '/dev/vg_test/test-snap1', fmt=qcow2 size=10485760 backing_file=/dev/vg_test/test-base backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ qemu-img create -f qcow2 -F qcow2 -b /dev/vg_test/test-snap1 /dev/vg_test/test-snap2
Formatting '/dev/vg_test/test-snap2', fmt=qcow2 size=10485760 backing_file=/dev/vg_test/test-snap1 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+ ln -s /dev/vg_test/test-base /tmp/base.qcow2
+ ln -s /dev/vg_test/test-snap1 /tmp/snap1
+ ln -s /dev/vg_test/test-snap2 /tmp/snap2
+ virsh define rhv.xml
Domain rhv defined from rhv.xml

+ virsh start rhv
Domain rhv started

+ virsh snapshot-create-as --disk-only --no-metadata rhv --diskspec vda,stype=block,file=/tmp/snap1 --reuse-external
Domain snapshot 1585646507 created
+ virsh snapshot-create-as --disk-only --no-metadata rhv --diskspec vda,stype=block,file=/tmp/snap2 --reuse-external
Domain snapshot 1585646507 created
+ echo current vm disk xml is:
current vm disk xml is:
+ virsh dumpxml rhv
+ awk '/<disk/,/<\/disk/'
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' copy_on_read='off'/>
      <source dev='/tmp/snap2' index='3'/>
      <backingStore type='block' index='2'>
        <format type='qcow2'/>
        <source dev='/tmp/snap1'/>
        <backingStore type='block' index='1'>
          <format type='qcow2'/>
          <source dev='/tmp/base.qcow2'/>
          <backingStore/>
        </backingStore>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
+ echo commit1 '?'
commit1 ?
+ read

+ virsh blockcommit rhv vda --top 'vda[2]' --base 'vda[1]'
Block Commit started
+ echo current vm disk xml is:
current vm disk xml is:
+ virsh dumpxml rhv
+ awk '/<disk/,/<\/disk/'
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' copy_on_read='off'/>
      <source dev='/tmp/snap2' index='3'/>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/tmp/base.qcow2'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
+ echo commit2 '?'
commit2 ?
+ read

+ virsh blockcommit rhv vda --top 'vda[3]' --base 'vda[1]' --active --wait --pivot
Successfully pivoted

Nothing wrong.
Comment 15 errata-xmlrpc 2020-05-05 09:57:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017