Bug 1972145

Summary: [snapshot-create-as] When snapshot creation failed, the snapshot's target block device will be removded
Product: Red Hat Enterprise Linux 9 Reporter: yisun
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED CURRENTRELEASE QA Contact: yisun
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.0CC: jdenemar, meili, pkrempa, virt-maint, xuzhang, yicui, yisun
Target Milestone: betaKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-7.5.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-07 21:57:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 7.5.0
Embargoed:
Attachments:
Description Flags
libvirtd-debug.log none

Description yisun 2021-06-15 10:31:28 UTC
Created attachment 1791217 [details]
libvirtd-debug.log

Description:
[snapshot-create-as] When snapshot creation failed, the snapshot's target block device will be removded

Versions:
libvirt-7.4.0-1.el9.x86_64
qemu-kvm-6.0.0-5.el9.x86_64

How reproducible:
100%

Steps:
1. prepare a running vm with a disk, here we use vdb
[root@dell-per730-61 ~]# virsh dumpxml avocado-vt-vm1 
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
...

2. make sure you have a extra block device can be used on host, here we can use a usb disk or prepare a iscsi lun
[root@dell-per730-61 ~]# lsscsi
[0:0:0:0]    disk    SanDisk  Ultra USB 3.0    1.00  /dev/sdb 

[root@dell-per730-61 ~]# ll /dev/sdb
brw-rw----. 1 root disk 8, 16 Jun 15 06:10 /dev/sdb

3. create sanpshot with stype=file to trigger an error
[root@dell-per730-61 ~]# virsh snapshot-create-as avocado-vt-vm1 s1 --disk-only --diskspec vdb,file=/dev/sdb,stype=file
error: internal error: unable to execute QEMU command 'blockdev-add': 'file' driver requires '/dev/sdb' to be a regular file

4. check the /dev/sdb, it's gone
[root@dell-per730-61 ~]# ll /dev/sdb
ls: cannot access '/dev/sdb': No such file or directory

[root@dell-per730-61 ~]# lsscsi
[0:0:0:0]    disk    SanDisk  Ultra USB 3.0    1.00  -     


Additional steps to test mem snapshot target, it's also removed when snapshot failed
5. prepare a block device sdc
[root@dell-per730-61 ~]# lsscsi
...
[0:0:0:0]    disk    SanDisk  Ultra USB 3.0    1.00  /dev/sdb 
[12:0:0:0]   disk    LIO-ORG  device.logical-  4.0   /dev/sdc 
...

6. ln /tmp/vdb.s1 to a block device to trigger the error
[root@dell-per730-61 ~]# ln -s /dev/sdb /tmp/vdb.s1

[root@dell-per730-61 ~]# virsh snapshot-create-as avocado-vt-vm1 s1 --diskspec vdb,file=/tmp/vdb.s1,stype=file --memspec file=/dev/sdc
error: internal error: unable to execute QEMU command 'blockdev-add': 'file' driver requires '/tmp/vdb.s1' to be a regular file

7. sdc gone after cmd failed, sdc is the mem snapshot target device
[root@dell-per730-61 ~]# ll /dev/sdc
ls: cannot access '/dev/sdc': No such file or directory

[root@dell-per730-61 ~]# lsscsi
[0:0:0:0]    disk    SanDisk  Ultra USB 3.0    1.00  /dev/sdb 
[12:0:0:0]   disk    LIO-ORG  device.logical-  4.0   -        


Expected result:
When snapshot failed, libvirt should not remove the target block device

Actual result:
block device removed from host when snapshot failed.

Comment 1 Peter Krempa 2021-06-21 07:15:27 UTC
Fixed upstream:

commit b396e9dd9ddd9745ac90ab549912de7bc5cc0b17
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 16:56:56 2021 +0200

    qemuSnapshotCreateActiveExternal: Don't unlink memory snapshot image if it was existing before
    
    When writing the memory snapshot into an existing file don't remove it
    if the snapshot fails later.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit b30a8ee67d355f5258ece062bacaa102428e6d69
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 16:54:18 2021 +0200

    conf: snapshot: rename variable holding memory snapshot file location
    
    'file' is too generic to know what's going on. Rename it to
    'memorysnapshotfile'.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 308aafe289ae377d6508964512d41df2712c50b3
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 15:43:03 2021 +0200

    qemuSnapshotPrepareDiskExternal: Refactor existing file check
    
    Use the snapshot disk type from the definition now that we validate that
    it matches.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>


commit 919b1296032a58b27995ac2d13b035c9be86c17b
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 15:42:53 2021 +0200

    qemuSnapshotPrepareDiskExternal: Enforce match between snapshot type and existing file type
    
    The code executed later when creating a snapshot makes all decisions
    based on the configured type rather than the actual type of the existing
    file, while the check whether the file exists is based solely on the
    on-disk type.
    
    Since a block device is allowed to exist even when not reusing existing
    files in contrast to regular files this creates a potential for a block
    device to squeak past the check but then be influenced by other code
    executed later. Specifically this is a problem when creating a snapshot
    with the following XML:
    
      <domainsnapshot>
        <disks>
          <disk name='vdb' type='file'>
            <source file='/dev/sdb'/>
          </disk>
        </disks>
      </domainsnapshot>
    
    If the snapshot creation fails, '/dev/sdb' will be removed because it's
    considered to be a regular file by the cleanup code.
    
    Add a check that will force that the configured type matches the on-disk
    state.
    
    Additional supporting reason is that qemu stopped to accept block
    devices with the 'file' backend, thus the above configuration will not
    work any more. This allows us to fail sooner.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1972145
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 66adff17a81e9114b19b7467ce3a23e0f7ad5c3c
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 14:57:06 2021 +0200

    qemuSnapshotPrepareDiskExternal: Reject creation of block devices sooner
    
    In case when the snapshot target is of VIR_STORAGE_TYPE_BLOCK type and
    doesn't exist libvirt won't be able to create it. Reject such a config
    sooner.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit a96cc845d7518c0540234ca231b94e8171fbc5ab
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 14:57:44 2021 +0200

    qemuSnapshotPrepareDiskExternal: Avoid condition squashing
    
    Separate the 'else if' branches into nested conditions so that it's more
    obvious when we'll be adding additional checks later.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 006821a809e5f7876d14145b1dd7cd79bf95c7ff
Author: Peter Krempa <pkrempa>
Date:   Wed Jun 16 14:49:26 2021 +0200

    qemuSnapshotPrepareDiskExternal: Move temp variables into the block using them
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>


v7.4.0-212-gb396e9dd9d

Comment 2 yisun 2021-06-30 09:18:45 UTC
pre-test with: libvirt-7.5.0-1.fc34.x86_64

[root@yisun-fed34 ~]# lsscsi
[0:0:0:0]    cd/dvd  QEMU     QEMU DVD-ROM     2.5+  /dev/sr0 
[2:0:0:0]    disk    LIO-ORG  device.logical-  4.0   /dev/sda 

[root@yisun-fed34 ~]# virsh domblklist pc
 Target   Source
---------------------------------------------
 vda      /var/lib/libvirt/images/vda.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2


[root@yisun-fed34 ~]# virsh snapshot-create-as pc s1 --disk-only --diskspec vdb,file=/dev/sda,stype=file
error: unsupported configuration: mismatch between configured type for snapshot disk 'vdb' and the type of existing file '/dev/sda'

[root@yisun-fed34 ~]# ll /dev/sda
brw-rw----. 1 root disk 8, 0 Jun 30 09:15 /dev/sda

[root@yisun-fed34 ~]# virsh snapshot-create-as pc s1 --disk-only --diskspec vdb,file=/dev/sda,stype=block
Domain snapshot s1 created

Comment 5 yisun 2021-07-20 13:22:23 UTC
Verified on libvirt-7.5.0-1.el9.x86_64

1. have a running vm
[root@ibm-x3250m6-10 images]# virsh domstate vm1
running

[root@ibm-x3250m6-10 images]# virsh domblklist vm1
 Target   Source
----------------------------------------------------------------
 vda      /var/lib/libvirt/images/RHEL-9.0-x86_64-latest.qcow2
 vdb      /var/lib/libvirt/images/vdb.qcow2

2. have local block devices sda and sdb
[root@ibm-x3250m6-10 images]# lsscsi
[0:0:0:0]    disk    ATA      ST1000NX0423     LE43  /dev/sda 
[6:0:0:0]    disk    LIO-ORG  device.logical-  4.0   /dev/sdb 

3. create disk only snapshot for vm1, but use wrong config '--diskspec vdb,file=/dev/sdb,stype=file'
[root@ibm-x3250m6-10 images]# virsh snapshot-create-as vm1 s1 --disk-only --diskspec vdb,file=/dev/sdb,stype=file
error: unsupported configuration: mismatch between configured type for snapshot disk 'vdb' and the type of existing file '/dev/sdb'

4. /dev/sdb still existing as block device
[root@ibm-x3250m6-10 images]# ll /dev/sdb
brw-rw----. 1 root disk 8, 16 Jul 20 09:05 /dev/sdb

5. create disk + mem snapshot, but use wrong config '--diskspec vdb,file=/dev/sda,stype=file --memspec file=/dev/sdb'
[root@ibm-x3250m6-10 images]# virsh snapshot-create-as vm1 s1 --diskspec vdb,file=/dev/sda,stype=file --memspec file=/dev/sdb
error: unsupported configuration: mismatch between configured type for snapshot disk 'vdb' and the type of existing file '/dev/sda'

6. /dev/sdb still existing as block device
[root@ibm-x3250m6-10 images]# ll /dev/sdb 
brw-rw----. 1 root disk 8, 16 Jul 20 09:07 /dev/sdb