Bug 1008350

Summary: blockcommit with --delete option is not functional
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: zhengqin <zsong>
Component: libvirtAssignee: Pavel Mores <pmores>
Status: CLOSED ERRATA QA Contact: yisun
Severity: medium Docs Contact:
Priority: medium    
Version: 8.0CC: dyuan, eblake, jdenemar, jfehlig, jgao, lmen, lyan, mzhan, pmores, tmo7452, xuzhang
Target Milestone: rc   
Target Release: 8.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-6.0.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1001475 Environment:
Last Closed: 2020-05-05 09:43:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1001475    
Bug Blocks:    

Description zhengqin 2013-09-16 08:54:58 UTC
+++ This bug was initially created as a clone of Bug #1001475 +++

Description of problem:
blockcommit with --delete option is not functional

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 6.5 Beta
libvirt-0.10.2-23.el6.x86_64
kernel-2.6.32-414.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.398.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. prepare a guest foo.

2. blockcommit with --delete option.

# virsh blockcommit foo hda --delete
error: unsupported flags (0x2) in function qemuDomainBlockCommit

Expected result:
Successfully remove files in the backing chain or report an explicit error when not success.

--- Additional comment from RHEL Product and Program Management on 2013-08-27 02:53:08 EDT ---

Since this bug report was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Alex Jia on 2013-08-27 03:05:43 EDT ---

(In reply to Hao Liu from comment #0)
> # virsh blockcommit foo hda --delete
> error: unsupported flags (0x2) in function qemuDomainBlockCommit

Need to check VIR_DOMAIN_BLOCK_COMMIT_DELETE flags, I will commit a patch to fix this.

--- Additional comment from Alex Jia on 2013-08-27 04:05:27 EDT ---

Patch on upstream:
https://www.redhat.com/archives/libvir-list/2013-August/msg01333.html

--- Additional comment from Eric Blake on 2013-08-27 08:04:56 EDT ---

Moving to 6.6.  The flag is a case of known future expansion, and as we're already past the feature freeze for 6.5, it isn't going to change in 6.5.


This issue also occurs on RHEL7

Comment 6 Liang 2017-07-05 16:28:41 UTC
Just wondering if there are any update for this issue? Thanks.

Comment 7 tmo7452 2018-08-03 07:51:15 UTC
Yearly check-in here. I hit this bug while writing a hot backup script. It has to actually compare the output text against "Successfully pivoted" and conditionally run "rm -f <snapshot>", which makes me very uncomfortable.

Comment 8 Jaroslav Suchanek 2019-04-24 12:26:35 UTC
This bug is going to be addressed in next major release.

Comment 9 Pavel Mores 2019-11-20 09:59:11 UTC
Posted a proposed fix: https://www.redhat.com/archives/libvir-list/2019-November/msg00912.html

Comment 10 Pavel Mores 2019-12-11 11:23:12 UTC
Fixed by

7d484ede20 qemu: block: enable the snapshot image deletion feature
73532dadd2 qemu: block: store the delete flag in libvirtd's status XML
9e5c98e84f qemu: block: use the delete flag to delete snapshot images if requested
cb03fd9340 qemu: block: propagate the delete flag to where it can actually be used

v5.10.0-175-gcb03fd9340

Comment 12 gaojianan 2020-01-20 03:01:19 UTC
As we can see that we have added the --delete function to delete the backing store, why not just delete the display in the snapshot-list directly?
If there is a backing-chain like base<--s1<--s2<--s3<--s4
After we execute "virsh blockcommit demo vdb --wait --verbose  --top /tmp/vdb.s3  --delete --active --pivot" with --delete to delete snapshot s3,but we can still find the snap s3 in the snapshot list.
As i think,we can clear it together with the image.
How do you think of this idea?

Comment 13 gaojianan 2020-01-20 03:46:39 UTC
(In reply to gaojianan from comment #12)
> As we can see that we have added the --delete function to delete the backing
> store, why not just delete the display in the snapshot-list directly?
> If there is a backing-chain like base<--s1<--s2<--s3<--s4
> After we execute "virsh blockcommit demo vdb --wait --verbose  --top
> /tmp/vdb.s3  --delete --active --pivot" with --delete to delete snapshot
> s3,but we can still find the snap s3 in the snapshot list.
> As i think,we can clear it together with the image.
> How do you think of this idea?

Maybe i have known my wrong,if there are several disks in the guest,but we can onlt delete 1 of the disk's snapshot image,but other disks still use the snapshot name.

Comment 14 Pavel Mores 2020-01-22 16:16:37 UTC
I agree that the fact that a snapshot still comes up in snapshot-list output even after the snapshot has been committed and deleted might seem confusing (I was confused myself when I first saw it).

However, the explanation is roughly that the blockcommit command really operates on QEMU level, it just manipulates image chains without understanding the wider context.  In particular, it has no knowledge of libvirt's snapshot metadata (the XMLs normally stored under /var/lib/libvirt/qemu/snapshot) which is what the snapshot-list command uses to list snapshots.

To finish the operation and make the snapshot disappear from snapshot-list, you currently have to issue 'snapshot-delete --metadata' to remove the snapshot metadata left behind by blockcommit.

Obviously this is suboptimal, there should be a single command to handle the whole operation.  That command should be snapshot-delete which however can't do it at the time as its support of external snapshots is incomplete.  However, there's a BZ requesting to fix just that (https://bugzilla.redhat.com/show_bug.cgi?id=1519001) and chances are it might be fixed soon(ish).

Comment 15 yisun 2020-02-21 03:35:05 UTC
Test result: PASS
[root@hp-dl320eg8-05 bz1008350]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-4"
qemu-kvm-4.2.0-10.module+el8.2.0+5740+c3dff59e.x86_64
libvirt-6.0.0-5.module+el8.2.0+5765+64816f89.x86_64


Test scenarios:
1. Commit from middle to middle
2. Commit from middle to base
3. Commit from top to middle
4. Commit from top to base
5. Abort block job and see if files still exist ('blockcommit --timeout' or 'blockjob --abort')
Since snapshots' metadata is not operated by this fix according to above comments, we'll skip snapshot metadata check in this bug verification.



1. Commit from middle to middle
[root@hp-dl320eg8-05 bz1008350]# virsh domblklist vm1
 Target   Source
---------------------------------------------
 vda      /var/lib/libvirt/images/vda.qcow2

[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:05 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:05 vda.snap1
-rw-------. 1 qemu qemu     196768 Feb 20 22:05 vda.snap2
-rw-------. 1 qemu qemu     196768 Feb 20 22:05 vda.snap3
-rw-------. 1 qemu qemu   32112640 Feb 20 22:05 vda.snap4

[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap3 --base /var/lib/libvirt/images/vda.snap2 --delete --verbose --wait
Block commit: [100 %]
Commit complete
[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:05 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:05 vda.snap1
-rw-------. 1 qemu qemu     196768 Feb 20 22:05 vda.snap2
-rw-------. 1 qemu qemu   41025536 Feb 20 22:06 vda.snap4
<==== snap3 deleted as expected

[root@hp-dl320eg8-05 bz1008350]# qemu-img info -U /var/lib/libvirt/images/vda.snap4 --backing-chain
image: /var/lib/libvirt/images/vda.snap4
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 70.9 MiB
cluster_size: 65536
backing file: /var/lib/libvirt/images/vda.snap2
...
<==== backing chain is correct

2. Commit from middle to base (start from a clean vm)
[root@hp-dl320eg8-05 bz1008350]# virsh start vm1
Domain vm1 started



[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap3 --base /var/lib/libvirt/images/vda.qcow2 --delete --verbose --wait
Block commit: [100 %]
Commit complete


[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:11 vda.qcow2
-rw-------. 1 qemu qemu    1835008 Feb 20 22:12 vda.snap4
<==== snap1,2,3 deleted as expected

[root@hp-dl320eg8-05 bz1008350]# qemu-img info -U /var/lib/libvirt/images/vda.snap4 --backing-chain
image: /var/lib/libvirt/images/vda.snap4
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 3.44 MiB
cluster_size: 65536
backing file: /var/lib/libvirt/images/vda.qcow2
<==== backing chain is correct

3. Commit from top to middle (start from a clean vm)
[root@hp-dl320eg8-05 bz1008350]# virsh start vm1
Domain vm1 started

[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap4 --base /var/lib/libvirt/images/vda.snap2 --delete --verbose --wait --active
Block commit: [100 %]
Now in synchronized phase

[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:15 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:15 vda.snap1
-rw-------. 1 qemu qemu   34144256 Feb 20 22:16 vda.snap2
-rw-------. 1 qemu qemu     196768 Feb 20 22:15 vda.snap3
-rw-------. 1 qemu qemu   34144256 Feb 20 22:16 vda.snap4
<==== file still exist since vm still using snap4 for now, need a pivot action

[root@hp-dl320eg8-05 bz1008350]# virsh blockjob vm1 vda --pivot

[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:15 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:15 vda.snap1
-rw-------. 1 qemu qemu   35520512 Feb 20 22:16 vda.snap2
<==== now files deleted as expected


[root@hp-dl320eg8-05 bz1008350]# qemu-img info -U /var/lib/libvirt/images/vda.snap2 --backing-chain
image: /var/lib/libvirt/images/vda.snap2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 34.8 MiB
cluster_size: 65536
backing file: /var/lib/libvirt/images/vda.snap1
...
<==== correct backing chain

4. Commit from top to base (start from a clean vm)
[root@hp-dl320eg8-05 bz1008350]# virsh start vm1
Domain vm1 started

[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap4 --delete --verbose --wait --active --pivot
Block commit: [100 %]
Successfully pivoted

[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:20 vda.qcow2
<==== snaps 1,2,3,4 deleted as expected

5. Abort block job and see if file still exists ('blockcommit --timeout' or 'blockjob --abort') (start from a clean vm)
[root@hp-dl320eg8-05 bz1008350]# virsh start vm1
Domain vm1 started


[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

In vm, create some data to make sure the blockcommit will take long time
[root@localhost ~]# dd if=/dev/urandom of=/tmp/bigfile bs=1M count=99
99+0 records in
99+0 records out
103809024 bytes (104 MB, 99 MiB) copied, 0.457437 s, 227 MB/s
[root@localhost ~]# sync

[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap4 --delete --verbose --wait --active --pivot --timeout 2 --bandwidth 1
Block commit: [  2 %]
Commit aborted
[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:25 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:22 vda.snap1
-rw-------. 1 qemu qemu     196768 Feb 20 22:22 vda.snap2
-rw-------. 1 qemu qemu     196768 Feb 20 22:22 vda.snap3
-rw-------. 1 qemu qemu  107806720 Feb 20 22:25 vda.snap4
<==== snap files still exist, as expected

[root@hp-dl320eg8-05 bz1008350]# qemu-img info -U /var/lib/libvirt/images/vda.snap4 --backing-chain
image: /var/lib/libvirt/images/vda.snap4
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 115 MiB
cluster_size: 65536
backing file: /var/lib/libvirt/images/vda.snap3
...
<==== correct backing chain

6. Abort block job and see if file still exists ('blockjob --abort') (start from a clean vm)
[root@hp-dl320eg8-05 bz1008350]# virsh start vm1
Domain vm1 started

[root@hp-dl320eg8-05 bz1008350]# for i in {1..4}; do virsh snapshot-create-as vm1 snap$i --disk-only; done
Domain snapshot snap1 created
Domain snapshot snap2 created
Domain snapshot snap3 created
Domain snapshot snap4 created

In vm, create some data to make sure the blockcommit will take long time
[root@localhost ~]#  dd if=/dev/urandom of=/tmp/bigfile bs=1M count=99
99+0 records in
99+0 records out
103809024 bytes (104 MB, 99 MiB) copied, 0.461168 s, 225 MB/s
[root@localhost ~]# sync

[root@hp-dl320eg8-05 ~]# virsh blockjob vm1 vda
Active Block Commit: [  4 %]    Bandwidth limit: 1048576 bytes/s (1.000 MiB/s)

In another terminal [root@hp-dl320eg8-05 ~]# virsh blockjob vm1 vda --abort

Back to original terminal
[root@hp-dl320eg8-05 bz1008350]# virsh blockcommit vm1 vda --top  /var/lib/libvirt/images/vda.snap4 --delete --verbose --wait --active --pivot --bandwidth 1
Block commit: [ 17 %]
Commit aborted

[root@hp-dl320eg8-05 bz1008350]# ll /var/lib/libvirt/images/ | grep vda
-rw-r--r--. 1 qemu qemu 2612920603 Feb 20 22:29 vda.qcow2
-rw-------. 1 qemu qemu     196768 Feb 20 22:28 vda.snap1
-rw-------. 1 qemu qemu     196768 Feb 20 22:28 vda.snap2
-rw-------. 1 qemu qemu     196768 Feb 20 22:28 vda.snap3
-rw-------. 1 qemu qemu  139526144 Feb 20 22:30 vda.snap4
<==== all snap files exist, as expected

[root@hp-dl320eg8-05 bz1008350]# virsh domblklist vm1
 Target   Source
---------------------------------------------
 vda      /var/lib/libvirt/images/vda.snap4

[root@hp-dl320eg8-05 bz1008350]# qemu-img info -U /var/lib/libvirt/images/vda.snap4 --backing-chain
image: /var/lib/libvirt/images/vda.snap4
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 137 MiB
cluster_size: 65536
backing file: /var/lib/libvirt/images/vda.snap3
...
<==== backing chain correct

Comment 17 errata-xmlrpc 2020-05-05 09:43:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017