Bug 1277705 - Fix handling of thin write correctly in active-commit and drive-mirror (including qemu-img commit)
Summary: Fix handling of thin write correctly in active-commit and drive-mirror (inclu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: Qianqian Zhu
URL:
Whiteboard:
: 1277706 (view as bug list)
Depends On:
Blocks: 1305606 734120 1288337 1459730
TreeView+ depends on / blocked
 
Reported: 2015-11-03 21:11 UTC by Yaniv Kaul
Modified: 2017-06-08 00:54 UTC (History)
15 users (show)

Fixed In Version: QEMU 2.6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1459730 (view as bug list)
Environment:
Last Closed: 2016-11-07 20:51:26 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Yaniv Kaul 2015-11-03 21:11:46 UTC
Description of problem:
Per my email discussion with Richard:

> 1.When sparsifying, virt-sparsify is only working on the current snap? Is
> that the intention?
virt-sparsify isn't really tested on snapshots at all.  It's expecting
a single base image that it can sparsify.

> 2. Perhaps it should at least warn that there's work to be done on the base?

It could check that, and should probably just stop if it encounters
that.  I can't think of a case where you'd want to sparsify a
snapshot.

Comment 2 Pino Toscano 2015-11-04 09:02:24 UTC
*** Bug 1277706 has been marked as a duplicate of this bug. ***

Comment 3 Richard W.M. Jones 2015-11-04 09:59:53 UTC
Just to be clear, this is only for --in-place mode, not copying mode.

Comment 4 Yaniv Kaul 2015-11-04 10:06:38 UTC
(In reply to Richard W.M. Jones from comment #3)
> Just to be clear, this is only for --in-place mode, not copying mode.

So what will happen in copy mode? It'll copy the whole disk? The snapshot tree?

Comment 5 Richard W.M. Jones 2015-11-04 10:12:18 UTC
(In reply to Yaniv Kaul from comment #4)
> (In reply to Richard W.M. Jones from comment #3)
> > Just to be clear, this is only for --in-place mode, not copying mode.
> 
> So what will happen in copy mode? It'll copy the whole disk? The snapshot
> tree?

It'll copy the whole disk.

Comment 6 Richard W.M. Jones 2015-11-04 10:12:33 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2015-November/msg00022.html

Comment 7 Richard W.M. Jones 2015-11-04 12:28:22 UTC
Discussion of this continued on the mailing list.  I'm dropping
the above patch for now.

Comment 9 Richard W.M. Jones 2015-11-04 21:03:41 UTC
Kevin identified a particular commit which causes the
'qemu-img commit' not to be propagated down to backing
files.

The problematic commit is:

https://lists.gnu.org/archive/html/qemu-stable/2015-06/msg00014.html

His comments are:

https://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg00817.html

Assuming this can be fixed in qemu, then it would be possible
to run:

virt-sparsify --in-place overlay.qcow2
qemu-img commit overlay.qcow2

which would propagate sparseness through the whole backing chain.

Or:

virt-sparsify --in-place overlay.qcow2
qemu-img commit -b base overlay.qcow2

to propagate sparseness through all layers from the overlay
down to 'base' (but not to lower layers).

Of course we will need to check that qemu gets fixed and
this really works, so let's leave this bug open for now.

Comment 10 Yaniv Kaul 2015-11-05 11:07:39 UTC
Richard, the above is great news. I wonder if I should rephrase the title of the bug to reflect that, or we can close the bug altogether (or move it to qemu-kvm).

However, I assume that unless it is carefully coordinated, this is uber dangerous - the backing file might be used by multiple snapshots, no? Some of which may be running.
oVirt can orchestrate it safely, of course, but generally, you need to ensure / warn / provide a '--force' option.

Comment 11 Richard W.M. Jones 2015-11-05 12:24:10 UTC
Agreed on both points.

Comment 14 Fam Zheng 2016-01-15 07:57:29 UTC
Upstream patches waiting to be merged:

https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg02139.html

Comment 16 Mike McCune 2016-03-28 22:35:49 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 18 Fam Zheng 2016-09-09 03:13:42 UTC
To verify this bug, please check that zero areas smaller than 10M are committed as 'zeroed' cluster in base:

qemu-img create /var/tmp/base.qcow2 -f qcow2 1G
qemu-io /var/tmp/base.qcow2 -c 'write -P 1 0 1G'
qemu-img create /var/tmp/top.qcow2 -f qcow2 -b /var/tmp/base.qcow2
qemu-io /var/tmp/top.qcow2 -c 'write -z 0 64k'
qemu-img commit /var/tmp/top.qcow2
qemu-img map --output=json /var/tmp/base.qcow2


Before fix:

Formatting '/var/tmp/base.qcow2', fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:02.97 (343.968 MiB/sec and 0.3359 ops/sec)
Formatting '/var/tmp/top.qcow2', fmt=qcow2 size=1073741824 backing_file='/var/tmp/base.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 0
64 KiB, 1 ops; 0.0233 sec (2.672 MiB/sec and 42.7515 ops/sec)
Image committed.
[{ "start": 0, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 327680},
{ "start": 536870912, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 537264128}]


After:

Formatting '/var/tmp/base.qcow2', fmt=qcow2 size=1073741824 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:03.54 (288.653 MiB/sec and 0.2819 ops/sec)
Formatting '/var/tmp/top.qcow2', fmt=qcow2 size=1073741824 backing_file=/var/tmp/base.qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
wrote 65536/65536 bytes at offset 0
64 KiB, 1 ops; 0.0278 sec (2.247 MiB/sec and 35.9492 ops/sec)
Image committed.
[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 65536, "length": 536805376, "depth": 0, "zero": false, "data": true, "offset": 393216},
{ "start": 536870912, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 537264128}]

Comment 19 Qianqian Zhu 2016-09-09 06:38:19 UTC
Verified with:
qemu-kvm-rhev-2.6.0-22.el7.x86_64
qemu-img-rhev-2.6.0-22.el7.x86_64
kernel-3.10.0-495.el7.x86_64

Steps:
qemu-img create /var/tmp/base.qcow2 -f qcow2 1G
qemu-io /var/tmp/base.qcow2 -c 'write -P 1 0 1G'
qemu-img create /var/tmp/top.qcow2 -f qcow2 -b /var/tmp/base.qcow2
qemu-io /var/tmp/top.qcow2 -c 'write -z 0 64k'
qemu-img commit /var/tmp/top.qcow2
qemu-img map --output=json /var/tmp/base.qcow2

Results:
[{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false},
{ "start": 65536, "length": 536805376, "depth": 0, "zero": false, "data": true, "offset": 393216},
{ "start": 536870912, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 537264128}]

Compared with the result on:
qemu-img-rhev-2.3.0-6.el7.x86_64
qemu-kvm-rhev-2.3.0-6.el7.x86_64

[{ "start": 0, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 327680},
{ "start": 536870912, "length": 536870912, "depth": 0, "zero": false, "data": true, "offset": 537264128}]

Moving to Verified.

Comment 21 errata-xmlrpc 2016-11-07 20:51:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html


Note You need to log in before you can comment on or make changes to this bug.