(In reply to Peter Krempa from comment #0)
> Additional info:
> Using 'detect-zeroes' to sparsify a file was suggested to be used to keep
> sparsness of storage files on prior to nfs 4.2 in 1297255, which is
> impossible to implement without this fixed.

I can't see a comment suggesting this in bug 1297255, but either case, this approach isn't going to work for NFS < 4.2. Old NFS versions just don't support thin provisioning at all, so qemu can neither punch holes nor get information about the allocation status of blocks.

The reason why 'qemu-img convert' can be more efficient is because it knows that it just created the image file and that any parts of the file that it doesn't write to will read as zero. The best we could do to allow the same in the mirror block job would be adding a flag in the QMP command that allows libvirt to tell essentially "I promise this whole image reads as zeros, you can just skip any write_zeroes and discard commands without impacting the result".

Would you think that libvirt has enough information to pass this option, and would it solve what you need?

Preferably, of course, everyone would just switch to NFS 4.2, but I'm told that this is unlikely to be a viable solution...

(In reply to Kevin Wolf from comment #3)
> (In reply to Peter Krempa from comment #0)
> > Additional info:
> > Using 'detect-zeroes' to sparsify a file was suggested to be used to keep
> > sparsness of storage files on prior to nfs 4.2 in 1297255, which is
> > impossible to implement without this fixed.
> 
> I can't see a comment suggesting this in bug 1297255, but either case, this
> approach isn't going to work for NFS < 4.2. Old NFS versions just don't
> support thin provisioning at all, so qemu can neither punch holes nor get
> information about the allocation status of blocks.
> 
> The reason why 'qemu-img convert' can be more efficient is because it knows
> that it just created the image file and that any parts of the file that it
> doesn't write to will read as zero. The best we could do to allow the same
> in the mirror block job would be adding a flag in the QMP command that
> allows libvirt to tell essentially "I promise this whole image reads as
> zeros, you can just skip any write_zeroes and discard commands without
> impacting the result".

Yes that is exactly what I had in mind. In libvirt we can offload this knowledge to the management applications as they usually pre-create the image and pass it in with the --reuse-external flag which translates to mode: existing.

> 
> Would you think that libvirt has enough information to pass this option, and
> would it solve what you need?

We can require users to certify this the same way as we require passing an image which has enough space.

> 
> Preferably, of course, everyone would just switch to NFS 4.2, but I'm told
> that this is unlikely to be a viable solution...

Yes, obviously. It actually works great there.

So I looked some more into it and the mirror block job actually already tries quite hard to avoid unnecessary writes. The reason that it is defeated here is that the setup chosen simply doesn't support thin provisioning:

* Only allocated blocks are initially marked as dirty (and will therefore be copied). However, the source filesystem doesn't support getting information whether a block is allocated or not. You have to read it and manually check whether it's zero.

* The target filesystem doesn't support efficient zero writes, so even if you know that you need to write zeros, it doesn't help you. Just skipping any zero writes is wrong because the guest keeps working on the image and it could first write non-zero data to it and later overwrite it with zeroes. If we copied the non-zero data, we must zero out it again even though the target had zeros initially.

* Both sides don't use a proper image format, but the raw file. This means that we only get the features that the file system already provides on its own. If qcow2 had been used for either source or target, the mirror block job would have a way to keep things sparse.

Reconstructing the allocation information while initially marking blocks as dirty isn't possible either, this would involve reading in the whole image (a second time) while the VM is blocked, which would take by far too long to be reasonable.

Possible solutions for the problem, in descending order of preferability:

1. Use qcow2 for the target image. (No changes in the QEMU/libvirt necessary)

2. Get the concurrent guest writes out of the way and use qemu-img. This could be achieve by an image fleecing setup (add a temporary overlay image, blockdev-backup from the active guest image to the temporary overlay and start the built-in NBD server for the overlay) and qemu-img convert from an NBD source. (Changes might be necessary in libvirt (not sure), but not QEMU)

3. Add an option to the mirror job that makes it manage another bitmap in memory that tracks for each block in the target image whether it has already been written to or is still in its original state. Zero writes could then be skipped if the block had never written to. (Changes necessary to every layer in the virt stack)

(In reply to Kevin Wolf from comment #12)
> Possible solutions for the problem, in descending order of preferability:
> 
> 1. Use qcow2 for the target image. (No changes in the QEMU/libvirt necessary)
> 
> 2. Get the concurrent guest writes out of the way and use qemu-img. This
> could be achieve by an image fleecing setup (add a temporary overlay image,
> blockdev-backup from the active guest image to the temporary overlay and
> start the built-in NBD server for the overlay) and qemu-img convert from an
> NBD source. (Changes might be necessary in libvirt (not sure), but not QEMU)
> 
> 3. Add an option to the mirror job that makes it manage another bitmap in
> memory that tracks for each block in the target image whether it has already
> been written to or is still in its original state. Zero writes could then be
> skipped if the block had never written to. (Changes necessary to every layer
> in the virt stack)

I should add another option (#0): upgrade to NFS 4.2+, as this is a limitation of NFS < 4.1.

Anyway, option #3 (this BZ) is not feasible anytime soon. It has a high cost to implement and would take too long to be worth it (by then, the affected customer might be able to upgrade to NFS 4.2+). So I'm closing this BZ. Please consider the workarounds above.

For some extra technical details, please consult comment #12 in full.