Bug 2226862

On 4k disk, there is bug about 4k alignment issue. not sure if they are same reason
Bug 2141964 - Guest hit EXT4-fs error on host 4K disk when repeatedly hot-plug/unplug running IO disk

Thank you @qing.wang - I can reproduce it with the qemu build from that BZ though - qemu-kvm-6.2.0-37.module+el8.9.0+19491+15e62c0a

Can't reproduce on RHEL 9 with nested setup and logical_block_size=physical_block_size=4096 for kvm host

Reproduce on 4k host 

Red Hat Enterprise Linux release 8.9 Beta (Ootpa)
4.18.0-504.el8.x86_64
qemu-kvm-6.2.0-37.module+el8.9.0+19491+15e62c0a.x86_64
seabios-bin-1.16.0-3.module+el8.9.0+18724+20190c23.noarch
edk2-ovmf-20220126gitbb1bba3d77-5.el8.noarch
libvirt-8.0.0-21.module+el8.9.0+19166+e262ca96.x86_64
virtio-win-prewhql-0.1-240.iso


Steps
1. Create image files

      qemu-img create -f qcow2 /home/kvm_autotest_root/images/mstg1.qcow2 1.1G
      qemu-img create -f raw /home/kvm_autotest_root/images/mstg2.raw 1.2G
      qemu-img create -f qcow2 /home/kvm_autotest_root/images/mstg3.qcow2 1.3G

2. Boot VM
/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35 \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
 -blockdev '{"driver":"file","filename":"/home/kvm_autotest_root/images/mstg1.qcow2","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
 -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage"}' \
 -device scsi-hd,bus=scsi0.0,scsi-id=0,share-rw=on,drive=libvirt-2-format,id=scsi0-0-0-1,write-cache=on \
\
-blockdev '{"driver":"file","filename":"/home/kvm_autotest_root/images/mstg3.qcow2","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
 -blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
 -device scsi-hd,bus=scsi0.0,scsi-id=0,share-rw=on,drive=libvirt-3-format,id=scsi0-0-0-3,write-cache=on \
\
 -blockdev '{"driver":"file","filename":"/home/kvm_autotest_root/images/mstg2.raw","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
 -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
 -device virtio-blk-pci,bus=pcie-root-port-6,addr=0x0,share-rw=on,drive=libvirt-1-format,id=virtio-disk0,write-cache=on \
\
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \


VM failed :(qemu) qemu-kvm: -device scsi-hd,bus=scsi0.0,scsi-id=0,share-rw=on,drive=libvirt-3-format,id=scsi0-0-0-3,write-cache=on: Cannot get 'write' permission without 'resize': Image size is not a multiple of request alignment


But I think that is an invalid test

1. set raw format on qcow2 file, it makes the wrong disk size in guest even if it may boot successfully on non-4k disk host.
(it display 192.5K in guest,this usage is wrong )
the share-rw only works on raw format, So the normal usage is using raw format file instead of qcow2 file backend.


2. workaround: alloc disk space in full in advance if it has to use qcow2 format backend.
qemu-img create  -o preallocation=full -f qcow2 /home/kvm_autotest_root/images/mstg3.qcow2 1.3G
(qcow2 is meaningless)

3. alternative convert qcow2 file to raw then start to share-rw using.

Hi,kwolf, what do you think?

(In reply to qing.wang from comment #4)
> Reproduce on 4k host 
> 

> But I think that is an invalid test
> 
> 1. set raw format on qcow2 file, it makes the wrong disk size in guest even
> if it may boot successfully on non-4k disk host.
> (it display 192.5K in guest,this usage is wrong )
> the share-rw only works on raw format, So the normal usage is using raw
> format file instead of qcow2 file backend.
> 
> 
Yep
> 2. workaround: alloc disk space in full in advance if it has to use qcow2
> format backend.
> qemu-img create  -o preallocation=full -f qcow2
> /home/kvm_autotest_root/images/mstg3.qcow2 1.3G
> (qcow2 is meaningless)
> 
> 3. alternative convert qcow2 file to raw then start to share-rw using.
> 
> Hi,kwolf, what do you think?

If we want to support qcow2 format, I think this requires a change in libvirt. Currently, it wouldn't allow me to set qcow2 format:

"error: unsupported configuration: shared access for disk 'vdb' requires use of supported storage format",

snippet

<disk device="disk" type="file"><target bus="virtio" dev="vdb" /><source file="/var/lib/avocado/data/avocado-vt/test.qcow2" /><driver name="qemu" type="qcow2"/><shareable /></disk>

qemu cannot and will never report that an image is not in “raw format”, because there is no such format.  Any file is technically a valid image in raw format.

Setting the format to “raw” will have qemu pass the image through to the guest as-is.  If qemu were to fail because some format has been detected on the disk, that would be extremely problematic when a guest application gets to write a fake image format header (like a qcow2 header) into the image file.  (This is why probing for the image format is seen as a security problem today.  The guest could write a fake image header, and thus make qemu interpret the image format the next time the VM is booted, e.g. reading a backing file at an arbitrary path.)

So qemu adhering to the user’s choice and not complaining about it is entirely correct.

---

Now, as far as the error message goes: It does tell the problem, namely “Image size is not a multiple of request alignment”.  The cause is that qemu-img create doesn’t use O_DIRECT (there is no way to set the cache mode), so when creating an image, the alignment requirements are ignored (because everything goes through the page cache).  Normally, this won’t cause problems: For raw images, the file size is just the guest disk size, so it’s on you to have it match the disk alignment.  For qcow2 images, when created, the underlying file will generally be misaligned, but when you open it with O_DIRECT for the first time, it will simply be resized to match once it’s opened with O_DIRECT for the first time, and you write to the end of file.

However, you are attaching a qcow2 image (that is thus not aligned) as a raw image.  qemu will again try to auto-align the end of the image file, but here, it fails, because through virtio-blk, you cannot resize images, and so the image cannot be resized upon a request from it.  In more technical terms, virtio-blk doesn’t take the permission to resize, and without that permission, the block layer cannot do a resize (which is what the error message says).  If there were a qcow2 driver involved, that would take the permission to resize, which is why it would work then.

So because qemu can’t do the resize automatically, the user would need to do it (growing the image with qemu-img resize such that its size becomes a multiple of the host block size).  But in this case, clearly, that’s not the problem, the problem is that a qcow2 image is used as a raw image, which was not intended and is wrong.

---

Finally, it looks to me like the root of the problem was that it was impossible to attach the image correctly as a qcow2 image because libvirt refused to do so while enabling sharing.  That looks correct to me.

On the lower level, I expect <shareable/> to translate into share-rw=on.  This is a setting that tells qemu that the guest is fine with the disk contents being modified by an outside source.  But the guest is only one stakeholder in this: The qcow2 driver internally keeps metadata caches, and if another party were to write to the same image file concurrently, the cache contents will begin to differ from what’s on disk, and this will lead to image corruption.  Therefore, a qcow2 image can never be shared between VMs (unless all of them are configured to access it read-only).

You can technically use share-rw=on with qcow2 images, but only under the harsh restriction that the image file is still opened only once.  This can only be the case if all disks that access the image are part of the same VM, and all of them reference the same --blockdev.  I understand that libvirt’s <shareable/> allows sharing images between VMs, though, which breaks the restriction.

---

As a conclusion, let me go through the list of additional info from comment 0:

A) When executing the scenario on a host with block size 512, the vm starts without a problem.

I’m not sure what this is supposed to mean.  If you are attaching a qcow2 image as a raw image, the guest will not see the data that is on the qcow2 image, it will see the qcow2 metadata and data.  So there will be a problem sooner or later.

Disregarding that, though, qemu internally tracks image size in units of 512 byte sectors, so that’s why qemu won’t see a misalignment unless the alignment requirement exceeds 512.

B) This doesn't seem to have worked in earlier RHEL version either (I confirmed for 8.6).

(Nothing I can add)

C) I have found this on s390x but assume that this should be reproducible on other archs.

I believe all of this is architecture-independent code, yes.

D) Luckily, I can't set format qcow2, libvirt informs me I can't use that format "error: unsupported configuration: shared access for disk 'vdb' requires use of supported storage format
Failed. Try again? [y,n,i,f,?]: 
" - this concluded I could use a raw image.

Yes, you can use a raw image, but only if it’s actually a raw image (see (E)).  Of course you cannot create a qcow2 image and attach it as a raw image, though.

E) When I instead create the image with -f raw I can start the machine without a problem.

Yes.  You can share raw images between VMs, and the alignment works out (1G is aligned to 4k).

F) Given that the error message doesn't reveal the root cause but has a workaround I set medium severity

I understand you mean the root cause to be “this is a qcow2 image, not a raw image”, but indeed it is a raw image.  So if the user tells qemu that it is a raw image, qemu is in no place to question it.

(A side note: If you really must share qcow2 images between VMs, the correct way is to use a central qemu-storage-daemon instance.  You open the qcow2 image there, and then export it to all VMs that need access to it, e.g. via vhost-user-blk.)

(In reply to Hanna Czenczek from comment #9)
> (A side note: If you really must share qcow2 images between VMs, the correct
> way is to use a central qemu-storage-daemon instance.  You open the qcow2
> image there, and then export it to all VMs that need access to it, e.g. via
> vhost-user-blk.)

Do you mean I should create a pool for the image? I don't know what the qemu-storage-daemon is used for from user perspective.

(In reply to Hanna Czenczek from comment #8)
> qemu cannot and will never report that an image is not in “raw format”,
> because there is no such format.  Any file is technically a valid image in
> raw format.
> 
> Setting the format to “raw” will have qemu pass the image through to the
> guest as-is.  If qemu were to fail because some format has been detected on
> the disk, that would be extremely problematic when a guest application gets
> to write a fake image format header (like a qcow2 header) into the image
> file.  (This is why probing for the image format is seen as a security
> problem today.  The guest could write a fake image header, and thus make
> qemu interpret the image format the next time the VM is booted, e.g. reading
> a backing file at an arbitrary path.)
> 
> So qemu adhering to the user’s choice and not complaining about it is
> entirely correct.
> 
> ---
> 
> Now, as far as the error message goes: It does tell the problem, namely
> “Image size is not a multiple of request alignment”.  The cause is that
> qemu-img create doesn’t use O_DIRECT (there is no way to set the cache
> mode), so when creating an image, the alignment requirements are ignored
> (because everything goes through the page cache).  Normally, this won’t
> cause problems: For raw images, the file size is just the guest disk size,
> so it’s on you to have it match the disk alignment.  For qcow2 images, when
> created, the underlying file will generally be misaligned, but when you open
> it with O_DIRECT for the first time, it will simply be resized to match once
> it’s opened with O_DIRECT for the first time, and you write to the end of
> file.
> 
> However, you are attaching a qcow2 image (that is thus not aligned) as a raw
> image.  qemu will again try to auto-align the end of the image file, but
> here, it fails, because through virtio-blk, you cannot resize images, and so
> the image cannot be resized upon a request from it.  In more technical
> terms, virtio-blk doesn’t take the permission to resize, and without that
> permission, the block layer cannot do a resize (which is what the error
> message says).  If there were a qcow2 driver involved, that would take the
> permission to resize, which is why it would work then.
> 
> So because qemu can’t do the resize automatically, the user would need to do
> it (growing the image with qemu-img resize such that its size becomes a
> multiple of the host block size).  But in this case, clearly, that’s not the
> problem, the problem is that a qcow2 image is used as a raw image, which was
> not intended and is wrong.
> 
> ---
> 
> Finally, it looks to me like the root of the problem was that it was
> impossible to attach the image correctly as a qcow2 image because libvirt
> refused to do so while enabling sharing.  That looks correct to me.
> 
> On the lower level, I expect <shareable/> to translate into share-rw=on. 
> This is a setting that tells qemu that the guest is fine with the disk
> contents being modified by an outside source.  But the guest is only one
> stakeholder in this: The qcow2 driver internally keeps metadata caches, and
> if another party were to write to the same image file concurrently, the
> cache contents will begin to differ from what’s on disk, and this will lead
> to image corruption.  Therefore, a qcow2 image can never be shared between
> VMs (unless all of them are configured to access it read-only).
> 
> You can technically use share-rw=on with qcow2 images, but only under the
> harsh restriction that the image file is still opened only once.  This can
> only be the case if all disks that access the image are part of the same VM,
> and all of them reference the same --blockdev.  I understand that libvirt’s
> <shareable/> allows sharing images between VMs, though, which breaks the
> restriction.
> 
> ---
> 
> As a conclusion, let me go through the list of additional info from comment
> 0:
> 
> A) When executing the scenario on a host with block size 512, the vm starts
> without a problem.
> 
> I’m not sure what this is supposed to mean.  If you are attaching a qcow2
> image as a raw image, the guest will not see the data that is on the qcow2
> image, it will see the qcow2 metadata and data.  So there will be a problem
> sooner or later.
> 
> Disregarding that, though, qemu internally tracks image size in units of 512
> byte sectors, so that’s why qemu won’t see a misalignment unless the
> alignment requirement exceeds 512.

We have a test case that passed if host has 512 blocksize. I assume it just uses the image as raw ignoring and overwriting any qcow2 specific data. The test starts the VM, formats the disk and writes to it. All without issues AFAIK.
> 
> B) This doesn't seem to have worked in earlier RHEL version either (I
> confirmed for 8.6).
> 
> (Nothing I can add)
> 
> C) I have found this on s390x but assume that this should be reproducible on
> other archs.
> 
> I believe all of this is architecture-independent code, yes.
> 
> D) Luckily, I can't set format qcow2, libvirt informs me I can't use that
> format "error: unsupported configuration: shared access for disk 'vdb'
> requires use of supported storage format
> Failed. Try again? [y,n,i,f,?]: 
> " - this concluded I could use a raw image.
> 
> Yes, you can use a raw image, but only if it’s actually a raw image (see
> (E)).  Of course you cannot create a qcow2 image and attach it as a raw
> image, though.
> 
> E) When I instead create the image with -f raw I can start the machine
> without a problem.
> 
> Yes.  You can share raw images between VMs, and the alignment works out (1G
> is aligned to 4k).
> 
> F) Given that the error message doesn't reveal the root cause but has a
> workaround I set medium severity
> 
> I understand you mean the root cause to be “this is a qcow2 image, not a raw
> image”, but indeed it is a raw image.  So if the user tells qemu that it is
> a raw image, qemu is in no place to question it.

This makes sense to me now. Thank you. Please feel free to close this BZ.

(In reply to smitterl from comment #10)
> (In reply to Hanna Czenczek from comment #9)
> > (A side note: If you really must share qcow2 images between VMs, the correct
> > way is to use a central qemu-storage-daemon instance.  You open the qcow2
> > image there, and then export it to all VMs that need access to it, e.g. via
> > vhost-user-blk.)
> 
> Do you mean I should create a pool for the image? I don't know what the
> qemu-storage-daemon is used for from user perspective.

I don’t know how it would look on the libvirt layer (I don’t know how far the QSD support is, BZ 1884667 is still open), but down at the qemu layer, it can look something like this (all run parallel, just start the daemon before the qemu instances):

$ qemu-storage-daemon \
    --blockdev qcow2,node-name=qcow2-node,file.driver=file,file.filename=arch.qcow2,file.cache.direct=on \
    --export vhost-user-blk,id=exp0,node-name=qcow2-node,writable=on,addr.type=unix,addr.path=/tmp/exp0.sock \
    --export vhost-user-blk,id=exp1,node-name=qcow2-node,writable=on,addr.type=unix,addr.path=/tmp/exp1.sock

$ qemu-system-x86_64 \
    --chardev socket,id=vhost-user,path=/tmp/exp0.sock \
    --device vhost-user-blk-pci,chardev=vhost-user \
    --enable-kvm \
    -m 2g \
    --object memory-backend-file,id=mem,size=2g,mem-path=/dev/shm,share=on \
    --numa node,memdev=mem

$ qemu-system-x86_64 \
    --chardev socket,id=vhost-user,path=/tmp/exp1.sock \
    --device vhost-user-blk-pci,chardev=vhost-user \
    --enable-kvm \
    -m 2g \
    --object memory-backend-file,id=mem,size=2g,mem-path=/dev/shm,share=on \
    --numa node,memdev=mem

I’m closing the BZ, but if you’d like to continue the discussion, feel free to do it here or send me an email!

(In reply to Hanna Czenczek from comment #12)
> (In reply to smitterl from comment #10)
> > (In reply to Hanna Czenczek from comment #9)
> > > (A side note: If you really must share qcow2 images between VMs, the correct
> > > way is to use a central qemu-storage-daemon instance.  You open the qcow2
> > > image there, and then export it to all VMs that need access to it, e.g. via
> > > vhost-user-blk.)
> > 
> > Do you mean I should create a pool for the image? I don't know what the
> > qemu-storage-daemon is used for from user perspective.
> 
> I don’t know how it would look on the libvirt layer (I don’t know how far
> the QSD support is, BZ 1884667 is still open), but down at the qemu layer,
> it can look something like this (all run parallel, just start the daemon
> before the qemu instances):
> 
> $ qemu-storage-daemon \
>     --blockdev
> qcow2,node-name=qcow2-node,file.driver=file,file.filename=arch.qcow2,file.
> cache.direct=on \
>     --export
> vhost-user-blk,id=exp0,node-name=qcow2-node,writable=on,addr.type=unix,addr.
> path=/tmp/exp0.sock \
>     --export
> vhost-user-blk,id=exp1,node-name=qcow2-node,writable=on,addr.type=unix,addr.
> path=/tmp/exp1.sock
> 
> $ qemu-system-x86_64 \
>     --chardev socket,id=vhost-user,path=/tmp/exp0.sock \
>     --device vhost-user-blk-pci,chardev=vhost-user \
>     --enable-kvm \
>     -m 2g \
>     --object memory-backend-file,id=mem,size=2g,mem-path=/dev/shm,share=on \
>     --numa node,memdev=mem
> 
> $ qemu-system-x86_64 \
>     --chardev socket,id=vhost-user,path=/tmp/exp1.sock \
>     --device vhost-user-blk-pci,chardev=vhost-user \
>     --enable-kvm \
>     -m 2g \
>     --object memory-backend-file,id=mem,size=2g,mem-path=/dev/shm,share=on \
>     --numa node,memdev=mem
> 
> I’m closing the BZ, but if you’d like to continue the discussion, feel free
> to do it here or send me an email!
Thanks for the info.