Bug 2034791

There is no the issue in qemu-kvm-6.1.0-8.el9. So mark this bug with regression.

And the complete core dump file is:
http://fileshare.englab.nay.redhat.com/pub/section2/kvm/timao/bugs/2034791/core.qemu-kvm.0.ddd914dce3784b99b6e97036d1122b13.358780.1640144233000000.zst

I did a very fast check, and the issue should be introduced by commit 0347a8fd4c ("block/rbd: implement bdrv_co_block_status"): https://gitlab.com/qemu-project/qemu/-/commit/0347a8fd4c3faaedf119be04c197804be40a384b
So it seems to be a regression in QEMU 6.2.
​
Looking at the code I can't see an obvious error.
The only doubt I have is that the qemu_rbd_diff_iterate_cb() callback is invoked by librbd with the length of the entire region allocated, not limited to the length requested by the caller, but I'll have to check.

*** Bug 2036193 has been marked as a duplicate of this bug. ***

Also hit this issue when do online snapshot whose backend is rbd image.

I easily reproduce locally on my Fedora 34 machine (QEMU - git: b5a3d8bc9146b, librbd1-16.2.7-2.fc34) following these steps:
  qemu-img create -f raw rbd:rbd/base.img 1G
  rbd map base.img --pool rbd
​
  # create a parition
  echo 'type=83' | sfdisk /dev/rbd0
  mkfs.ext4 /dev/rbd0p1
​
  mount /dev/rbd0p1 mnt
  dd if=/dev/urandom of=mnt/data bs=1M count=100 conv=fsync
  md5sum mnt/data
​
  rbd unmap /dev/rbd0
​
  qemu-img create -f qcow2 -F raw -b rbd:rbd/base.img sn.qcow2
​
  qemu-system-x86_64 -m 512M -smp 2 -M q35,accel=kvm \
    -drive file=/path/to/fedora35-vm.qcow2,format=qcow2,if=virtio \
    -drive file=/path/to/sn.qcow2,format=qcow2,if=virtio
 
  guest$ mount /dev/vdb1 /mnt
  # ../block/rbd.c:1355: int qemu_rbd_co_block_status(BlockDriverState *, _Bool, int64_t, int64_t, int64_t *, int64_t *, BlockDriverState **): Assertion `req.bytes <= bytes' failed
​
I put some logs and QEMU crashes the first time qemu_rbd_co_block_status() is invoked:
  qemu_rbd_co_block_status(..., offset=1052672, bytes=61440, ...)
  qemu_rbd_diff_iterate_cb(offs=2105344, len=61440, exists=1, ...)
  
  # qemu_rbd_diff_iterate_cb set req->bytes = offs - req->offs;
​
  # qemu_rbd_co_block_status, when rbd_diff_iterate2() returns
  r=-9000 req.offs=1052672 req.bytes=1052672 req.exists 0
  
  # assert(req.bytes <= bytes) fails because req.bytes=1052672 and bytes=61440
​
I think the assert (or the callback that set req->bytes = offs - req->offs) is wrong, since the bytes passed to rbd_diff_iterate2() I guess are interpreted as the length of the first allocated area.
​
However removing the assert and setting *pnum = min(req->bytes, bytes) causes the image (sn.qcow2) to be corrupted.
​
I need to better understand the interaction between qcow2 and rbd when used as a backing file, I don't know if in this case it makes sense that qemu_rbd_co_block_status() is called

Moving to Ilya. Please double check if this needs BZ and clones need a product/component adjustment.

The following patches just merged upstream fix this issue:
  9e302f64bb ("block/rbd: fix handling of holes in .bdrv_co_block_status")
  fc176116cd ("block/rbd: workaround for ceph issue #53784")

I'll backport them ASAP.

Test with qemu-kvm-6.2.0-7.el9, live snapshot works on with rbd data disk.

Test Steps:
 1. Start guest with qemu cmdline:
    /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/monitor-qmpmonitor1-20220208-065943-cYtOtaK4,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/monitor-catch_monitor-20220208-065943-cYtOtaK4,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idiZM03j \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/serial-serial0-20220208-065943-cYtOtaK4,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220208-065943-cYtOtaK4,path=/tmp/avocado_j6oh18rs/seabios-20220208-065943-cYtOtaK4,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220208-065943-cYtOtaK4,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=rbd_image1,driver=rbd,auto-read-only=on,discard=unmap,pool=rbd,image=rhel900-64-virtio-scsi.raw,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=rbd_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=rbd_data,driver=rbd,auto-read-only=on,discard=unmap,pool=rbd,image=data.raw,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_data,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=rbd_data \
    -device scsi-hd,id=data,drive=drive_data,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:d6:2c:c9:d8:23,id=idmofnQG,netdev=idzeOXI2,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idzeOXI2,vhost=on,vhostfd=22,fd=19  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

 2. Create snapshot node
     {'execute': 'blockdev-create', 'arguments': {'options': {'driver': 'file', 'filename': '/root/avocado/data/avocado-vt/sn1.qcow2', 'size': 104857600}, 'job-id': 'file_sn1'}, 'id': 'Gw0rF7YP'}
     {'execute': 'job-dismiss', 'arguments': {'id': 'file_sn1'}, 'id': '0FHPEJZb'}
     {'execute': 'blockdev-add', 'arguments': {'node-name': 'file_sn1', 'driver': 'file', 'filename': '/root/avocado/data/avocado-vt/sn1.qcow2', 'aio': 'threads', 'auto-read-only': True, 'discard': 'unmap'}, 'id': 'DxsXHF6s'}
     {'execute': 'blockdev-create', 'arguments': {'options': {'driver': 'qcow2', 'file': 'file_sn1', 'size': 104857600}, 'job-id': 'drive_sn1'}, 'id': 'NWk7K7bW'}
     {'execute': 'job-dismiss', 'arguments': {'id': 'drive_sn1'}, 'id': 'YnsgqFkq'}
     {'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_sn1', 'driver': 'qcow2', 'file': 'file_sn1', 'read-only': False}, 'id': 'Gz4raC2f'}
     {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_data', 'overlay': 'drive_sn1'}, 'id': 'XPhP9EPE'}

 3. After snapshot, shutdown vm, then restart it with snapshot
    /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/monitor-qmpmonitor1-20220208-065947-NfkPqkxt,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/monitor-catch_monitor-20220208-065947-NfkPqkxt,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idXYT98I \
    -chardev socket,wait=off,server=on,path=/tmp/avocado_j6oh18rs/serial-serial0-20220208-065947-NfkPqkxt,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220208-065947-NfkPqkxt,path=/tmp/avocado_j6oh18rs/seabios-20220208-065947-NfkPqkxt,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220208-065947-NfkPqkxt,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=rbd_image1,driver=rbd,auto-read-only=on,discard=unmap,pool=rbd,image=rhel900-64-virtio-scsi.raw,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=rbd_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=file_sn1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/root/avocado/data/avocado-vt/sn1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_sn1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_sn1 \
    -device scsi-hd,id=sn1,drive=drive_sn1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:d6:2c:c9:d8:24,id=idMBuGGV,netdev=idibO6Jo,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idibO6Jo,vhost=on,vhostfd=29,fd=21  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

After step3, guest start with snapshot image successfully.


Test with qemu-kvm-6.2.0-7.el9, block mirror and backup all works on with rbd data disk.
 (1/2) Host_RHEL.m9.u0.ceph.raw.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_full_mirror.simple_test.q35: PASS (121.10 s)
 (2/2) Host_RHEL.m9.u0.ceph.raw.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_full_backup_multi_disks.with_data_plane.q35: PASS (123.64 s)

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Verified this bug as below.


Tested with:
qemu-kvm-6.2.0-7.el9
kernel-5.14.0-54.kpq0.el9.x86_64


Steps:
1. Create snapshot file over RBD base system image
# qemu-img create -f qcow2 -b rbd:rbd/rhel860-64-virtio-scsi.raw -F raw /home/kvm_autotest_root/images/snA.qcow2

2. Boot the guest from the snapshot file
# /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -chardev socket,path=/tmp/avocado_chxfaduo/monitor-qmpmonitor1-20220208-225523-enugzswL,server=on,wait=off,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/tmp/avocado_chxfaduo/monitor-catch_monitor-20220208-225523-enugzswL,server=on,wait=off,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idEd2X7B \
    -chardev socket,path=/tmp/avocado_chxfaduo/serial-serial0-20220208-225523-enugzswL,server=on,wait=off,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220208-225523-enugzswL,path=/tmp/avocado_chxfaduo/seabios-20220208-225523-enugzswL,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220208-225523-enugzswL,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_snA,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/snA.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_snA,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_snA \
    -device scsi-hd,id=snA,drive=drive_snA,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:2f:40:54:79:81,id=idjzk0KY,netdev=idb0HWuQ,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idb0HWuQ,vhost=on,vhostfd=25,fd=18  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5


Results:
Boot up guest successfully.

According to comment 16, set this bug as verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: qemu-kvm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2307