Test on qemu-kvm-4.2.0-29.module+el8.2.1+7712+3c3fe332.2 for 20 times, can't reproduce this issue.
Details: Create snapshot chain with length 11 and iothread enable by auto test, can't hit this issue. ... -object iothread,id=iothread0 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel821-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,w{'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn8', 'overlay': 'drive_sn9'}, 'id': 'sXpwrBBw'} rite-cache=on \ -blockdev node-name=file_sn10,driver=file,aio=threads,filename=/var/lib/avocado/data/avocado-vt/sn10.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_sn10,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_sn10 \ -device scsi-hd,id=sn10,drive=drive_sn10,write-cache=on \ ... {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_data1', 'overlay': 'drive_sn0'}, 'id': 'wuOhlHNJ'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn0', 'overlay': 'drive_sn1'}, 'id': 'PgAFMAuP'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn1', 'overlay': 'drive_sn2'}, 'id': 'uxVo0BLz'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn2', 'overlay': 'drive_sn3'}, 'id': 'euSdVYL4'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn3', 'overlay': 'drive_sn4'}, 'id': 'nCzebCYL'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn4', 'overlay': 'drive_sn5'}, 'id': 'VIifEExP'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn5', 'overlay': 'drive_sn6'}, 'id': 'BsvGDV6o'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn6', 'overlay': 'drive_sn7'}, 'id': 'iwMUuaNx'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn7', 'overlay': 'drive_sn8'}, 'id': '5rZQu3wq'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn8', 'overlay': 'drive_sn9'}, 'id': 'sXpwrBBw'} {'execute': 'blockdev-snapshot', 'arguments': {'node': 'drive_sn9', 'overlay': 'drive_sn10'}, 'id': 'pf0xpTfc'} (01/20) repeat1.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (168.90 s) (02/20) repeat2.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (171.83 s) (03/20) repeat3.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (165.35 s) (04/20) repeat4.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (164.45 s) (05/20) repeat5.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (164.74 s) (06/20) repeat6.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (166.24 s) (07/20) repeat7.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (171.87 s) (08/20) repeat8.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (173.18 s) (09/20) repeat9.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (162.99 s) (10/20) repeat10.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (165.92 s) (11/20) repeat11.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (168.26 s) (12/20) repeat12.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (167.72 s) (13/20) repeat13.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (170.33 s) (14/20) repeat14.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (169.58 s) (15/20) repeat15.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (171.56 s) (16/20) repeat16.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (173.00 s) (17/20) repeat17.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (170.65 s) (18/20) repeat18.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (173.43 s) (19/20) repeat19.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (167.46 s) (20/20) repeat20.Host_RHEL.m8.u2.product_av.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.2.1.x86_64.io-github-autotest-qemu.blockdev_snapshot_chains.q35: PASS (164.03 s)
Fixing the product to RHEL Advanced Virtualization
To clarify, we are talking about RHV snapshots operations, not RHEL-KVM snapshots. It is when RHV takes the qcow2 image and creates another one and links the two under same vm disk, if to explain in one sentence.
Sergio, anything else we need from the customer to help reproducing this or maybe something to deploy on customer's site? Or you have enough information via the coredump?
(In reply to Marina Kalinin from comment #6) > To clarify, we are talking about RHV snapshots operations, not RHEL-KVM > snapshots. > It is when RHV takes the qcow2 image and creates another one and links the > two under same vm disk, if to explain in one sentence. Hi Niny, Could you please have a try? Thanks.
Try to reproduce it again, still fail to reproduct it. Test env: kernel version:4.18.0-193.el8.x86_64 qemu-kvm version:qemu-kvm-4.2.0-29.module+el8.2.1+7712+3c3fe332.2 Guest: RHEL7.6 Machine type: pc Driveformat: virtio_blk Data plane: enable Test Steps: 1. Start guest with qemu cmd: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine pc-i440fx-rhel7.6.0 \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -m 30720 \ -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2 \ -cpu 'Cascadelake-Server',+kvm_pv_unhalt \ -chardev socket,server=on,id=qmp_id_qmpmonitor1,wait=off,path=/tmp/monitor-qmpmonitor1-20210510-045626-P2QqGJ5E \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,server=on,id=qmp_id_catch_monitor,wait=off,path=/tmp/monitor-catch_monitor-20210510-045626-P2QqGJ5E \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idcKGxjw \ -chardev socket,server=on,id=chardev_serial0,wait=off,path=/tmp/serial-serial0-20210510-045626-P2QqGJ5E \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20210510-045626-P2QqGJ5E,path=/tmp/seabios-20210510-045626-P2QqGJ5E,server=on,wait=off \ -device isa-debugcon,chardev=seabioslog_id_20210510-045626-P2QqGJ5E,iobase=0x402 \ -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object iothread,id=iothread0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel76-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pci.0,addr=0x4,iothread=iothread0,scsi=off,write-cache=on \ -device virtio-net-pci,mac=9a:0a:a8:f0:c5:08,id=id2V3RTk,netdev=id0s6sUn,bus=pci.0,addr=0x6 \ -netdev tap,id=id0s6sUn,vhost=on \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ 2. Create a snapshot node, and do snapshot. #create snapshot node {'execute':'qmp_capabilities' } {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn1','size':21474836480},'job-id':'job1'}} {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn1','filename':'/root/sn1'}} {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn1','size':21474836480,'backing-file':'/home/kvm_autotest_root/images/rhel76-64-virtio.qcow2','backing-fmt':'qcow2'},'job-id':'job2'}} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn1','file':'drive_sn1','backing':null}} {'execute':'job-dismiss','arguments':{'id':'job1'}} {'execute':'job-dismiss','arguments':{'id':'job2'}} {'execute':'blockdev-snapshot','arguments':{'node':'drive_image1','overlay':'sn1'}} 3. Create target snapshot image and backing image. #qemu-img create -f qcow2 mirror.qcow2 20G #qemu-img create -f qcow2 mirror_sn.qcow2 -b mirror.qcow2 20G 4. DD file in guest. #dd if=/dev/urandom of=test bs=1M count=1000 5. Add target snapshot mirror node and do mirror with sync=top. {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_mirror_sn','filename':'/home/mirror_sn.qcow2'}} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'mirror_sn','file':'drive_mirror_sn','backing':null}} {"execute": "blockdev-mirror", "arguments": {"sync": "top", "device": "sn1","target":"mirror_sn","job-id": "j1"}} {"timestamp": {"seconds": 1620645242, "microseconds": 859775}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}} {"timestamp": {"seconds": 1620645242, "microseconds": 859855}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}} {"return": {}} {"timestamp": {"seconds": 1620645242, "microseconds": 884363}, "event": "JOB_STATUS_CHANGE", "data": {"status": "ready", "id": "j1"}} {"timestamp": {"seconds": 1620645242, "microseconds": 884430}, "event": "BLOCK_JOB_READY", "data": {"device": "j1", "len": 21757952, "offset": 21757952, "speed": 0, "type": "mirror"}} 6. Convert backing image from src to target. #qemu-img convert /home/kvm_autotest_root/images/rhel76-64-virtio.qcow2 /home/mirror.qcow2 -O qcow2 7. Do snapshot in dst. {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_mirror','filename':'/home/mirror.qcow2'}} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'mirror','file':'drive_mirror'}} {"execute":"blockdev-snapshot","arguments":{"node":"mirror","overlay":"mirror_sn"}} 8. Complete mirror job {"execute":"job-complete","arguments":{"id":"j1"}} {"return": {}} {"timestamp": {"seconds": 1620645305, "microseconds": 139017}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}} {"timestamp": {"seconds": 1620645305, "microseconds": 139062}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}} {"timestamp": {"seconds": 1620645305, "microseconds": 139161}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 22085632, "offset": 22085632, "speed": 0, "type": "mirror"}} {"timestamp": {"seconds": 1620645305, "microseconds": 139201}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}} {"timestamp": {"seconds": 1620645305, "microseconds": 139230}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}} 9. Check block info (qemu)info block mirror_sn: json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "/home/mirror.qcow2"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/home/mirror_sn.qcow2"}} (qcow2) Attached to: /machine/peripheral/image1/virtio-backend Cache mode: writeback Backing file: /home/mirror.qcow2 (chain depth: 1) 10. Quit vm (qemu)system_powerdown After step10, vm shutdown successfully. Repeat 10 times from step1 to step10, can't reproduce this issue. Hi, Sergio Can you help to check if the reproduce scenario is correct, thanks. And Hi, Anitha Do we have updates from customer now? Is this case still active? Thanks. I noticed the issue in customer portal is closed.
*** Bug 1967895 has been marked as a duplicate of this bug. ***
Run regression test on qemu-kvm-6.0.0-21.module+el8.5.0+11555+e0ab0d09, all test pass. Logs link: 1. virtio_blk+iothread+storage_vm_migration_blockdev: http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/?jobid=5505620 2. virtio_scsi+iothread+storage_vm_migration_blockdev: http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/?jobid=5500885
As comment 26, set bug's status to "Verified".
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4684