Bug 1650493 - [upstream] qemu core dumped when doing incremental live backup without "bitmap" parameter by mistake in a transaction mode((blockdev-backup/block-dirty-bitmap-add/x-block-dirty-bitmap-merge) - Fast Train
Summary: [upstream] qemu core dumped when doing incremental live backup without "bitma...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: 8.1
Assignee: John Snow
QA Contact: aihua liang
URL:
Whiteboard:
: 1655864 (view as bug list)
Depends On:
Blocks: 1655864 1658426
TreeView+ depends on / blocked
 
Reported: 2018-11-16 10:42 UTC by Gu Nini
Modified: 2019-11-06 07:12 UTC (History)
15 users (show)

Fixed In Version: qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1655864 1658426 (view as bug list)
Environment:
Last Closed: 2019-11-06 07:12:03 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
gdb_debug_info_all_threads-11162018 (16.75 KB, text/plain)
2018-11-16 10:42 UTC, Gu Nini
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:12:47 UTC

Description Gu Nini 2018-11-16 10:42:51 UTC
Created attachment 1506382 [details]
gdb_debug_info_all_threads-11162018

Description of problem:

After the full backup, when do the differential backup in a transaction mode(block-dirty-bitmap-add/x-block-dirty-bitmap-merge/blockdev-backup) but lack the "bitmap" parameter by a mistake, there would be following core dump issue:


[root@ibm-p8-rhevm-13 home]# ./vm00.sh
QEMU 3.0.91 monitor - type 'help' for more information
(qemu) double free or corruption (!prev)
./vm00.sh: line 25: 95505 Aborted                 (core dumped) /home/qemu/ppc64-softmmu/qemu-system-ppc64 -name 'avocado-vt-vm1' -machine pseries -object secret,id=sec0,data=redhat -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado1,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -blockdev node-name=disk0,file.driver=file,driver=qcow2,file.filename=/home/rhel80-ppc64le-upstream.qcow2,file.auto-read-only=on -device scsi-hd,drive=disk0,id=image0,bootindex=0 -blockdev node-name=disk1,file.driver=file,driver=qcow2,file.filename=/home/hd1 -device scsi-hd,drive=disk1,id=image1,bootindex=1 -device virtio-net-pci,mac=9a:78:79:7a:7b:6a,id=id8e5D72,vectors=4,netdev=idrYUYaH,bus=pci.0,addr=0x3 -netdev tap,id=idrYUYaH,vhost=on -m 1024 -smp 2,maxcpus=2,cores=2,threads=1,sockets=1 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :20 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio


Version-Release number of selected component (if applicable):
Host kernel: 4.18.0-32.el8.ppc64le
qemu: upstream v3.1.0-rc1-dirty


How reproducible:
100%

Steps to Reproduce:
1. Boot up the guest with a system disk and a data disk, try to do differental live backup for the data disk as following steps:
    -blockdev node-name=disk0,file.driver=file,driver=qcow2,file.filename=/home/rhel80-ppc64le-upstream.qcow2,file.auto-read-only=on \
    -device scsi-hd,drive=disk0,id=image0,bootindex=0 \
    -blockdev node-name=disk1,file.driver=file,driver=qcow2,file.filename=/home/hd1 \
    -device scsi-hd,drive=disk1,id=image1,bootindex=1 \

2. Create an image used for the full backup with blockdev-create:
echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options": {"driver":"file","filename":"/home/fullbackup.qcow2","size":2147483648},"job-id":"job1"}} {"execute":"blockdev-add","arguments":{"driver":"file","node-name":"fullbackup","filename":"/home/fullbackup.qcow2"}}' | nc -U /var/tmp/avocado1
sleep 2
echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options":{"driver":"qcow2","file":"fullbackup","size":2147483648},"job-id":"job2"}}' | nc -U /var/tmp/avocado1
sleep 4
echo -e "{'execute':'qmp_capabilities'} { 'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'fbk','file':'fullbackup'}}" | nc -U /var/tmp/avocado1
sleep 4
echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job1"}}' | nc -U /var/tmp/avocado1
echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job2"}}' | nc -U /var/tmp/avocado1

3. Do full backup and create the 1st bitmap at the same time:
{ "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "disk1", "target": "fbk", "sync": "full","job-id":"j1"} }, {"type": "block-dirty-bitmap-add", "data": { "node": "disk1", "name": "bitmap1" } } ] } }

4. Create an image used for the differental backup with blockdev-create:
echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options": {"driver":"file","filename":"/home/inc.qcow2","size":2147483648},"job-id":"job1"}} {"execute":"blockdev-add","arguments":{"driver":"file","node-name":"incremental","filename":"/home/inc.qcow2"}}' | nc -U /var/tmp/avocado1
sleep 2
echo -e "{'execute':'qmp_capabilities'} {'execute': 'blockdev-create', 'arguments': {'job-id': 'job2', 'options': {'driver': 'qcow2', 'file': 'incremental', 'size': 2147483648, 'backing-fmt':'qcow2','backing-file':'/home/fullbackup.qcow2'}}}" | nc -U /var/tmp/avocado1
sleep 4
echo -e "{'execute':'qmp_capabilities'} { 'execute': 'blockdev-add','arguments':{'driver':'qcow2','node-name':'inc','file':'incremental','backing':'fbk'}}" | nc -U /var/tmp/avocado1
sleep 4
echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job1"}}' | nc -U /var/tmp/avocado1
echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job2"}}' | nc -U /var/tmp/avocado1

5. Create the 2nd bitmap, do dirty bitmap merge and do differential live backup in a transaction mode at the same time(but missed "bitmap":"bitmap_tmp" by mistake):
{ "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": { "node": "disk1", "name": "bitmap_tmp" } }, {"type": "x-block-dirty-bitmap-merge", "data": { "node": "disk1", "src_name":"bitmap1","dst_name": "bitmap_tmp" } }, {"type": "blockdev-backup", "data": { "device": "disk1", "target": "inc", "sync": "incremental", "job-id":"j1"} }  ] } }


Actual results:
Guest core dumped as that in  the description part.

Expected results:
There is error prompt about missing the bitmap parameter.

Additional info:
If do step 5 with "bitmap":"bitmap_tmp" in 'blockdev-backup' cmd, the core dump won't occur and the block job finishes normally.

Comment 2 aihua liang 2018-12-04 09:17:41 UTC
*** Bug 1655864 has been marked as a duplicate of this bug. ***

Comment 3 John Snow 2018-12-12 00:48:15 UTC
This is actually merge failing to undo changes on failure.

In your second transaction, you do:
- bitmap-add
- bitmap-merge
- blockdev-backup.

Of course, blockdev-backup is wrong, so we abort the transaction -- but merge fails to abort and panics.

What happens here:
- we undo "add", which deletes bitmap_tmp
- we undo "merge" which restores bitmap_tmp's original bit pattern before the transaction, but ... oops! we already deleted it.

Ah, I know which bug this is -- I've already submitted a patch for this upstream (by coincidence), but didn't realize it was able to cause problems in practice. Thank you for this report!
Can you please clone this for qemu-kvm-rhev 7.7 as well, too?

Upstream reference: https://lists.gnu.org/archive/html/qemu-devel/2018-12/msg01317.html

Comment 4 Gu Nini 2018-12-12 03:00:54 UTC
(In reply to John Snow from comment #3)

Thanks John, cloned as bz1658426.

Comment 8 aihua liang 2019-06-06 09:50:31 UTC
Verified with qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64, the bug has been fixed, set its status to "Verified".

Test steps:
  1.Start guest with qemu cmds:
     /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idn20piu  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-221944-MrlxVzia,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190602-221944-MrlxVzia,path=/var/tmp/seabios-20190602-221944-MrlxVzia,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190602-221944-MrlxVzia,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -blockdev driver=file,filename=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2,node-name=file_node \
    -blockdev driver=qcow2,node-name=drive_image1,file=file_node \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie.0-root-port-5,addr=0x0 \
    -blockdev driver=file,filename=/home/data,node-name=file_data \
    -blockdev driver=qcow2,node-name=drive_data1,file=file_data \
    -device scsi-hd,id=data1,drive=drive_data1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:33:34:35:36:37,id=idj01pFr,vectors=4,netdev=idMgbx8B,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idMgbx8B,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \


  2.Create target inc image
     {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}}
     {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}}
     {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480},'job-id':'job2'}}
     {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i'}}

  3.Add a bitmap
     { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}}
{"return": {}}

  4.Do incremental live backup without "bitmap" in transaction mode
     { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn1", "sync": "incremental", "job-id":"j1" } },{"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap1"}}]}}
{"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for \"incremental\" sync mode"}}

Comment 10 errata-xmlrpc 2019-11-06 07:12:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.