Bug 1650493
Summary: | [upstream] qemu core dumped when doing incremental live backup without "bitmap" parameter by mistake in a transaction mode((blockdev-backup/block-dirty-bitmap-add/x-block-dirty-bitmap-merge) - Fast Train | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Gu Nini <ngu> | ||||
Component: | qemu-kvm | Assignee: | John Snow <jsnow> | ||||
Status: | CLOSED ERRATA | QA Contact: | aihua liang <aliang> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 8.0 | CC: | aliang, areis, chayang, coli, ddepaula, eblake, jinzhao, jsnow, juzhang, kanderso, knoel, mtessun, qzhang, rbalakri, virt-maint | ||||
Target Milestone: | rc | ||||||
Target Release: | 8.1 | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | |||||||
: | 1655864 1658426 (view as bug list) | Environment: | |||||
Last Closed: | 2019-11-06 07:12:03 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1655864, 1658426 | ||||||
Attachments: |
|
*** Bug 1655864 has been marked as a duplicate of this bug. *** This is actually merge failing to undo changes on failure. In your second transaction, you do: - bitmap-add - bitmap-merge - blockdev-backup. Of course, blockdev-backup is wrong, so we abort the transaction -- but merge fails to abort and panics. What happens here: - we undo "add", which deletes bitmap_tmp - we undo "merge" which restores bitmap_tmp's original bit pattern before the transaction, but ... oops! we already deleted it. Ah, I know which bug this is -- I've already submitted a patch for this upstream (by coincidence), but didn't realize it was able to cause problems in practice. Thank you for this report! Can you please clone this for qemu-kvm-rhev 7.7 as well, too? Upstream reference: https://lists.gnu.org/archive/html/qemu-devel/2018-12/msg01317.html (In reply to John Snow from comment #3) Thanks John, cloned as bz1658426. Verified with qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64, the bug has been fixed, set its status to "Verified". Test steps: 1.Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-221944-MrlxVzia,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-221944-MrlxVzia,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idn20piu \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-221944-MrlxVzia,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190602-221944-MrlxVzia,path=/var/tmp/seabios-20190602-221944-MrlxVzia,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190602-221944-MrlxVzia,iobase=0x402 \ -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \ -blockdev driver=file,filename=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2,node-name=file_node \ -blockdev driver=qcow2,node-name=drive_image1,file=file_node \ -device scsi-hd,id=image1,drive=drive_image1 \ -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie.0-root-port-5,addr=0x0 \ -blockdev driver=file,filename=/home/data,node-name=file_data \ -blockdev driver=qcow2,node-name=drive_data1,file=file_data \ -device scsi-hd,id=data1,drive=drive_data1 \ -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -device virtio-net-pci,mac=9a:33:34:35:36:37,id=idj01pFr,vectors=4,netdev=idMgbx8B,bus=pcie.0-root-port-4,addr=0x0 \ -netdev tap,id=idMgbx8B,vhost=on \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ 2.Create target inc image {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}} {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}} {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480},'job-id':'job2'}} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i'}} 3.Add a bitmap { "execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}} {"return": {}} 4.Do incremental live backup without "bitmap" in transaction mode { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn1", "sync": "incremental", "job-id":"j1" } },{"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap1"}}]}} {"error": {"class": "GenericError", "desc": "must provide a valid bitmap name for \"incremental\" sync mode"}} Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723 |
Created attachment 1506382 [details] gdb_debug_info_all_threads-11162018 Description of problem: After the full backup, when do the differential backup in a transaction mode(block-dirty-bitmap-add/x-block-dirty-bitmap-merge/blockdev-backup) but lack the "bitmap" parameter by a mistake, there would be following core dump issue: [root@ibm-p8-rhevm-13 home]# ./vm00.sh QEMU 3.0.91 monitor - type 'help' for more information (qemu) double free or corruption (!prev) ./vm00.sh: line 25: 95505 Aborted (core dumped) /home/qemu/ppc64-softmmu/qemu-system-ppc64 -name 'avocado-vt-vm1' -machine pseries -object secret,id=sec0,data=redhat -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado1,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6 -blockdev node-name=disk0,file.driver=file,driver=qcow2,file.filename=/home/rhel80-ppc64le-upstream.qcow2,file.auto-read-only=on -device scsi-hd,drive=disk0,id=image0,bootindex=0 -blockdev node-name=disk1,file.driver=file,driver=qcow2,file.filename=/home/hd1 -device scsi-hd,drive=disk1,id=image1,bootindex=1 -device virtio-net-pci,mac=9a:78:79:7a:7b:6a,id=id8e5D72,vectors=4,netdev=idrYUYaH,bus=pci.0,addr=0x3 -netdev tap,id=idrYUYaH,vhost=on -m 1024 -smp 2,maxcpus=2,cores=2,threads=1,sockets=1 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :20 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn,once=c -enable-kvm -monitor stdio Version-Release number of selected component (if applicable): Host kernel: 4.18.0-32.el8.ppc64le qemu: upstream v3.1.0-rc1-dirty How reproducible: 100% Steps to Reproduce: 1. Boot up the guest with a system disk and a data disk, try to do differental live backup for the data disk as following steps: -blockdev node-name=disk0,file.driver=file,driver=qcow2,file.filename=/home/rhel80-ppc64le-upstream.qcow2,file.auto-read-only=on \ -device scsi-hd,drive=disk0,id=image0,bootindex=0 \ -blockdev node-name=disk1,file.driver=file,driver=qcow2,file.filename=/home/hd1 \ -device scsi-hd,drive=disk1,id=image1,bootindex=1 \ 2. Create an image used for the full backup with blockdev-create: echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options": {"driver":"file","filename":"/home/fullbackup.qcow2","size":2147483648},"job-id":"job1"}} {"execute":"blockdev-add","arguments":{"driver":"file","node-name":"fullbackup","filename":"/home/fullbackup.qcow2"}}' | nc -U /var/tmp/avocado1 sleep 2 echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options":{"driver":"qcow2","file":"fullbackup","size":2147483648},"job-id":"job2"}}' | nc -U /var/tmp/avocado1 sleep 4 echo -e "{'execute':'qmp_capabilities'} { 'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'fbk','file':'fullbackup'}}" | nc -U /var/tmp/avocado1 sleep 4 echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job1"}}' | nc -U /var/tmp/avocado1 echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job2"}}' | nc -U /var/tmp/avocado1 3. Do full backup and create the 1st bitmap at the same time: { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "disk1", "target": "fbk", "sync": "full","job-id":"j1"} }, {"type": "block-dirty-bitmap-add", "data": { "node": "disk1", "name": "bitmap1" } } ] } } 4. Create an image used for the differental backup with blockdev-create: echo -e '{"execute":"qmp_capabilities"} {"execute":"blockdev-create","arguments":{"options": {"driver":"file","filename":"/home/inc.qcow2","size":2147483648},"job-id":"job1"}} {"execute":"blockdev-add","arguments":{"driver":"file","node-name":"incremental","filename":"/home/inc.qcow2"}}' | nc -U /var/tmp/avocado1 sleep 2 echo -e "{'execute':'qmp_capabilities'} {'execute': 'blockdev-create', 'arguments': {'job-id': 'job2', 'options': {'driver': 'qcow2', 'file': 'incremental', 'size': 2147483648, 'backing-fmt':'qcow2','backing-file':'/home/fullbackup.qcow2'}}}" | nc -U /var/tmp/avocado1 sleep 4 echo -e "{'execute':'qmp_capabilities'} { 'execute': 'blockdev-add','arguments':{'driver':'qcow2','node-name':'inc','file':'incremental','backing':'fbk'}}" | nc -U /var/tmp/avocado1 sleep 4 echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job1"}}' | nc -U /var/tmp/avocado1 echo -e '{"execute":"qmp_capabilities"} {"execute":"job-dismiss","arguments":{"id":"job2"}}' | nc -U /var/tmp/avocado1 5. Create the 2nd bitmap, do dirty bitmap merge and do differential live backup in a transaction mode at the same time(but missed "bitmap":"bitmap_tmp" by mistake): { "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": { "node": "disk1", "name": "bitmap_tmp" } }, {"type": "x-block-dirty-bitmap-merge", "data": { "node": "disk1", "src_name":"bitmap1","dst_name": "bitmap_tmp" } }, {"type": "blockdev-backup", "data": { "device": "disk1", "target": "inc", "sync": "incremental", "job-id":"j1"} } ] } } Actual results: Guest core dumped as that in the description part. Expected results: There is error prompt about missing the bitmap parameter. Additional info: If do step 5 with "bitmap":"bitmap_tmp" in 'blockdev-backup' cmd, the core dump won't occur and the block job finishes normally.