Bug 1683498 - Failed to do incremental live backup on multi-disks in transaction mode with one disk's data_plane enabled
Summary: Failed to do incremental live backup on multi-disks in transaction mode with ...
Status: POST
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: John Snow
QA Contact: aihua liang
URL:
Whiteboard:
Keywords:
: 1683514 (view as bug list)
Depends On:
Blocks: 1683514
TreeView+ depends on / blocked
 
Reported: 2019-02-27 02:32 UTC by aihua liang
Modified: 2019-07-09 01:21 UTC (History)
8 users (show)

(edit)
Clone Of:
: 1683514 (view as bug list)
(edit)
Last Closed:


Attachments (Terms of Use)

Description aihua liang 2019-02-27 02:32:59 UTC
Description of problem:
  Failed to do incremental live backup on multi-disks in transaction mode with one disk's data_plane enabled

Version-Release number of selected component (if applicable):
   kernel version:3.10.0-993.el7.x86_64
   qemu-kvm version:qemu-kvm-rhev-2.12.0-23.el7.x86_64

How reproducible:
  100%

Steps to Reproduce:
1.Start guest in paused status with two disks, one disk with data_plane enabled.
    /usr/libexec/qemu-kvm \
    -S \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id8Ec4Bn  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel77-64-virtio.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,iothread=iothread0 \
    -blockdev driver=file,node-name=file_data,filename=/home/data1.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_data,node-name=drive_data1,auto-read-only=on \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
    -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0  \
    -netdev tap,id=idVpZZ6A,vhost=on \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \

2. Create images sn1, sn3 for full backup
     {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}}
     {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}}
     {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480},'job-id':'job2'}}
     {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i'}}
     {'execute':'job-dismiss','arguments':{'id':'job1'}}
     {'execute':'job-dismiss','arguments':{'id':'job2'}}

3. Do full backup and create bitmaps: bitmap1 and bitmap2
    { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn1", "sync": "full", "job-id":"j1" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap1" } },{"type": "blockdev-backup", "data": { "device": "drive_data1", "target": "sn3", "sync": "full", "job-id":"j2" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_data1", "name": "bitmap2"}}]}}

4. Continue guest, wait for some minutes, then create images sn2,sn4 for incremental live backup.
    (qemu)cont
    {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}}
    {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}}
    {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480,'backing-file':'/root/sn$j','backing-fmt':'qcow2'},'job-id':'job2'}}
    {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i','backing':'sn$j'}}
    {'execute':'job-dismiss','arguments':{'id':'job1'}}
    {'execute':'job-dismiss','arguments':{'id':'job2'}}

5. Do incremental live backup.
   { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn2", "sync": "incremental", "bitmap":"bitmap1", "job-id":"j1" } },{"type": "blockdev-backup", "data": { "device": "drive_data1", "target": "sn4", "sync": "incremental", "bitmap":"bitmap2", "job-id":"j2"}}]}}   

Actual results:
 After step5, incremental live backup failed with info:
   {"error": {"class": "GenericError", "desc": "Backup between two IO threads is not implemented"}}

Expected results:
   Incremental live backup can executed successfully.


Additional info:
   When disable data_plane on disk, incremental live backup can executed successfully.
   Don't hit this issue when do incremental live backup on only one disk.

Comment 2 John Snow 2019-03-15 15:28:30 UTC
I think this is supposed to be filed against qemu-kvm-rhev; changing component. (Keeping as new because I still need to investigate.)

Comment 3 John Snow 2019-05-06 20:13:49 UTC
Confirmed upstream and I have a tentative fix.

Comment 4 John Snow 2019-05-06 20:56:34 UTC
This might be the appropriate fix: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg01243.html

Comment 5 John Snow 2019-06-05 23:09:37 UTC
fix merged upstream; but we've missed the deadline for 7.7.

Comment 7 John Snow 2019-06-06 21:02:51 UTC
Staged upstream: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg02613.html

Comment 8 John Snow 2019-07-08 20:33:19 UTC
I think we only need one copy of this bug now, for 8.1-AV (So after I post this, I will go close #1683514)

Expected to be picked up in rebase to qemu 4.1 with commit:

d81e1efbea7d19c2f142d300df56538c73800590 blockdev-backup: don't check aio_context too early

Comment 9 John Snow 2019-07-08 20:34:32 UTC
*** Bug 1683514 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.