Bug 1683498

I think this is supposed to be filed against qemu-kvm-rhev; changing component. (Keeping as new because I still need to investigate.)

Confirmed upstream and I have a tentative fix.

This might be the appropriate fix: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg01243.html

fix merged upstream; but we've missed the deadline for 7.7.

Staged upstream: https://lists.gnu.org/archive/html/qemu-devel/2019-05/msg02613.html

I think we only need one copy of this bug now, for 8.1-AV (So after I post this, I will go close #1683514)

Expected to be picked up in rebase to qemu 4.1 with commit:

d81e1efbea7d19c2f142d300df56538c73800590 blockdev-backup: don't check aio_context too early

*** Bug 1683514 has been marked as a duplicate of this bug. ***

Verified on qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1.x86_64, the problem has been resolved, set bug's status to "Verified".

Test steps:
  1.Start guest in paused status with two disks, one disk with data_plane enabled.
    /usr/libexec/qemu-kvm \
    -S \
    -name 'avocado-vt-vm1' \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190123-032240-rOoB4cgD,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id8Ec4Bn  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190123-032240-rOoB4cgD,path=/var/tmp/seabios-20190123-032240-rOoB4cgD,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190123-032240-rOoB4cgD,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -object iothread,id=iothread0 \
    -blockdev driver=file,node-name=file_base,filename=/home/kvm_autotest_root/images/rhel77-64-virtio.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,iothread=iothread0 \
    -blockdev driver=file,node-name=file_data,filename=/home/data1.qcow2,auto-read-only=on \
    -blockdev driver=qcow2,file=file_data,node-name=drive_data1,auto-read-only=on \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
    -device virtio-net-pci,mac=9a:39:3a:3b:3c:3d,id=id1JNQsL,vectors=4,netdev=idVpZZ6A,bus=pci.0  \
    -netdev tap,id=idVpZZ6A,vhost=on \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \

2. Create images sn1, sn3 for full backup
     {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}}
     {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}}
     {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480},'job-id':'job2'}}
     {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i'}}
     {'execute':'job-dismiss','arguments':{'id':'job1'}}
     {'execute':'job-dismiss','arguments':{'id':'job2'}}

3. Do full backup and create bitmaps: bitmap1 and bitmap2
    { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn1", "sync": "full", "job-id":"j1" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_image1", "name": "bitmap1" } },{"type": "blockdev-backup", "data": { "device": "drive_data1", "target": "sn3", "sync": "full", "job-id":"j2" } }, {"type": "block-dirty-bitmap-add", "data": { "node": "drive_data1", "name": "bitmap2"}}]}}

4. Continue guest, wait for some minutes, then create images sn2,sn4 for incremental live backup.
    (qemu)cont
    {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn$i','size':21474836480},'job-id':'job1'}}
    {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn$i','filename':'/root/sn$i'}}
    {'execute':'blockdev-create','arguments':{'options': {'driver': 'qcow2','file':'drive_sn$i','size':21474836480,'backing-file':'/root/sn$j','backing-fmt':'qcow2'},'job-id':'job2'}}
    {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn$i','file':'drive_sn$i','backing':'sn$j'}}
    {'execute':'job-dismiss','arguments':{'id':'job1'}}
    {'execute':'job-dismiss','arguments':{'id':'job2'}}

5. Do incremental live backup.
   { "execute": "transaction", "arguments": { "actions": [ {"type": "blockdev-backup", "data": { "device": "drive_image1", "target": "sn2", "sync": "incremental", "bitmap":"bitmap1", "job-id":"j1" } },{"type": "blockdev-backup", "data": { "device": "drive_data1", "target": "sn4", "sync": "incremental", "bitmap":"bitmap2", "job-id":"j2"}}]}}
   tamp": {"seconds": 1565935317, "microseconds": 578619}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j2", "len": 2147483648, "offset": 2147483648, "speed": 0, "type": "backup"}}
{"timestamp": {"seconds": 1565935317, "microseconds": 578658}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j2"}}
{"timestamp": {"seconds": 1565935317, "microseconds": 578697}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j2"}}

{"timestamp": {"seconds": 1565935319, "microseconds": 353422}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}}
{"timestamp": {"seconds": 1565935319, "microseconds": 353466}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}}
{"timestamp": {"seconds": 1565935319, "microseconds": 353499}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "backup"}}
{"timestamp": {"seconds": 1565935319, "microseconds": 353556}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
{"timestamp": {"seconds": 1565935319, "microseconds": 353603}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}

6. Rebase sn3 to sn1, rebase sn4 to sn2, then start with sn2,sn4.
   #qemu-img rebase -u -b /root/sn1 /root/sn3
   #qemu-img rebase -u -b /root/sn2 /root/sn4
 
   ...
   -object iothread,id=iothread0 \
    -blockdev driver=file,node-name=file_base,filename=/root/sn3,auto-read-only=on \
    -blockdev driver=qcow2,file=file_base,node-name=drive_image1,auto-read-only=on \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,iothread=iothread0 \
    -blockdev driver=file,node-name=file_data,filename=/root/sn4,auto-read-only=on \
    -blockdev driver=qcow2,file=file_data,node-name=drive_data1,auto-read-only=on \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
   ...

After step6, vm can start successfully.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723