Bug 1750322

Summary: qemu aborts in blockCommit: qemu-kvm: block.c:3486 [rhel-7.7.z]
Product: Red Hat Enterprise Linux 7 Reporter: RAD team bot copy to z-stream <autobot-eus-copy>
Component: qemu-kvm-rhevAssignee: Kevin Wolf <kwolf>
Status: CLOSED ERRATA QA Contact: aihua liang <aliang>
Severity: high Docs Contact:
Priority: high    
Version: 7.7CC: aefrat, aliang, bcholler, bugs, coli, gwatson, jinzhao, juzhang, kwolf, mkalinin, mrezanin, mtessun, nsoffer, virt-maint
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-33.el7_7.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1711643 Environment:
Last Closed: 2019-10-22 15:24:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description RAD team bot copy to z-stream 2019-09-09 10:26:19 UTC 
This bug has been copied from bug #1711643 and has been proposed to be backported to 7.7 z-stream (EUS).
Comment 7 aihua liang 2019-09-19 08:04:46 UTC 
Can reproduce it on qemu-kvm-rhev-2.12.0-29.el7.x86_64.
 
 Reproduce steps:
   1.Create images sn1 ~ sn5 
     for i in range (1,6)
     #qemu-img create -f qcow2 /root/sn$i 20G

   2.Start guest with qemu cmds:
      /usr/libexec/qemu-kvm \
    -object throttle-group,x-bps-read=4096,id=throttle0 \
    -blockdev file,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,node-name=file_base \
    -blockdev qcow2,file=file_base,node-name=base \
    -blockdev throttle,throttle-group=throttle0,file=base,node-name=throttled \
    -blockdev file,filename=/root/sn1,node-name=root_sn1 \
    -blockdev qcow2,file=root_sn1,node-name=sn1,backing=throttled \
    -blockdev file,filename=/root/sn2,node-name=root_sn2 \
    -blockdev qcow2,file=root_sn2,node-name=sn2,backing=sn1 \
    -blockdev file,filename=/root/sn3,node-name=root_sn3 \
    -blockdev qcow2,file=root_sn3,node-name=sn3,backing=sn2 \
    -blockdev file,filename=/root/sn4,node-name=root_sn4 \
    -blockdev qcow2,file=root_sn4,node-name=sn4,backing=sn3 \
    -blockdev file,filename=/root/sn5,node-name=root_sn5 \
    -blockdev qcow2,file=root_sn5,node-name=sn5,backing=sn4 \
    -device ide-hd,drive=sn5 \
    -m 8192  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -vnc :0 \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

  3.During guest boot, do block commit
    {'execute':'block-commit', 'arguments': {'device': 'sn5', 'top-node': 'sn4','job-id':'j1'}}
{"timestamp": {"seconds": 1568879801, "microseconds": 692671}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
Connection closed by foreign host.

  After step3:
    (qemu) qemu-kvm: block.c:3508: bdrv_replace_node: Assertion `!({ _Static_assert(!(sizeof(*&from->in_flight) > 8), "not expecting: " "sizeof(*&from->in_flight) > ATOMIC_REG_SIZE"); __atomic_load_n(&from->in_flight, 0); })' failed.
tat.txt: Line 26: 30297 Aborted               (Coredump)/usr/libexec/qemu-kvm -object throttle-group,x-bps-read=4096,id=throttle0 -blockdev file,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,node-name=file_base...
Comment 8 aihua liang 2019-09-19 09:16:07 UTC 
Verified on qemu-kvm-rhev-2.12.0-33.el7_7.4.x86_64, the issue has been fixed, but there's a new issue.
   
   1.Create images sn1 ~ sn5 
     for i in range (1,6)
     #qemu-img create -f qcow2 /root/sn$i 20G

   2.Start guest with qemu cmds:
      /usr/libexec/qemu-kvm \
    -object throttle-group,x-bps-read=4096,id=throttle0 \
    -blockdev file,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,node-name=file_base \
    -blockdev qcow2,file=file_base,node-name=base \
    -blockdev throttle,throttle-group=throttle0,file=base,node-name=throttled \
    -blockdev file,filename=/root/sn1,node-name=root_sn1 \
    -blockdev qcow2,file=root_sn1,node-name=sn1,backing=throttled \
    -blockdev file,filename=/root/sn2,node-name=root_sn2 \
    -blockdev qcow2,file=root_sn2,node-name=sn2,backing=sn1 \
    -blockdev file,filename=/root/sn3,node-name=root_sn3 \
    -blockdev qcow2,file=root_sn3,node-name=sn3,backing=sn2 \
    -blockdev file,filename=/root/sn4,node-name=root_sn4 \
    -blockdev qcow2,file=root_sn4,node-name=sn4,backing=sn3 \
    -blockdev file,filename=/root/sn5,node-name=root_sn5 \
    -blockdev qcow2,file=root_sn5,node-name=sn5,backing=sn4 \
    -device ide-hd,drive=sn5 \
    -m 8192  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -vnc :0 \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

  3.During guest boot, do block commit from sn4 to base.
    {'execute':'block-commit', 'arguments': {'device': 'sn5', 'top-node': 'sn4','job-id':'j1'}}
{"timestamp": {"seconds": 1568881186, "microseconds": 588417}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "j1"}}
{"timestamp": {"seconds": 1568881186, "microseconds": 646349}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "j1"}}
{"return": {}}
{"timestamp": {"seconds": 1568881187, "microseconds": 347398}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "j1"}}
{"timestamp": {"seconds": 1568881187, "microseconds": 347460}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "j1"}}
{"timestamp": {"seconds": 1568881187, "microseconds": 347739}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "j1", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "commit"}}
{"timestamp": {"seconds": 1568881187, "microseconds": 347800}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
{"timestamp": {"seconds": 1568881187, "microseconds": 347847}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}

   4. Do block commit from sn5 to base.
    {'execute':'block-commit', 'arguments': {'device': 'sn5','job-id':'j1'}}
Connection closed by foreign host.
  
   After step4, qemu coredump with info:
     (qemu) qemu-kvm: block/throttle.c:214: throttle_co_drain_end: Assertion `tgm->io_limits_disabled' failed.
tat.txt: line 26: 30790 Aborted               (Coredump)/usr/libexec/qemu-kvm -object throttle-group,x-bps-read=4096,id=throttle0 -blockdev file,filename=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2,node-name=file_base -blockdev qcow2,file=file_base,node-name=base -blockdev throttle,throttle-group=throttle0,file=base,node-name=throttled ...

    #gdb -c core.30790
    (gdb) bt
#0  0x00007f7171f64377 in raise () at /lib64/libc.so.6
#1  0x00007f7171f65a68 in abort () at /lib64/libc.so.6
#2  0x00007f7171f5d196 in __assert_fail_base () at /lib64/libc.so.6
#3  0x00007f7171f5d242 in  () at /lib64/libc.so.6
#4  0x000055f83ce3bdd1 in throttle_co_drain_end (bs=<optimized out>) at block/throttle.c:214
#5  0x000055f83ce26bce in bdrv_drain_invoke_entry (opaque=0x55f841c3ea80) at block/io.c:197
#6  0x000055f83cec5d6a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>)
    at util/coroutine-ucontext.c:116
#7  0x00007f7171f76180 in __start_context () at /lib64/libc.so.6
#8  0x00007ffe8e928340 in  ()
#9  0x0000000000000000 in  ()
Comment 9 aihua liang 2019-09-19 09:24:05 UTC 
Will track the new issue by a new bug, and set this bug's status to "Verified".
Comment 11 errata-xmlrpc 2019-10-22 15:24:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3179