Bug 1776808

Summary: qemu-kvm crashes when Windows VM is migrated with multiqueue
Product: Red Hat Enterprise Linux 8 Reporter: Dr. David Alan Gilbert <dgilbert>
Component: qemu-kvmAssignee: Dr. David Alan Gilbert <dgilbert>
qemu-kvm sub component: General QA Contact: Li Xiaohui <xiaohli>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: ddepaula, hhuang, jinzhao, juzhang, rbalakri, rduda, rh-spice-bugs, virt-maint, xiaohli
Version: 8.2Keywords: Regression, TestBlocker
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-2.12.0-92.module+el8.2.0+5014+5115d99d Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1775251 Environment:
Last Closed: 2020-04-28 15:33:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1775251    
Bug Blocks:    

Comment 1 Dr. David Alan Gilbert 2019-11-26 12:26:06 UTC
Since 8.x !AV is the same code as RHEL7-rhev, clone for the same fix.

Comment 3 Li Xiaohui 2019-11-27 06:00:15 UTC
Hi Dave,
I test on the rhel8.2-Non-AV hosts(kernel-4.18.0-151.el8.x86_64&qemu-img-2.12.0-90.module+el8.2.0+4652+138f0dc3.x86_64), couldn't hit this issue.
But could reproduce bz on rhel 7.8 hosts(kernel-3.10.0-1111.el7.x86_64&qemu-kvm-rhev-2.12.0-38.el7.x86_64) using same commands as below:
**********************************
/usr/libexec/qemu-kvm  \
-name "mouse-vm" \
-machine q35 \
-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-m 8G \
-smp 8 \
-nodefaults  \
-device VGA \
-device pcie-root-port,id=root0,slot=0,addr=0x3 \
-device pcie-root-port,id=root1,slot=1,addr=0x4 \
-device pcie-root-port,id=root2,slot=2,addr=0x5 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=root1 \
-device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0,channel=0,scsi-id=0,lun=0,bootindex=0 \
-device virtio-net-pci,mac=9a:8a:8b:8c:8d:8e,id=net0,vectors=4,netdev=tap0,bus=root2,mq=on \
-blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/mnt/nfs/win10-64-virtio-scsi-tm1120.qcow2,node-name=drive_sys1 \
-blockdev driver=qcow2,node-name=drive_image1,file=drive_sys1 \
-netdev tap,id=tap0,vhost=on,queues=3 \
-vnc :10 \
-rtc base=utc,clock=host \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm  \
-qmp tcp:0:3333,server,nowait \
-serial tcp:0:4444,server,nowait \
-monitor stdio \

Comment 4 Li Xiaohui 2019-11-27 06:18:12 UTC
test on same rhel8.2-non-AV hosts with qemu-img-2.12.0-91.module+el8.2.0+4916+02633b66.x86_64, didn't reproduce.

Comment 5 Li Xiaohui 2019-11-27 08:44:19 UTC
Ok, I can reproduce bz on qemu-img-2.12.0-91.module+el8.2.0+4916+02633b66.x86_64 with "-smp 2 -netdev tap,id=tap0,vhost=on,queues=3 \ -device virtio-net-pci,mac=9a:8a:8b:8c:8d:8e,id=net0,vectors=4,netdev=tap0,bus=root2,mq=on"

Comment 8 Li Xiaohui 2019-12-10 10:52:28 UTC
differ from clone bz 1775251(always reproduce with mq=on&queues=$num), it didn't always reproduce on rhel8.2-non-av, so here do migration by running 100 times base_test_common script with mq=on&queues=$num in command line. 
Will check the result after finishing running.

Comment 9 Li Xiaohui 2019-12-12 10:14:21 UTC
Ok, found the right reproduction on rhel8.2 hosts: 
1) always hit this issue to start migration when the win guest is starting(just the moment that guest is starting the NIC up)
2) couldn't reproduce bz to start migration after guest has started.

Run base_test_common_win script with mq=on&queues=3:
1)on qemu-kvm-2.12.0-91.module+el8.2.0+4916+02633b66.x86_64, after tried 3 times, all failed with qemu core dump;
2)on qemu-img-2.12.0-92.module+el8.2.0+5014+5115d99d.x86_64, run 30 times, all Pass:
**********************************************************************************************
RESULTS [RHEL7-47114-WIN-Q35-BLOCKDEV]:
==>TOTAL : 30
==>PASS : 30 
   1: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
   2: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
   3: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
   4: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
  ...
   28: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
   29: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
   30: BASE-TEST-COMMON-WIN-Migration basic precopy test without setting downtime and speed (1 min 40 sec)
==>ERROR : 0 
==>RUN TIME : 40 min 3 sec 
==>TEST LOG : /home/ipa/test_logs/rhel7_47114_win_q35_blockdev-2019-12-12-04:17:59 
**********************************************************************************************

So make this bz verified via above tests

Comment 10 Ademar Reis 2020-02-05 23:09:04 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 12 errata-xmlrpc 2020-04-28 15:33:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1587