Description of problem: [RHEL8.1.1][vfio-vf] hotplug/hotunplug a vf，guest aborted with coredump（Segmentation fault (core dumped) /usr/libexec/qemu-kvm） Version-Release number of selected component (if applicable): [root@ibm-p9wr-22 ppc-sriov-vfio-at]# uname -r 4.18.0-147.3.1.el8_1.ppc64le [root@ibm-p9wr-22 ppc-sriov-vfio-at]# rpm -qa|grep qemu ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch qemu-kvm-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-rbd-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-curl-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-img-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-core-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-rbd-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-guest-agent-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-curl-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-ssh-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-debugsource-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-common-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-iscsi-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-ssh-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-core-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-tests-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-tests-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-common-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-kvm-block-iscsi-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-img-debuginfo-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le qemu-guest-agent-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le How reproducible: 5/5 Steps to Reproduce: 1.[root@ibm-p9wr-22 ppc-sriov-vfio-at]# ./main -v -s sr-iov.sh --only 22 [INFO] Started at Tue Dec 17 22:28:16 EST 2019 [INFO] Logging to local directory: /tmp/test_logs/2019-12-17_222816 #################################################################################################### Running test suite: sr-iov.sh ONLY=22 ===== 22:28:16 (1576639696) #################################################################################################### [INFO] Run sr-iov /home/ppc-sriov-vfio-at/sr-iov.sh Only run test: 22 excepting tests: ==================================================================================================== Check and set up test environment ===== 22:28:16 (1576639696) ==================================================================================================== [COMMAND] Check host env QEMU emulator version 4.1.0 (qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers [INFO] HOST kernel version: 4.18.0-147.3.1.el8_1.ppc64le [SUCCEED] Host is health [COMMAND] Setup env for migration test [INFO] Cannot ping Dest host [INFO] Will skip all migration related cases [COMMAND] qemu-img check /home/rhel811-ppc64le-virtio-scsi.qcow2 No errors were found on the image. 79315/327680 = 24.21% allocated, 8.78% fragmented, 0.00% compressed clusters Image end offset: 5199429632 [COMMAND] Check whether SRIOV is enabled for Infiniband device: lspci | grep -i "Infiniband" 0030:01:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] Capabilities: [180] Single Root I/O Virtualization (SR-IOV) ......................... ==================================================================================================== test 22: RHEL7-11385 - [SR-IOV] 500 times Hotplug/Hotunplug ===== 22:28:26 (1576639706) ==================================================================================================== [STEP] =======> 1. bring up VFs by following instructions **How To **in **Setup** [COMMAND] Write 8 to sriov_numvfs sysfs file to enable VFs: echo 8 > /sys/bus/pci/devices/0030:01:00.0/sriov_numvfs 0030:01:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:01.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] VFs bring up successfully on PF 0030:01:00.0 [STEP] =======> 2. unbind VFs from host kernel driver to vfio-pci [COMMAND] Unbind from host kernel driver: echo 0030:01:00.5 > /sys/bus/pci/devices/0030:01:00.5/driver/unbind [COMMAND] Bind device to vfio-pci: echo "15b3 1018" > /sys/bus/pci/drivers/vfio-pci/new_id [COMMAND] echo 15b3 1018 > /sys/bus/pci/drivers/vfio-pci/remove_id Kernel driver in use: vfio-pci [STEP] =======> 3. record output of free -m[INFO] memory used: 5215 free: 513542 [STEP] =======> 4. start a guest without VF assigned [COMMAND] Local host: boot guest via: qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:33094,server,nowait -qmp tcp:0:43094,server,nowait -serial tcp:0:53094,server,nowait -cpu host -nodefaults -m 4096 -machine ic-mode=xics -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/home/rhel811-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [COMMAND] {"execute": "cont"} {"return": {}} {"timestamp": {"seconds": 1576639724, "microseconds": 506478}, "event": "RESUME"} [COMMAND] Wait until guest boots up successfully [INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++ localhost login: Password: Last login: Wed Dec 18 06:29:27 on tty1 [root@localhost ~]# [INFO] GUEST kernel version: 4.18.0-147.3.1.el8_1.ppc64le [SUCCEED] Boot up and login guest successfully [STEP] =======> 5. hot plug a VF, 6. hot unplug this VF, 7.repeats step 4-5 for 500 times[INFO] Current 1 [COMMAND] {"execute":"device_add", "arguments":{"driver":"vfio-pci","host":"0030:01:00.5","id":"hostnet_VF"}} {"return": {}} [COMMAND] [Guest]# lspci | grep Inf 0001:00:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] all VFs can bring up in guest, actual: 1 <expected: 1> [COMMAND] {"execute":"device_del", "arguments":{"id":"hostnet_VF"}} {"return": {}} /home/ppc-sriov-vfio-at/qemu_cli/sr-iov: line 28: 27604 Segmentation fault (core dumped) /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:$QMP_SHELL_PORT,server,nowait -qmp tcp:0:$QMP_EVENT_PORT,server,nowait -serial tcp:0:$SERIAL_PORT,server,nowait -cpu host -nodefaults -m 4096 -machine ic-mode=xics -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=$SYSTEM_IMAGE,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [FAIL] QMP didn't generate <DEVICE_DELETED> event in more than 300 sec Dumping logs to /tmp/test_logs/2019-12-17_222816/sr-iov.test_22* sr-iov: FAIL: test_22: QMP can't get "DEVICE_DELETED" event [FAIL] Qemu process aborted/crashed! [FAIL] Fail to shutdown guest [COMMAND] {"execute": "quit"} ERROR: Could not connect to localhost:33094 [INFO] Qemu-kvm process <pid: 27604> exited FAIL 22 (397s) Actual results: Qemu process aborted/crashed! Expected results: guest run well Additional info: [root@ibm-p9wr-22 ppc-sriov-vfio-at]# lsprop /sys/firmware/devicetree/base/ibm,firmware-versions skiboot "v6.2-176-g261ca8e779e5" bmc-firmware-version "2.03" occ "12c8088" hostboot "0b3f4e1-p64efa43" buildroot "2018.11.3-12-g222837a" capp-ucode "p9-dd2-v4" machine-xml "e3e9aef" hostboot-binaries "hw021419a.930" sbe "1410677" hcode "hw031219a.940" petitboot "v1.10.2" phandle 00000794 (1940) version "witherspoon-OP9-v2.2-9.56" linux "4.19.26-openpower1-p278e5c6" name "ibm,firmware-versions"
[root@ibm-p9wr-22 home]# gdb core.27604 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-6.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "ppc64le-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... [New LWP 27612] [New LWP 27604] [New LWP 27605] [New LWP 27611] [New LWP 27609] [New LWP 27613] [New LWP 27789] Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le.debug...done. done. warning: Ignoring non-absolute filename: <linux-vdso64.so.1> Missing separate debuginfo for linux-vdso64.so.1 Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/49/3fb1c1eb944a249163bfea939cabf2e9be3f7b [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/power9/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:33094,server,nowait '. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000001234f99dc in notifier_remove (notifier=0x1514e1458) at util/notify.c:31 31 QLIST_REMOVE(notifier, node); [Current thread is 1 (Thread 0x7fff8d47e820 (LWP 27612))] (gdb) bt #0 0x00000001234f99dc in notifier_remove (notifier=0x1514e1458) at util/notify.c:31 #1 0x00000001230d04e8 in kvm_irqchip_remove_change_notifier (n=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/accel/kvm/kvm-all.c:1353 #2 0x000000012313d3cc in vfio_exitfn (pdev=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/vfio/pci.c:3097 #3 0x00000001232ce26c in pci_qdev_unrealize (dev=<optimized out>, errp=<optimized out>) at hw/pci/pci.c:1127 #4 0x00000001232a9e8c in device_set_realized (obj=0x1514e0810, value=<optimized out>, errp=0x0) at hw/core/qdev.c:914 #5 0x00000001233a9dc8 in property_set_bool (obj=0x1514e0810, v=<optimized out>, name=<optimized out>, opaque=0x150fbf390, errp=0x0) at qom/object.c:2079 #6 0x00000001233acc28 in object_property_set (obj=0x1514e0810, v=0x7fff7c201470, name=0x1235cc2d0 "realized", errp=0x0) at qom/object.c:1271 #7 0x00000001233b075c in object_property_set_qobject (obj=0x1514e0810, value=<optimized out>, name=0x1235cc2d0 "realized", errp=0x0) at qom/qom-qobject.c:26 #8 0x00000001233acf80 in object_property_set_bool (obj=0x1514e0810, value=<optimized out>, name=0x1235cc2d0 "realized", errp=0x0) at qom/object.c:1337 #9 0x000000012317dff0 in spapr_pci_unplug (plug_handler=<optimized out>, plugged_dev=0x1514e0810, errp=0x123a08c30 <error_abort>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_pci.c:1656 #10 0x00000001232aff20 in hotplug_handler_unplug (plug_handler=0x15197a930, plugged_dev=0x1514e0810, errp=0x123a08c30 <error_abort>) at hw/core/hotplug.c:56 #11 0x0000000123182e24 in spapr_phb_remove_pci_device_cb (dev=0x1514e0810) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_pci.c:1501 #12 0x0000000123184a74 in spapr_drc_release (drc=0x15197a0c0) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_drc.c:400 #13 0x0000000123186068 in drc_isolate_physical (drc=0x15197a0c0) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_drc.c:69 #14 0x0000000123186dcc in rtas_set_isolation_state (state=0, idx=1073807360) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_drc.c:936 #15 rtas_set_indicator (cpu=<optimized out>, spapr=<optimized out>, token=<optimized out>, nargs=<optimized out>, args=<optimized out>, nret=<optimized out>, rets=24119656) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_drc.c:1003 #16 0x000000012317ca0c in spapr_rtas_call (cpu=<optimized out>, spapr=<optimized out>, token=<optimized out>, nargs=<optimized out>, args=<optimized out>, nret=<optimized out>, rets=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_rtas.c:370 #17 0x000000012317625c in h_rtas (cpu=0x1512c7930, spapr=0x150efa400, opcode=<optimized out>, args=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_hcall.c:1077 #18 0x0000000123179f6c in spapr_hypercall (cpu=0x1512c7930, opcode=61440, args=0x7fff8cc50030) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/hw/ppc/spapr_hcall.c:1863 #19 0x000000012322e508 in kvm_arch_handle_exit (cs=0x1512c7930, run=0x7fff8cc50000) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/target/ppc/kvm.c:1742 #20 0x00000001230d533c in kvm_cpu_exec (cpu=0x1512c7930) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/accel/kvm/kvm-all.c:2365 #21 0x00000001230ac7c8 in qemu_kvm_cpu_thread_fn (arg=<optimized out>) at /usr/src/debug/qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1.ppc64le/cpus.c:1285 #22 0x00000001234e7970 in qemu_thread_start (args=0x15130ea60) at util/qemu-thread-posix.c:502 #23 0x00007fff90c08ae0 in start_thread (arg=0x7fff91a70000 <_rtld_local>) at pthread_create.c:486 #24 0x00007fff90b1e8f8 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:82 (gdb)
do the same operation on rhelav8.2 has this bug . Therefore, change the version information to 8. 2 [STEP] =======> 1. bring up VFs by following instructions **How To **in **Setup** [COMMAND] Write 3 to sriov_numvfs sysfs file to enable VFs: echo 3 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs 0000:01:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] VFs bring up successfully on PF 0000:01:00.0 [STEP] =======> 2. unbind VFs from host kernel driver to vfio-pci [COMMAND] Unbind from host kernel driver: echo 0000:01:00.3 > /sys/bus/pci/devices/0000:01:00.3/driver/unbind [COMMAND] Bind device to vfio-pci: echo "15b3 1018" > /sys/bus/pci/drivers/vfio-pci/new_id [COMMAND] echo 15b3 1018 > /sys/bus/pci/drivers/vfio-pci/remove_id Kernel driver in use: vfio-pci [STEP] =======> 3. record output of free -m[INFO] memory used: 20978 free: 476224 [STEP] =======> 4. start a guest without VF assigned [COMMAND] Local host: boot guest via: qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:35735,server,nowait -qmp tcp:0:45735,server,nowait -serial tcp:0:55735,server,nowait -cpu host -nodefaults -machine accel=kvm,kvm-type=HV -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/root/rhel820-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [COMMAND] {"execute": "cont"} {"return": {}} {"timestamp": {"seconds": 1576657975, "microseconds": 512124}, "event": "RESUME"} [COMMAND] Wait until guest boots up successfully [INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++ qemu-kvm: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM Falling back to kernel-irqchip=off localhost login: Password: Last login: Wed Dec 18 11:33:52 on tty1[root@localhost ~]# [INFO] GUEST kernel version: 4.18.0-160.el8.ppc64le [SUCCEED] Boot up and login guest successfully [STEP] =======> 5. hot plug a VF, 6. hot unplug this VF, 7.repeats step 4-5 for 500 times[INFO] Current 1 [COMMAND] {"execute":"device_add", "arguments":{"driver":"vfio-pci","host":"0000:01:00.3","id":"hostnet_VF"}} {"return": {}} [COMMAND] [Guest]# lspci | grep Inf 0001:00:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] all VFs can bring up in guest, actual: 1 <expected: 1> [COMMAND] {"execute":"device_del", "arguments":{"id":"hostnet_VF"}} {"return": {}} /home/ppc-sriov-vfio-at/qemu_cli/sr-iov: line 28: 20216 Segmentation fault (core dumped) /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:$QMP_SHELL_PORT,server,nowait -qmp tcp:0:$QMP_EVENT_PORT,server,nowait -serial tcp:0:$SERIAL_PORT,server,nowait -cpu host -nodefaults -machine accel=kvm,kvm-type=HV -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=$SYSTEM_IMAGE,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1
Looks like a regression caused by the fix for bug 1739299.
I haven't been able to reproduce this so far. I'm not sure if that's because I'm using a PF rather than a VF, or because my scripts need some kind of refinement. Minjia, in the meantime could you check: 1) Does the problem occur with the latest RHEL-AV-8.2 qemu package (based on qemu-4.2.0 rather than qemu-4.1.0)/ 2) Does the problem occur with the package I built at bug 1784261 comment 14. I'm not sure if this bug will be fixed by the same patch, but it's possible it might be.
(In reply to David Gibson from comment #5) > I haven't been able to reproduce this so far. I'm not sure if that's > because I'm using a PF rather than a VF, or because my scripts need some > kind of refinement. > > Minjia, in the meantime could you check: > > 1) Does the problem occur with the latest RHEL-AV-8.2 qemu package (based on > qemu-4.2.0 rather than qemu-4.1.0)/ I test on qemu-4.2.0: [root@ibm-p9b-03 ~]# rpm -qa|grep qemu qemu-kvm-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-iscsi-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-core-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-rbd-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-guest-agent-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-guest-agent-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-debugsource-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-common-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-rbd-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-curl-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch qemu-kvm-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-ssh-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-iscsi-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-tests-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-common-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-core-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-ssh-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-img-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-tests-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-kvm-block-curl-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le qemu-img-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > > 2) Does the problem occur with the package I built at bug 1784261 comment > 14. I'm not sure if this bug will be fixed by the same patch, but it's > possible it might be. I will update the test result later
I think it is a duplicate of Bug 1782678 - qemu core dump after hot-unplugging the XXV710/XL710 PF
(In reply to Minjia Cai from comment #6) > (In reply to David Gibson from comment #5) > > I haven't been able to reproduce this so far. I'm not sure if that's > > because I'm using a PF rather than a VF, or because my scripts need some > > kind of refinement. > > > > Minjia, in the meantime could you check: > > > > 1) Does the problem occur with the latest RHEL-AV-8.2 qemu package (based on > > qemu-4.2.0 rather than qemu-4.1.0)/ > I test on qemu-4.2.0: > [root@ibm-p9b-03 ~]# rpm -qa|grep qemu > qemu-kvm-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-iscsi-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-core-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-rbd-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-guest-agent-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-guest-agent-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-debugsource-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-common-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-rbd-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-curl-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch > qemu-kvm-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-ssh-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-iscsi-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-tests-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-common-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-core-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-ssh-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-img-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-tests-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-kvm-block-curl-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > qemu-img-debuginfo-4.2.0-2.scrmod+el8.2.0+5137+ec04dc0c.ppc64le > > > > > 2) Does the problem occur with the package I built at bug 1784261 comment > > 14. I'm not sure if this bug will be fixed by the same patch, but it's > > possible it might be. > > I will update the test result later [COMMAND] Check whether SRIOV is enabled for Infiniband device: lspci | grep -i "Infiniband" 0000:01:00.0 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 0000:01:00.1 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] Capabilities: [180] Single Root I/O Virtualization (SR-IOV) Capabilities: [180] Single Root I/O Virtualization (SR-IOV) ......................... ==================================================================================================== test 22: RHEL7-11385 - [SR-IOV] 500 times Hotplug/Hotunplug ===== 00:25:15 (1576733115) ==================================================================================================== [STEP] =======> 1. bring up VFs by following instructions **How To **in **Setup** [COMMAND] Write 7 to sriov_numvfs sysfs file to enable VFs: echo 7 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs 0000:01:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:00.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0000:01:01.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] VFs bring up successfully on PF 0000:01:00.0 [STEP] =======> 2. unbind VFs from host kernel driver to vfio-pci [COMMAND] Unbind from host kernel driver: echo 0000:01:00.4 > /sys/bus/pci/devices/0000:01:00.4/driver/unbind [COMMAND] Bind device to vfio-pci: echo "15b3 1018" > /sys/bus/pci/drivers/vfio-pci/new_id [COMMAND] echo 15b3 1018 > /sys/bus/pci/drivers/vfio-pci/remove_id Kernel driver in use: vfio-pci [STEP] =======> 3. record output of free -m[INFO] memory used: 21276 free: 496954 [STEP] =======> 4. start a guest without VF assigned [COMMAND] Local host: boot guest via: qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:32125,server,nowait -qmp tcp:0:42125,server,nowait -serial tcp:0:52125,server,nowait -cpu host -nodefaults -machine accel=kvm,kvm-type=HV -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/root/rhel820-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [COMMAND] {"execute": "cont"} {"return": {}} {"timestamp": {"seconds": 1576733131, "microseconds": 319517}, "event": "RESUME"} [COMMAND] Wait until guest boots up successfully [INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++ qemu-kvm: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM Falling back to kernel-irqchip=off localhost login: Password: Last login: Thu Dec 19 08:26:27 on tty1[root@localhost ~]# [INFO] GUEST kernel version: 4.18.0-160.el8.ppc64le [SUCCEED] Boot up and login guest successfully [STEP] =======> 5. hot plug a VF, 6. hot unplug this VF, 7.repeats step 4-5 for 500 times[INFO] Current 1 [COMMAND] {"execute":"device_add", "arguments":{"driver":"vfio-pci","host":"0000:01:00.4","id":"hostnet_VF"}} {"return": {}} [COMMAND] [Guest]# lspci | grep Inf 0001:00:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] all VFs can bring up in guest, actual: 1 <expected: 1> [COMMAND] {"execute":"device_del", "arguments":{"id":"hostnet_VF"}} {"return": {}} /home/ppc-sriov-vfio-at/qemu_cli/sr-iov: line 28: 15963 Segmentation fault (core dumped) /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:$QMP_SHELL_PORT,server,nowait -qmp tcp:0:$QMP_EVENT_PORT,server,nowait -serial tcp:0:$SERIAL_PORT,server,nowait -cpu host -nodefaults -machine accel=kvm,kvm-type=HV -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=$SYSTEM_IMAGE,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [FAIL] QMP didn't generate <DEVICE_DELETED> event in more than 300 sec Dumping logs to /tmp/test_logs/2019-12-19_002504/sr-iov.test_22* sr-iov: FAIL: test_22: QMP can't get "DEVICE_DELETED" event [FAIL] Qemu process aborted/crashed! [FAIL] Fail to shutdown guest the host: [root@ibm-p9b-03 ppc-sriov-vfio-at]# rpm -qa|grep qemu qemu-kvm-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-rbd-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-curl-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-common-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch qemu-kvm-debugsource-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-common-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-iscsi-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-ssh-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-core-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-tests-4.1.0-19.el8.dwg201912191508.ppc64le qemu-img-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-iscsi-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-ssh-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-core-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-guest-agent-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-curl-4.1.0-19.el8.dwg201912191508.ppc64le qemu-img-4.1.0-19.el8.dwg201912191508.ppc64le qemu-guest-agent-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-block-rbd-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le qemu-kvm-tests-debuginfo-4.1.0-19.el8.dwg201912191508.ppc64le
Minjia, Now that bug 1782678 is fixed, can you retest and see if this problem is also fixed?
(In reply to David Gibson from comment #9) > Minjia, > > Now that bug 1782678 is fixed, can you retest and see if this problem is > also fixed? ok,I retest.that bug is fixed now,host: [root@ibm-p9wr-22 ppc-sriov-vfio-at]# rpm -qa|grep qemu qemu-kvm-common-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-tests-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-guest-agent-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le libvirt-daemon-driver-qemu-4.5.0-37.module+el8.2.0+5221+0a85e35d.ppc64le qemu-kvm-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-ssh-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-common-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-img-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le ipxe-roms-qemu-20181214-5.git133f4c47.el8.noarch qemu-kvm-debugsource-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-rbd-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-core-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-rbd-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-iscsi-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-guest-agent-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-curl-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-img-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-ssh-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-curl-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-core-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-block-iscsi-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le qemu-kvm-tests-debuginfo-4.2.0-6.module+el8.2.0+5451+991cea0d.ppc64le test 22: RHEL7-11385 - [SR-IOV] 500 times Hotplug/Hotunplug ===== 03:58:06 (1579510686) ==================================================================================================== [STEP] =======> 1. bring up VFs by following instructions **How To **in **Setup** [COMMAND] Write 6 to sriov_numvfs sysfs file to enable VFs: echo 6 > /sys/bus/pci/devices/0030:01:00.0/sriov_numvfs 0030:01:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] 0030:01:00.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] VFs bring up successfully on PF 0030:01:00.0 [STEP] =======> 2. unbind VFs from host kernel driver to vfio-pci [COMMAND] Unbind from host kernel driver: echo 0030:01:00.2 > /sys/bus/pci/devices/0030:01:00.2/driver/unbind [COMMAND] Bind device to vfio-pci: echo "15b3 1018" > /sys/bus/pci/drivers/vfio-pci/new_id [COMMAND] echo 15b3 1018 > /sys/bus/pci/drivers/vfio-pci/remove_id Kernel driver in use: vfio-pci [STEP] =======> 3. record output of free -m[INFO] memory used: 11114 free: 486932 [STEP] =======> 4. start a guest without VF assigned [COMMAND] Local host: boot guest via: qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:32346,server,nowait -qmp tcp:0:42346,server,nowait -serial tcp:0:52346,server,nowait -cpu host -nodefaults -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/home/rhel820-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1 [COMMAND] {"execute": "cont"} {"return": {}} {"timestamp": {"seconds": 1579510722, "microseconds": 631833}, "event": "RESUME"} [COMMAND] Wait until guest boots up successfully [INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++ qemu-kvm: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM Falling back to kernel-irqchip=off localhost login: Password: Last login: Mon Jan 20 11:59:22 on tty1[root@localhost ~]# [INFO] GUEST kernel version: 4.18.0-167.el8.ppc64le [SUCCEED] Boot up and login guest successfully [STEP] =======> 5. hot plug a VF, 6. hot unplug this VF, 7.repeats step 4-5 for 500 times[INFO] Current 1 [COMMAND] {"execute":"device_add", "arguments":{"driver":"vfio-pci","host":"0030:01:00.2","id":"hostnet_VF"}} {"return": {}} [COMMAND] [Guest]# lspci | grep Inf 0001:00:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [SUCCEED] all VFs can bring up in guest, actual: 1 <expected: 1> [COMMAND] {"execute":"device_del", "arguments":{"id":"hostnet_VF"}} {"return": {}} {"timestamp": {"seconds": 1579510799, "microseconds": 278058}, "event": "DEVICE_DELETED", "data": {"device": "hostnet_VF", "path": "/machine/peripheral/hostnet_VF"}} [COMMAND] [Guest]# lspci 00:00.0 VGA compatible controller: Device 1234:1111 (rev 02) 00:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI 00:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [SUCCEED] After hot unplug, lspci doesn't output info about this VF(s)
Great, sounds good. Closing this as a dupe. *** This bug has been marked as a duplicate of bug 1782678 ***