Bug 1784261
| Summary: | [RHEL8.1.1][p8]XICS is failing to set up on p8 machines while booting with vfio-pci device | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Minjia Cai <micai> |
| Component: | qemu-kvm | Assignee: | David Gibson <dgibson> |
| Status: | CLOSED ERRATA | QA Contact: | Minjia Cai <micai> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 8.1 | CC: | ailan, dgibson, knoel, lvivier, micai, mtessun, ngu, qzhang, rbalakri, virt-maint, xuma |
| Target Milestone: | rc | Keywords: | Regression |
| Target Release: | 8.1 | Flags: | pm-rhel:
mirror+
|
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-4.1.0-21.module+el8.1.1+5388+fd51bfbc | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-02-04 18:29:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Minjia Cai
2019-12-17 06:01:42 UTC
in the guest:
Red Hat Enterprise Linux 8.0 (Ootpa)
Kernel 4.18.0-147.3.1.el8_1.ppc64le on an ppc64le
Activate the web console with: systemctl enable --now cockpit.socket
localhost login: root
root
Password: kvmautotest
Last login: Thu Dec 12 08:58:41 on tty1
[root@localhost ~]#
[root@localhost ~]# cat /proc/interrupts
cat /proc/interrupts
CPU0 CPU1
16: 7645 11855 XICS 2 Edge IPI
18: 0 0 XICS 4610 Level virtio1
20: 0 0 XICS 4096 Edge RAS_EPOW
21: 0 0 XICS 4097 Edge RAS_HOTPLUG
256: 0 4 XICS 4352 Edge hvc_console
258: 0 0 XICS 4864 Edge shpchp
259: 0 0 XICS 4866 Edge virtio0-config
260: 0 0 XICS 4867 Edge virtio0-control
261: 0 0 XICS 4868 Edge virtio0-event
262: 12026 42 XICS 4869 Edge virtio0-request
263: 0 0 XICS 4870 Edge mlx5_pages_eq@pci:0000:00:03.0
264: 1033 0 XICS 4871 Edge mlx5_cmd_eq@pci:0000:00:03.0
265: 1 0 XICS 4872 Edge mlx5_async_eq@pci:0000:00:03.0
267: 0 0 XICS 4874 Edge mlx5_comp0@pci:0000:00:03.0
268: 0 0 XICS 4875 Edge mlx5_comp1@pci:0000:00:03.0
LOC: 4655 5336 Local timer interrupts for timer event device
BCT: 0 0 Broadcast timer interrupts for timer event device
LOC: 4 1 Local timer interrupts for others
SPU: 0 0 Spurious interrupts
PMI: 0 0 Performance monitoring interrupts
MCE: 0 0 Machine check exceptions
NMI: 0 0 System Reset interrupts
WDG: 0 0 Watchdog soft-NMI interrupts
DBL: 0 1 Doorbell interrupts
[root@localhost ~]#
in the host:
[root@ibm-p8-kvm-02-qe home]# lspci |grep Inf
0000:01:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
0000:01:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
This seems to be the consequence of the fix for BZ 1739299. Could you confirme this worked well on P8 with previous version of qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1 ? Thanks (In reply to Laurent Vivier from comment #3) > This seems to be the consequence of the fix for BZ 1739299. > > Could you confirme this worked well on P8 with previous version of > qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1 ? > > Thanks on p8 with previous of qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1,the result such as the following，but it works well on p9（5172）： [root@ibm-p8-kvm-02-qe home]# rpm -qa|grep qemu qemu-kvm-debugsource-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-rbd-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-core-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-rbd-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-guest-agent-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-guest-agent-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-ssh-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-img-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-ssh-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-tests-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-core-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch qemu-kvm-common-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-curl-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-tests-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-curl-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-img-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-iscsi-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-iscsi-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-common-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le [root@ibm-p8-kvm-02-qe mdeng]# /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:30325,server,nowait -qmp tcp:0:40325,server,nowait -serial tcp:0:50325,server,nowait -cpu host -nodefaults -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/home/vmt/kar/vt_test_images/rhel811-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device vfio-pci,host=0000:01:00.0,id=hostnet_PF qemu-kvm: -device vfio-pci,host=0000:01:00.0,id=hostnet_PF: warning: vfio 0000:01:00.0: failed to setup resample irqfd: Resource temporarily unavailable qemu-kvm: warning: vfio 0000:01:00.0: failed to setup resample irqfd: Resource temporarily unavailable The log in comment 4 isn't showing this bug. The error message there is caused by bug 1739299 and is expected. This bug is specific to this error message: (qemu) qemu-kvm: warning: kernel_irqchip allowed but unavailable: KVM_SET_DEVICE_ATTR failed: Group 1 attr 0x0000000000001000: Invalid argument At rhelav8.2.0, this problem was not encountered. So the version that found the problem was 8.1. root@ibm-p8-kvm-02-qe micai]# rpm -qa|grep qemu qemu-kvm-debugsource-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-rbd-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-core-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-rbd-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-guest-agent-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-guest-agent-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-ssh-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-img-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-ssh-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-tests-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-core-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch qemu-kvm-common-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-curl-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-tests-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-curl-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-img-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-iscsi-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le qemu-kvm-block-iscsi-debuginfo-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le [root@ibm-p8-kvm-02-qe home]# /usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:38950,server,nowait -qmp tcp:0:48950,server,nowait -serial tcp:0:58950,server,nowait -cpu host -nodefaults -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/home/vmt/kar/vt_test_images/rhel811-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device vfio-pci,host=0000:01:00.1,id=hostnet_PF -monitor stdio QEMU 4.2.0 monitor - type 'help' for more information (qemu) (qemu) cont (qemu) This problem is a regression from qemu-kvm-4.1.0-18.module+el8.1.1+5150+45ce6c40.ppc64le, and also doesn't appear upstream. Downstream biscect log: git bisect start # good: [070b89db1cb52e09ee81609013104bd4ea7774af] Update to qemu-kvm-4.1.0-18.module+el8.1.1+5150+45ce6c40 git bisect good 070b89db1cb52e09ee81609013104bd4ea7774af # bad: [4915198943b5ec85c77dba397c9449a8786eb623] Update to qemu-kvm-4.1.0-19.module+el8.1.1+5172+e3ff58a1 git bisect bad 4915198943b5ec85c77dba397c9449a8786eb623 # good: [247b02f6fc33a701645a625ec7508b93e6b03cd1] spapr: Add return value to spapr_irq_check() git bisect good 247b02f6fc33a701645a625ec7508b93e6b03cd1 # good: [0d618a2cc368871a290baa61d372f39eedae4909] spapr, xics, xive: Match signatures for XICS and XIVE KVM connect routines git bisect good 0d618a2cc368871a290baa61d372f39eedae4909 # good: [062790a398ff1ccbba0acc367e8a07dd0452b5ea] spapr: Move SpaprIrq::nr_xirqs to SpaprMachineClass git bisect good 062790a398ff1ccbba0acc367e8a07dd0452b5ea # good: [b733a1a12ba4137fe59580366e5f60557ae69b77] vfio/pci: Respond to KVM irqchip change notifier git bisect good b733a1a12ba4137fe59580366e5f60557ae69b77 # bad: [092bd0788903e610077a30a7e02d62997f72df87] spapr: Work around spurious warnings from vfio INTx initialization git bisect bad 092bd0788903e610077a30a7e02d62997f72df87 # good: [b17f068d044bc0859e0d56fa1adb418760cdcccd] spapr: Handle irq backend changes with VFIO PCI devices git bisect good b17f068d044bc0859e0d56fa1adb418760cdcccd # first bad commit: [092bd0788903e610077a30a7e02d62997f72df87] spapr: Work around spurious warnings from vfio INTx initialization The problem doesn't show up if I go to the corresponding commit upstream. I think in backporting the fix for bug 1739299, I must have missed something which causes this error. Problem also doesn't occur with qemu-kvm-4.2.0-4.scrmod+el8.2.0+5264+b06bacec.ppc64le.rpm, so this is strictly an 8.1.1 problem. Flagging for blocker consideration, since this is a regression in a fairly important area (this bug means that VMs on POWER8 hosts won't get KVM accelerated interrupt controller). I think this will be addressed by upstream patch 4c3539d4910 "spapr/irq: Only claim VALID interrupts at the KVM level". Building a backport at: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=25448793 Based on the following test pass results, change the bug status to verify status：
Check host env
QEMU emulator version 4.1.0 (qemu-kvm-4.1.0-21.module+el8.1.1+5388+fd51bfbc)
HOST kernel version: 4.18.0-147.4.1.el8_1.ppc64le
[COMMAND] Setup env for migration test
[INFO] Cannot ping Dest host
[INFO] Will skip all migration related cases
[COMMAND] qemu-img check /root/test_home/micai/rhel811-ppc64le-virtio.qcow2
No errors were found on the image.
79631/327680 = 24.30% allocated, 7.37% fragmented, 0.00% compressed clusters
Image end offset: 5220139008
[COMMAND] Check whether PCI device <Infiniband> exist or not: lspci | grep -i "Infiniband"
0000:01:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
0000:01:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
====================================================================================================
test 1: RHEL7-11412 - [vfio] Test on BDF indication (Boot a guest with bus) ===== 00:32:31 (1578461551)
====================================================================================================
[STEP] =======> 1. Assume user wants to access PCI device 0000:06:00.0. Check which IOMMU group this PF is in, and check devices in this group[INFO] All devices in this group: 0000:01:00.0 0000:01:00.1
[STEP] =======> 2. Unbind all devices belong to same group number outputted from above and bind them to vfio-pci
[COMMAND] Unbind from host kernel driver: echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
[COMMAND] Bind device to vfio-pci: echo "15b3 1017" > /sys/bus/pci/drivers/vfio-pci/new_id
[COMMAND] echo 15b3 1017 > /sys/bus/pci/drivers/vfio-pci/remove_id
Kernel driver in use: vfio-pci
[COMMAND] Unbind from host kernel driver: echo 0000:01:00.1 > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
[COMMAND] Bind device to vfio-pci: echo "15b3 1017" > /sys/bus/pci/drivers/vfio-pci/new_id
[COMMAND] echo 15b3 1017 > /sys/bus/pci/drivers/vfio-pci/remove_id
Kernel driver in use: vfio-pci
[STEP] =======> 3. Boot guest with one PF assigned by: (with bus)
[COMMAND] Local host: boot guest via:
qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name
/usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:36534,server,nowait -qmp tcp:0:46534,server,nowait -serial tcp:0:56534,server,nowait -cpu host -nodefaults -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/root/test_home/micai/rhel811-ppc64le-virtio.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device vfio-pci,host=0000:01:00.0,id=hostnet_PF
[COMMAND] {"execute": "cont"}
{"return": {}}
{"timestamp": {"seconds": 1578461568, "microseconds": 137588}, "event": "RESUME"}
[COMMAND] Wait until guest boots up successfully
[INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++
localhost login:
Password: Last login: Wed Jan 8 08:33:21 on tty1
[root@localhost ~]#
[INFO] GUEST kernel version: 4.18.0-147.4.1.el8_1.ppc64le
[SUCCEED] Boot up and login guest successfully
I found this problem on rhelav8.2. can I clone this bug on rhelav8.2?
[COMMAND] Local host: boot guest via:
qemu-kvm: warning: global mc146818rtc.lost_tick_policy has invalid class name
/usr/libexec/qemu-kvm -S -name qemukvm_at_Guest -qmp tcp:0:32346,server,nowait -qmp tcp:0:42346,server,nowait -serial tcp:0:52346,server,nowait -cpu host -nodefaults -m 4096 -smp 2,sockets=2,cores=1,threads=1 -vnc :1 -vga std -enable-kvm -uuid b18fdd6c-a213-4022-9ca4-5d07225e40b0 -rtc base=localtime,clock=host,driftfix=slew -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -device virtio-scsi-pci,id=scsi0,bus=pci.0 -drive file=/home/rhel820-ppc64le-virtio-scsi.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -device spapr-pci-host-bridge,index=1
[COMMAND] {"execute": "cont"}
{"return": {}}
{"timestamp": {"seconds": 1579510722, "microseconds": 631833}, "event": "RESUME"}
[COMMAND] Wait until guest boots up successfully
[INFO] +++++++++++++++++ Guest Console Log +++++++++++++++++
qemu-kvm: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM
Falling back to kernel-irqchip=off
I'm not sure what the problem is you're referring to. Is it just the "qemu-kvm: warning: kernel_irqchip allowed but unavailable: IRQ_XIVE capability must be present for KVM; Falling back to kernel-irqchip=off" message? That message is expected on Boston hosts for reasons that we can't really fix, and is mostly harmless (things won't perform as well, but will still work). (In reply to David Gibson from comment #23) > I'm not sure what the problem is you're referring to. > > Is it just the "qemu-kvm: warning: kernel_irqchip allowed but unavailable: > IRQ_XIVE capability must be present for KVM; Falling back to > kernel-irqchip=off" message? > > That message is expected on Boston hosts for reasons that we can't really > fix, and is mostly harmless (things won't perform as well, but will still > work). got it，ignore above.I will recheck it.thanks Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0404 |