Bug 1961178
| Summary: | [aarch64] [kernel] Add support for pvpanic-pci | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Eric Auger <eric.auger> |
| Component: | kernel | Assignee: | Eric Auger <eric.auger> |
| kernel sub component: | KVM | QA Contact: | Zhenyu Zhang <zhenyzha> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | drjones, gshan, hkrzesin, jiji, lcapitulino, mlangsdo, qzhang, virt-maint |
| Version: | 9.0 | Keywords: | TestOnly, Triaged |
| Target Milestone: | beta | Flags: | pm-rhel:
mirror+
|
| Target Release: | 9.0 Beta | ||
| Hardware: | aarch64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | kernel-5.13.0-0.rc7.51.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-02 23:02:45 UTC | Type: | Feature Request |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1747467, 1924294, 1961326 | ||
|
Description
Eric Auger
2021-05-17 12:39:50 UTC
I noticed os-build branch has the pvpanic-pci related commits But at the moment, wrt configs, we have common/generic/CONFIG_PVPANIC_PCI:# CONFIG_PVPANIC_PCI is not set common/generic/CONFIG_PVPANIC_MMIO:# CONFIG_PVPANIC_MMIO is not set ark/generic/x86/x86_64/CONFIG_PVPANIC:CONFIG_PVPANIC=y ark/generic/CONFIG_PVPANIC:# CONFIG_PVPANIC is not set x86 most probably wants ark/generic/x86/x86_64/CONFIG_PVPANIC_MMIO:CONFIG_PVPANIC_MMIO=y ark/generic/x86/x86_64/CONFIG_PVPANIC_PIC:# CONFIG_PVPANIC_PCI is not set aarch64 most probably wants ark/generic/arm/aarch64/CONFIG_PVPANIC_MMIO:# CONFIG_PVPANIC_MMIO is not set ark/generic/arm/aarch64/CONFIG_PVPANIC_PIC:CONFIG_PVPANIC_PCI=y Since this is in POST, it needs an ITR and a ITM, please. Eric, what would be a good test case for this? Qunfang, can you help with the ITM? (In reply to Luiz Capitulino from comment #4) > Eric, what would be a good test case for this? > > Qunfang, can you help with the ITM? There is no libvirt integration yet. So this must be tested at qemu level. The downstream qemu patch enabling the compilation of the pvpanic-pci device is not merged I am afraid so the simplest is to use upstream qemu. launch with -device pvpanic-pci -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait open a qmp shell sudo /home/augere/UPSTREAM/qemu5/scripts/qmp/qmp-shell /home/augere/TEST/QEMU/qmp-sock Generate a panic in the guest: echo 1 > /proc/sys/kernel/sysrq echo c > /proc/sysrq-trigger in the qmp shell you should get the following messages: {"timestamp": {"seconds": 1613122186, "microseconds": 141729}, "event": "GUEST_PANICKED", "data": {"action": "pause"}} {"timestamp": {"seconds": 1613122186, "microseconds": 141833}, "event": "GUEST_PANICKED", "data": {"action": "poweroff"}} {"timestamp": {"seconds": 1613122186, "microseconds": 141896}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-panic"}} After triggerting the guest panic, I needed to return in the qmp shell to get the qmp messages visible. Thanks Eric Adding NEEDINFO for Qunfang for ITM. (In reply to Luiz Capitulino from comment #6) > Adding NEEDINFO for Qunfang for ITM. Thanks Luiz and Eric, provided ITM and qa_ack+. Only merged in ark atm so shifting DTM/ITM Hello Eric, I noticed that the DTM has expired. Do we need to postpone? Thanks Zhenyu Shifted the DTM and ITM by 2 weeks. Unfortunately I do master when the ark changes will be backported to rhel9 and the bug will be moved to MODIFIED. Update test results:
The qmp message is not displayed
Test Environment:
Host kernel: 5.13.0-0.rc7.51.el9.aarch64
Guest kernel: 5.13.0-0.rc4.33.el9.aarch64
QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9)
1. Boot guest with -device pvpanic-pci,id=pvpanic0
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox on \
-blockdev node-name=file_aavmf_code,driver=file,filename=/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_aavmf_code,driver=raw,read-only=on,file=file_aavmf_code \
-blockdev node-name=file_aavmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel900-aarch64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_aavmf_vars,driver=raw,read-only=off,file=file_aavmf_vars \
-machine virt,gic-version=host,memory-backend=mem-machine_mem,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \
-nodefaults \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device virtio-gpu-pci,bus=pcie-root-port-1,addr=0x0 \
-m 8192 \
-object memory-backend-ram,size=8192M,id=mem-machine_mem \
-smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \
-cpu 'host' \
-chardev socket,path=/tmp/monitor-qmpmonitor1,wait=off,id=qmp_id_qmpmonitor1,server=on \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-serial unix:'/tmp/serial-serial0',server=on,wait=off \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-2,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-3,addr=0x0 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-aarch64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
-device virtio-net-pci,mac=9a:3d:8d:a3:aa:5d,rombar=0,id=idqiNvUU,netdev=idSd7tcm,bus=pcie-root-port-4,addr=0x0 \
-netdev tap,id=idSd7tcm \
-vnc :20 \
-rtc base=utc,clock=host,driftfix=slew \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x2,chassis=6 \
-device pcie-root-port,id=pcie_extra_root_port_1,addr=0x2.0x1,bus=pcie.0,chassis=7 \
-device pvpanic-pci,id=pvpanic0 \
-monitor stdio
2. Check the info qtree to contain the dump the device 'pvpanic-pci'
(qemu) info qtree
......
dev: gpex-pcihost, id ""
gpio-out "sysbus-irq" 4
x-config-reg-migration-enabled = true
mmio ffffffffffffffff/0000000020000000
mmio ffffffffffffffff/ffffffffffffffff
mmio 000000003eff0000/0000000000010000
bus: pcie.0
type PCIE
dev: pvpanic-pci, id "pvpanic0" ----------------------------- Verify info qtree to contain the dump the device 'pvpanic-pci'
events = 3 (0x3)
addr = 03.0
romfile = ""
romsize = 4294967295 (0xffffffff)
rombar = 1 (0x1)
multifunction = false
x-pcie-lnksta-dllla = true
x-pcie-extcap-init = true
failover_pair_id = ""
acpi-index = 0 (0x0)
class Class 0880, addr 00:03.0, pci id 1b36:0011 (sub 1af4:1100)
bar 0: mem at 0xffffffffffffffff [0x0]
......
3. open a qmp
# nc -U /tmp/monitor-qmpmonitor1
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 6}, "package": "qemu-kvm-6.0.0-5.el9"}, "capabilities": ["oob"]}}
{"execute": "qmp_capabilities", "id": "LW24Xd4G"}
{"return": {}, "id": "LW24Xd4G"}
4. Check not using kdump, generate a panic in the guest:
[root@dhcp19-243-26 ~]# systemctl status kdump
○ kdump.service - Crash recovery kernel arming
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor pre>
Active: inactive (dead)
Condition: start condition failed at Thu 2021-06-24 17:24:09 CST; 1min 28s ago
└─ ConditionKernelCommandLine=crashkernel was not met
Jun 24 17:24:09 dhcp19-243-26.khw4.lab.eng.bos.redhat.com systemd[1]: Condition>
[root@dhcp19-243-26 ~]#
[root@dhcp19-243-26 ~]# echo 1 > /proc/sys/kernel/sysrq
[root@dhcp19-243-26 ~]# echo c > /proc/sysrq-trigger
[ 337.528530] sysrq: Trigger a crash
[ 337.529711] Kernel panic - not syncing: sysrq triggered crash
[ 337.531593] CPU: 5 PID: 1738 Comm: bash Tainted: G X --------- --- 5.13.0-0.rc7.51.el9.aarch64 #1
[ 337.534922] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 337.537160] Call trace:
[ 337.537966] dump_backtrace+0x0/0x1f0
[ 337.539177] show_stack+0x24/0x30
[ 337.540381] dump_stack+0xd0/0x128
[ 337.541589] panic+0x158/0x370
[ 337.542685] sysrq_handle_crash+0x28/0x30
[ 337.544113] __handle_sysrq+0x94/0x1a0
[ 337.545454] write_sysrq_trigger+0x94/0x1ac
[ 337.546947] proc_reg_write+0xb4/0xf0
[ 337.548258] vfs_write+0xfc/0x2bc
[ 337.549449] ksys_write+0x74/0x100
[ 337.550666] __arm64_sys_write+0x28/0x3c
[ 337.552090] invoke_syscall.constprop.0+0x58/0xf0
[ 337.553764] el0_svc_common.constprop.0+0x5c/0x164
[ 337.555462] do_el0_svc+0x34/0xcc
[ 337.556652] el0_svc+0x2c/0x90
[ 337.557745] el0_sync_handler+0xa4/0x150
[ 337.559142] el0_sync+0x198/0x1c0
[ 337.560335] SMP: stopping secondary CPUs
[ 337.562147] Kernel Offset: disabled
[ 337.563401] CPU features: 0x00000211,20000846
[ 337.564958] Memory Limit: none
[ 337.566063] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
================================
The qmp message is not displayed
{"return": {}, "id": "LW24Xd4G"}
Other info:
With -qmp unix:'/tmp/monitor-qmpmonitor1',server,nowait
The qmp message is not displayed too
{"return": {}, "id": "LW24Xd4G"}
Zhenyu Herton filled "Fixed In Version: kernel-5.13.0-0.rc7.51.el9" so I guess your guest does not feature pvpanic-pci? Thanks Eric (In reply to Eric Auger from comment #15) > Zhenyu > > Herton filled "Fixed In Version: kernel-5.13.0-0.rc7.51.el9" > so I guess your guest does not feature pvpanic-pci? > > Thanks > > Eric Hi Eric, Sorry for the inconvenience. In fact, the guest is 5.13.0-0.rc7.51.el9.aarch64, and I made a mistake in the version information above. You can see the version information displayed by panic in the guest. [ 337.531593] CPU: 5 PID: 1738 Comm: bash Tainted: G X --------- --- 5.13.0-0.rc7.51.el9.aarch64 #1 Correct Test Environment: Host kernel: 5.13.0-0.rc7.51.el9.aarch64 Guest kernel: 5.13.0-0.rc7.51.el9.aarch64 QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9) Test again the qmp message is not displayed When pvpanic-pci is loaded, the verification is successful.
QMP: Verify it generates a new QEVENT_GUEST_PANICKED event successfully.
The guest auto quit after panic.
[root@dhcp19-243-26 ~]# sudo modprobe pvpanic-pci
[ 52.605967] pvpanic-pci 0000:00:03.0: enabling device (0000 -> 0002)
[root@dhcp19-243-26 ~]# ls /sys/bus/pci/drivers/pvpanic-pci/
0000:00:03.0 bind module new_id remove_id uevent unbind
[root@dhcp19-243-26 ~]# echo 1 > /proc/sys/kernel/sysrq
[root@dhcp19-243-26 ~]# echo c > /proc/sysrq-trigger
{"timestamp": {"seconds": 1624876165, "microseconds": 731678}, "event": "GUEST_PANICKED", "data": {"action": "poweroff"}}
{"timestamp": {"seconds": 1624876165, "microseconds": 731785}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-panic"}}
[root@dhcp19-243-26 ~]# modinfo pvpanic-pci
modinfo pvpanic-pci
filename: /lib/modules/5.13.0-0.rc7.51.el9.aarch64/kernel/drivers/misc/pvpanic/pvpanic-pci.ko.xz
license: GPL
description: pvpanic device driver
author: Mihai Carabas <mihai.carabas>
rhelversion: 8.99
srcversion: 81599BBE1790A0ACF5E63B9
depends: pvpanic
intree: Y
name: pvpanic_pci
vermagic: 5.13.0-0.rc7.51.el9.aarch64 SMP mod_unload modversions aarch64
sig_id: PKCS#7
signer: Red Hat Enterprise Linux kernel signing key
sig_key: 6A:3D:0F:64:1A:FB:C1:76:91:D9:0B:C0:46:99:93:DC:E3:C6:5C:0D
sig_hashalgo: sha256
signature: 65:73:D9:B4:0E:93:99:91:8B:1A:53:1D:C0:32:98:4F:60:E3:8E:37:
60:2B:54:3E:A8:C4:0A:01:F7:63:3F:0E:15:4F:02:CE:94:56:CF:05:
28:7D:95:31:95:7B:9A:51:64:0C:77:FF:1A:86:29:2C:DB:FE:0F:01:
04:26:07:94:0A:42:DF:E3:88:69:45:42:2E:F3:26:E0:AC:91:C5:39:
6C:5D:6C:80:C6:49:B8:34:C3:62:C4:FE:EA:7E:29:20:46:EF:1E:EB:
15:0C:E0:3E:D3:ED:6D:76:85:FA:F2:25:59:51:A9:7A:EE:AF:67:0E:
0B:77:69:44:C8:3D:D9:A0:3E:F1:B4:50:78:FD:31:D2:55:6D:30:6E:
90:69:25:DD:6B:21:53:FF:54:3A:04:59:D4:54:95:07:96:B0:1B:13:
AF:02:5A:01:09:87:C3:10:D7:B4:73:E3:77:8E:F2:D4:BC:40:78:62:
EF:D8:57:DD:67:D7:7E:5B:50:5E:E1:67:FB:2D:C4:41:EC:3D:60:E5:
C0:DF:74:4C:59:20:76:AB:BB:F8:17:C0:D9:EE:66:EE:4E:23:D8:53:
AA:5E:DF:35:A8:27:51:7B:B0:6C:D8:2B:43:27:43:92:A3:1E:B0:34:
B8:11:BB:95:49:9E:B3:11:C9:AA:7F:C2:CD:E5:C2:7A:6A:44:32:10:
97:26:FA:39:14:73:36:B2:C2:E7:96:0A:E2:67:D7:2C:66:46:89:19:
B3:24:D1:03:35:AC:50:0B:5D:5A:8E:0F:0A:6B:AE:F5:07:C6:2F:2A:
B3:CE:AE:3B:79:B6:76:AF:96:55:10:62:DD:E0:5E:37:2C:F4:66:C9:
71:24:AF:95:25:FD:0D:4D:DA:7C:F3:F8:9F:CA:DC:53:F4:EC:97:11:
98:C4:37:F3:6E:C3:4C:C7:DF:50:E9:64:B3:D2:EA:AF:61:E5:89:66:
36:46:25:A9:01:C4:B2:BE:9D:DF:33:12:88:E8:AF:F3:CA:CE:0B:6A:
1B:6C:02:B0
Test Environment:
Host kernel: 5.13.0-0.rc7.51.el9.aarch64
Guest kernel: 5.13.0-0.rc4.33.el9.aarch64
QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9)
Refreshing the need info as when the pvpanic-pci module is loaded, it is confirmed the use case works. The module loading issue will be handled in a separate BZ as currently under discussion upstream. See [PATCH v2] misc/pvpanic-pci: Allow automatic loading So to me you should move this BZ to VERIFIED. According to the test result of Comment 18, set the bug to VERIFIED. The module loading issue is tracked by Bug 1977192-[aarch64][kernel]pvpanic-pci module is not loaded automatically |