RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1961178 - [aarch64] [kernel] Add support for pvpanic-pci
Summary: [aarch64] [kernel] Add support for pvpanic-pci
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: kernel
Version: 9.0
Hardware: aarch64
OS: Linux
medium
medium
Target Milestone: beta
: 9.0 Beta
Assignee: Eric Auger
QA Contact: Zhenyu Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1747467 1924294 1961326
TreeView+ depends on / blocked
 
Reported: 2021-05-17 12:39 UTC by Eric Auger
Modified: 2022-05-09 08:54 UTC (History)
8 users (show)

Fixed In Version: kernel-5.13.0-0.rc7.51.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-02 23:02:45 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Eric Auger 2021-05-17 12:39:50 UTC
For the aarch64 virt stack we would like to support pvpanic. pvpanic-pci has landed upstream in 5.13:

[GIT PULL] Char/Misc driver changes for 5.13-rc1
https://lore.kernel.org/lkml/YIa0d685cQZ7a2mR@kroah.com/

We would need the following commits to be cherry-picked (from the most recent to the oldest)

391e2415e966  misc/pvpanic: Make some symbols static
642fa28bb5ee  misc/pvpanic: fix return value check in pvpanic_pci_probe()
db3a4f0abefd  misc/pvpanic: add PCI driver
b3c0f8774668  misc/pvpanic: probe multiple instances
6861d27cf590  misc/pvpanic: split-up generic and platform dependent code

Also the following configs shall be set for aarch64

CONFIG_PVPANIC=y
# CONFIG_PVPANIC_MMIO is not set
CONFIG_PVPANIC_PCI=y


PATCH v8] add support for pci in the pvpanic driver
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2524169.html#

Testing Hints from the above cover letter:

Run a VM with latest qemu which has the pvpanic-pci device:
qemu-system-aarch64  -device pvpanic-pci

Generate a panic in the guest:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

Observe in the QMP console the panic events received by the device:

{"timestamp": {"seconds": 1613122186, "microseconds": 141729}, "event":
"GUEST_PANICKED", "data": {"action": "pause"}}

{"timestamp": {"seconds": 1613122186, "microseconds": 141833}, "event":
"GUEST_PANICKED", "data": {"action": "poweroff"}}

{"timestamp": {"seconds": 1613122186, "microseconds": 141896}, "event":
"SHUTDOWN", "data": {"guest": true, "reason": "guest-panic"}}

Comment 1 Eric Auger 2021-05-17 15:55:24 UTC
I noticed os-build branch has the pvpanic-pci related commits

But at the moment, wrt configs, we have

common/generic/CONFIG_PVPANIC_PCI:# CONFIG_PVPANIC_PCI is not set
common/generic/CONFIG_PVPANIC_MMIO:# CONFIG_PVPANIC_MMIO is not set
ark/generic/x86/x86_64/CONFIG_PVPANIC:CONFIG_PVPANIC=y
ark/generic/CONFIG_PVPANIC:# CONFIG_PVPANIC is not set

x86 most probably wants
ark/generic/x86/x86_64/CONFIG_PVPANIC_MMIO:CONFIG_PVPANIC_MMIO=y
ark/generic/x86/x86_64/CONFIG_PVPANIC_PIC:# CONFIG_PVPANIC_PCI is not set

aarch64 most probably wants
ark/generic/arm/aarch64/CONFIG_PVPANIC_MMIO:# CONFIG_PVPANIC_MMIO is not set
ark/generic/arm/aarch64/CONFIG_PVPANIC_PIC:CONFIG_PVPANIC_PCI=y

Comment 3 Mark Langsdorf 2021-06-03 15:18:30 UTC
Since this is in POST, it needs an ITR and a ITM, please.

Comment 4 Luiz Capitulino 2021-06-03 16:40:35 UTC
Eric, what would be a good test case for this?

Qunfang, can you help with the ITM?

Comment 5 Eric Auger 2021-06-03 19:13:46 UTC
(In reply to Luiz Capitulino from comment #4)
> Eric, what would be a good test case for this?
> 
> Qunfang, can you help with the ITM?

There is no libvirt integration yet. So this must be tested at qemu level. The downstream qemu patch enabling the compilation of the pvpanic-pci device is not merged I am afraid so the simplest is to use upstream qemu.

launch with -device pvpanic-pci   -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait
open a qmp shell
sudo /home/augere/UPSTREAM/qemu5/scripts/qmp/qmp-shell /home/augere/TEST/QEMU/qmp-sock

Generate a panic in the guest:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

in the qmp shell you should get the following messages:
{"timestamp": {"seconds": 1613122186, "microseconds": 141729}, "event":
"GUEST_PANICKED", "data": {"action": "pause"}}

{"timestamp": {"seconds": 1613122186, "microseconds": 141833}, "event":
"GUEST_PANICKED", "data": {"action": "poweroff"}}

{"timestamp": {"seconds": 1613122186, "microseconds": 141896}, "event":
"SHUTDOWN", "data": {"guest": true, "reason": "guest-panic"}}


After triggerting the guest panic, I needed to return in the qmp shell to get the qmp messages visible.

Thanks

Eric

Comment 6 Luiz Capitulino 2021-06-04 12:12:05 UTC
Adding NEEDINFO for Qunfang for ITM.

Comment 7 Qunfang Zhang 2021-06-07 02:13:42 UTC
(In reply to Luiz Capitulino from comment #6)
> Adding NEEDINFO for Qunfang for ITM.

Thanks Luiz and Eric, provided ITM and qa_ack+.

Comment 8 Eric Auger 2021-06-14 13:50:24 UTC
Only merged in ark atm so shifting DTM/ITM

Comment 9 Zhenyu Zhang 2021-06-22 01:12:30 UTC
Hello Eric,

I noticed that the DTM has expired.
Do we need to postpone?

Thanks

Zhenyu

Comment 10 Eric Auger 2021-06-22 16:19:15 UTC
Shifted the DTM and ITM by 2 weeks. Unfortunately I do master when the ark changes will be backported to rhel9 and the bug will be moved to MODIFIED.

Comment 11 Zhenyu Zhang 2021-06-24 10:44:39 UTC
Update test results:
The qmp message is not displayed

Test Environment:
Host kernel: 5.13.0-0.rc7.51.el9.aarch64
Guest kernel: 5.13.0-0.rc4.33.el9.aarch64
QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9)

1. Boot guest with -device pvpanic-pci,id=pvpanic0
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-blockdev node-name=file_aavmf_code,driver=file,filename=/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_aavmf_code,driver=raw,read-only=on,file=file_aavmf_code \
-blockdev node-name=file_aavmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel900-aarch64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
-blockdev node-name=drive_aavmf_vars,driver=raw,read-only=off,file=file_aavmf_vars \
-machine virt,gic-version=host,memory-backend=mem-machine_mem,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device virtio-gpu-pci,bus=pcie-root-port-1,addr=0x0 \
-m 8192 \
-object memory-backend-ram,size=8192M,id=mem-machine_mem  \
-smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
-cpu 'host' \
-chardev socket,path=/tmp/monitor-qmpmonitor1,wait=off,id=qmp_id_qmpmonitor1,server=on  \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-serial unix:'/tmp/serial-serial0',server=on,wait=off \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-2,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-3,addr=0x0 \
-blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-aarch64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
-device virtio-net-pci,mac=9a:3d:8d:a3:aa:5d,rombar=0,id=idqiNvUU,netdev=idSd7tcm,bus=pcie-root-port-4,addr=0x0  \
-netdev tap,id=idSd7tcm  \
-vnc :20  \
-rtc base=utc,clock=host,driftfix=slew \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x2,chassis=6 \
-device pcie-root-port,id=pcie_extra_root_port_1,addr=0x2.0x1,bus=pcie.0,chassis=7 \
-device pvpanic-pci,id=pvpanic0 \
-monitor stdio 

2. Check the info qtree to contain the dump the device 'pvpanic-pci'
(qemu) info qtree 
......
  dev: gpex-pcihost, id ""
    gpio-out "sysbus-irq" 4
    x-config-reg-migration-enabled = true
    mmio ffffffffffffffff/0000000020000000
    mmio ffffffffffffffff/ffffffffffffffff
    mmio 000000003eff0000/0000000000010000
    bus: pcie.0
      type PCIE
      dev: pvpanic-pci, id "pvpanic0"     -----------------------------  Verify info qtree to contain the dump the device 'pvpanic-pci' 
        events = 3 (0x3)
        addr = 03.0
        romfile = ""
        romsize = 4294967295 (0xffffffff)
        rombar = 1 (0x1)
        multifunction = false
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        failover_pair_id = ""
        acpi-index = 0 (0x0)
        class Class 0880, addr 00:03.0, pci id 1b36:0011 (sub 1af4:1100)
        bar 0: mem at 0xffffffffffffffff [0x0]
......

3. open a qmp
# nc -U /tmp/monitor-qmpmonitor1
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 6}, "package": "qemu-kvm-6.0.0-5.el9"}, "capabilities": ["oob"]}}
{"execute": "qmp_capabilities", "id": "LW24Xd4G"}
{"return": {}, "id": "LW24Xd4G"}



4. Check not using kdump, generate a panic in the guest:
[root@dhcp19-243-26 ~]# systemctl status kdump
○ kdump.service - Crash recovery kernel arming
     Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor pre>
     Active: inactive (dead)
  Condition: start condition failed at Thu 2021-06-24 17:24:09 CST; 1min 28s ago
             └─ ConditionKernelCommandLine=crashkernel was not met

Jun 24 17:24:09 dhcp19-243-26.khw4.lab.eng.bos.redhat.com systemd[1]: Condition>

[root@dhcp19-243-26 ~]# 
[root@dhcp19-243-26 ~]# echo 1 > /proc/sys/kernel/sysrq

[root@dhcp19-243-26 ~]# echo c > /proc/sysrq-trigger

[  337.528530] sysrq: Trigger a crash
[  337.529711] Kernel panic - not syncing: sysrq triggered crash
[  337.531593] CPU: 5 PID: 1738 Comm: bash Tainted: G               X --------- ---  5.13.0-0.rc7.51.el9.aarch64 #1
[  337.534922] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  337.537160] Call trace:
[  337.537966]  dump_backtrace+0x0/0x1f0
[  337.539177]  show_stack+0x24/0x30
[  337.540381]  dump_stack+0xd0/0x128
[  337.541589]  panic+0x158/0x370
[  337.542685]  sysrq_handle_crash+0x28/0x30
[  337.544113]  __handle_sysrq+0x94/0x1a0
[  337.545454]  write_sysrq_trigger+0x94/0x1ac
[  337.546947]  proc_reg_write+0xb4/0xf0
[  337.548258]  vfs_write+0xfc/0x2bc
[  337.549449]  ksys_write+0x74/0x100
[  337.550666]  __arm64_sys_write+0x28/0x3c
[  337.552090]  invoke_syscall.constprop.0+0x58/0xf0
[  337.553764]  el0_svc_common.constprop.0+0x5c/0x164
[  337.555462]  do_el0_svc+0x34/0xcc
[  337.556652]  el0_svc+0x2c/0x90
[  337.557745]  el0_sync_handler+0xa4/0x150
[  337.559142]  el0_sync+0x198/0x1c0
[  337.560335] SMP: stopping secondary CPUs
[  337.562147] Kernel Offset: disabled
[  337.563401] CPU features: 0x00000211,20000846
[  337.564958] Memory Limit: none
[  337.566063] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---

================================

The qmp message is not displayed
{"return": {}, "id": "LW24Xd4G"}

Other info:
With -qmp unix:'/tmp/monitor-qmpmonitor1',server,nowait 
The qmp message is not displayed too
{"return": {}, "id": "LW24Xd4G"}

Comment 15 Eric Auger 2021-06-24 14:03:31 UTC
Zhenyu

Herton filled "Fixed In Version: kernel-5.13.0-0.rc7.51.el9"
so I guess your guest does not feature pvpanic-pci?

Thanks

Eric

Comment 16 Zhenyu Zhang 2021-06-25 05:17:22 UTC
(In reply to Eric Auger from comment #15)
> Zhenyu
> 
> Herton filled "Fixed In Version: kernel-5.13.0-0.rc7.51.el9"
> so I guess your guest does not feature pvpanic-pci?
> 
> Thanks
> 
> Eric

Hi Eric,

Sorry for the inconvenience.
In fact, the guest is 5.13.0-0.rc7.51.el9.aarch64, and I made a mistake in the version information above.
You can see the version information displayed by panic in the guest.
[  337.531593] CPU: 5 PID: 1738 Comm: bash Tainted: G               X --------- ---  5.13.0-0.rc7.51.el9.aarch64 #1


Correct Test Environment:
Host kernel: 5.13.0-0.rc7.51.el9.aarch64
Guest kernel: 5.13.0-0.rc7.51.el9.aarch64
QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9)


Test again the qmp message is not displayed

Comment 18 Zhenyu Zhang 2021-06-28 10:42:44 UTC
When pvpanic-pci is loaded, the verification is successful.
QMP: Verify it generates a new QEVENT_GUEST_PANICKED event successfully.
The guest auto quit after panic.

[root@dhcp19-243-26 ~]# sudo modprobe pvpanic-pci
[   52.605967] pvpanic-pci 0000:00:03.0: enabling device (0000 -> 0002)

[root@dhcp19-243-26 ~]# ls /sys/bus/pci/drivers/pvpanic-pci/
0000:00:03.0  bind  module  new_id  remove_id  uevent  unbind

[root@dhcp19-243-26 ~]# echo 1 > /proc/sys/kernel/sysrq
[root@dhcp19-243-26 ~]# echo c > /proc/sysrq-trigger

{"timestamp": {"seconds": 1624876165, "microseconds": 731678}, "event": "GUEST_PANICKED", "data": {"action": "poweroff"}}
{"timestamp": {"seconds": 1624876165, "microseconds": 731785}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-panic"}}

[root@dhcp19-243-26 ~]# modinfo pvpanic-pci
modinfo pvpanic-pci
filename:       /lib/modules/5.13.0-0.rc7.51.el9.aarch64/kernel/drivers/misc/pvpanic/pvpanic-pci.ko.xz
license:        GPL
description:    pvpanic device driver 
author:         Mihai Carabas <mihai.carabas>
rhelversion:    8.99
srcversion:     81599BBE1790A0ACF5E63B9
depends:        pvpanic
intree:         Y
name:           pvpanic_pci
vermagic:       5.13.0-0.rc7.51.el9.aarch64 SMP mod_unload modversions aarch64
sig_id:         PKCS#7
signer:         Red Hat Enterprise Linux kernel signing key
sig_key:        6A:3D:0F:64:1A:FB:C1:76:91:D9:0B:C0:46:99:93:DC:E3:C6:5C:0D
sig_hashalgo:   sha256
signature:      65:73:D9:B4:0E:93:99:91:8B:1A:53:1D:C0:32:98:4F:60:E3:8E:37:
		60:2B:54:3E:A8:C4:0A:01:F7:63:3F:0E:15:4F:02:CE:94:56:CF:05:
		28:7D:95:31:95:7B:9A:51:64:0C:77:FF:1A:86:29:2C:DB:FE:0F:01:
		04:26:07:94:0A:42:DF:E3:88:69:45:42:2E:F3:26:E0:AC:91:C5:39:
		6C:5D:6C:80:C6:49:B8:34:C3:62:C4:FE:EA:7E:29:20:46:EF:1E:EB:
		15:0C:E0:3E:D3:ED:6D:76:85:FA:F2:25:59:51:A9:7A:EE:AF:67:0E:
		0B:77:69:44:C8:3D:D9:A0:3E:F1:B4:50:78:FD:31:D2:55:6D:30:6E:
		90:69:25:DD:6B:21:53:FF:54:3A:04:59:D4:54:95:07:96:B0:1B:13:
		AF:02:5A:01:09:87:C3:10:D7:B4:73:E3:77:8E:F2:D4:BC:40:78:62:
		EF:D8:57:DD:67:D7:7E:5B:50:5E:E1:67:FB:2D:C4:41:EC:3D:60:E5:
		C0:DF:74:4C:59:20:76:AB:BB:F8:17:C0:D9:EE:66:EE:4E:23:D8:53:
		AA:5E:DF:35:A8:27:51:7B:B0:6C:D8:2B:43:27:43:92:A3:1E:B0:34:
		B8:11:BB:95:49:9E:B3:11:C9:AA:7F:C2:CD:E5:C2:7A:6A:44:32:10:
		97:26:FA:39:14:73:36:B2:C2:E7:96:0A:E2:67:D7:2C:66:46:89:19:
		B3:24:D1:03:35:AC:50:0B:5D:5A:8E:0F:0A:6B:AE:F5:07:C6:2F:2A:
		B3:CE:AE:3B:79:B6:76:AF:96:55:10:62:DD:E0:5E:37:2C:F4:66:C9:
		71:24:AF:95:25:FD:0D:4D:DA:7C:F3:F8:9F:CA:DC:53:F4:EC:97:11:
		98:C4:37:F3:6E:C3:4C:C7:DF:50:E9:64:B3:D2:EA:AF:61:E5:89:66:
		36:46:25:A9:01:C4:B2:BE:9D:DF:33:12:88:E8:AF:F3:CA:CE:0B:6A:
		1B:6C:02:B0

Test Environment:
Host kernel: 5.13.0-0.rc7.51.el9.aarch64
Guest kernel: 5.13.0-0.rc4.33.el9.aarch64
QEMU emulator version 6.0.0 (qemu-kvm-6.0.0-5.el9)

Comment 19 Eric Auger 2021-06-29 07:25:07 UTC
Refreshing the need info as when the pvpanic-pci module is loaded, it is confirmed the use case works. The module loading issue will be handled in a separate BZ as currently under discussion upstream. See [PATCH v2] misc/pvpanic-pci: Allow automatic loading

Comment 20 Eric Auger 2021-06-29 07:26:37 UTC
So to me you should move this BZ to VERIFIED.

Comment 21 Zhenyu Zhang 2021-06-29 08:29:26 UTC
According to the test result of Comment 18, set the bug to VERIFIED.

The module loading issue is tracked by Bug 1977192-[aarch64][kernel]pvpanic-pci module is not loaded automatically


Note You need to log in before you can comment on or make changes to this bug.