Bug 2055123
Summary: | [Q35] Failed to hot-plug a device whose membar > 2M into the vm | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Yanghang Liu <yanghliu> |
Component: | edk2 | Assignee: | Gerd Hoffmann <kraxel> |
Status: | CLOSED ERRATA | QA Contact: | Yanghang Liu <yanghliu> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 9.0 | CC: | ailan, berrange, chayang, imammedo, jinzhao, juzhang, kraxel, mhou, mrezanin, mst, nilal, pbonzini, vgoyal, virt-maint, xuwei, yalzhang, yanghliu, yfu, yicui, ymankad, yuma |
Target Milestone: | rc | Keywords: | CustomerScenariosInitiative, Triaged |
Target Release: | 9.3 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | edk2-20230524-2.el9 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2023-11-07 08:24:29 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 2174749 | ||
Bug Blocks: |
Description
Yanghang Liu
2022-02-16 10:35:03 UTC
> Additional info:
> (1) Not only PF, but all pci devices with a memory bar larger than 2M can reproduce this problem
>
> (2)
> This problem can be fixed after adding the following setup into the vm cfg:
>
> -global pcie-root-port.pref64-reserve=64M
> or
> <qemu:commandline>
> <qemu:arg value='-global'/>
> <qemu:arg value='pcie-root-port.pref64-reserve=64M'/>
> </qemu:commandline>
1.Test "Hot-plug the pci-testdev whose membar=4M into the vm" scenario without "-global pcie-root-port.pref64-reserve=64M" in the vm cfg:
# virsh qemu-monitor-command rhel90 --hmp "device_add pci-testdev,membar=4M,bus=pci.4"
The related dmesg in the vm:
# dmesg
[ 91.926757] pci 0000:04:00.0: [1b36:0005] type 00 class 0x00ff00
[ 91.930049] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 91.933174] pci 0000:04:00.0: reg 0x14: [io 0x0000-0x00ff]
[ 91.936015] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x003fffff 64bit pref]
[ 91.941166] pci 0000:04:00.0: BAR 2: no space for [mem size 0x00400000 64bit pref]
[ 91.944529] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x00400000 64bit pref]
[ 91.947251] pci 0000:04:00.0: BAR 0: assigned [mem 0xc1000000-0xc1000fff]
[ 91.949654] pci 0000:04:00.0: BAR 1: assigned [io 0x6000-0x60ff]
2. Test "Hot-plug the pci-testdev whose membar=4M into the vm" scenario with "-global pcie-root-port.pref64-reserve=64M" in the vm cfg:
# virsh qemu-monitor-command rhel90 --hmp "device_add pci-testdev,membar=4M,bus=pci.4"
The related dmesg in the vm:
# dmesg
[ 45.943065] pci 0000:04:00.0: [1b36:0005] type 00 class 0x00ff00
[ 45.946711] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00000fff]
[ 45.950183] pci 0000:04:00.0: reg 0x14: [io 0x0000-0x00ff]
[ 45.952725] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x003fffff 64bit pref]
[ 45.958099] pci 0000:04:00.0: BAR 2: assigned [mem 0x80c000000-0x80c3fffff 64bit pref]
[ 45.962333] pci 0000:04:00.0: BAR 0: assigned [mem 0xc1000000-0xc1000fff]
[ 45.965884] pci 0000:04:00.0: BAR 1: assigned [io 0x6000-0x60ff]
Hi Michael, May I ask if there is any chance that we can fix this issue on current 9.1 ? If so, could you please help set the ITR ? The reason I ask this is that QE thinks "hotplug a device whose membar > 2M into a vm" is a very basic scenario and maybe a blocker for other related tests, so it is better if this bug can be fixed with priority. (In reply to Yanghang Liu from comment #2) > Hi Michael, > > May I ask if there is any chance that we can fix this issue on current 9.1 ? > If so, could you please help set the ITR ? > > The reason I ask this is that QE thinks "hotplug a device whose membar > 2M > into a vm" is a very basic scenario and maybe a blocker for other related > tests, so it is better if this bug can be fixed with priority. There is no proven way to fix it in so far (so 9.1 is out of question). At the moment you can either specify reservation hints explicitly or switch to native PCI-E hotplug to get around the issue. PS: (I'm looking at allowing IO resource reallocation which might fix the problem with default ACPI hotplug but it's still work in progress and there is no guaranties that it will work out in the end) > There is no proven way to fix it in so far (so 9.1 is out of question).
> At the moment you can either specify reservation hints explicitly or switch
> to
> native PCI-E hotplug to get around the issue.
FYI: Latest edk2 (2022-11+) scales the default 64-bit pci io window
size and the default 64-bit pci bridge window sizes with the available
physical address space.
Right now only rawhide has rpms. f37, f36, c9s should get the edk2
rebase soon.
*** Bug 2152130 has been marked as a duplicate of this bug. *** (In reply to Gerd Hoffmann from comment #4) > FYI: Latest edk2 (2022-11+) scales the default 64-bit pci io window > size and the default 64-bit pci bridge window sizes with the available > physical address space. > > Right now only rawhide has rpms. f37, f36, c9s should get the edk2 > rebase soon. Hi Igor, May I ask if we have a conclusion about which package this bug will eventually be fixed on ? Is it qemu-kvm or edk2-ovmf? We may need to change this bug's component based on your feedback. (In reply to Yanghang Liu from comment #8) > (In reply to Gerd Hoffmann from comment #4) > > > FYI: Latest edk2 (2022-11+) scales the default 64-bit pci io window > > size and the default 64-bit pci bridge window sizes with the available > > physical address space. > > > > Right now only rawhide has rpms. f37, f36, c9s should get the edk2 > > rebase soon. > > Hi Igor, > > May I ask if we have a conclusion about which package this bug will > eventually be fixed on ? edk2-ovmf. We tried for 9.2, but ran into live migration problems (see bug 2171860), so we had do back out. Will come for real in 9.3, with luck in 9.2.z too. This bug can still be reproduced in the edk2-ovmf-20221207gitfff6d81270b5-8.el9_2.noarch. The main check point: [1] start a Q35 + OVMF domain [2] hot-plug a MT2892 PF into the domain [3] check the PF status in the domain # ifconfig <-- I can not get any PF info here # dmesg ... [ 47.599819] pci 0000:04:00.0: [15b3:101d] type 00 class 0x020000 [ 47.600128] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x01ffffff 64bit pref] [ 47.600597] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x000fffff pref] [ 47.600778] pci 0000:04:00.0: Max Payload Size set to 128 (was 256, max 512) [ 47.602917] pci 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:00:02.3 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link) [ 47.604415] pci 0000:04:00.0: BAR 0: no space for [mem size 0x02000000 64bit pref] [ 47.604419] pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x02000000 64bit pref] [ 47.604425] pci 0000:04:00.0: BAR 6: assigned [mem 0x80200000-0x802fffff pref] [ 47.936797] mlx5_core 0000:04:00.0: Missing registers BAR, aborting [ 47.936801] mlx5_core 0000:04:00.0: mlx5_pci_init:839:(pid 4556): error requesting BARs, aborting [ 47.937043] mlx5_core 0000:04:00.0: probe_one:1679:(pid 4556): mlx5_pci_init failed with error code -19 (In reply to Gerd Hoffmann from comment #9) > > > > May I ask if we have a conclusion about which package this bug will > > eventually be fixed on ? > > edk2-ovmf. We tried for 9.2, but ran into live migration problems > (see bug 2171860), so we had do back out. Will come for real > in 9.3, with luck in 9.2.z too. Hi Gerd, Thanks for the info. I have moved the bug's component to edk2 and setup the ITR to 9.3.0. Feel free to correct me if anything changes. QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. Hi Gerd, My test result shows edk2-20230524-1.el9 can not fix my issue, could you please help check it ? Test env: 5.14.0-334.el9.x86_64 qemu-kvm-8.0.0-6.el9.x86_64 edk2-ovmf-20230524-1.el9.noarch libvirt-9.5.0-0rc1.1.el9.x86_64 Test result: FAILED Test step: (1) start a domain # virt-install --machine=q35 --noreboot --name=rhel93 --memory=4096 --vcpus=4 --graphics type=vnc,port=5993,listen=0.0.0.0 --boot=uefi --network bridge=switch,model=virtio,mac=52:54:00:00:93:93 --import --noautoconsole --disk path=/home/images/RHEL93.qcow2,bus=virtio,cache=none,format=qcow2,io=threads,size=20 --osinfo detect=on,require=off (2) hot-plug a MT2892 PF into domain # lspci -s 60:00.0 60:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] # /bin/virsh attach-device rhel93 /tmp/device/0000:87:00.0.xml Device attached successfully (3) check the PF status in the domain # dmesg [ 111.800926] pci 0000:04:00.0: [15b3:101d] type 00 class 0x020000 [ 111.801232] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x01ffffff 64bit pref] [ 111.801511] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x000fffff pref] [ 111.801636] pci 0000:04:00.0: Max Payload Size set to 128 (was 256, max 512) [ 111.803490] pci 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:00:02.3 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link) [ 111.807277] pci 0000:04:00.0: BAR 0: no space for [mem size 0x02000000 64bit pref] [ 111.807282] pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x02000000 64bit pref] [ 111.807288] pci 0000:04:00.0: BAR 6: assigned [mem 0x82400000-0x824fffff pref] [ 112.171492] mlx5_core 0000:04:00.0: Missing registers BAR, aborting [ 112.171495] mlx5_core 0000:04:00.0: mlx5_pci_init:839:(pid 1397): error requesting BARs, aborting [ 112.171706] mlx5_core 0000:04:00.0: probe_one:1687:(pid 1397): mlx5_pci_init failed with error code -19 (4)repeat the above test result with QL41112 PF, the QL41112 PF failed to be hot-plugged into domain as well. The related guest dmesg: # dmesg [ 40.045152] pci 0000:04:00.0: [1077:8070] type 00 class 0x020000 [ 40.045467] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x0001ffff 64bit pref] [ 40.045568] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x007fffff 64bit pref] [ 40.045666] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit pref] [ 40.045728] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 40.045843] pci 0000:04:00.0: Max Payload Size set to 128 (was 256, max 512) [ 40.051771] pci 0000:04:00.0: BAR 2: no space for [mem size 0x00800000 64bit pref] [ 40.051776] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x00800000 64bit pref] [ 40.051781] pci 0000:04:00.0: BAR 6: assigned [mem 0x82400000-0x8247ffff pref] [ 40.051784] pci 0000:04:00.0: BAR 0: assigned [mem 0x800300000-0x80031ffff 64bit pref] [ 40.051979] pci 0000:04:00.0: BAR 4: assigned [mem 0x800320000-0x80032ffff 64bit pref] [ 40.175261] QLogic FastLinQ 4xxxx Core Module qed [ 40.201075] qede init: QLogic FastLinQ 4xxxx Ethernet Driver qede [ 40.201223] qede 0000:04:00.0: enabling device (0000 -> 0002) [ 40.203280] [qed_init_pci:299()]No memory region found in bar #2 [ 40.203510] [qed_probe:516()]init pci failed Additional info: I have tried this issue with edk2-ovmf-20230301gitf80f052277c8-3.el9.bz2174749.20230515.1346.noarch before, but the MT2892/QL41112 PF can be hot-plugged into the domain at that time. The details is in https://bugzilla.redhat.com/show_bug.cgi?id=2174749#c16 The test log at that time : http://10.73.72.41/log/bug/Bug2174749/2023_05_15_08:58:13_MT2892 http://10.73.72.41/log/bug/Bug2174749/2023_05_15_09:01:27_QL41112 Move the status to ASSIGNED (In reply to Yanghang Liu from comment #16) > Hi Gerd, > > My test result shows edk2-20230524-1.el9 can not fix my issue, could you > please help check it ? edk2-20230524-1.el9 has not yet the patches to re-enable the dynamic mmio window. Latest dynamic mmio window scratch build is here: https://bugzilla.redhat.com/show_bug.cgi?id=2174749#c42 Alternatively wait for the next edk2 build which should arrive next week when mirek is back from PTO. (In reply to Gerd Hoffmann from comment #23) > Alternatively wait for the next edk2 build which should > arrive next week when mirek is back from PTO. edk2-20230524-2.el9 is available now. Test env: edk2-ovmf-20230524-2.el9.noarch Test device: MT2892, QL41112,82599ES,E810, XXV710 Test result: 60:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] Subsystem: Mellanox Technologies Device 0083 Flags: bus master, fast devsel, latency 0, IRQ 65, NUMA node 0, IOMMU group 3 Memory at bc000000 (64-bit, prefetchable) [size=32M] Expansion ROM at b8900000 [disabled] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [180] Single Root I/O Virtualization (SR-IOV) Capabilities: [1c0] Secondary PCI Express Capabilities: [230] Access Control Services Capabilities: [320] Lane Margining at the Receiver <?> Capabilities: [370] Physical Layer 16.0 GT/s <?> Capabilities: [420] Data Link Feature <?> Kernel driver in use: mlx5_core Kernel modules: mlx5_core 60:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function Subsystem: Mellanox Technologies Device 0083 Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 132 Memory at be800000 (64-bit, prefetchable) [virtual] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [9c] MSI-X: Enable+ Count=12 Masked- Capabilities: [100] Vendor Specific Information: ID=0000 Rev=0 Len=00c <?> Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: mlx5_core Kernel modules: mlx5_core 3b:00.0 Ethernet controller: QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller (rev 02) Subsystem: QLogic Corp. 10GE 2P QL41112HxCU-DE Adapter Flags: bus master, fast devsel, latency 0, IRQ 117, NUMA node 0, IOMMU group 6 Memory at ac020000 (64-bit, prefetchable) [size=128K] Memory at ab800000 (64-bit, prefetchable) [size=8M] Memory at ac050000 (64-bit, prefetchable) [size=64K] Expansion ROM at aca00000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+ Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable+ Count=129 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [168] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [178] Power Budgeting <?> Capabilities: [188] Alternative Routing-ID Interpretation (ARI) Capabilities: [198] Secondary PCI Express Capabilities: [1b8] Single Root I/O Virtualization (SR-IOV) Capabilities: [1f8] Transaction Processing Hints Capabilities: [284] Latency Tolerance Reporting Capabilities: [28c] Vendor Specific Information: ID=0002 Rev=3 Len=100 <?> Capabilities: [38c] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Capabilities: [3c4] Precision Time Measurement Capabilities: [3d0] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?> Capabilities: [424] Physical Resizable BAR Kernel driver in use: qede Kernel modules: qede 3b:02.0 Ethernet controller: QLogic Corp. FastLinQ QL41000 Series Gigabit Ethernet Controller (SR-IOV VF) (rev 02) Subsystem: QLogic Corp. 10GE 2P QL41112HxCU-DE Adapter Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 133 Memory at ac360000 (64-bit, prefetchable) [virtual] [size=32K] Memory at ac840000 (64-bit, prefetchable) [virtual] [size=4K] Memory at ac720000 (64-bit, prefetchable) [virtual] [size=8K] Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable+ Count=16 Masked- Capabilities: [100] Alternative Routing-ID Interpretation (ARI) Capabilities: [110] Transaction Processing Hints Kernel driver in use: qede Kernel modules: qede d8:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Flags: bus master, fast devsel, latency 0, IRQ 115, NUMA node 1, IOMMU group 185 Memory at ee880000 (64-bit, non-prefetchable) [size=512K] I/O ports at e020 [size=32] Memory at eeb04000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at eed80000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [e0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-c3-d0-3c Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: ixgbe Kernel modules: ixgbe d8:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) Subsystem: Intel Corporation Device 7a11 Flags: bus master, fast devsel, latency 0, NUMA node 1, IOMMU group 187 Memory at eeb08000 (64-bit, non-prefetchable) [virtual] [size=16K] Memory at eec08000 (64-bit, non-prefetchable) [virtual] [size=16K] Capabilities: [70] MSI-X: Enable+ Count=3 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: ixgbevf Kernel modules: ixgbevf 3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02) Subsystem: Intel Corporation Ethernet Network Adapter E810-C-Q2 Flags: bus master, fast devsel, latency 0, IRQ 114, NUMA node 0, IOMMU group 65 Memory at ae000000 (64-bit, prefetchable) [size=32M] Memory at b2010000 (64-bit, prefetchable) [size=64K] Expansion ROM at ab000000 [disabled] [size=1M] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=1024 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [e0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [148] Alternative Routing-ID Interpretation (ARI) Capabilities: [150] Device Serial Number 6c-fe-54-ff-ff-47-62-20 Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Capabilities: [1a0] Transaction Processing Hints Capabilities: [1b0] Access Control Services Capabilities: [1d0] Secondary PCI Express Capabilities: [200] Data Link Feature <?> Capabilities: [210] Physical Layer 16.0 GT/s <?> Capabilities: [250] Lane Margining at the Receiver <?> Kernel driver in use: ice Kernel modules: ice 3b:01.0 Ethernet controller: Intel Corporation Ethernet Adaptive Virtual Function (rev 02) Subsystem: Intel Corporation Device 0000 Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 188 Memory at b1000000 (64-bit, prefetchable) [virtual] [size=128K] Memory at b2220000 (64-bit, prefetchable) [virtual] [size=16K] Capabilities: [70] MSI-X: Enable+ Count=17 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [148] Alternative Routing-ID Interpretation (ARI) Capabilities: [1a0] Transaction Processing Hints Capabilities: [1d0] Access Control Services Kernel driver in use: iavf Kernel modules: iavf 87:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02) Subsystem: Intel Corporation Device 0000 Flags: bus master, fast devsel, latency 0, NUMA node 1, IOMMU group 189 Memory at d6400000 (64-bit, prefetchable) [virtual] [size=64K] Memory at d6910000 (64-bit, prefetchable) [virtual] [size=16K] Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [1a0] Transaction Processing Hints Capabilities: [1d0] Access Control Services Kernel driver in use: iavf Kernel modules: iavf 41:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02) Subsystem: Intel Corporation Ethernet 25G 2P XXV710 Adapter Flags: bus master, fast devsel, latency 0, IRQ 109, IOMMU group 40 Memory at 91000000 (64-bit, prefetchable) [size=16M] Memory at 92808000 (64-bit, prefetchable) [size=32K] Expansion ROM at 92b00000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable+ Count=129 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [e0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 40-eb-b5-ff-ff-fe-fd-3c Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Capabilities: [1a0] Transaction Processing Hints Capabilities: [1b0] Access Control Services Capabilities: [1d0] Secondary PCI Express Kernel driver in use: i40e Kernel modules: i40e Test result: 2023-07-14 18:14:55 | PASS - hot plug 1 MT2892 pf into rhel93 domain 2023-07-14 18:17:06 | PASS - hot plug 2 MT2892 pf into rhel93 domain 2023-07-14 18:09:48 | PASS - hot plug 1 MT2892 vf into rhel93 domain 2023-07-14 18:12:50 | PASS - hot plug 7 MT2892 vf into rhel93 domain 2023-07-14 18:22:19 | PASS - hot plug 1 QL41112 pf into rhel93 domain 2023-07-14 18:27:46 | PASS - hot plug 2 QL41112 pf into rhel93 domain 2023-07-17 11:08:12 | PASS - hot plug 1 QL41112 vf into rhel93 domain 2023-07-17 11:14:37 | PASS - hot plug 7 QL41112 vf into rhel93 domain 2023-07-14 18:20:39 | PASS - hot plug 1 82599ES vf into rhel93 domain 2023-07-14 18:23:32 | PASS - hot plug 7 82599ES vf into rhel93 domain 2023-07-14 18:16:27 | PASS - hot plug 1 82599ES pf into rhel93 domain 2023-07-14 18:18:39 | PASS - hot plug 2 82599ES pf into rhel93 domain 2023-07-14 18:11:45 | PASS - hot plug 1 E810 vf into rhel93 domain 2023-07-14 18:14:36 | PASS - hot plug 7 E810 vf into rhel93 domain 2023-07-14 18:07:52 | PASS - hot plug 1 E810 pf into rhel93 domain 2023-07-14 18:09:52 | PASS - hot plug 2 E810 pf into rhel93 domain 2023-07-17 10:57:10 | PASS - hot plug 1 XXV710 pf into rhel93 domain 2023-07-17 11:02:17 | PASS - hot plug 2 XXV710 pf into rhel93 domain 2023-07-17 11:07:24 | PASS - hot plug 1 XXV710 vf into rhel93 domain 2023-07-17 11:13:18 | PASS - hot plug 7 XXV710 vf into rhel93 domain Test env: edk2-ovmf-20230524-2.el9.noarch Test device: SFC9220 1a:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02) Subsystem: Solarflare Communications SFN8522-R2 8000 Series 10G Adapter Flags: bus master, fast devsel, latency 0, IRQ 117, NUMA node 0, IOMMU group 33 I/O ports at 4100 [size=256] Memory at 9e000000 (64-bit, non-prefetchable) [size=8M] Memory at a6904000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at a6a40000 [disabled] [size=256K] Capabilities: [40] Power Management version 3 Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable+ Count=32 Masked- Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [148] Device Serial Number 00-0f-53-ff-ff-4d-8c-30 Capabilities: [158] Alternative Routing-ID Interpretation (ARI) Capabilities: [168] Secondary PCI Express Capabilities: [198] Single Root I/O Virtualization (SR-IOV) Capabilities: [1d8] Transaction Processing Hints Capabilities: [26c] L1 PM Substates Kernel driver in use: sfc Kernel modules: sfc 1a:00.2 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (Virtual Function) (rev 02) Subsystem: Solarflare Communications Device 8017 Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 190 Memory at a2800000 (64-bit, non-prefetchable) [virtual] [size=1M] Memory at a6908000 (64-bit, non-prefetchable) [virtual] [size=16K] Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- Capabilities: [100] Alternative Routing-ID Interpretation (ARI) Capabilities: [110] Transaction Processing Hints Kernel driver in use: sfc Kernel modules: sfc Test result: 2023-07-14 18:43:02 | FAIL - hot plug 1 SFC9220 pf to rhel93 domain 2023-07-14 18:48:56 | FAIL - hot plug 2 SFC9220 pf into rhel93 2023-07-17 11:04:35 | PASS - hot plug 1 SFC9220 vf to rhel93 domain 2023-07-17 11:10:41 | PASS - hot plug 7 SFC9220 vf into rhel93 > 1a:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G > Ethernet Controller (rev 02) > Subsystem: Solarflare Communications SFN8522-R2 8000 Series 10G > Adapter > Memory at 9e000000 (64-bit, non-prefetchable) [size=8M] > Memory at a6904000 (64-bit, non-prefetchable) [size=16K] > Test result: > 2023-07-14 18:43:02 | FAIL - hot plug 1 SFC9220 pf to rhel93 domain > 2023-07-14 18:48:56 | FAIL - hot plug 2 SFC9220 pf into rhel93 As expected. The big (8M) non-prefetchable bar continues to need manual configuration of the pcie root port (mem-reserve property). > 2023-07-17 11:04:35 | PASS - hot plug 1 SFC9220 vf to rhel93 domain > 2023-07-17 11:10:41 | PASS - hot plug 7 SFC9220 vf into rhel93 Good. (In reply to Gerd Hoffmann from comment #29) > > 1a:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G > > Ethernet Controller (rev 02) > > Subsystem: Solarflare Communications SFN8522-R2 8000 Series 10G > > Adapter > > > Memory at 9e000000 (64-bit, non-prefetchable) [size=8M] > > Memory at a6904000 (64-bit, non-prefetchable) [size=16K] > > > Test result: > > 2023-07-14 18:43:02 | FAIL - hot plug 1 SFC9220 pf to rhel93 domain > > 2023-07-14 18:48:56 | FAIL - hot plug 2 SFC9220 pf into rhel93 > > As expected. The big (8M) non-prefetchable bar continues to need manual > configuration of the pcie root port (mem-reserve property). Hi Gerd :) Thanks for the confirmation. In my tests, I found the mem-reserve property of pcie root port is not exposed via libvirt. May I ask if you have any suggestions about we open a libvirt bug to request for enabling it ? I am testing to see if SPC9220 PF can be hot-plugged into the VM (whose pcie-root-port has the mem-reserve property set) via qemu-kvm cmdline now. The related qemu-kvm cmd is like: -device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","mem-reserve":16777216,"bus":"pcie.0","addr":"0x2.0x4"}' \ I will update my test result in the comment soon. (In reply to Yanghang Liu from comment #28) > Test env: edk2-ovmf-20230524-2.el9.noarch > > Test device: SFC9220 > > 1a:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G > Ethernet Controller (rev 02) > Subsystem: Solarflare Communications SFN8522-R2 8000 Series 10G > Adapter > Flags: bus master, fast devsel, latency 0, IRQ 117, NUMA node 0, > IOMMU group 33 > I/O ports at 4100 [size=256] > Memory at 9e000000 (64-bit, non-prefetchable) [size=8M] > Memory at a6904000 (64-bit, non-prefetchable) [size=16K] > Expansion ROM at a6a40000 [disabled] [size=256K] > Capabilities: [40] Power Management version 3 > Capabilities: [70] Express Endpoint, MSI 00 > Capabilities: [b0] MSI-X: Enable+ Count=32 Masked- > Capabilities: [d0] Vital Product Data > Capabilities: [100] Advanced Error Reporting > Capabilities: [148] Device Serial Number 00-0f-53-ff-ff-4d-8c-30 > Capabilities: [158] Alternative Routing-ID Interpretation (ARI) > Capabilities: [168] Secondary PCI Express > Capabilities: [198] Single Root I/O Virtualization (SR-IOV) > Capabilities: [1d8] Transaction Processing Hints > Capabilities: [26c] L1 PM Substates > Kernel driver in use: sfc > Kernel modules: sfc > > 1a:00.2 Ethernet controller: Solarflare Communications SFC9220 10/40G > Ethernet Controller (Virtual Function) (rev 02) > Subsystem: Solarflare Communications Device 8017 > Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group > 190 > Memory at a2800000 (64-bit, non-prefetchable) [virtual] [size=1M] > Memory at a6908000 (64-bit, non-prefetchable) [virtual] [size=16K] > Capabilities: [70] Express Endpoint, MSI 00 > Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- > Capabilities: [100] Alternative Routing-ID Interpretation (ARI) > Capabilities: [110] Transaction Processing Hints > Kernel driver in use: sfc > Kernel modules: sfc > > > > Test result: > 2023-07-14 18:43:02 | FAIL - hot plug 1 SFC9220 pf to rhel93 domain > 2023-07-14 18:48:56 | FAIL - hot plug 2 SFC9220 pf into rhel93 > 2023-07-17 11:04:35 | PASS - hot plug 1 SFC9220 vf to rhel93 domain > 2023-07-17 11:10:41 | PASS - hot plug 7 SFC9220 vf into rhel93 Test with SFC9220 PF + 16M pcie-root-port. Test result: PASS Test step: (1) bind the two PFs' driver to vfio-pci (2) start a Q35 + OVMF VM /usr/libexec/qemu-kvm \ -name guest=rhel93,debug-threads=on \ -blockdev '{"driver":"file","filename":"/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/rhel93_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \ -machine pc-q35-rhel9.2.0,usb=off,smm=on,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \ -accel kvm \ -cpu host,migratable=on \ -m 8192 \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \ -overcommit mem-lock=off \ -smp 4,sockets=4,dies=1,cores=1,threads=1 \ -uuid ce70e79f-8854-490a-8b0b-f5261a9b8bad \ -no-user-config \ -nodefaults \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \ -device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \ -device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","mem-reserve":16777216,"bus":"pcie.0","addr":"0x2.0x4"}' \ -device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","mem-reserve":16777216,"bus":"pcie.0","addr":"0x2.0x5"}' \ -device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \ -device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \ -device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \ -device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \ -device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \ -device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \ -device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \ -device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \ -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/images/RHEL93.qcow2", "cache": {"direct": true, "no-flush": false}}' \ -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \ -device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 1, "write-cache": "on", "bus": "pci.2", "addr": "0x0"}' \ -netdev '{"type":"tap","vhost":true,"id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:41:5b:56","bus":"pci.1","addr":"0x0"}' \ -vnc 0.0.0.0:93 \ -device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.3","addr":"0x0"}' \ -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \ -device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.4","addr":"0x0"}' \ -monitor stdio \ -qmp tcp:0:5555,server,nowait \ (3) hot-plug two SFC9220 PFs into the VM (whose pcie-root-port has setup mem-reserve property to 16M) (qemu) device_add vfio-pci,host=0000:1a:00.0,id=hostdev0,bus=pci.5 (qemu) device_add vfio-pci,host=0000:1a:00.1,id=hostdev1,bus=pci.6 (4) check if the two PFs have been hot-plugged into the VM # ifconfig enp5s0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.200.103 netmask 255.255.255.0 broadcast 192.168.200.255 inet6 2001::6e74:5d13:9b7c:d348 prefixlen 64 scopeid 0x0<global> inet6 fe80::2d03:dfb1:ac5e:4d3c prefixlen 64 scopeid 0x20<link> ether 00:0f:53:4d:8c:30 txqueuelen 1000 (Ethernet) RX packets 7 bytes 1110 (1.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 23 bytes 2674 (2.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 22 enp6s0np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.200.171 netmask 255.255.255.0 broadcast 192.168.200.255 inet6 fe80::158a:d2dd:a6c2:714e prefixlen 64 scopeid 0x20<link> inet6 2001::b8b5:3c3b:ab95:8698 prefixlen 64 scopeid 0x0<global> ether 00:0f:53:4d:8c:31 txqueuelen 1000 (Ethernet) RX packets 2 bytes 468 (468.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 20 bytes 2314 (2.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # lspci 05:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02) 06:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02) # dmesg [ 48.567503] pci 0000:05:00.0: [1924:0a03] type 00 class 0x020000 <--- The first PF is hot-plugged into VM at this time [ 48.568112] pci 0000:05:00.0: reg 0x10: [io 0x0000-0x00ff] [ 48.568511] pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x007fffff 64bit] [ 48.569045] pci 0000:05:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit] [ 48.569212] pci 0000:05:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref] [ 48.569459] pci 0000:05:00.0: Max Payload Size set to 128 (was 256, max 1024) [ 48.571327] pci 0000:05:00.0: supports D1 D2 [ 48.576328] pci 0000:05:00.0: BAR 2: assigned [mem 0x81000000-0x817fffff 64bit] [ 48.576669] pci 0000:05:00.0: BAR 6: assigned [mem 0x81800000-0x8183ffff pref] [ 48.576674] pci 0000:05:00.0: BAR 4: assigned [mem 0x81840000-0x81843fff 64bit] [ 48.577034] pci 0000:05:00.0: BAR 0: assigned [io 0x5000-0x50ff] [ 48.639886] Solarflare NET driver [ 48.640922] sfc 0000:05:00.0: Solarflare NIC detected [ 48.648894] sfc 0000:05:00.0: Part Number : SFN8522 [ 48.648946] sfc 0000:05:00.0: enabling device (0000 -> 0003) [ 48.661751] sfc 0000:05:00.0: no PTP support [ 48.734037] sfc 0000:05:00.0 enp5s0np0: renamed from eth0 [ 48.762177] sfc 0000:05:00.0 enp5s0np0: link up at 10000Mbps full-duplex (MTU 1500) [ 48.838016] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0np0: link becomes ready [ 152.924900] pci 0000:06:00.0: [1924:0a03] type 00 class 0x020000 <--- The second PF is hot-plugged into VM at this time [ 152.925306] pci 0000:06:00.0: reg 0x10: [io 0x0000-0x00ff] [ 152.925646] pci 0000:06:00.0: reg 0x18: [mem 0x00000000-0x007fffff 64bit] [ 152.925907] pci 0000:06:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit] [ 152.926030] pci 0000:06:00.0: reg 0x30: [mem 0x00000000-0x0003ffff pref] [ 152.926189] pci 0000:06:00.0: Max Payload Size set to 128 (was 256, max 1024) [ 152.927515] pci 0000:06:00.0: supports D1 D2 [ 152.944474] pci 0000:06:00.0: BAR 2: assigned [mem 0x80000000-0x807fffff 64bit] [ 152.944603] pci 0000:06:00.0: BAR 6: assigned [mem 0x80800000-0x8083ffff pref] [ 152.944605] pci 0000:06:00.0: BAR 4: assigned [mem 0x80840000-0x80843fff 64bit] [ 152.944733] pci 0000:06:00.0: BAR 0: assigned [io 0x7000-0x70ff] [ 152.946150] sfc 0000:06:00.0: Solarflare NIC detected [ 152.959655] sfc 0000:06:00.0: Part Number : SFN8522 [ 152.962646] sfc 0000:06:00.0: enabling device (0000 -> 0003) [ 152.970826] sfc 0000:06:00.0: no PTP support [ 152.998568] sfc 0000:06:00.0 enp6s0np1: renamed from eth0 [ 153.014551] sfc 0000:06:00.0 enp6s0np1: link up at 10000Mbps full-duplex (MTU 1500) [ 153.098966] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0np1: link becomes ready (5) reboot the VM and check the PFs info against # ifconfig enp5s0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.200.103 netmask 255.255.255.0 broadcast 192.168.200.255 inet6 fe80::2d03:dfb1:ac5e:4d3c prefixlen 64 scopeid 0x20<link> inet6 2001::6e74:5d13:9b7c:d348 prefixlen 64 scopeid 0x0<global> ether 00:0f:53:4d:8c:30 txqueuelen 1000 (Ethernet) RX packets 7 bytes 1110 (1.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 26 bytes 2866 (2.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 22 enp6s0np1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.200.171 netmask 255.255.255.0 broadcast 192.168.200.255 inet6 2001::b8b5:3c3b:ab95:8698 prefixlen 64 scopeid 0x0<global> inet6 fe80::158a:d2dd:a6c2:714e prefixlen 64 scopeid 0x20<link> ether 00:0f:53:4d:8c:31 txqueuelen 1000 (Ethernet) RX packets 10 bytes 1302 (1.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22 bytes 2520 (2.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 23 # dmesg [ 2.303562] virtio_net virtio1 enp1s0: renamed from eth0 [ 4.848376] sfc 0000:05:00.0: Solarflare NIC detected [ 4.854592] sfc 0000:05:00.0: Part Number : SFN8522 [ 4.863097] sfc 0000:05:00.0: no PTP support [ 4.963889] sfc 0000:06:00.0: Solarflare NIC detected [ 4.968948] sfc 0000:06:00.0: Part Number : SFN8522 [ 4.974428] sfc 0000:06:00.0: no PTP support [ 5.013352] sfc 0000:05:00.0 enp5s0np0: renamed from eth0 [ 5.114354] sfc 0000:06:00.0 enp6s0np1: renamed from eth1 [ 7.054082] sfc 0000:05:00.0 enp5s0np0: link up at 10000Mbps full-duplex (MTU 1500) [ 7.127095] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0np0: link becomes ready [ 7.135970] sfc 0000:06:00.0 enp6s0np1: link up at 10000Mbps full-duplex (MTU 1500) [ 8.137395] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0np1: link becomes ready > May I ask if you have any suggestions about we open a libvirt bug to request
> for enabling it ?
Yes, makes sense. Even with the dynamic mmio window removing the need
for manual configuration in most cases there are some corner cases like
this left which will need it.
(In reply to Gerd Hoffmann from comment #32) > > May I ask if you have any suggestions about we open a libvirt bug to request > > for enabling it ? > > Yes, makes sense. Even with the dynamic mmio window removing the need > for manual configuration in most cases there are some corner cases like > this left which will need it. Thanks Gerd. I have opened a libvirt bug to tracking the support: Bug 2224472 - [RFE] Request to expose pcie-root-port's mem-reserve option Verify this bug based on the test result in Comment 26, Comment 28, Comment 29 and Comment 31. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: edk2 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6330 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |