Red Hat Bugzilla – Bug 1390346
PCI: Reserve MMIO space over 4G for PCI hotplug
Last modified: 2018-04-10 20:15:02 EDT
QEMU reserves only 32-bit memory range for hotplug. The range can be rather limited and not enough to hot-plug devices with large BARs. Add a parameter to QEMU to reserve a 64-bit range in CRS starting after the memory ranged reserved for memory hotplug. Check that all the range is addressable by VM's CPU.
Hi Marcel Could you provide some details on what actual use for the case or how can QE test it? Thanks Jing Zhao
(In reply to jingzhao from comment #1) > Hi Marcel > > Could you provide some details on what actual use for the case or how can > QE test it? > > Thanks > Jing Zhao This is a little tricky, you have to create a configuration the has several PCI devices such as there is little MMIO range space in the 32-bit area. Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The hotplug will fail. Be aware you need a 64bit guest. After the feature will be implemented the hotplug should work by reserving the range from >4G address space. Thanks, Marcel
(In reply to Marcel Apfelbaum from comment #2) > (In reply to jingzhao from comment #1) > > Hi Marcel > > > > Could you provide some details on what actual use for the case or how can > > QE test it? > > > > Thanks > > Jing Zhao > > This is a little tricky, you have to create a configuration the has several > PCI devices such as there is little MMIO range space in the 32-bit area. --- Marcel, sorry, I have no idea about the confirguration, could you share an example with me? > Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The > hotplug will fail. ---Marcel, about the ivshmem-plain, you mean the following command and hotplug ivshmem-plain device? -object memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \ -device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \ Thanks Jing Zhao > Be aware you need a 64bit guest. > > After the feature will be implemented the hotplug should work by reserving > the range from >4G address space. > > Thanks, > Marcel
(In reply to jingzhao from comment #3) > (In reply to Marcel Apfelbaum from comment #2) > > (In reply to jingzhao from comment #1) > > > Hi Marcel > > > > > > Could you provide some details on what actual use for the case or how can > > > QE test it? > > > > > > Thanks > > > Jing Zhao > > > > This is a little tricky, you have to create a configuration the has several > > PCI devices such as there is little MMIO range space in the 32-bit area. > > --- Marcel, sorry, I have no idea about the confirguration, could you share > an example with me? > Sure, start with an usual command line ... (no need for a lot of devices) > > Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The > > hotplug will fail. > > ---Marcel, about the ivshmem-plain, you mean the following command and > hotplug ivshmem-plain device? > yes > -object > memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \ > -device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \ > ... and try to hotplug it and it will fail. You don't have 4G space in 32-bit memory space and we can't take it from 64-bit space today. This feature aims to solve the problem. Thanks, Marcel > Thanks > Jing Zhao > > > Be aware you need a 64bit guest. > > > > After the feature will be implemented the hotplug should work by reserving > > the range from >4G address space. > > > > Thanks, > > Marcel
Hi Marcel Could you help to check the reproduced steps. According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64. 1. Boot guest with qemu command line[1] 2. Hotplug memory through qmp (qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share (qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1 3. Check the device through "info qtree" and "lspci" in guest Actual Result: Error message in dmesg: [ 175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] [ 175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] [ 175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04] [ 175.531027] pcieport 0000:03:00.0: bridge window [io 0x2000-0x2fff] [ 175.533280] pcieport 0000:03:00.0: bridge window [mem 0xfc400000-0xfc5fffff] Thanks Jing Zhao
qemu command line [1] Hi Marcel Could you help to check the reproduced steps. According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64. 1. Boot guest with qemu command line[1] 2. Hotplug memory through qmp (qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share (qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1 3. Check the device through "info qtree" and "lspci" in guest Actual Result: Error message in dmesg: [ 175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] [ 175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] [ 175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04] [ 175.531027] pcieport 0000:03:00.0: bridge window [io 0x2000-0x2fff] [ 175.533280] pcieport 0000:03:00.0: bridge window [mem 0xfc400000-0xfc5fffff] [1] /usr/libexec/qemu-kvm \ -M q35 \ -cpu SandyBridge \ -nodefaults -rtc base=utc \ -m 4G \ -smp 2,sockets=2,cores=1,threads=1 \ -enable-kvm \ -name rhel7.4 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -serial unix:/tmp/console,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -chardev file,path=/home/test/seabios.log,id=seabios \ -device isa-debugcon,chardev=seabios,iobase=0x402 \ -qmp tcp::8887,server,nowait \ -vga qxl \ -spice port=5932,disable-ticketing \ -device ioh3420,id=root.0,slot=1 \ -drive file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \ -device ioh3420,id=root.1,slot=2 \ -device x3130-upstream,bus=root.1,id=upstream1 \ -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \ -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \ -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \ -device ioh3420,id=root.2,slot=3 \ -netdev tap,id=hostnet1 \ -device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \ -monitor stdio \ Thanks Jing Zhao
(In reply to jingzhao from comment #6) > qemu command line [1] > > Hi Marcel > Hi, > Could you help to check the reproduced steps. Sure > According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 > and qemu-kvm-rhev-2.8.0-5.el7.x86_64. > > 1. Boot guest with qemu command line[1] > > 2. Hotplug memory through qmp > hmp, not qmp :) > (qemu) object_add > memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share > (qemu) device_add > ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1 > > 3. Check the device through "info qtree" and "lspci" in guest > > Actual Result: > > Error message in dmesg: > > [ 175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 > 64bit pref] > [ 175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size > 0x100000000 64bit pref] Yes, the issue is reproduced correctly. Thanks, Marcel > [ 175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04] > [ 175.531027] pcieport 0000:03:00.0: bridge window [io 0x2000-0x2fff] > [ 175.533280] pcieport 0000:03:00.0: bridge window [mem > 0xfc400000-0xfc5fffff] > > [1] > /usr/libexec/qemu-kvm \ > -M q35 \ > -cpu SandyBridge \ > -nodefaults -rtc base=utc \ > -m 4G \ > -smp 2,sockets=2,cores=1,threads=1 \ > -enable-kvm \ > -name rhel7.4 \ > -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ > -smbios type=1,manufacturer='Red Hat',product='RHEV > Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3- > e63fcdb67170 \ > -k en-us \ > -serial unix:/tmp/console,server,nowait \ > -boot menu=on \ > -bios /usr/share/seabios/bios.bin \ > -chardev file,path=/home/test/seabios.log,id=seabios \ > -device isa-debugcon,chardev=seabios,iobase=0x402 \ > -qmp tcp::8887,server,nowait \ > -vga qxl \ > -spice port=5932,disable-ticketing \ > -device ioh3420,id=root.0,slot=1 \ > -drive > file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2, > cache=none,werror=stop,rerror=stop \ > -device > virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable- > legacy=on,disable-modern=off,bootindex=1 \ > -device ioh3420,id=root.1,slot=2 \ > -device x3130-upstream,bus=root.1,id=upstream1 \ > -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \ > -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \ > -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \ > -device ioh3420,id=root.2,slot=3 \ > -netdev tap,id=hostnet1 \ > -device > virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \ > -monitor stdio \ > > > Thanks > Jing Zhao
Patch posted upstream: - https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg02089.html
Fix included in qemu-kvm-rhev-2.10.0-7.el7
Do we need anything in libvirt to allow this?
(In reply to Yaniv Kaul from comment #17) > Do we need anything in libvirt to allow this? We will need something from libvirt, but with low priority. We decided to use a 2G MMIO window for PCI hot-plug for I440FX machine and a 32G window for Q35. If someone would want to hot-plug a device with larger BARs, they would need to add a parameter to QEMU command line It would be a corner case, but we need to handle it anyway. Thanks, Marcel
(In reply to jingzhao from comment #19) > Hi Marcel > > tested the issue with latest qemu-kvm-rhev-2.10.0-10.el7.x86_64 according > to comment 7, also can see the error message, the same error when hotplug to > pcie-root-port > > [root@localhost ~]# diff tmp tmp1 > 711a712,739 > > [ 109.296785] pciehp 0000:03:00.0:pcie204: Slot(0): Attention button pressed > > [ 109.296795] pciehp 0000:03:00.0:pcie204: Slot(0): Card present > > [ 109.296825] pciehp 0000:03:00.0:pcie204: Slot(0) Powering on due to button press > > [ 110.398812] pci 0000:04:00.0: [1af4:1110] type 00 class 0x050000 > > [ 110.398865] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x000000ff] > > [ 110.398931] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref] > > [ 110.399469] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] > > [ 110.399471] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] > > [ 110.399473] pci 0000:04:00.0: BAR 0: assigned [mem 0xfc400000-0xfc4000ff] > > [ 110.399611] pcieport 0000:03:00.0: PCI bridge to [bus 04] > > [ 110.399617] pcieport 0000:03:00.0: bridge window [io 0x2000-0x2fff] > > [ 110.400109] pcieport 0000:03:00.0: bridge window [mem 0xfc400000-0xfc5fffff] > > [ 110.400404] pcieport 0000:03:00.0: bridge window [mem 0xfe400000-0xfe5fffff 64bit pref] > > [ 110.401001] PCI: No. 2 try to assign unassigned res > > [ 110.401004] pcieport 0000:03:00.0: resource 15 [mem 0xfe400000-0xfe5fffff 64bit pref] released > > [ 110.401005] pcieport 0000:03:00.0: PCI bridge to [bus 04] > > [ 110.402072] pcieport 0000:03:00.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 04] add_size 100000000 add_align 100000000 > > [ 110.402074] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 > > [ 110.402075] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 > > [ 110.402078] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x200000000 64bit pref] > > [ 110.402079] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref] > > [ 110.402081] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x100000000 64bit pref] > > [ 110.402082] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref] > > [ 110.402084] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] > > [ 110.402085] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] > > [ 110.402086] pcieport 0000:03:00.0: PCI bridge to [bus 04] > > [ 110.402092] pcieport 0000:03:00.0: bridge window [io 0x2000-0x2fff] > > [ 110.402536] pcieport 0000:03:00.0: bridge window [mem 0xfc400000-0xfc5fffff] > [root@localhost ~]# lspci > 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM > Controller > 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic > card (rev 04) > 00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express > Root Port 0 (rev 02) > 00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:04.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:05.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port > 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface > Controller (rev 02) > 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 > port SATA Controller [AHCI mode] (rev 02) > 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev > 02) > 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) > 02:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream) > (rev 02) > 03:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch > (Downstream) (rev 01) > 03:01.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch > (Downstream) (rev 01) > 03:02.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch > (Downstream) (rev 01) > 04:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01) > 07:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) > > > Could you help to check ? Should work... please add the dmesg, I want to see the pci windows. Also, please don't use switches/intel io port, use pcie-root-port Thanks, Marcel > > Thanks > Jing
Created attachment 1366514 [details] dmesg log
Summarized the test result 1.Reproduced on qemu-kvm-rhev-2.10.0-1.el7.x86_64 checked the test result: [root@localhost home]# lspci -v -s 08:00.0 08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01) Subsystem: Red Hat, Inc. QEMU Virtual Machine Physical Slot: 4 Flags: fast devsel Memory at fc600000 (32-bit, non-prefetchable) [size=256] Memory at <unassigned> (64-bit, prefetchable) Kernel modules: virtio_pci dmesg result: > [ 253.942692] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed > [ 253.942700] pciehp 0000:00:05.0:pcie004: Slot(4): Card present > [ 253.942726] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press > [ 255.044112] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000 > [ 255.044152] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff] > [ 255.044213] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref] > [ 255.046090] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] > [ 255.046092] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] > [ 255.046094] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff] > [ 255.046863] pcieport 0000:00:05.0: PCI bridge to [bus 08] > [ 255.046869] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] > [ 255.049238] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] > [ 255.050798] pcieport 0000:00:05.0: bridge window [mem 0xfe600000-0xfe7fffff 64bit pref] > [ 255.053912] PCI: No. 2 try to assign unassigned res > [ 255.053917] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released > [ 255.053919] pcieport 0000:00:05.0: PCI bridge to [bus 08] > [ 255.062433] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000 > [ 255.062437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 > [ 255.062438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 > [ 255.062443] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x200000000 64bit pref] > [ 255.062445] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref] > [ 255.062447] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x100000000 64bit pref] > [ 255.062448] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref] > [ 255.062450] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] > [ 255.062451] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] > [ 255.062452] pcieport 0000:00:05.0: PCI bridge to [bus 08] > [ 255.062465] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] > [ 255.064884] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] 2. Re-tested environment kernel-3.10.0-820.el7.x86_64 qemu-kvm-rhev-2.10.0-12.el7.x86_64 seabios-1.11.0-1.el7.x86_64 3. Re-tested result: 1) Boot guest with qemu command line [1] 2) Hotplug memory through qmp {'execute': 'object-add', 'arguments': {'id': 'shmmem-shmem0', 'qom-type': 'memory-backend-ram', 'props': {'policy': 'default', 'size': 4294967296}}} {"return": {}} {'execute': 'device_add', 'arguments':{'id': 'shmem0','driver': 'ivshmem-plain', 'memdev': 'shmmem-shmem0', 'bus':'root.3'}} {"return": {}} 3). Checked test result in guest [root@localhost home]# lspci -v -s 08:00.0 08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01) Subsystem: Red Hat, Inc. QEMU Virtual Machine Physical Slot: 4 Flags: fast devsel Memory at fc600000 (32-bit, non-prefetchable) [size=256] Memory at 200000000 (64-bit, prefetchable) [size=4G] Kernel modules: virtio_pci [root@localhost home]# lspci -vvv -t -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Red Hat, Inc. QXL paravirtual graphic card +-02.0-[01]----00.0 Red Hat, Inc. Virtio block device +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]-- | +-01.0-[05]-- | \-02.0-[06]-- +-04.0-[07]----00.0 Red Hat, Inc. Virtio network device +-05.0-[08-10]----00.0 Red Hat, Inc. Inter-VM shared memory +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller dmesg result: [ 107.253288] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000 [ 107.253329] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 107.253391] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref] [ 107.254011] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] [ 107.254013] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] [ 107.254015] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff] [ 107.254179] pcieport 0000:00:05.0: PCI bridge to [bus 08] [ 107.254185] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] [ 107.254668] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] [ 107.254999] pcieport 0000:00:05.0: bridge window [mem 0xfe600000-0xfe7fffff 64bit pref] [ 107.255637] PCI: No. 2 try to assign unassigned res [ 107.255641] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released [ 107.255642] pcieport 0000:00:05.0: PCI bridge to [bus 08] [ 107.256789] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000 [ 107.256792] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 [ 107.256793] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 [ 107.256796] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref] [ 107.256798] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref] [ 107.257237] pcieport 0000:00:05.0: PCI bridge to [bus 08] [ 107.257245] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] [ 107.257741] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] [ 107.258076] pcieport 0000:00:05.0: bridge window [mem 0x200000000-0x3ffffffff 64bit pref] Compared the reproduce result, hotplug successfully 4) boot guest against through qemu command line [2] 5) Hotplug memory through qmp and check result in guest [ 112.823897] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed [ 112.823907] pciehp 0000:00:05.0:pcie004: Slot(4): Card present [ 112.823935] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press [ 113.925972] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000 [ 113.926012] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 113.926073] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref] [ 113.926651] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref] [ 113.926653] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref] [ 113.926654] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff] [ 113.926816] pcieport 0000:00:05.0: PCI bridge to [bus 08-10] [ 113.926823] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] [ 113.927312] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] [ 113.927629] pcieport 0000:00:05.0: bridge window [mem 0xfe600000-0xfe7fffff 64bit pref] [ 113.928276] PCI: No. 2 try to assign unassigned res [ 113.928280] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released [ 113.928281] pcieport 0000:00:05.0: PCI bridge to [bus 08-10] [ 113.929434] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08-10] add_size 100000000 add_align 100000000 [ 113.929437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 [ 113.929438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000 [ 113.929442] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref] [ 113.929443] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref] [ 113.929908] pcieport 0000:00:05.0: PCI bridge to [bus 08-10] [ 113.929915] pcieport 0000:00:05.0: bridge window [io 0x6000-0x6fff] [ 113.930410] pcieport 0000:00:05.0: bridge window [mem 0xfc600000-0xfc7fffff] [ 113.930736] pcieport 0000:00:05.0: bridge window [mem 0x200000000-0x3ffffffff 64bit pref] [root@localhost ~]# lspci -vvv -t -[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Red Hat, Inc. QXL paravirtual graphic card +-02.0-[01]----00.0 Red Hat, Inc. Virtio block device +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]-- | +-01.0-[05]-- | \-02.0-[06]-- +-04.0-[07]----00.0 Red Hat, Inc. Virtio network device +-05.0-[08-10]----00.0 Red Hat, Inc. Inter-VM shared memory +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller [root@localhost ~]# lspci -v -s 08:00.0 08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01) Subsystem: Red Hat, Inc. QEMU Virtual Machine Physical Slot: 4 Flags: fast devsel Memory at fc600000 (32-bit, non-prefetchable) [size=256] Memory at 200000000 (64-bit, prefetchable) [size=4G] Kernel modules: virtio_pci Addition info: /usr/libexec/qemu-kvm \ -M q35 \ -cpu SandyBridge \ -nodefaults -rtc base=utc \ -m 4G \ -smp 2,sockets=2,cores=1,threads=1 \ -enable-kvm \ -name rhel7.4 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -serial unix:/tmp/console,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -chardev file,path=/home/test/seabios.log,id=seabios \ -device isa-debugcon,chardev=seabios,iobase=0x402 \ -qmp tcp::8887,server,nowait \ -vga qxl \ -spice port=5932,disable-ticketing \ -device pcie-root-port,id=root.0,slot=1 \ -drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \ -device pcie-root-port,id=root.1,slot=2 \ -device x3130-upstream,bus=root.1,id=upstream1 \ -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \ -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \ -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \ -device pcie-root-port,id=root.2,slot=3 \ -netdev tap,id=hostnet1 \ -device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \ -device pcie-root-port,id=root.3,slot=4\ -monitor stdio \ [2] /usr/libexec/qemu-kvm \ -M q35 \ -cpu SandyBridge \ -nodefaults -rtc base=utc \ -m 4G \ -smp 2,sockets=2,cores=1,threads=1 \ -enable-kvm \ -name rhel7.4 \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \ -k en-us \ -serial unix:/tmp/console,server,nowait \ -boot menu=on \ -bios /usr/share/seabios/bios.bin \ -chardev file,path=/home/test/seabios.log,id=seabios \ -device isa-debugcon,chardev=seabios,iobase=0x402 \ -qmp tcp::8887,server,nowait \ -vga qxl \ -spice port=5932,disable-ticketing \ -device pcie-root-port,id=root.0,slot=1 \ -drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \ -device pcie-root-port,id=root.1,slot=2 \ -device x3130-upstream,bus=root.1,id=upstream1 \ -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \ -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \ -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \ -device pcie-root-port,id=root.2,slot=3 \ -netdev tap,id=hostnet1 \ -device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \ -device pcie-root-port,id=root.3,slot=4,mem-reserve=4G \ -monitor stdio \ Thanks Jing
Created attachment 1367064 [details] dmesg with mem-reserver=4G
Created attachment 1370294 [details] dmesg when used "mem-reserved=4G" on ovmf
Created attachment 1372003 [details] dmesg that used upstream seabios with "mem-reserved=4G"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1104