Bug 1390346

Summary: PCI: Reserve MMIO space over 4G for PCI hotplug
Product: Red Hat Enterprise Linux 7 Reporter: Marcel Apfelbaum <marcel>
Component: qemu-kvm-rhevAssignee: Marcel Apfelbaum <marcel>
Status: CLOSED ERRATA QA Contact: jingzhao <jinzhao>
Severity: low Docs Contact:
Priority: low    
Version: 7.4CC: ailan, chayang, ehabkost, jinzhao, juzhang, marcel, mtessun, virt-maint
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.10.0-7.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1408813 1529618 (view as bug list) Environment:
Last Closed: 2018-04-11 00:12:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1408813, 1410580, 1410589    
Attachments:
Description Flags
dmesg log
none
dmesg with mem-reserver=4G
none
dmesg when used "mem-reserved=4G" on ovmf
none
dmesg that used upstream seabios with "mem-reserved=4G" none

Description Marcel Apfelbaum 2016-10-31 18:38:01 UTC 
QEMU reserves only 32-bit memory range for hotplug. The range can be rather limited and not enough to hot-plug devices with large BARs.

Add a parameter to QEMU to reserve a 64-bit range in CRS
starting after the memory ranged reserved for memory hotplug.

Check that all the range is addressable by VM's CPU.
Comment 1 jingzhao 2017-02-16 06:06:46 UTC 
Hi Marcel

  Could you provide some details on what actual use for the case or how can QE test it?

Thanks
Jing Zhao
Comment 2 Marcel Apfelbaum 2017-02-16 14:05:42 UTC 
(In reply to jingzhao from comment #1)
> Hi Marcel
> 
>   Could you provide some details on what actual use for the case or how can
> QE test it?
> 
> Thanks
> Jing Zhao

This is a little tricky, you have to create a configuration the has several PCI devices such as there is little MMIO range space in the 32-bit area. Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The hotplug will fail.
Be aware you need a 64bit guest.

After the feature will be implemented the hotplug should work by reserving the range from >4G address space.

Thanks,
Marcel
Comment 3 jingzhao 2017-02-21 06:22:12 UTC 
(In reply to Marcel Apfelbaum from comment #2)
> (In reply to jingzhao from comment #1)
> > Hi Marcel
> > 
> >   Could you provide some details on what actual use for the case or how can
> > QE test it?
> > 
> > Thanks
> > Jing Zhao
> 
> This is a little tricky, you have to create a configuration the has several
> PCI devices such as there is little MMIO range space in the 32-bit area.

--- Marcel, sorry, I have no idea about the confirguration, could you share an example with me?

> Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The
> hotplug will fail.

---Marcel, about the ivshmem-plain, you mean the following command and hotplug ivshmem-plain device?

-object memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \
-device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \

Thanks
Jing Zhao

> Be aware you need a 64bit guest.
> 
> After the feature will be implemented the hotplug should work by reserving
> the range from >4G address space.
> 
> Thanks,
> Marcel
Comment 4 Marcel Apfelbaum 2017-03-01 13:36:21 UTC 
(In reply to jingzhao from comment #3)
> (In reply to Marcel Apfelbaum from comment #2)
> > (In reply to jingzhao from comment #1)
> > > Hi Marcel
> > > 
> > >   Could you provide some details on what actual use for the case or how can
> > > QE test it?
> > > 
> > > Thanks
> > > Jing Zhao
> > 
> > This is a little tricky, you have to create a configuration the has several
> > PCI devices such as there is little MMIO range space in the 32-bit area.
> 
> --- Marcel, sorry, I have no idea about the confirguration, could you share
> an example with me?
> 

Sure, start with an usual command line ... (no need for a lot of devices)

> > Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The
> > hotplug will fail.
> 
> ---Marcel, about the ivshmem-plain, you mean the following command and
> hotplug ivshmem-plain device?
> 

yes

> -object
> memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \
> -device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \
> 

... and try to hotplug it and it will fail. You don't have 4G space in 32-bit
memory space and we can't take it from 64-bit space today.

This feature aims to solve the problem.

Thanks,
Marcel

> Thanks
> Jing Zhao
> 
> > Be aware you need a 64bit guest.
> > 
> > After the feature will be implemented the hotplug should work by reserving
> > the range from >4G address space.
> > 
> > Thanks,
> > Marcel
Comment 5 jingzhao 2017-03-13 06:31:30 UTC 
Hi Marcel

 Could you help to check the reproduced steps.

 According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64.

 1. Boot guest with qemu command line[1]

 2. Hotplug memory through qmp

(qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
(qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1

 3. Check the device through "info qtree" and "lspci" in guest

Actual Result:

Error message in dmesg:

[  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
[  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
[  175.533280] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]

Thanks
Jing Zhao
Comment 6 jingzhao 2017-03-13 06:32:58 UTC 
qemu command line [1]

Hi Marcel

 Could you help to check the reproduced steps.

 According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64.

 1. Boot guest with qemu command line[1]

 2. Hotplug memory through qmp

(qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
(qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1

 3. Check the device through "info qtree" and "lspci" in guest

Actual Result:

Error message in dmesg:

[  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
[  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
[  175.533280] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]

[1]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device ioh3420,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-monitor stdio \


Thanks
Jing Zhao
Comment 7 Marcel Apfelbaum 2017-03-14 13:08:47 UTC 
(In reply to jingzhao from comment #6)
> qemu command line [1]
> 
> Hi Marcel
> 

Hi,

>  Could you help to check the reproduced steps.

Sure 

>  According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64
> and qemu-kvm-rhev-2.8.0-5.el7.x86_64.
> 
>  1. Boot guest with qemu command line[1]
> 
>  2. Hotplug memory through qmp
> 

hmp, not qmp :)

> (qemu) object_add
> memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
> (qemu) device_add
> ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1
> 
>  3. Check the device through "info qtree" and "lspci" in guest
> 
> Actual Result:
> 
> Error message in dmesg:
> 
> [  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000
> 64bit pref]
> [  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size
> 0x100000000 64bit pref]

Yes, the issue is reproduced correctly.

Thanks,
Marcel


> [  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> [  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> [  175.533280] pcieport 0000:03:00.0:   bridge window [mem
> 0xfc400000-0xfc5fffff]
> 
> [1]
> /usr/libexec/qemu-kvm \
> -M q35 \
> -cpu SandyBridge \
> -nodefaults -rtc base=utc \
> -m 4G \
> -smp 2,sockets=2,cores=1,threads=1 \
> -enable-kvm \
> -name rhel7.4 \
> -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
> -smbios type=1,manufacturer='Red Hat',product='RHEV
> Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-
> e63fcdb67170 \
> -k en-us \
> -serial unix:/tmp/console,server,nowait \
> -boot menu=on \
> -bios /usr/share/seabios/bios.bin \
> -chardev file,path=/home/test/seabios.log,id=seabios \
> -device isa-debugcon,chardev=seabios,iobase=0x402 \
> -qmp tcp::8887,server,nowait \
> -vga qxl \
> -spice port=5932,disable-ticketing \
> -device ioh3420,id=root.0,slot=1 \
> -drive
> file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,
> cache=none,werror=stop,rerror=stop \
> -device
> virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-
> legacy=on,disable-modern=off,bootindex=1 \
> -device ioh3420,id=root.1,slot=2 \
> -device x3130-upstream,bus=root.1,id=upstream1 \
> -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> -device ioh3420,id=root.2,slot=3 \
> -netdev tap,id=hostnet1 \
> -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
> -monitor stdio \
> 
> 
> Thanks
> Jing Zhao
Comment 13 Marcel Apfelbaum 2017-11-14 12:46:40 UTC 
Patch posted upstream:
  - https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg02089.html
Comment 15 Miroslav Rezanina 2017-11-22 15:20:30 UTC 
Fix included in qemu-kvm-rhev-2.10.0-7.el7
Comment 17 Yaniv Kaul 2017-12-04 11:50:24 UTC 
Do we need anything in libvirt to allow this?
Comment 18 Marcel Apfelbaum 2017-12-06 19:11:36 UTC 
(In reply to Yaniv Kaul from comment #17)
> Do we need anything in libvirt to allow this?

We will need something from libvirt, but with low priority.

We decided to use a 2G MMIO window for PCI hot-plug for I440FX machine
and a 32G window for Q35.

If someone would want to hot-plug a device with larger BARs, they
would need to add a parameter to QEMU command line
It would be a corner case, but we need to handle it anyway.

Thanks,
Marcel
Comment 20 Marcel Apfelbaum 2017-12-12 09:39:21 UTC 
(In reply to jingzhao from comment #19)
> Hi Marcel
> 
>   tested the issue with latest qemu-kvm-rhev-2.10.0-10.el7.x86_64 according
> to comment 7, also can see the error message, the same error when hotplug to
> pcie-root-port
> 
> [root@localhost ~]# diff tmp tmp1
> 711a712,739
> > [  109.296785] pciehp 0000:03:00.0:pcie204: Slot(0): Attention button pressed
> > [  109.296795] pciehp 0000:03:00.0:pcie204: Slot(0): Card present
> > [  109.296825] pciehp 0000:03:00.0:pcie204: Slot(0) Powering on due to button press
> > [  110.398812] pci 0000:04:00.0: [1af4:1110] type 00 class 0x050000
> > [  110.398865] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
> > [  110.398931] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
> > [  110.399469] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> > [  110.399471] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.399473] pci 0000:04:00.0: BAR 0: assigned [mem 0xfc400000-0xfc4000ff]
> > [  110.399611] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.399617] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> > [  110.400109] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]
> > [  110.400404] pcieport 0000:03:00.0:   bridge window [mem 0xfe400000-0xfe5fffff 64bit pref]
> > [  110.401001] PCI: No. 2 try to assign unassigned res
> > [  110.401004] pcieport 0000:03:00.0: resource 15 [mem 0xfe400000-0xfe5fffff 64bit pref] released
> > [  110.401005] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.402072] pcieport 0000:03:00.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 04] add_size 100000000 add_align 100000000
> > [  110.402074] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> > [  110.402075] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> > [  110.402078] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x200000000 64bit pref]
> > [  110.402079] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref]
> > [  110.402081] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x100000000 64bit pref]
> > [  110.402082] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.402084] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> > [  110.402085] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.402086] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.402092] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> > [  110.402536] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]
> [root@localhost ~]# lspci 
> 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
> 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic
> card (rev 04)
> 00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:04.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:05.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface
> Controller (rev 02)
> 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6
> port SATA Controller [AHCI mode] (rev 02)
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev
> 02)
> 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
> 02:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream)
> (rev 02)
> 03:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 03:01.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 03:02.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 04:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
> 07:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> 
> 
> Could you help to check ?

Should work... please add the dmesg, I want to see the pci windows.

Also, please don't use switches/intel io port, use pcie-root-port

Thanks,
Marcel


> 
> Thanks
> Jing
Comment 22 jingzhao 2017-12-12 10:00:34 UTC 
Created attachment 1366514 [details]
dmesg log
Comment 25 jingzhao 2017-12-13 02:48:49 UTC 
Summarized the test result

1.Reproduced  on qemu-kvm-rhev-2.10.0-1.el7.x86_64

checked the test result:

[root@localhost home]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at <unassigned> (64-bit, prefetchable)
	Kernel modules: virtio_pci

dmesg result:
> [  253.942692] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed
> [  253.942700] pciehp 0000:00:05.0:pcie004: Slot(4): Card present
> [  253.942726] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press
> [  255.044112] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
> [  255.044152] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
> [  255.044213] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
> [  255.046090] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> [  255.046092] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> [  255.046094] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
> [  255.046863] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.046869] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
> [  255.049238] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
> [  255.050798] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
> [  255.053912] PCI: No. 2 try to assign unassigned res
> [  255.053917] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
> [  255.053919] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.062433] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000
> [  255.062437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> [  255.062438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> [  255.062443] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x200000000 64bit pref]
> [  255.062445] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref]
> [  255.062447] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x100000000 64bit pref]
> [  255.062448] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref]
> [  255.062450] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> [  255.062451] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> [  255.062452] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.062465] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
> [  255.064884] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]


2. Re-tested environment 

kernel-3.10.0-820.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7.x86_64
seabios-1.11.0-1.el7.x86_64

3. Re-tested result:

1) Boot guest with qemu command line [1]

2) Hotplug memory through qmp
{'execute': 'object-add', 'arguments': {'id': 'shmmem-shmem0', 'qom-type': 'memory-backend-ram', 'props': {'policy': 'default', 'size': 4294967296}}}
{"return": {}}
{'execute': 'device_add', 'arguments':{'id': 'shmem0','driver': 'ivshmem-plain', 'memdev': 'shmmem-shmem0', 'bus':'root.3'}}
{"return": {}}

3). Checked test result in guest
[root@localhost home]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at 200000000 (64-bit, prefetchable) [size=4G]
	Kernel modules: virtio_pci

[root@localhost home]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]--
           |                               +-01.0-[05]--
           |                               \-02.0-[06]--
           +-04.0-[07]----00.0  Red Hat, Inc. Virtio network device
           +-05.0-[08-10]----00.0  Red Hat, Inc. Inter-VM shared memory
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

dmesg result:
[  107.253288] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
[  107.253329] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
[  107.253391] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
[  107.254011] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  107.254013] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  107.254015] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
[  107.254179] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.254185] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  107.254668] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  107.254999] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
[  107.255637] PCI: No. 2 try to assign unassigned res
[  107.255641] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
[  107.255642] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.256789] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000
[  107.256792] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  107.256793] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  107.256796] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref]
[  107.256798] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref]
[  107.257237] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.257245] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  107.257741] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  107.258076] pcieport 0000:00:05.0:   bridge window [mem 0x200000000-0x3ffffffff 64bit pref]

Compared the reproduce result, hotplug successfully

4) boot guest against through qemu command line [2]

5) Hotplug memory through qmp and check result in guest
[  112.823897] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed
[  112.823907] pciehp 0000:00:05.0:pcie004: Slot(4): Card present
[  112.823935] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press
[  113.925972] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
[  113.926012] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
[  113.926073] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
[  113.926651] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  113.926653] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  113.926654] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
[  113.926816] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.926823] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  113.927312] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  113.927629] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
[  113.928276] PCI: No. 2 try to assign unassigned res
[  113.928280] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
[  113.928281] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.929434] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08-10] add_size 100000000 add_align 100000000
[  113.929437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  113.929438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  113.929442] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref]
[  113.929443] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref]
[  113.929908] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.929915] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  113.930410] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  113.930736] pcieport 0000:00:05.0:   bridge window [mem 0x200000000-0x3ffffffff 64bit pref]


[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]--
           |                               +-01.0-[05]--
           |                               \-02.0-[06]--
           +-04.0-[07]----00.0  Red Hat, Inc. Virtio network device
           +-05.0-[08-10]----00.0  Red Hat, Inc. Inter-VM shared memory
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at 200000000 (64-bit, prefetchable) [size=4G]
	Kernel modules: virtio_pci


Addition info:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device pcie-root-port,id=root.0,slot=1 \
-drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device pcie-root-port,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device pcie-root-port,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-device pcie-root-port,id=root.3,slot=4\
-monitor stdio \

[2]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device pcie-root-port,id=root.0,slot=1 \
-drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device pcie-root-port,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device pcie-root-port,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-device pcie-root-port,id=root.3,slot=4,mem-reserve=4G \
-monitor stdio \


Thanks
Jing
Comment 26 jingzhao 2017-12-13 02:55:51 UTC 
Created attachment 1367064 [details]
dmesg with mem-reserver=4G
Comment 30 jingzhao 2017-12-20 06:14:22 UTC 
Created attachment 1370294 [details]
dmesg when used "mem-reserved=4G" on ovmf
Comment 33 jingzhao 2017-12-25 04:33:51 UTC 
Created attachment 1372003 [details]
dmesg that used upstream seabios with "mem-reserved=4G"
Comment 38 errata-xmlrpc 2018-04-11 00:12:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104