RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1390346 - PCI: Reserve MMIO space over 4G for PCI hotplug
Summary: PCI: Reserve MMIO space over 4G for PCI hotplug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: 7.4
Assignee: Marcel Apfelbaum
QA Contact: jingzhao
URL:
Whiteboard:
Depends On:
Blocks: 1408813 1410580 1410589
TreeView+ depends on / blocked
 
Reported: 2016-10-31 18:38 UTC by Marcel Apfelbaum
Modified: 2018-04-11 00:15 UTC (History)
8 users (show)

Fixed In Version: qemu-kvm-rhev-2.10.0-7.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1408813 1529618 (view as bug list)
Environment:
Last Closed: 2018-04-11 00:12:33 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
dmesg log (48.44 KB, text/plain)
2017-12-12 10:00 UTC, jingzhao 		no flags Details
dmesg with mem-reserver=4G (48.46 KB, text/plain)
2017-12-13 02:55 UTC, jingzhao 		no flags Details
dmesg when used "mem-reserved=4G" on ovmf (59.03 KB, text/plain)
2017-12-20 06:14 UTC, jingzhao 		no flags Details
dmesg that used upstream seabios with "mem-reserved=4G" (48.66 KB, text/plain)
2017-12-25 04:33 UTC, jingzhao 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1104 0 None None None 2018-04-11 00:15:02 UTC

Description Marcel Apfelbaum 2016-10-31 18:38:01 UTC 
QEMU reserves only 32-bit memory range for hotplug. The range can be rather limited and not enough to hot-plug devices with large BARs.

Add a parameter to QEMU to reserve a 64-bit range in CRS
starting after the memory ranged reserved for memory hotplug.

Check that all the range is addressable by VM's CPU.
Comment 1 jingzhao 2017-02-16 06:06:46 UTC 
Hi Marcel

  Could you provide some details on what actual use for the case or how can QE test it?

Thanks
Jing Zhao
Comment 2 Marcel Apfelbaum 2017-02-16 14:05:42 UTC 
(In reply to jingzhao from comment #1)
> Hi Marcel
> 
>   Could you provide some details on what actual use for the case or how can
> QE test it?
> 
> Thanks
> Jing Zhao

This is a little tricky, you have to create a configuration the has several PCI devices such as there is little MMIO range space in the 32-bit area. Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The hotplug will fail.
Be aware you need a 64bit guest.

After the feature will be implemented the hotplug should work by reserving the range from >4G address space.

Thanks,
Marcel
Comment 3 jingzhao 2017-02-21 06:22:12 UTC 
(In reply to Marcel Apfelbaum from comment #2)
> (In reply to jingzhao from comment #1)
> > Hi Marcel
> > 
> >   Could you provide some details on what actual use for the case or how can
> > QE test it?
> > 
> > Thanks
> > Jing Zhao
> 
> This is a little tricky, you have to create a configuration the has several
> PCI devices such as there is little MMIO range space in the 32-bit area.

--- Marcel, sorry, I have no idea about the confirguration, could you share an example with me?

> Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The
> hotplug will fail.

---Marcel, about the ivshmem-plain, you mean the following command and hotplug ivshmem-plain device?

-object memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \
-device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \

Thanks
Jing Zhao

> Be aware you need a 64bit guest.
> 
> After the feature will be implemented the hotplug should work by reserving
> the range from >4G address space.
> 
> Thanks,
> Marcel
Comment 4 Marcel Apfelbaum 2017-03-01 13:36:21 UTC 
(In reply to jingzhao from comment #3)
> (In reply to Marcel Apfelbaum from comment #2)
> > (In reply to jingzhao from comment #1)
> > > Hi Marcel
> > > 
> > >   Could you provide some details on what actual use for the case or how can
> > > QE test it?
> > > 
> > > Thanks
> > > Jing Zhao
> > 
> > This is a little tricky, you have to create a configuration the has several
> > PCI devices such as there is little MMIO range space in the 32-bit area.
> 
> --- Marcel, sorry, I have no idea about the confirguration, could you share
> an example with me?
> 

Sure, start with an usual command line ... (no need for a lot of devices)

> > Then try to hotplug a device with a huge BAR, lets say ivshmem-plain. The
> > hotplug will fail.
> 
> ---Marcel, about the ivshmem-plain, you mean the following command and
> hotplug ivshmem-plain device?
> 

yes

> -object
> memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share \
> -device ivshmem-plain,id=shmem0,memdev=shmmem-shmem0 \
> 

... and try to hotplug it and it will fail. You don't have 4G space in 32-bit
memory space and we can't take it from 64-bit space today.

This feature aims to solve the problem.

Thanks,
Marcel

> Thanks
> Jing Zhao
> 
> > Be aware you need a 64bit guest.
> > 
> > After the feature will be implemented the hotplug should work by reserving
> > the range from >4G address space.
> > 
> > Thanks,
> > Marcel
Comment 5 jingzhao 2017-03-13 06:31:30 UTC 
Hi Marcel

 Could you help to check the reproduced steps.

 According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64.

 1. Boot guest with qemu command line[1]

 2. Hotplug memory through qmp

(qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
(qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1

 3. Check the device through "info qtree" and "lspci" in guest

Actual Result:

Error message in dmesg:

[  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
[  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
[  175.533280] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]

Thanks
Jing Zhao
Comment 6 jingzhao 2017-03-13 06:32:58 UTC 
qemu command line [1]

Hi Marcel

 Could you help to check the reproduced steps.

 According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64 and qemu-kvm-rhev-2.8.0-5.el7.x86_64.

 1. Boot guest with qemu command line[1]

 2. Hotplug memory through qmp

(qemu) object_add memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
(qemu) device_add ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1

 3. Check the device through "info qtree" and "lspci" in guest

Actual Result:

Error message in dmesg:

[  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
[  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
[  175.533280] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]

[1]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device ioh3420,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-monitor stdio \


Thanks
Jing Zhao
Comment 7 Marcel Apfelbaum 2017-03-14 13:08:47 UTC 
(In reply to jingzhao from comment #6)
> qemu command line [1]
> 
> Hi Marcel
> 

Hi,

>  Could you help to check the reproduced steps.

Sure 

>  According to above comments, reproduce it with kernel-3.10.0-594.el7.x86_64
> and qemu-kvm-rhev-2.8.0-5.el7.x86_64.
> 
>  1. Boot guest with qemu command line[1]
> 
>  2. Hotplug memory through qmp
> 

hmp, not qmp :)

> (qemu) object_add
> memory-backend-file,id=shmmem-shmem0,size=4G,mem-path=/dev/shm/shmem0,share
> (qemu) device_add
> ivshmem-plain,id=shmem0,memdev=shmmem-shmem0,bus=downstream1
> 
>  3. Check the device through "info qtree" and "lspci" in guest
> 
> Actual Result:
> 
> Error message in dmesg:
> 
> [  175.530956] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000
> 64bit pref]
> [  175.530957] pci 0000:04:00.0: BAR 2: failed to assign [mem size
> 0x100000000 64bit pref]

Yes, the issue is reproduced correctly.

Thanks,
Marcel


> [  175.530958] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> [  175.531027] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> [  175.533280] pcieport 0000:03:00.0:   bridge window [mem
> 0xfc400000-0xfc5fffff]
> 
> [1]
> /usr/libexec/qemu-kvm \
> -M q35 \
> -cpu SandyBridge \
> -nodefaults -rtc base=utc \
> -m 4G \
> -smp 2,sockets=2,cores=1,threads=1 \
> -enable-kvm \
> -name rhel7.4 \
> -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
> -smbios type=1,manufacturer='Red Hat',product='RHEV
> Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-
> e63fcdb67170 \
> -k en-us \
> -serial unix:/tmp/console,server,nowait \
> -boot menu=on \
> -bios /usr/share/seabios/bios.bin \
> -chardev file,path=/home/test/seabios.log,id=seabios \
> -device isa-debugcon,chardev=seabios,iobase=0x402 \
> -qmp tcp::8887,server,nowait \
> -vga qxl \
> -spice port=5932,disable-ticketing \
> -device ioh3420,id=root.0,slot=1 \
> -drive
> file=/home/test/q35-seabios.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,
> cache=none,werror=stop,rerror=stop \
> -device
> virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-
> legacy=on,disable-modern=off,bootindex=1 \
> -device ioh3420,id=root.1,slot=2 \
> -device x3130-upstream,bus=root.1,id=upstream1 \
> -device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
> -device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
> -device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
> -device ioh3420,id=root.2,slot=3 \
> -netdev tap,id=hostnet1 \
> -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
> -monitor stdio \
> 
> 
> Thanks
> Jing Zhao
Comment 13 Marcel Apfelbaum 2017-11-14 12:46:40 UTC 
Patch posted upstream:
  - https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg02089.html
Comment 15 Miroslav Rezanina 2017-11-22 15:20:30 UTC 
Fix included in qemu-kvm-rhev-2.10.0-7.el7
Comment 17 Yaniv Kaul 2017-12-04 11:50:24 UTC 
Do we need anything in libvirt to allow this?
Comment 18 Marcel Apfelbaum 2017-12-06 19:11:36 UTC 
(In reply to Yaniv Kaul from comment #17)
> Do we need anything in libvirt to allow this?

We will need something from libvirt, but with low priority.

We decided to use a 2G MMIO window for PCI hot-plug for I440FX machine
and a 32G window for Q35.

If someone would want to hot-plug a device with larger BARs, they
would need to add a parameter to QEMU command line
It would be a corner case, but we need to handle it anyway.

Thanks,
Marcel
Comment 20 Marcel Apfelbaum 2017-12-12 09:39:21 UTC 
(In reply to jingzhao from comment #19)
> Hi Marcel
> 
>   tested the issue with latest qemu-kvm-rhev-2.10.0-10.el7.x86_64 according
> to comment 7, also can see the error message, the same error when hotplug to
> pcie-root-port
> 
> [root@localhost ~]# diff tmp tmp1
> 711a712,739
> > [  109.296785] pciehp 0000:03:00.0:pcie204: Slot(0): Attention button pressed
> > [  109.296795] pciehp 0000:03:00.0:pcie204: Slot(0): Card present
> > [  109.296825] pciehp 0000:03:00.0:pcie204: Slot(0) Powering on due to button press
> > [  110.398812] pci 0000:04:00.0: [1af4:1110] type 00 class 0x050000
> > [  110.398865] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
> > [  110.398931] pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
> > [  110.399469] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> > [  110.399471] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.399473] pci 0000:04:00.0: BAR 0: assigned [mem 0xfc400000-0xfc4000ff]
> > [  110.399611] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.399617] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> > [  110.400109] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]
> > [  110.400404] pcieport 0000:03:00.0:   bridge window [mem 0xfe400000-0xfe5fffff 64bit pref]
> > [  110.401001] PCI: No. 2 try to assign unassigned res
> > [  110.401004] pcieport 0000:03:00.0: resource 15 [mem 0xfe400000-0xfe5fffff 64bit pref] released
> > [  110.401005] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.402072] pcieport 0000:03:00.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 04] add_size 100000000 add_align 100000000
> > [  110.402074] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> > [  110.402075] pcieport 0000:03:00.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> > [  110.402078] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x200000000 64bit pref]
> > [  110.402079] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref]
> > [  110.402081] pcieport 0000:03:00.0: BAR 15: no space for [mem size 0x100000000 64bit pref]
> > [  110.402082] pcieport 0000:03:00.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.402084] pci 0000:04:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> > [  110.402085] pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> > [  110.402086] pcieport 0000:03:00.0: PCI bridge to [bus 04]
> > [  110.402092] pcieport 0000:03:00.0:   bridge window [io  0x2000-0x2fff]
> > [  110.402536] pcieport 0000:03:00.0:   bridge window [mem 0xfc400000-0xfc5fffff]
> [root@localhost ~]# lspci 
> 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
> 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic
> card (rev 04)
> 00:02.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:04.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:05.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
> 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface
> Controller (rev 02)
> 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6
> port SATA Controller [AHCI mode] (rev 02)
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev
> 02)
> 01:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
> 02:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream)
> (rev 02)
> 03:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 03:01.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 03:02.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch
> (Downstream) (rev 01)
> 04:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
> 07:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> 
> 
> Could you help to check ?

Should work... please add the dmesg, I want to see the pci windows.

Also, please don't use switches/intel io port, use pcie-root-port

Thanks,
Marcel


> 
> Thanks
> Jing
Comment 22 jingzhao 2017-12-12 10:00:34 UTC 
Created attachment 1366514 [details]
dmesg log
Comment 25 jingzhao 2017-12-13 02:48:49 UTC 
Summarized the test result

1.Reproduced  on qemu-kvm-rhev-2.10.0-1.el7.x86_64

checked the test result:

[root@localhost home]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at <unassigned> (64-bit, prefetchable)
	Kernel modules: virtio_pci

dmesg result:
> [  253.942692] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed
> [  253.942700] pciehp 0000:00:05.0:pcie004: Slot(4): Card present
> [  253.942726] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press
> [  255.044112] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
> [  255.044152] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
> [  255.044213] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
> [  255.046090] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> [  255.046092] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> [  255.046094] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
> [  255.046863] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.046869] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
> [  255.049238] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
> [  255.050798] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
> [  255.053912] PCI: No. 2 try to assign unassigned res
> [  255.053917] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
> [  255.053919] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.062433] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000
> [  255.062437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> [  255.062438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
> [  255.062443] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x200000000 64bit pref]
> [  255.062445] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x200000000 64bit pref]
> [  255.062447] pcieport 0000:00:05.0: BAR 15: no space for [mem size 0x100000000 64bit pref]
> [  255.062448] pcieport 0000:00:05.0: BAR 15: failed to assign [mem size 0x100000000 64bit pref]
> [  255.062450] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
> [  255.062451] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
> [  255.062452] pcieport 0000:00:05.0: PCI bridge to [bus 08]
> [  255.062465] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
> [  255.064884] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]


2. Re-tested environment 

kernel-3.10.0-820.el7.x86_64
qemu-kvm-rhev-2.10.0-12.el7.x86_64
seabios-1.11.0-1.el7.x86_64

3. Re-tested result:

1) Boot guest with qemu command line [1]

2) Hotplug memory through qmp
{'execute': 'object-add', 'arguments': {'id': 'shmmem-shmem0', 'qom-type': 'memory-backend-ram', 'props': {'policy': 'default', 'size': 4294967296}}}
{"return": {}}
{'execute': 'device_add', 'arguments':{'id': 'shmem0','driver': 'ivshmem-plain', 'memdev': 'shmmem-shmem0', 'bus':'root.3'}}
{"return": {}}

3). Checked test result in guest
[root@localhost home]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at 200000000 (64-bit, prefetchable) [size=4G]
	Kernel modules: virtio_pci

[root@localhost home]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]--
           |                               +-01.0-[05]--
           |                               \-02.0-[06]--
           +-04.0-[07]----00.0  Red Hat, Inc. Virtio network device
           +-05.0-[08-10]----00.0  Red Hat, Inc. Inter-VM shared memory
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

dmesg result:
[  107.253288] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
[  107.253329] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
[  107.253391] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
[  107.254011] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  107.254013] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  107.254015] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
[  107.254179] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.254185] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  107.254668] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  107.254999] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
[  107.255637] PCI: No. 2 try to assign unassigned res
[  107.255641] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
[  107.255642] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.256789] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08] add_size 100000000 add_align 100000000
[  107.256792] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  107.256793] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  107.256796] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref]
[  107.256798] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref]
[  107.257237] pcieport 0000:00:05.0: PCI bridge to [bus 08]
[  107.257245] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  107.257741] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  107.258076] pcieport 0000:00:05.0:   bridge window [mem 0x200000000-0x3ffffffff 64bit pref]

Compared the reproduce result, hotplug successfully

4) boot guest against through qemu command line [2]

5) Hotplug memory through qmp and check result in guest
[  112.823897] pciehp 0000:00:05.0:pcie004: Slot(4): Attention button pressed
[  112.823907] pciehp 0000:00:05.0:pcie004: Slot(4): Card present
[  112.823935] pciehp 0000:00:05.0:pcie004: Slot(4) Powering on due to button press
[  113.925972] pci 0000:08:00.0: [1af4:1110] type 00 class 0x050000
[  113.926012] pci 0000:08:00.0: reg 0x10: [mem 0x00000000-0x000000ff]
[  113.926073] pci 0000:08:00.0: reg 0x18: [mem 0x00000000-0xffffffff 64bit pref]
[  113.926651] pci 0000:08:00.0: BAR 2: no space for [mem size 0x100000000 64bit pref]
[  113.926653] pci 0000:08:00.0: BAR 2: failed to assign [mem size 0x100000000 64bit pref]
[  113.926654] pci 0000:08:00.0: BAR 0: assigned [mem 0xfc600000-0xfc6000ff]
[  113.926816] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.926823] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  113.927312] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  113.927629] pcieport 0000:00:05.0:   bridge window [mem 0xfe600000-0xfe7fffff 64bit pref]
[  113.928276] PCI: No. 2 try to assign unassigned res
[  113.928280] pcieport 0000:00:05.0: resource 15 [mem 0xfe600000-0xfe7fffff 64bit pref] released
[  113.928281] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.929434] pcieport 0000:00:05.0: bridge window [mem 0x100000000-0x1ffffffff 64bit pref] to [bus 08-10] add_size 100000000 add_align 100000000
[  113.929437] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x1ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  113.929438] pcieport 0000:00:05.0: res[15]=[mem 0x100000000-0x2ffffffff 64bit pref] res_to_dev_res add_size 100000000 min_align 100000000
[  113.929442] pcieport 0000:00:05.0: BAR 15: assigned [mem 0x200000000-0x3ffffffff 64bit pref]
[  113.929443] pci 0000:08:00.0: BAR 2: assigned [mem 0x200000000-0x2ffffffff 64bit pref]
[  113.929908] pcieport 0000:00:05.0: PCI bridge to [bus 08-10]
[  113.929915] pcieport 0000:00:05.0:   bridge window [io  0x6000-0x6fff]
[  113.930410] pcieport 0000:00:05.0:   bridge window [mem 0xfc600000-0xfc7fffff]
[  113.930736] pcieport 0000:00:05.0:   bridge window [mem 0x200000000-0x3ffffffff 64bit pref]


[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]--
           |                               +-01.0-[05]--
           |                               \-02.0-[06]--
           +-04.0-[07]----00.0  Red Hat, Inc. Virtio network device
           +-05.0-[08-10]----00.0  Red Hat, Inc. Inter-VM shared memory
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci -v -s 08:00.0
08:00.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)
	Subsystem: Red Hat, Inc. QEMU Virtual Machine
	Physical Slot: 4
	Flags: fast devsel
	Memory at fc600000 (32-bit, non-prefetchable) [size=256]
	Memory at 200000000 (64-bit, prefetchable) [size=4G]
	Kernel modules: virtio_pci


Addition info:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device pcie-root-port,id=root.0,slot=1 \
-drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device pcie-root-port,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device pcie-root-port,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-device pcie-root-port,id=root.3,slot=4\
-monitor stdio \

[2]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/console,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/test/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp::8887,server,nowait \
-vga qxl \
-spice port=5932,disable-ticketing \
-device pcie-root-port,id=root.0,slot=1 \
-drive file=/home/test/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device pcie-root-port,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-device pcie-root-port,id=root.2,slot=3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22,bus=root.2 \
-device pcie-root-port,id=root.3,slot=4,mem-reserve=4G \
-monitor stdio \


Thanks
Jing
Comment 26 jingzhao 2017-12-13 02:55:51 UTC 
Created attachment 1367064 [details]
dmesg with mem-reserver=4G
Comment 30 jingzhao 2017-12-20 06:14:22 UTC 
Created attachment 1370294 [details]
dmesg when used "mem-reserved=4G" on ovmf
Comment 33 jingzhao 2017-12-25 04:33:51 UTC 
Created attachment 1372003 [details]
dmesg that used upstream seabios with "mem-reserved=4G"
Comment 38 errata-xmlrpc 2018-04-11 00:12:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104
Note You need to log in before you can comment on or make changes to this bug.