Bug 1352808 - [Q35] Fail to assign [mem size 0x01000000 64bit pref] when hot-plug virtio block device
Summary: [Q35] Fail to assign [mem size 0x01000000 64bit pref] when hot-plug virtio bl...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Marcel Apfelbaum
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1245125 1353141 1355661 (view as bug list)
Depends On:
Blocks: 1288337
TreeView+ depends on / blocked
 
Reported: 2016-07-05 06:53 UTC by jingzhao
Modified: 2020-06-16 02:42 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-08 11:29:44 UTC
Target Upstream Version:


Attachments (Terms of Use)
dmesg log (51.22 KB, text/plain)
2016-07-05 06:56 UTC, jingzhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1365613 None CLOSED [PCI] The default MMIO range reserved by firmware for PCI bridges is not enough to hotplug virtio-1 devices 2019-09-11 07:22:08 UTC

Internal Links: 1365613

Description jingzhao 2016-07-05 06:53:43 UTC
Description of problem:
Guest prompt error messages when hot-plug virtio block devices 
[   90.401588] pcieport 0000:03:02.0: BAR 15: no space for [mem size 0x01000000 64bit pref]
[   90.401589] pcieport 0000:03:02.0: BAR 15: failed to assign [mem size 0x01000000 64bit pref]
[   90.401592] pcieport 0000:03:02.0: BAR 15: assigned [mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401594] pcieport 0000:03:02.0: BAR 15: [mem 0xfd800000-0xfdffffff 64bit pref] (failed to expand by 0x800000)
[   90.401595] pcieport 0000:03:02.0: failed to add 800000 res[15]=[mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401597] pci 0000:06:00.0: BAR 4: assigned [mem 0xfd800000-0xfdffffff 64bit pref]

Version-Release number of selected component (if applicable):
kernel-3.10.0-461.el7.x86_64
qemu-kvm-rhev-2.6.0-11.el7.x86_64
seabios-1.9.1-4.el7.x86_64

How reproducible:
3/3

Steps to Reproduce:
1. Boot guest with following cli:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.3 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-vga qxl \
-spice port=5932,disable-ticketing \
-qmp tcp:0:6666,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/rhel73.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-drive file=/home/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream1,drive=drive-virtio-disk1,id=virtio-disk1,disable-legacy=on,disable-modern=off \
-drive file=/home/block2.raw,if=none,id=drive-virtio-disk2,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream2,drive=drive-virtio-disk2,id=virtio-disk2,disable-legacy=on,disable-modern=off \
-device ioh3420,id=root.2,slot=3 \
-device virtio-net-pci,bus=root.2,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device ioh3420,id=root.3,slot=4 \
-monitor stdio \

2. Hot-plug devices to root port and downstream port
In qmp
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/block4.qcow2","format":"qcow2","id":"test31"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test31","id":"test31","disable-legacy":"on","disable-modern":"off","bus":"root.3"}}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/block3.raw","format":"raw","id":"test30"}}{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test30","id":"test30","disable-legacy":"on","disable-modern":"off","bus":"downstream3"}}
{"return": {}}


3. In hmp:
(qemu) info block
drive-virtio-disk0 (#block189): /home/rhel73.img (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk1 (#block321): /home/block1.qcow2 (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk2 (#block542): /home/block2.raw (raw)
    Cache mode:       writeback, direct

test31 (#block765): /home/block4.qcow2 (qcow2)
    Cache mode:       writeback

test30 (#block922): /home/block3.raw (raw)
    Cache mode:       writeback


4. In guest, check the dmesg


Actual results:
[   90.401588] pcieport 0000:03:02.0: BAR 15: no space for [mem size 0x01000000 64bit pref]
[   90.401589] pcieport 0000:03:02.0: BAR 15: failed to assign [mem size 0x01000000 64bit pref]
[   90.401592] pcieport 0000:03:02.0: BAR 15: assigned [mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401594] pcieport 0000:03:02.0: BAR 15: [mem 0xfd800000-0xfdffffff 64bit pref] (failed to expand by 0x800000)
[   90.401595] pcieport 0000:03:02.0: failed to add 800000 res[15]=[mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401597] pci 0000:06:00.0: BAR 4: assigned [mem 0xfd800000-0xfdffffff 64bit pref]


Expected results:
No error message and guest run successfully

Additional info:
No error messages when boot guest with these virtio block devices

Comment 1 jingzhao 2016-07-05 06:56:18 UTC
Created attachment 1176264 [details]
dmesg  log

Comment 3 Marcel Apfelbaum 2016-08-08 11:29:44 UTC
Hi,

This is actually not a problem. The error messages from dmesg result from the fact the firmware may assign less MMIO space than virtio-1 devices need. The virtio 1.0 devices need 8MB space while SeaBIOS (for example) reserves only 2MB.

Some guests, Linux for example, will try to re-balance the PCI root bus MMIO space and assign to pci-bridges/pcie-ports more memory space. Sometimes the guest OS will succeed and sometimes not.

The example you presented is actually a good one with a happy ending. The device runs OK, right?

Try the same with a i440fx machine and a pci-bridge while requesting virtio-1 mode and the Linux (on guest) will not even try to re-balance.

Again, the root-cause is virtio-1 mode that requests more MMIO space than reserved by the firmware. What I will take for this BZ is we need to see what to do about this discrepancy.

Thanks,
Marcel

Comment 4 Marcel Apfelbaum 2016-08-08 13:16:25 UTC
*** Bug 1353141 has been marked as a duplicate of this bug. ***

Comment 5 Marcel Apfelbaum 2016-08-08 14:18:30 UTC
*** Bug 1355661 has been marked as a duplicate of this bug. ***

Comment 6 Marcel Apfelbaum 2016-08-09 17:02:16 UTC
*** Bug 1245125 has been marked as a duplicate of this bug. ***

Comment 7 Marcel Apfelbaum 2016-08-09 17:22:32 UTC
I have decided to open BZ 1365613 to track the root cause of this BZ.

Thanks,
Marcel


Note You need to log in before you can comment on or make changes to this bug.