Bug 1352808

Summary: [Q35] Fail to assign [mem size 0x01000000 64bit pref] when hot-plug virtio block device
Product: Red Hat Enterprise Linux 7 Reporter: jingzhao <jinzhao>
Component: qemu-kvm-rhevAssignee: Marcel Apfelbaum <marcel>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: chayang, juzhang, knoel, marcel, virt-maint, yama, yanghliu, yduan, yiwei
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-08 11:29:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1288337    
Attachments:
Description Flags
dmesg log none

Description jingzhao 2016-07-05 06:53:43 UTC
Description of problem:
Guest prompt error messages when hot-plug virtio block devices 
[   90.401588] pcieport 0000:03:02.0: BAR 15: no space for [mem size 0x01000000 64bit pref]
[   90.401589] pcieport 0000:03:02.0: BAR 15: failed to assign [mem size 0x01000000 64bit pref]
[   90.401592] pcieport 0000:03:02.0: BAR 15: assigned [mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401594] pcieport 0000:03:02.0: BAR 15: [mem 0xfd800000-0xfdffffff 64bit pref] (failed to expand by 0x800000)
[   90.401595] pcieport 0000:03:02.0: failed to add 800000 res[15]=[mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401597] pci 0000:06:00.0: BAR 4: assigned [mem 0xfd800000-0xfdffffff 64bit pref]

Version-Release number of selected component (if applicable):
kernel-3.10.0-461.el7.x86_64
qemu-kvm-rhev-2.6.0-11.el7.x86_64
seabios-1.9.1-4.el7.x86_64

How reproducible:
3/3

Steps to Reproduce:
1. Boot guest with following cli:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.3 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-vga qxl \
-spice port=5932,disable-ticketing \
-qmp tcp:0:6666,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/rhel73.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-drive file=/home/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream1,drive=drive-virtio-disk1,id=virtio-disk1,disable-legacy=on,disable-modern=off \
-drive file=/home/block2.raw,if=none,id=drive-virtio-disk2,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream2,drive=drive-virtio-disk2,id=virtio-disk2,disable-legacy=on,disable-modern=off \
-device ioh3420,id=root.2,slot=3 \
-device virtio-net-pci,bus=root.2,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device ioh3420,id=root.3,slot=4 \
-monitor stdio \

2. Hot-plug devices to root port and downstream port
In qmp
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/block4.qcow2","format":"qcow2","id":"test31"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test31","id":"test31","disable-legacy":"on","disable-modern":"off","bus":"root.3"}}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/block3.raw","format":"raw","id":"test30"}}{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test30","id":"test30","disable-legacy":"on","disable-modern":"off","bus":"downstream3"}}
{"return": {}}


3. In hmp:
(qemu) info block
drive-virtio-disk0 (#block189): /home/rhel73.img (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk1 (#block321): /home/block1.qcow2 (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk2 (#block542): /home/block2.raw (raw)
    Cache mode:       writeback, direct

test31 (#block765): /home/block4.qcow2 (qcow2)
    Cache mode:       writeback

test30 (#block922): /home/block3.raw (raw)
    Cache mode:       writeback


4. In guest, check the dmesg


Actual results:
[   90.401588] pcieport 0000:03:02.0: BAR 15: no space for [mem size 0x01000000 64bit pref]
[   90.401589] pcieport 0000:03:02.0: BAR 15: failed to assign [mem size 0x01000000 64bit pref]
[   90.401592] pcieport 0000:03:02.0: BAR 15: assigned [mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401594] pcieport 0000:03:02.0: BAR 15: [mem 0xfd800000-0xfdffffff 64bit pref] (failed to expand by 0x800000)
[   90.401595] pcieport 0000:03:02.0: failed to add 800000 res[15]=[mem 0xfd800000-0xfdffffff 64bit pref]
[   90.401597] pci 0000:06:00.0: BAR 4: assigned [mem 0xfd800000-0xfdffffff 64bit pref]


Expected results:
No error message and guest run successfully

Additional info:
No error messages when boot guest with these virtio block devices

Comment 1 jingzhao 2016-07-05 06:56:18 UTC
Created attachment 1176264 [details]
dmesg  log

Comment 3 Marcel Apfelbaum 2016-08-08 11:29:44 UTC
Hi,

This is actually not a problem. The error messages from dmesg result from the fact the firmware may assign less MMIO space than virtio-1 devices need. The virtio 1.0 devices need 8MB space while SeaBIOS (for example) reserves only 2MB.

Some guests, Linux for example, will try to re-balance the PCI root bus MMIO space and assign to pci-bridges/pcie-ports more memory space. Sometimes the guest OS will succeed and sometimes not.

The example you presented is actually a good one with a happy ending. The device runs OK, right?

Try the same with a i440fx machine and a pci-bridge while requesting virtio-1 mode and the Linux (on guest) will not even try to re-balance.

Again, the root-cause is virtio-1 mode that requests more MMIO space than reserved by the firmware. What I will take for this BZ is we need to see what to do about this discrepancy.

Thanks,
Marcel

Comment 4 Marcel Apfelbaum 2016-08-08 13:16:25 UTC
*** Bug 1353141 has been marked as a duplicate of this bug. ***

Comment 5 Marcel Apfelbaum 2016-08-08 14:18:30 UTC
*** Bug 1355661 has been marked as a duplicate of this bug. ***

Comment 6 Marcel Apfelbaum 2016-08-09 17:02:16 UTC
*** Bug 1245125 has been marked as a duplicate of this bug. ***

Comment 7 Marcel Apfelbaum 2016-08-09 17:22:32 UTC
I have decided to open BZ 1365613 to track the root cause of this BZ.

Thanks,
Marcel