Bug 1365613

Summary:	[PCI] The default MMIO range reserved by firmware for PCI bridges is not enough to hotplug virtio-1 devices
Product:	Red Hat Enterprise Linux 7	Reporter:	Marcel Apfelbaum <marcel>
Component:	qemu-kvm-rhev	Assignee:	Marcel Apfelbaum <marcel>
Status:	CLOSED ERRATA	QA Contact:	jingzhao <jinzhao>
Severity:	high	Docs Contact:
Priority:	high
Version:	7.3	CC:	ailan, chayang, dyuan, ehabkost, jinzhao, juzhang, lmiksik, marcel, mst, sherold, virt-bugs, virt-maint, xfu, xuzhang, yanghliu, yanyang, yduan
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu-kvm-rhev-2.6.0-26.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1365619 1365623 (view as bug list)		Environment:
Last Closed:	2016-11-07 21:29:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1365619, 1365623, 1375166

Description Marcel Apfelbaum 2016-08-09 17:18:15 UTC

The virtio 1.0 devices need 8MB MMIO space while SeaBIOS(OVMF?) reserves only 2MB.

The reason the firmware reserves 'only' 2MB MMIO is because it is the minimum allowed by the PCI Spec. It will be unreasonable to default in firmware to a larger MMIO window because it is related only 2 virtio. What about assigned devices that may need more?

I propose the following solution. Add a mmio-window-size parameter to pci bridges and pass all the information to firmware using a para-virt channel like fwconfig.

The firmware can use this info to increase the mmio range for our devices.

We can default the mmio-window-size to 8MB for PCIe ports (which are seen by the firmware as PCI bridges). This will allow hot-plugging virtio-1 devices into PCIe ports with no problem.

Regarding the legacy pci-bridges, the default size is not so clear. I propose to
leave it as is with the following reasoning: On pc 'legacy' machines it is likely we don't need to hotplug virtio-1 'modern' devices, just hotplug virtio 0.9 devices instead.  

Side-note:
Some guests, Linux for example, will try to re-balance the PCI root bus MMIO space and assign to pci-bridges/pcie-ports more memory space. Sometimes the guest OS will succeed and sometimes not.

Comment 2 Marcel Apfelbaum 2016-09-06 10:53:46 UTC

See https://bugzilla.redhat.com/show_bug.cgi?id=1365623#c4 for the selected solution.

Comment 6 Marcel Apfelbaum 2016-09-08 09:33:28 UTC

Please follow https://bugzilla.redhat.com/show_bug.cgi?id=1365623
for more details.

Patch posted upstream:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg395607.html

virtio 8MB BAR is reduced to 16K


Thanks,
Marcel

Comment 7 Marcel Apfelbaum 2016-09-08 09:35:56 UTC

*** Bug 1365623 has been marked as a duplicate of this bug. ***

Comment 8 Laszlo Ersek 2016-09-08 09:39:15 UTC

*** Bug 1365619 has been marked as a duplicate of this bug. ***

Comment 9 Miroslav Rezanina 2016-09-20 12:29:28 UTC

Fix included in qemu-kvm-rhev-2.6.0-26.el7

Comment 11 jingzhao 2016-09-21 06:15:17 UTC

Reproduce the bz on qemu-kvm-rhev-2.6.0-23.el7.x86_64
1.Boot the guest with pci-bridge

2.Hot-plug virtio devices through qmp
{"execute":"__com.redhat_drive_add", "arguments": {"file":"/home/pci-bridge/block2.qcow2","format":"qcow2","id":"test2"
}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test2","id":"test2","bus":"bridge1","addr":"0x4"}}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments": {"file":"/home/pci-bridge/test.raw","format":"raw","id":"drive_datadisk0"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":"scsi1","bus":"bridge1","addr":"0x5"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"drive_datadisk0","id":"device_datadisk0","bus":"scsi1.0"}}
{"return": {}}
{"execute": "netdev_add", "arguments": { "type":"tap","id":"net1"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"virtio-net-pci","netdev":"net1","mac":"22:11:22:45:61:98","id":"net1","bus":"bridge1","addr":"0x6"}}
{"return": {}}

3.Check the virtio devices in guest
[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 440FX - 82441FX PMC [Natoma]
           +-01.0  Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
           +-01.1  Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
           +-01.3  Intel Corporation 82371AB/EB/MB PIIX4 ACPI
           +-02.0-[01]--+-01.0  Red Hat, Inc Virtio block device
           |            +-02.0  Red Hat, Inc Virtio network device
           |            +-03.0  Device 1234:1111
           |            +-04.0  Red Hat, Inc Virtio block device
           |            +-05.0  Red Hat, Inc Virtio SCSI
           |            \-06.0  Red Hat, Inc Virtio network device
           \-03.0  Red Hat, Inc Virtio block device

[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:01.0/virtio1/features 
0010101001110000000000000000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:04.0/virtio3/features 
0010101001110000000000000000110000000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:05.0/virtio4/features 
0110000000000000000000000000110000000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:06.0/virtio5/features 
1100011111111111111101010001110000000000000000000000000000000000

[root@localhost ~]# lspci -vvv 
01:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
	Subsystem: Red Hat, Inc Device 0002
	Physical Slot: 1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c000 [size=64]
	Region 1: Memory at fe850000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd000000 (64-bit, prefetchable) [size=8M]
	Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00400000 multiplier=00001000
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:02.0 Ethernet controller: Red Hat, Inc Virtio network device
	Subsystem: Red Hat, Inc Device 0001
	Physical Slot: 2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c040 [size=32]
	Region 1: Memory at fe851000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd800000 (64-bit, prefetchable) [size=8M]
	Expansion ROM at fe800000 [disabled] [size=256K]
	Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00400000 multiplier=00001000
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:03.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller])
	Subsystem: Red Hat, Inc Device 1100
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: Memory at fc000000 (32-bit, prefetchable) [size=16M]
	Region 2: Memory at fe852000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at fe840000 [disabled] [size=64K]
	Kernel driver in use: bochs-drm
	Kernel modules: bochs_drm

01:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
	Subsystem: Red Hat, Inc Device 0002
	Physical Slot: 4-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 10
	Region 0: I/O ports at c080 [size=64]
	Region 1: Memory at fe853000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at <unassigned> (64-bit, prefetchable)
	Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00400000 multiplier=00001000
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:05.0 SCSI storage controller: Red Hat, Inc Virtio SCSI
	Subsystem: Red Hat, Inc Device 0008
	Physical Slot: 5-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c0c0 [size=64]
	Region 1: Memory at fe854000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at <unassigned> (64-bit, prefetchable)
	Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00400000 multiplier=00001000
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:06.0 Ethernet controller: Red Hat, Inc Virtio network device
	Subsystem: Red Hat, Inc Device 0001
	Physical Slot: 6-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c060 [size=32]
	Region 1: Memory at fe855000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at <unassigned> (64-bit, prefetchable)
	[virtual] Expansion ROM at fe880000 [disabled] [size=256K]
	Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00400000 multiplier=00001000
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci


Verified the bz on qemu-kvm-rhev-2.6.0-26.el7.x86_64
1.Boot guest with pci-bridge
2.Hot-plug virtio devices through qmp (same with the reproduce step2)
3.Check the info in hmp
(qemu) info block
drive-virtio-disk0 (#block182): /home/pci-bridge/big.img (qcow2)
    Cache mode:       writeback, direct

drive-virtio-disk1 (#block367): /home/pci-bridge/block1.qcow2 (qcow2)
    Cache mode:       writeback, direct

test2 (#block579): /home/pci-bridge/block2.qcow2 (qcow2)
    Cache mode:       writeback

drive_datadisk0 (#block754): /home/pci-bridge/test.raw (raw)
    Cache mode:       writeback
(qemu) info network 
virtio-net-pci.0: index=0,type=nic,model=virtio-net-pci,macaddr=9a:6a:6b:6c:6d:6e
 \ tap10: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net1: index=0,type=nic,model=virtio-net-pci,macaddr=22:11:22:45:61:98
 \ net1: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
4.Check the virtio devices info in guest
1)check the devices which hot-plug work well
[root@localhost ~]# ping 10.66.4.211 -I eth1
PING 10.66.4.211 (10.66.4.211) from 10.66.5.77 eth1: 56(84) bytes of data.
64 bytes from 10.66.4.211: icmp_seq=1 ttl=64 time=1.72 ms
64 bytes from 10.66.4.211: icmp_seq=2 ttl=64 time=0.970 ms
64 bytes from 10.66.4.211: icmp_seq=3 ttl=64 time=0.970 ms
^C
--- 10.66.4.211 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.970/1.220/1.721/0.355 ms
[root@localhost ~]# dd if=/dev/zero of=/dev/sda bs=1M count=500
dd: error writing ‘/dev/sda’: No space left on device
101+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0848982 s, 1.2 GB/s
[root@localhost ~]# dd if=/dev/zero of=/dev/vdc bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.653543 s, 1.6 GB/s

2)check the topology in guest through lspci
[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 440FX - 82441FX PMC [Natoma]
           +-01.0  Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
           +-01.1  Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
           +-01.3  Intel Corporation 82371AB/EB/MB PIIX4 ACPI
           +-02.0-[01]--+-01.0  Red Hat, Inc Virtio block device
           |            +-02.0  Red Hat, Inc Virtio network device
           |            +-03.0  Device 1234:1111
           |            +-04.0  Red Hat, Inc Virtio block device
           |            +-05.0  Red Hat, Inc Virtio SCSI
           |            \-06.0  Red Hat, Inc Virtio network device
           \-03.0  Red Hat, Inc Virtio block device

3)check the virtio driver
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:01.0/virtio1/features 
0010101001110000000000000000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:02.0/virtio2/features 
1100011111111111111101010000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:04.0/virtio3/features 
0010101001110000000000000000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:05.0/virtio4/features 
0110000000000000000000000000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:01\:06.0/virtio5/features 
1100011111111111111101010000110010000000000000000000000000000000

4)Check the memory size in guest through lspci -vvv
[root@localhost ~]# lspci -vvv
01:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
	Subsystem: Red Hat, Inc Device 0002
	Physical Slot: 1
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c000 [size=64]
	Region 1: Memory at fe850000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd000000 (64-bit, prefetchable) [size=16K]
	Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00001000 multiplier=00000004
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:02.0 Ethernet controller: Red Hat, Inc Virtio network device
	Subsystem: Red Hat, Inc Device 0001
	Physical Slot: 2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c040 [size=32]
	Region 1: Memory at fe851000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd004000 (64-bit, prefetchable) [size=16K]
	Expansion ROM at fe800000 [disabled] [size=256K]
	Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00001000 multiplier=00000004
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:03.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller])
	Subsystem: Red Hat, Inc Device 1100
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: Memory at fc000000 (32-bit, prefetchable) [size=16M]
	Region 2: Memory at fe852000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at fe840000 [disabled] [size=64K]
	Kernel driver in use: bochs-drm
	Kernel modules: bochs_drm

01:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
	Subsystem: Red Hat, Inc Device 0002
	Physical Slot: 4-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 10
	Region 0: I/O ports at c080 [size=64]
	Region 1: Memory at fe853000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd008000 (64-bit, prefetchable) [size=16K]
	Capabilities: [98] MSI-X: Enable+ Count=2 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00001000 multiplier=00000004
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:05.0 SCSI storage controller: Red Hat, Inc Virtio SCSI
	Subsystem: Red Hat, Inc Device 0008
	Physical Slot: 5-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c0c0 [size=64]
	Region 1: Memory at fe854000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd00c000 (64-bit, prefetchable) [size=16K]
	Capabilities: [98] MSI-X: Enable+ Count=4 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00001000 multiplier=00000004
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

01:06.0 Ethernet controller: Red Hat, Inc Virtio network device
	Subsystem: Red Hat, Inc Device 0001
	Physical Slot: 6-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 11
	Region 0: I/O ports at c060 [size=32]
	Region 1: Memory at fe855000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd010000 (64-bit, prefetchable) [size=16K]
	[virtual] Expansion ROM at fe880000 [disabled] [size=256K]
	Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
		Vector table: BAR=1 offset=00000000
		PBA: BAR=1 offset=00000800
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
		BAR=0 offset=00000000 size=00000000
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
		BAR=4 offset=00003000 size=00001000 multiplier=00000004
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
		BAR=4 offset=00002000 size=00001000
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
		BAR=4 offset=00001000 size=00001000
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
		BAR=4 offset=00000000 size=00001000
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

Focus on the result

1. the virtio driver that devices through hot-plug belong to virtio 1.0, please check the step4.3
2. Check the Memory size, like follwing

01:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
..........
Region 4: Memory at fd000000 (64-bit, prefetchable) [size=8M] ---> Region 4: Memory at fd000000 (64-bit, prefetchable) [size=16K]

01:06.0 Ethernet controller: Red Hat, Inc Virtio network device
............
Region 4: Memory at <unassigned> (64-bit, prefetchable)  -->  Region 4: Memory at fd010000 (64-bit, prefetchable) [size=16K]

Add info:

/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.3 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-nodefaults \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/pci-bridge/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-qmp tcp:0:6666,server,nowait \
-device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1 \
-drive file=/home/pci-bridge/big.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e,bus=bridge1,addr=0x2 -netdev tap,id=tap10 \
-drive file=/home/pci-bridge/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk1,id=virtio-disk1,bus=bridge1,addr=0x1 \
-device VGA,id=video,bus=bridge1,addr=0x3 \
-vnc :2 \
-monitor stdio \

Comment 12 jingzhao 2016-09-21 06:35:43 UTC

For q35 machine type

Reproduce the bz on qemu-kvm-rhev-2.6.0-23.el7.x86_64
1.Boot guest with q35 machine type
2.Hot plug virtio devices to the root port or downstream port
{"execute":"__com.redhat_drive_add", "arguments": {"file":"/home/pci-bridge/block4.qcow2","format":"qcow2","id":"test2"
}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test2","id":"test2","bus":"root.3"}}
{"return": {}}
{"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/pci-bridge/block3.raw","format":"raw","id":"test30"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test30","id":"test30","bus":"downstream3"}}
{"return": {}}
3.Check the info in the guest
[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]----00.0  Red Hat, Inc Virtio block device
           |                               +-01.0-[05]----00.0  Red Hat, Inc Virtio block device
           |                               \-02.0-[06]----00.0  Red Hat, Inc Virtio block device
           +-04.0-[07]----00.0  Red Hat, Inc Virtio network device
           +-05.0-[08]----00.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

[root@localhost ~]# lspci -vvv
.........
06:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
	Subsystem: Red Hat, Inc Device 1100
	Physical Slot: 0-2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 21
	Region 1: Memory at f8000000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fd000000 (64-bit, prefetchable) [size=8M]

...............
08:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
	Subsystem: Red Hat, Inc Device 1100
	Physical Slot: 4
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 21
	Region 1: Memory at f8600000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at c0000000 (64-bit, prefetchable) [size=8M]

4.Check the dmesg in guest
[root@localhost ~]# cat tmp |grep fail
[   97.261810] pci 0000:08:00.0: BAR 4: failed to assign [mem size 0x00800000 64bit pref]
[  124.416292] pci 0000:06:00.0: BAR 4: failed to assign [mem size 0x00800000 64bit pref]
[  124.449778] pcieport 0000:03:02.0: BAR 15: failed to assign [mem size 0x01000000 64bit pref]
[  124.449782] pcieport 0000:03:02.0: BAR 15: [mem 0xfd000000-0xfd7fffff 64bit pref] (failed to expand by 0x800000)
[  124.449783] pcieport 0000:03:02.0: failed to add 800000 res[15]=[mem 0xfd000000-0xfd7fffff 64bit pref]

Verified the bz on qemu-kvm-rhev-2.6.0-26.el7.x86_64
1.Boot guest with q35 machine type
2.Hot plug virtio devices to the root port or downstream port (same with reproduce step2)
3.Check the info in guest
[root@localhost ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0   30G  0 disk 
├─vda1        252:1    0    1G  0 part /boot
└─vda2        252:2    0   29G  0 part 
  ├─rhel-root 253:0    0   26G  0 lvm  /
  └─rhel-swap 253:1    0    3G  0 lvm  [SWAP]
vdb           252:16   0    5G  0 disk 
vdc           252:32   0    8G  0 disk 
vdd           252:48   0    1G  0 disk 
vde           252:64   0  500M  0 disk 

[root@localhost ~]# dd if=/dev/zero of=/dev/vdd bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.276003 s, 1.9 GB/s
[root@localhost ~]# dd if=/dev/zero of=/dev/vde bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.249241 s, 2.1 GB/s

[root@localhost ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0-[01]----00.0  Red Hat, Inc Virtio block device
           +-03.0-[02-06]----00.0-[03-06]--+-00.0-[04]----00.0  Red Hat, Inc Virtio block device
           |                               +-01.0-[05]----00.0  Red Hat, Inc Virtio block device
           |                               \-02.0-[06]----00.0  Red Hat, Inc Virtio block device
           +-04.0-[07]----00.0  Red Hat, Inc Virtio network device
           +-05.0-[08]----00.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:08\:00.0/virtio4/features 
0010101001110000000000000000110010000000000000000000000000000000
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:06\:00.0/virtio5/features 
0010101001110000000000000000110010000000000000000000000000000000
[root@localhost ~]# lspci -vvv 
........
06:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
	Subsystem: Red Hat, Inc Device 1100
	Physical Slot: 0-2
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 21
	Region 1: Memory at fc000000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fe000000 (64-bit, prefetchable) [size=16K]
.........

08:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
	Subsystem: Red Hat, Inc Device 1100
	Physical Slot: 4
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 21
	Region 1: Memory at fc600000 (32-bit, non-prefetchable) [size=4K]
	Region 4: Memory at fe600000 (64-bit, prefetchable) [size=16K]

4. No error message through dmesg in guest

Add info:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.3 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-chardev file,path=/home/seabios.log,id=seabios \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-vga qxl \
-spice port=5932,disable-ticketing \
-qmp tcp:0:6666,server,nowait \
-device ioh3420,id=root.0,slot=1 \
-drive file=/home/pci-bridge/big.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=root.0,drive=drive-virtio-disk0,id=virtio-disk0,disable-legacy=on,disable-modern=off,bootindex=1 \
-device ioh3420,id=root.1,slot=2 \
-device x3130-upstream,bus=root.1,id=upstream1 \
-device xio3130-downstream,bus=upstream1,id=downstream1,chassis=1 \
-device xio3130-downstream,bus=upstream1,id=downstream2,chassis=2 \
-device xio3130-downstream,bus=upstream1,id=downstream3,chassis=3 \
-drive file=/home/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream1,drive=drive-virtio-disk1,id=virtio-disk1,disable-legacy=on,disable-modern=off \
-drive file=/home/block2.raw,if=none,id=drive-virtio-disk2,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=downstream2,drive=drive-virtio-disk2,id=virtio-disk2,disable-legacy=on,disable-modern=off \
-device ioh3420,id=root.2,slot=3 \
-device virtio-net-pci,bus=root.2,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device ioh3420,id=root.3,slot=4 \
-monitor stdio \

Comment 13 jingzhao 2016-09-21 07:26:14 UTC

Hi Marcel

  1.Steps of comment11 and comment 12, right? please correct it if there have something wrong.

  2.About tests of comment 11 and comment 12, is it enough? If not, please share suggestions for verification.

Thanks for your help
Jing Zhao

Comment 14 Michael S. Tsirkin 2016-09-21 12:30:50 UTC

I think that's enough.

Comment 17 errata-xmlrpc 2016-11-07 21:29:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html