Hide Forgot
Add support for multiple PCIe Root Complexes as defined in PCI Express spec.
A solution is posted upstream, see: http://comments.gmane.org/gmane.comp.emulators.qemu/378868 The maintainer did not merge the code yet, since is not part of QEMU 2.5 it will be backported once merged.
Fix included in qemu-kvm-rhev-2.5.0-3.el7
Tested it with following steps kernel-3.10.0-402.el7.x86_64 qemu-kvm-rhev-2.6.0-1.el7.x86_64 seabios-1.9.1-4.el7.x86_64 test steps: 1. Add nic to pxb bus, boot vm with following command: /usr/libexec/qemu-kvm \ -M q35 \ -monitor stdio \ -S -vnc :20 \ -cpu SandyBridge \ -m 2G \ -vga std \ -smp 4,sockets=4,cores=1,threads=1 \ -object memory-backend-ram,size=1024M,id=ram-node0 \ -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ -object memory-backend-ram,size=1024M,id=ram-node1 \ -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \ -device ioh3420,bus=bridge1,id=root.0,slot=1 \ -device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \ -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \ -device ioh3420,bus=bridge2,id=root.1,slot=2 \ -drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \ -device virtio-blk-pci,drive=drive0,bus=root.1 \ 2.Using nic with connected to pxb-pcie and install rhel7.3 guest Test result: 1. rhel7.3 guest install successfully 2. check the info through "info qtree" (qemu) info qtree bus: main-system-bus type System dev: pxb-host, id "" bus: bridge2 type pxb-pcie-bus dev: ioh3420, id "root.1" power_controller_present = true chassis = 0 (0x0) slot = 2 (0x2) port = 0 (0x0) aer_log_max = 8 (0x8) addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000) bus: root.1 type PCIE dev: virtio-blk-pci, id "" class = 0 (0x0) ioeventfd = true vectors = 2 (0x2) virtio-pci-bus-master-bug-migration = true disable-legacy = false disable-modern = true migrate-extra = true modern-pio-notify = false x-disable-pcie = false addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002) bar 0: i/o at 0xc000 [0xc03f] bar 1: mem at 0xfe600000 [0xfe600fff] bus: virtio-bus type virtio-pci-bus dev: virtio-blk-device, id "" drive = "drive0" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 4294967295 (0xffffffff) cyls = 16383 (0x3fff) heads = 16 (0x10) secs = 63 (0x3f) serial = "" config-wce = true scsi = false request-merging = true indirect_desc = true event_idx = true notify_on_empty = true any_layout = false dev: pxb-host, id "" bus: bridge1 type pxb-pcie-bus dev: ioh3420, id "root.0" power_controller_present = true chassis = 0 (0x0) slot = 1 (0x1) port = 0 (0x0) aer_log_max = 8 (0x8) addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000) bus: root.0 type PCIE dev: virtio-net-pci, id "" ioeventfd = false vectors = 3 (0x3) virtio-pci-bus-master-bug-migration = true disable-legacy = false disable-modern = true migrate-extra = true modern-pio-notify = false x-disable-pcie = false addr = 00.0 romfile = "pxe-virtio.rom" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001) bar 0: i/o at 0xd000 [0xd01f] bar 1: mem at 0xfe840000 [0xfe840fff] bar 6: mem at 0xffffffffffffffff [0x3fffe] bus: virtio-bus type virtio-pci-bus dev: virtio-net-device, id "" csum = true guest_csum = true gso = true guest_tso4 = true guest_tso6 = true guest_ecn = true guest_ufo = true guest_announce = true host_tso4 = true host_tso6 = true host_ecn = true host_ufo = true mrg_rxbuf = true status = true ctrl_vq = true ctrl_rx = true ctrl_vlan = true ctrl_rx_extra = true ctrl_mac_addr = true ctrl_guest_offloads = true mq = false mac = "9a:6a:6b:6c:6d:6e" vlan = <null> netdev = "tap10" x-txtimer = 150000 (0x249f0) x-txburst = 256 (0x100) tx = "" indirect_desc = true event_idx = true notify_on_empty = true any_layout = true dev: kvm-ioapic, id "" gpio-in "" 24 gsi_base = 0 (0x0) mmio 00000000fec00000/0000000000001000 dev: q35-pcihost, id "" MCFG = 2952790016 (0xb0000000) pci-hole64-size = 0 (0 B) short_root_bus = 0 (0x0) bus: pcie.0 type PCIE dev: pxb-pcie, id "bridge2" bus_nr = 40 (0x28) numa_node = 1 (0x1) addr = 03.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100) dev: pxb-pcie, id "bridge1" bus_nr = 8 (0x8) numa_node = 0 (0x0) addr = 02.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100) dev: VGA, id "" vgamem_mb = 16 (0x10) mmio = true qemu-extended-regs = false addr = 01.0 romfile = "vgabios-stdvga.bin" rombar = 1 (0x1) multifunction = false command_serr_enable = true class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100) bar 0: mem at 0xfd000000 [0xfdffffff] bar 2: mem at 0xfea10000 [0xfea10fff] bar 6: mem at 0xffffffffffffffff [0xfffe] dev: ICH9 SMB, id "" addr = 1f.3 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100) bar 4: i/o at 0x700 [0x73f] bus: i2c type i2c-bus dev: smbus-eeprom, id "" address = 87 (0x57) dev: smbus-eeprom, id "" address = 86 (0x56) dev: smbus-eeprom, id "" address = 85 (0x55) dev: smbus-eeprom, id "" address = 84 (0x54) dev: smbus-eeprom, id "" address = 83 (0x53) dev: smbus-eeprom, id "" address = 82 (0x52) dev: smbus-eeprom, id "" address = 81 (0x51) dev: smbus-eeprom, id "" address = 80 (0x50) dev: ich9-ahci, id "" addr = 1f.2 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100) bar 4: i/o at 0xe040 [0xe05f] bar 5: mem at 0xfea11000 [0xfea11fff] bus: ide.5 type IDE bus: ide.4 type IDE bus: ide.3 type IDE bus: ide.2 type IDE dev: ide-cd, id "" drive = "ide1-cd0" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 512 (0x200) ver = "2.5+" wwn = 0 (0x0) serial = "QM00005" model = "" unit = 0 (0x0) bus: ide.1 type IDE bus: ide.0 type IDE dev: ICH9-LPC, id "" noreboot = true addr = 1f.0 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100) bus: isa.0 type ISA dev: i8257, id "" base = 192 (0xc0) page-base = 136 (0x88) pageh-base = -1 (0xffffffffffffffff) dshift = 1 (0x1) dev: i8257, id "" base = 0 (0x0) page-base = 128 (0x80) pageh-base = -1 (0xffffffffffffffff) dshift = 0 (0x0) dev: port92, id "" dev: vmmouse, id "" dev: vmport, id "" dev: i8042, id "" isa irqs 1,12 dev: isa-serial, id "" index = 0 (0x0) iobase = 1016 (0x3f8) irq = 4 (0x4) chardev = "serial0" wakeup = 0 (0x0) isa irq 4 dev: isa-pcspk, id "" iobase = 97 (0x61) dev: kvm-pit, id "" gpio-in "" 1 iobase = 64 (0x40) lost_tick_policy = "delay" dev: mc146818rtc, id "" base_year = 0 (0x0) lost_tick_policy = "discard" isa irq 8 dev: kvm-i8259, id "" iobase = 160 (0xa0) elcr_addr = 1233 (0x4d1) elcr_mask = 222 (0xde) master = false dev: kvm-i8259, id "" iobase = 32 (0x20) elcr_addr = 1232 (0x4d0) elcr_mask = 248 (0xf8) master = true dev: mch, id "" addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100) dev: fw_cfg_io, id "" iobase = 1296 (0x510) dma_iobase = 1300 (0x514) dma_enabled = true dev: kvmclock, id "" dev: kvmvapic, id "" 3. check the info through "lspci" in guest [root@localhost ~]# lspci -t -vvv -+-[0000:28]---00.0-[29]----00.0 Red Hat, Inc Virtio block device +-[0000:08]---00.0-[09]----00.0 Red Hat, Inc Virtio network device \-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Device 1234:1111 +-02.0 Red Hat, Inc. Device 000b +-03.0 Red Hat, Inc. Device 000b +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller [root@localhost ~]# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Device 1234:1111 00:02.0 Host bridge: Red Hat, Inc. Device 000b 00:03.0 Host bridge: Red Hat, Inc. Device 000b 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 09:00.0 Ethernet controller: Red Hat, Inc Virtio network device 28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device Marcel, I think it's the expected result, could you check it? Thanks Jing
(In reply to jingzhao from comment #4) > Tested it with following steps > kernel-3.10.0-402.el7.x86_64 > qemu-kvm-rhev-2.6.0-1.el7.x86_64 > seabios-1.9.1-4.el7.x86_64 > > test steps: > 1. Add nic to pxb bus, boot vm with following command: > /usr/libexec/qemu-kvm \ > -M q35 \ > -monitor stdio \ > -S -vnc :20 \ > -cpu SandyBridge \ > -m 2G \ > -vga std \ > -smp 4,sockets=4,cores=1,threads=1 \ > -object memory-backend-ram,size=1024M,id=ram-node0 \ > -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ > -object memory-backend-ram,size=1024M,id=ram-node1 \ > -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ > -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \ > -device ioh3420,bus=bridge1,id=root.0,slot=1 \ > -device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev > tap,id=tap10 \ > -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \ > -device ioh3420,bus=bridge2,id=root.1,slot=2 \ > -drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \ > -device virtio-blk-pci,drive=drive0,bus=root.1 \ > > 2.Using nic with connected to pxb-pcie and install rhel7.3 guest > > Test result: > 1. rhel7.3 guest install successfully > 2. check the info through "info qtree" > (qemu) info qtree > bus: main-system-bus > type System > dev: pxb-host, id "" > bus: bridge2 > type pxb-pcie-bus > dev: ioh3420, id "root.1" > power_controller_present = true > chassis = 0 (0x0) > slot = 2 (0x2) > port = 0 (0x0) > aer_log_max = 8 (0x8) > addr = 00.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000) > bus: root.1 > type PCIE > dev: virtio-blk-pci, id "" > class = 0 (0x0) > ioeventfd = true > vectors = 2 (0x2) > virtio-pci-bus-master-bug-migration = true > disable-legacy = false > disable-modern = true > migrate-extra = true > modern-pio-notify = false > x-disable-pcie = false > addr = 00.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub > 1af4:0002) > bar 0: i/o at 0xc000 [0xc03f] > bar 1: mem at 0xfe600000 [0xfe600fff] > bus: virtio-bus > type virtio-pci-bus > dev: virtio-blk-device, id "" > drive = "drive0" > logical_block_size = 512 (0x200) > physical_block_size = 512 (0x200) > min_io_size = 0 (0x0) > opt_io_size = 0 (0x0) > discard_granularity = 4294967295 (0xffffffff) > cyls = 16383 (0x3fff) > heads = 16 (0x10) > secs = 63 (0x3f) > serial = "" > config-wce = true > scsi = false > request-merging = true > indirect_desc = true > event_idx = true > notify_on_empty = true > any_layout = false > dev: pxb-host, id "" > bus: bridge1 > type pxb-pcie-bus > dev: ioh3420, id "root.0" > power_controller_present = true > chassis = 0 (0x0) > slot = 1 (0x1) > port = 0 (0x0) > aer_log_max = 8 (0x8) > addr = 00.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000) > bus: root.0 > type PCIE > dev: virtio-net-pci, id "" > ioeventfd = false > vectors = 3 (0x3) > virtio-pci-bus-master-bug-migration = true > disable-legacy = false > disable-modern = true > migrate-extra = true > modern-pio-notify = false > x-disable-pcie = false > addr = 00.0 > romfile = "pxe-virtio.rom" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub > 1af4:0001) > bar 0: i/o at 0xd000 [0xd01f] > bar 1: mem at 0xfe840000 [0xfe840fff] > bar 6: mem at 0xffffffffffffffff [0x3fffe] > bus: virtio-bus > type virtio-pci-bus > dev: virtio-net-device, id "" > csum = true > guest_csum = true > gso = true > guest_tso4 = true > guest_tso6 = true > guest_ecn = true > guest_ufo = true > guest_announce = true > host_tso4 = true > host_tso6 = true > host_ecn = true > host_ufo = true > mrg_rxbuf = true > status = true > ctrl_vq = true > ctrl_rx = true > ctrl_vlan = true > ctrl_rx_extra = true > ctrl_mac_addr = true > ctrl_guest_offloads = true > mq = false > mac = "9a:6a:6b:6c:6d:6e" > vlan = <null> > netdev = "tap10" > x-txtimer = 150000 (0x249f0) > x-txburst = 256 (0x100) > tx = "" > indirect_desc = true > event_idx = true > notify_on_empty = true > any_layout = true > dev: kvm-ioapic, id "" > gpio-in "" 24 > gsi_base = 0 (0x0) > mmio 00000000fec00000/0000000000001000 > dev: q35-pcihost, id "" > MCFG = 2952790016 (0xb0000000) > pci-hole64-size = 0 (0 B) > short_root_bus = 0 (0x0) > bus: pcie.0 > type PCIE > dev: pxb-pcie, id "bridge2" > bus_nr = 40 (0x28) > numa_node = 1 (0x1) > addr = 03.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100) > dev: pxb-pcie, id "bridge1" > bus_nr = 8 (0x8) > numa_node = 0 (0x0) > addr = 02.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100) > dev: VGA, id "" > vgamem_mb = 16 (0x10) > mmio = true > qemu-extended-regs = false > addr = 01.0 > romfile = "vgabios-stdvga.bin" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100) > bar 0: mem at 0xfd000000 [0xfdffffff] > bar 2: mem at 0xfea10000 [0xfea10fff] > bar 6: mem at 0xffffffffffffffff [0xfffe] > dev: ICH9 SMB, id "" > addr = 1f.3 > romfile = "" > rombar = 1 (0x1) > multifunction = true > command_serr_enable = true > class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100) > bar 4: i/o at 0x700 [0x73f] > bus: i2c > type i2c-bus > dev: smbus-eeprom, id "" > address = 87 (0x57) > dev: smbus-eeprom, id "" > address = 86 (0x56) > dev: smbus-eeprom, id "" > address = 85 (0x55) > dev: smbus-eeprom, id "" > address = 84 (0x54) > dev: smbus-eeprom, id "" > address = 83 (0x53) > dev: smbus-eeprom, id "" > address = 82 (0x52) > dev: smbus-eeprom, id "" > address = 81 (0x51) > dev: smbus-eeprom, id "" > address = 80 (0x50) > dev: ich9-ahci, id "" > addr = 1f.2 > romfile = "" > rombar = 1 (0x1) > multifunction = true > command_serr_enable = true > class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100) > bar 4: i/o at 0xe040 [0xe05f] > bar 5: mem at 0xfea11000 [0xfea11fff] > bus: ide.5 > type IDE > bus: ide.4 > type IDE > bus: ide.3 > type IDE > bus: ide.2 > type IDE > dev: ide-cd, id "" > drive = "ide1-cd0" > logical_block_size = 512 (0x200) > physical_block_size = 512 (0x200) > min_io_size = 0 (0x0) > opt_io_size = 0 (0x0) > discard_granularity = 512 (0x200) > ver = "2.5+" > wwn = 0 (0x0) > serial = "QM00005" > model = "" > unit = 0 (0x0) > bus: ide.1 > type IDE > bus: ide.0 > type IDE > dev: ICH9-LPC, id "" > noreboot = true > addr = 1f.0 > romfile = "" > rombar = 1 (0x1) > multifunction = true > command_serr_enable = true > class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100) > bus: isa.0 > type ISA > dev: i8257, id "" > base = 192 (0xc0) > page-base = 136 (0x88) > pageh-base = -1 (0xffffffffffffffff) > dshift = 1 (0x1) > dev: i8257, id "" > base = 0 (0x0) > page-base = 128 (0x80) > pageh-base = -1 (0xffffffffffffffff) > dshift = 0 (0x0) > dev: port92, id "" > dev: vmmouse, id "" > dev: vmport, id "" > dev: i8042, id "" > isa irqs 1,12 > dev: isa-serial, id "" > index = 0 (0x0) > iobase = 1016 (0x3f8) > irq = 4 (0x4) > chardev = "serial0" > wakeup = 0 (0x0) > isa irq 4 > dev: isa-pcspk, id "" > iobase = 97 (0x61) > dev: kvm-pit, id "" > gpio-in "" 1 > iobase = 64 (0x40) > lost_tick_policy = "delay" > dev: mc146818rtc, id "" > base_year = 0 (0x0) > lost_tick_policy = "discard" > isa irq 8 > dev: kvm-i8259, id "" > iobase = 160 (0xa0) > elcr_addr = 1233 (0x4d1) > elcr_mask = 222 (0xde) > master = false > dev: kvm-i8259, id "" > iobase = 32 (0x20) > elcr_addr = 1232 (0x4d0) > elcr_mask = 248 (0xf8) > master = true > dev: mch, id "" > addr = 00.0 > romfile = "" > rombar = 1 (0x1) > multifunction = false > command_serr_enable = true > class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100) > dev: fw_cfg_io, id "" > iobase = 1296 (0x510) > dma_iobase = 1300 (0x514) > dma_enabled = true > dev: kvmclock, id "" > dev: kvmvapic, id "" > > 3. check the info through "lspci" in guest > [root@localhost ~]# lspci -t -vvv > -+-[0000:28]---00.0-[29]----00.0 Red Hat, Inc Virtio block device > +-[0000:08]---00.0-[09]----00.0 Red Hat, Inc Virtio network device > \-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM > Controller > +-01.0 Device 1234:1111 > +-02.0 Red Hat, Inc. Device 000b > +-03.0 Red Hat, Inc. Device 000b > +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface > Controller > +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port > SATA Controller [AHCI mode] > \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller > [root@localhost ~]# lspci > 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM > Controller > 00:01.0 VGA compatible controller: Device 1234:1111 > 00:02.0 Host bridge: Red Hat, Inc. Device 000b > 00:03.0 Host bridge: Red Hat, Inc. Device 000b > 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface > Controller (rev 02) > 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 > port SATA Controller [AHCI mode] (rev 02) > 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev > 02) > 08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express > Root Port 0 (rev 02) > 09:00.0 Ethernet controller: Red Hat, Inc Virtio network device > 28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express > Root Port 0 (rev 02) > 29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device > > Marcel, I think it's the expected result, could you check it? > > Thanks > Jing The testing scenario is good, but you also need to check that the devices are now on the correct numa node. You can check this by looking at cat /sys/bus/pci/devices/<device>/numa_node And of course, I would check the device is working correctly :) An assigned device on a pxb-pcie is also a good test. Thanks, Marcel
Verfied it with fully test steps kernel-3.10.0-402.el7.x86_64 qemu-kvm-rhev-2.6.0-2.el7.x86_64 seabios-1.9.1-4.el7.x86_64 1. Add nic to pxb bus, boot vm with following command: /usr/libexec/qemu-kvm \ -M q35 \ -monitor stdio \ -S -vnc :20 \ -cpu SandyBridge \ -m 2G \ -vga std \ -smp 4,sockets=4,cores=1,threads=1 \ -object memory-backend-ram,size=1024M,id=ram-node0 \ -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ -object memory-backend-ram,size=1024M,id=ram-node1 \ -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \ -device ioh3420,bus=bridge1,id=root.0,slot=1 \ -device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \ -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \ -device ioh3420,bus=bridge2,id=root.1,slot=2 \ -drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \ -device virtio-blk-pci,drive=drive0,bus=root.1 \ 2.Using nic with connected to pxb-pcie and install rhel7.3 guest Test result: 1. rhel7.3 guest install successfully 2. check the info through "info qtree" (qemu) info qtree bus: main-system-bus type System dev: pxb-host, id "" bus: bridge2 type pxb-pcie-bus dev: ioh3420, id "root.1" power_controller_present = true chassis = 0 (0x0) slot = 2 (0x2) port = 0 (0x0) aer_log_max = 8 (0x8) addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000) bus: root.1 type PCIE dev: virtio-blk-pci, id "" class = 0 (0x0) ioeventfd = true vectors = 2 (0x2) virtio-pci-bus-master-bug-migration = true disable-legacy = false disable-modern = true migrate-extra = true modern-pio-notify = false x-disable-pcie = false addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002) bar 0: i/o at 0xc000 [0xc03f] bar 1: mem at 0xfe600000 [0xfe600fff] bus: virtio-bus type virtio-pci-bus dev: virtio-blk-device, id "" drive = "drive0" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 4294967295 (0xffffffff) cyls = 16383 (0x3fff) heads = 16 (0x10) secs = 63 (0x3f) serial = "" config-wce = true scsi = false request-merging = true indirect_desc = true event_idx = true notify_on_empty = true any_layout = false dev: pxb-host, id "" bus: bridge1 type pxb-pcie-bus dev: ioh3420, id "root.0" power_controller_present = true chassis = 0 (0x0) slot = 1 (0x1) port = 0 (0x0) aer_log_max = 8 (0x8) addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000) bus: root.0 type PCIE dev: virtio-net-pci, id "" ioeventfd = false vectors = 3 (0x3) virtio-pci-bus-master-bug-migration = true disable-legacy = false disable-modern = true migrate-extra = true modern-pio-notify = false x-disable-pcie = false addr = 00.0 romfile = "pxe-virtio.rom" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001) bar 0: i/o at 0xd000 [0xd01f] bar 1: mem at 0xfe840000 [0xfe840fff] bar 6: mem at 0xffffffffffffffff [0x3fffe] bus: virtio-bus type virtio-pci-bus dev: virtio-net-device, id "" csum = true guest_csum = true gso = true guest_tso4 = true guest_tso6 = true guest_ecn = true guest_ufo = true guest_announce = true host_tso4 = true host_tso6 = true host_ecn = true host_ufo = true mrg_rxbuf = true status = true ctrl_vq = true ctrl_rx = true ctrl_vlan = true ctrl_rx_extra = true ctrl_mac_addr = true ctrl_guest_offloads = true mq = false mac = "9a:6a:6b:6c:6d:6e" vlan = <null> netdev = "tap10" x-txtimer = 150000 (0x249f0) x-txburst = 256 (0x100) tx = "" indirect_desc = true event_idx = true notify_on_empty = true any_layout = true dev: kvm-ioapic, id "" gpio-in "" 24 gsi_base = 0 (0x0) mmio 00000000fec00000/0000000000001000 dev: q35-pcihost, id "" MCFG = 2952790016 (0xb0000000) pci-hole64-size = 0 (0 B) short_root_bus = 0 (0x0) bus: pcie.0 type PCIE dev: pxb-pcie, id "bridge2" bus_nr = 40 (0x28) numa_node = 1 (0x1) addr = 03.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100) dev: pxb-pcie, id "bridge1" bus_nr = 8 (0x8) numa_node = 0 (0x0) addr = 02.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100) dev: VGA, id "" vgamem_mb = 16 (0x10) mmio = true qemu-extended-regs = false addr = 01.0 romfile = "vgabios-stdvga.bin" rombar = 1 (0x1) multifunction = false command_serr_enable = true class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100) bar 0: mem at 0xfd000000 [0xfdffffff] bar 2: mem at 0xfea10000 [0xfea10fff] bar 6: mem at 0xffffffffffffffff [0xfffe] dev: ICH9 SMB, id "" addr = 1f.3 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100) bar 4: i/o at 0x700 [0x73f] bus: i2c type i2c-bus dev: smbus-eeprom, id "" address = 87 (0x57) dev: smbus-eeprom, id "" address = 86 (0x56) dev: smbus-eeprom, id "" address = 85 (0x55) dev: smbus-eeprom, id "" address = 84 (0x54) dev: smbus-eeprom, id "" address = 83 (0x53) dev: smbus-eeprom, id "" address = 82 (0x52) dev: smbus-eeprom, id "" address = 81 (0x51) dev: smbus-eeprom, id "" address = 80 (0x50) dev: ich9-ahci, id "" addr = 1f.2 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100) bar 4: i/o at 0xe040 [0xe05f] bar 5: mem at 0xfea11000 [0xfea11fff] bus: ide.5 type IDE bus: ide.4 type IDE bus: ide.3 type IDE bus: ide.2 type IDE dev: ide-cd, id "" drive = "ide1-cd0" logical_block_size = 512 (0x200) physical_block_size = 512 (0x200) min_io_size = 0 (0x0) opt_io_size = 0 (0x0) discard_granularity = 512 (0x200) ver = "2.5+" wwn = 0 (0x0) serial = "QM00005" model = "" unit = 0 (0x0) bus: ide.1 type IDE bus: ide.0 type IDE dev: ICH9-LPC, id "" noreboot = true addr = 1f.0 romfile = "" rombar = 1 (0x1) multifunction = true command_serr_enable = true class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100) bus: isa.0 type ISA dev: i8257, id "" base = 192 (0xc0) page-base = 136 (0x88) pageh-base = -1 (0xffffffffffffffff) dshift = 1 (0x1) dev: i8257, id "" base = 0 (0x0) page-base = 128 (0x80) pageh-base = -1 (0xffffffffffffffff) dshift = 0 (0x0) dev: port92, id "" dev: vmmouse, id "" dev: vmport, id "" dev: i8042, id "" isa irqs 1,12 dev: isa-serial, id "" index = 0 (0x0) iobase = 1016 (0x3f8) irq = 4 (0x4) chardev = "serial0" wakeup = 0 (0x0) isa irq 4 dev: isa-pcspk, id "" iobase = 97 (0x61) dev: kvm-pit, id "" gpio-in "" 1 iobase = 64 (0x40) lost_tick_policy = "delay" dev: mc146818rtc, id "" base_year = 0 (0x0) lost_tick_policy = "discard" isa irq 8 dev: kvm-i8259, id "" iobase = 160 (0xa0) elcr_addr = 1233 (0x4d1) elcr_mask = 222 (0xde) master = false dev: kvm-i8259, id "" iobase = 32 (0x20) elcr_addr = 1232 (0x4d0) elcr_mask = 248 (0xf8) master = true dev: mch, id "" addr = 00.0 romfile = "" rombar = 1 (0x1) multifunction = false command_serr_enable = true class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100) dev: fw_cfg_io, id "" iobase = 1296 (0x510) dma_iobase = 1300 (0x514) dma_enabled = true dev: kvmclock, id "" dev: kvmvapic, id "" 3. check the info through "lspci" in guest [root@localhost ~]# lspci -t -vvv -+-[0000:28]---00.0-[29]----00.0 Red Hat, Inc Virtio block device +-[0000:08]---00.0-[09]----00.0 Red Hat, Inc Virtio network device \-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Device 1234:1111 +-02.0 Red Hat, Inc. Device 000b +-03.0 Red Hat, Inc. Device 000b +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller [root@localhost ~]# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Device 1234:1111 00:02.0 Host bridge: Red Hat, Inc. Device 000b 00:03.0 Host bridge: Red Hat, Inc. Device 000b 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 09:00.0 Ethernet controller: Red Hat, Inc Virtio network device 28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) 29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device 4. Check the devices on the correct numa node [root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 0 [root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 1 [root@localhost ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2673.html