Bug 1272939 - [PCIe] Add multi-root support for Q35
[PCIe] Add multi-root support for Q35
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Marcel Apfelbaum
Virtualization Bugs
:
Depends On:
Blocks: 1305606 1313485 1227278
  Show dependency treegraph
 
Reported: 2015-10-19 05:23 EDT by Marcel Apfelbaum
Modified: 2016-11-07 15:46 EST (History)
12 users (show)

See Also:
Fixed In Version: qemu-kvm-rhev-2.5.0-3.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-07 15:46:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Marcel Apfelbaum 2015-10-19 05:23:10 EDT
Add support for multiple PCIe Root Complexes as defined in PCI Express spec.
Comment 1 Marcel Apfelbaum 2015-12-22 04:38:29 EST
A solution is posted upstream, see:
http://comments.gmane.org/gmane.comp.emulators.qemu/378868

The maintainer did not merge the code yet, since is not part of QEMU 2.5 it will be backported once merged.
Comment 2 Miroslav Rezanina 2016-03-23 05:53:59 EDT
Fix included in qemu-kvm-rhev-2.5.0-3.el7
Comment 4 jingzhao 2016-05-17 22:43:21 EDT
Tested it with following steps
kernel-3.10.0-402.el7.x86_64
qemu-kvm-rhev-2.6.0-1.el7.x86_64
seabios-1.9.1-4.el7.x86_64

test steps:
1. Add nic to pxb bus, boot vm with following command:
/usr/libexec/qemu-kvm \
-M q35 \
-monitor stdio \
-S -vnc :20 \
-cpu SandyBridge \
-m 2G \
-vga std \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device ioh3420,bus=bridge1,id=root.0,slot=1 \
-device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
-device ioh3420,bus=bridge2,id=root.1,slot=2 \
-drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
-device virtio-blk-pci,drive=drive0,bus=root.1 \

2.Using nic with connected to pxb-pcie and install rhel7.3 guest

Test result:
1. rhel7.3 guest install successfully
2. check the info through "info qtree"
(qemu) info qtree
bus: main-system-bus
  type System
  dev: pxb-host, id ""
    bus: bridge2
      type pxb-pcie-bus
      dev: ioh3420, id "root.1"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 2 (0x2)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.1
          type PCIE
          dev: virtio-blk-pci, id ""
            class = 0 (0x0)
            ioeventfd = true
            vectors = 2 (0x2)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002)
            bar 0: i/o at 0xc000 [0xc03f]
            bar 1: mem at 0xfe600000 [0xfe600fff]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-blk-device, id ""
                drive = "drive0"
                logical_block_size = 512 (0x200)
                physical_block_size = 512 (0x200)
                min_io_size = 0 (0x0)
                opt_io_size = 0 (0x0)
                discard_granularity = 4294967295 (0xffffffff)
                cyls = 16383 (0x3fff)
                heads = 16 (0x10)
                secs = 63 (0x3f)
                serial = ""
                config-wce = true
                scsi = false
                request-merging = true
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = false
  dev: pxb-host, id ""
    bus: bridge1
      type pxb-pcie-bus
      dev: ioh3420, id "root.0"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 1 (0x1)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.0
          type PCIE
          dev: virtio-net-pci, id ""
            ioeventfd = false
            vectors = 3 (0x3)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = "pxe-virtio.rom"
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001)
            bar 0: i/o at 0xd000 [0xd01f]
            bar 1: mem at 0xfe840000 [0xfe840fff]
            bar 6: mem at 0xffffffffffffffff [0x3fffe]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-net-device, id ""
                csum = true
                guest_csum = true
                gso = true
                guest_tso4 = true
                guest_tso6 = true
                guest_ecn = true
                guest_ufo = true
                guest_announce = true
                host_tso4 = true
                host_tso6 = true
                host_ecn = true
                host_ufo = true
                mrg_rxbuf = true
                status = true
                ctrl_vq = true
                ctrl_rx = true
                ctrl_vlan = true
                ctrl_rx_extra = true
                ctrl_mac_addr = true
                ctrl_guest_offloads = true
                mq = false
                mac = "9a:6a:6b:6c:6d:6e"
                vlan = <null>
                netdev = "tap10"
                x-txtimer = 150000 (0x249f0)
                x-txburst = 256 (0x100)
                tx = ""
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = true
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 0 (0 B)
    short_root_bus = 0 (0x0)
    bus: pcie.0
      type PCIE
      dev: pxb-pcie, id "bridge2"
        bus_nr = 40 (0x28)
        numa_node = 1 (0x1)
        addr = 03.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
      dev: pxb-pcie, id "bridge1"
        bus_nr = 8 (0x8)
        numa_node = 0 (0x0)
        addr = 02.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
      dev: VGA, id ""
        vgamem_mb = 16 (0x10)
        mmio = true
        qemu-extended-regs = false
        addr = 01.0
        romfile = "vgabios-stdvga.bin"
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
        bar 0: mem at 0xfd000000 [0xfdffffff]
        bar 2: mem at 0xfea10000 [0xfea10fff]
        bar 6: mem at 0xffffffffffffffff [0xfffe]
      dev: ICH9 SMB, id ""
        addr = 1f.3
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
        bar 4: i/o at 0x700 [0x73f]
        bus: i2c
          type i2c-bus
          dev: smbus-eeprom, id ""
            address = 87 (0x57)
          dev: smbus-eeprom, id ""
            address = 86 (0x56)
          dev: smbus-eeprom, id ""
            address = 85 (0x55)
          dev: smbus-eeprom, id ""
            address = 84 (0x54)
          dev: smbus-eeprom, id ""
            address = 83 (0x53)
          dev: smbus-eeprom, id ""
            address = 82 (0x52)
          dev: smbus-eeprom, id ""
            address = 81 (0x51)
          dev: smbus-eeprom, id ""
            address = 80 (0x50)
      dev: ich9-ahci, id ""
        addr = 1f.2
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
        bar 4: i/o at 0xe040 [0xe05f]
        bar 5: mem at 0xfea11000 [0xfea11fff]
        bus: ide.5
          type IDE
        bus: ide.4
          type IDE
        bus: ide.3
          type IDE
        bus: ide.2
          type IDE
          dev: ide-cd, id ""
            drive = "ide1-cd0"
            logical_block_size = 512 (0x200)
            physical_block_size = 512 (0x200)
            min_io_size = 0 (0x0)
            opt_io_size = 0 (0x0)
            discard_granularity = 512 (0x200)
            ver = "2.5+"
            wwn = 0 (0x0)
            serial = "QM00005"
            model = ""
            unit = 0 (0x0)
        bus: ide.1
          type IDE
        bus: ide.0
          type IDE
      dev: ICH9-LPC, id ""
        noreboot = true
        addr = 1f.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
        bus: isa.0
          type ISA
          dev: i8257, id ""
            base = 192 (0xc0)
            page-base = 136 (0x88)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 1 (0x1)
          dev: i8257, id ""
            base = 0 (0x0)
            page-base = 128 (0x80)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 0 (0x0)
          dev: port92, id ""
          dev: vmmouse, id ""
          dev: vmport, id ""
          dev: i8042, id ""
            isa irqs 1,12
          dev: isa-serial, id ""
            index = 0 (0x0)
            iobase = 1016 (0x3f8)
            irq = 4 (0x4)
            chardev = "serial0"
            wakeup = 0 (0x0)
            isa irq 4
          dev: isa-pcspk, id ""
            iobase = 97 (0x61)
          dev: kvm-pit, id ""
            gpio-in "" 1
            iobase = 64 (0x40)
            lost_tick_policy = "delay"
          dev: mc146818rtc, id ""
            base_year = 0 (0x0)
            lost_tick_policy = "discard"
            isa irq 8
          dev: kvm-i8259, id ""
            iobase = 160 (0xa0)
            elcr_addr = 1233 (0x4d1)
            elcr_mask = 222 (0xde)
            master = false
          dev: kvm-i8259, id ""
            iobase = 32 (0x20)
            elcr_addr = 1232 (0x4d0)
            elcr_mask = 248 (0xf8)
            master = true
      dev: mch, id ""
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
  dev: fw_cfg_io, id ""
    iobase = 1296 (0x510)
    dma_iobase = 1300 (0x514)
    dma_enabled = true
  dev: kvmclock, id ""
  dev: kvmvapic, id ""

3. check the info through "lspci" in guest
[root@localhost ~]# lspci -t -vvv
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Device 1234:1111
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device

Marcel, I think it's the expected result, could you check it?

Thanks
Jing
Comment 5 Marcel Apfelbaum 2016-05-19 15:03:07 EDT
(In reply to jingzhao from comment #4)
> Tested it with following steps
> kernel-3.10.0-402.el7.x86_64
> qemu-kvm-rhev-2.6.0-1.el7.x86_64
> seabios-1.9.1-4.el7.x86_64
> 
> test steps:
> 1. Add nic to pxb bus, boot vm with following command:
> /usr/libexec/qemu-kvm \
> -M q35 \
> -monitor stdio \
> -S -vnc :20 \
> -cpu SandyBridge \
> -m 2G \
> -vga std \
> -smp 4,sockets=4,cores=1,threads=1 \
> -object memory-backend-ram,size=1024M,id=ram-node0 \
> -numa node,nodeid=0,cpus=0,memdev=ram-node0 \
> -object memory-backend-ram,size=1024M,id=ram-node1 \
> -numa node,nodeid=1,cpus=1,memdev=ram-node1 \
> -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
> -device ioh3420,bus=bridge1,id=root.0,slot=1 \
> -device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev
> tap,id=tap10 \
> -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
> -device ioh3420,bus=bridge2,id=root.1,slot=2 \
> -drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
> -device virtio-blk-pci,drive=drive0,bus=root.1 \
> 
> 2.Using nic with connected to pxb-pcie and install rhel7.3 guest
> 
> Test result:
> 1. rhel7.3 guest install successfully
> 2. check the info through "info qtree"
> (qemu) info qtree
> bus: main-system-bus
>   type System
>   dev: pxb-host, id ""
>     bus: bridge2
>       type pxb-pcie-bus
>       dev: ioh3420, id "root.1"
>         power_controller_present = true
>         chassis = 0 (0x0)
>         slot = 2 (0x2)
>         port = 0 (0x0)
>         aer_log_max = 8 (0x8)
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
>         bus: root.1
>           type PCIE
>           dev: virtio-blk-pci, id ""
>             class = 0 (0x0)
>             ioeventfd = true
>             vectors = 2 (0x2)
>             virtio-pci-bus-master-bug-migration = true
>             disable-legacy = false
>             disable-modern = true
>             migrate-extra = true
>             modern-pio-notify = false
>             x-disable-pcie = false
>             addr = 00.0
>             romfile = ""
>             rombar = 1 (0x1)
>             multifunction = false
>             command_serr_enable = true
>             class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub
> 1af4:0002)
>             bar 0: i/o at 0xc000 [0xc03f]
>             bar 1: mem at 0xfe600000 [0xfe600fff]
>             bus: virtio-bus
>               type virtio-pci-bus
>               dev: virtio-blk-device, id ""
>                 drive = "drive0"
>                 logical_block_size = 512 (0x200)
>                 physical_block_size = 512 (0x200)
>                 min_io_size = 0 (0x0)
>                 opt_io_size = 0 (0x0)
>                 discard_granularity = 4294967295 (0xffffffff)
>                 cyls = 16383 (0x3fff)
>                 heads = 16 (0x10)
>                 secs = 63 (0x3f)
>                 serial = ""
>                 config-wce = true
>                 scsi = false
>                 request-merging = true
>                 indirect_desc = true
>                 event_idx = true
>                 notify_on_empty = true
>                 any_layout = false
>   dev: pxb-host, id ""
>     bus: bridge1
>       type pxb-pcie-bus
>       dev: ioh3420, id "root.0"
>         power_controller_present = true
>         chassis = 0 (0x0)
>         slot = 1 (0x1)
>         port = 0 (0x0)
>         aer_log_max = 8 (0x8)
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
>         bus: root.0
>           type PCIE
>           dev: virtio-net-pci, id ""
>             ioeventfd = false
>             vectors = 3 (0x3)
>             virtio-pci-bus-master-bug-migration = true
>             disable-legacy = false
>             disable-modern = true
>             migrate-extra = true
>             modern-pio-notify = false
>             x-disable-pcie = false
>             addr = 00.0
>             romfile = "pxe-virtio.rom"
>             rombar = 1 (0x1)
>             multifunction = false
>             command_serr_enable = true
>             class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub
> 1af4:0001)
>             bar 0: i/o at 0xd000 [0xd01f]
>             bar 1: mem at 0xfe840000 [0xfe840fff]
>             bar 6: mem at 0xffffffffffffffff [0x3fffe]
>             bus: virtio-bus
>               type virtio-pci-bus
>               dev: virtio-net-device, id ""
>                 csum = true
>                 guest_csum = true
>                 gso = true
>                 guest_tso4 = true
>                 guest_tso6 = true
>                 guest_ecn = true
>                 guest_ufo = true
>                 guest_announce = true
>                 host_tso4 = true
>                 host_tso6 = true
>                 host_ecn = true
>                 host_ufo = true
>                 mrg_rxbuf = true
>                 status = true
>                 ctrl_vq = true
>                 ctrl_rx = true
>                 ctrl_vlan = true
>                 ctrl_rx_extra = true
>                 ctrl_mac_addr = true
>                 ctrl_guest_offloads = true
>                 mq = false
>                 mac = "9a:6a:6b:6c:6d:6e"
>                 vlan = <null>
>                 netdev = "tap10"
>                 x-txtimer = 150000 (0x249f0)
>                 x-txburst = 256 (0x100)
>                 tx = ""
>                 indirect_desc = true
>                 event_idx = true
>                 notify_on_empty = true
>                 any_layout = true
>   dev: kvm-ioapic, id ""
>     gpio-in "" 24
>     gsi_base = 0 (0x0)
>     mmio 00000000fec00000/0000000000001000
>   dev: q35-pcihost, id ""
>     MCFG = 2952790016 (0xb0000000)
>     pci-hole64-size = 0 (0 B)
>     short_root_bus = 0 (0x0)
>     bus: pcie.0
>       type PCIE
>       dev: pxb-pcie, id "bridge2"
>         bus_nr = 40 (0x28)
>         numa_node = 1 (0x1)
>         addr = 03.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
>       dev: pxb-pcie, id "bridge1"
>         bus_nr = 8 (0x8)
>         numa_node = 0 (0x0)
>         addr = 02.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
>       dev: VGA, id ""
>         vgamem_mb = 16 (0x10)
>         mmio = true
>         qemu-extended-regs = false
>         addr = 01.0
>         romfile = "vgabios-stdvga.bin"
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
>         bar 0: mem at 0xfd000000 [0xfdffffff]
>         bar 2: mem at 0xfea10000 [0xfea10fff]
>         bar 6: mem at 0xffffffffffffffff [0xfffe]
>       dev: ICH9 SMB, id ""
>         addr = 1f.3
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
>         bar 4: i/o at 0x700 [0x73f]
>         bus: i2c
>           type i2c-bus
>           dev: smbus-eeprom, id ""
>             address = 87 (0x57)
>           dev: smbus-eeprom, id ""
>             address = 86 (0x56)
>           dev: smbus-eeprom, id ""
>             address = 85 (0x55)
>           dev: smbus-eeprom, id ""
>             address = 84 (0x54)
>           dev: smbus-eeprom, id ""
>             address = 83 (0x53)
>           dev: smbus-eeprom, id ""
>             address = 82 (0x52)
>           dev: smbus-eeprom, id ""
>             address = 81 (0x51)
>           dev: smbus-eeprom, id ""
>             address = 80 (0x50)
>       dev: ich9-ahci, id ""
>         addr = 1f.2
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
>         bar 4: i/o at 0xe040 [0xe05f]
>         bar 5: mem at 0xfea11000 [0xfea11fff]
>         bus: ide.5
>           type IDE
>         bus: ide.4
>           type IDE
>         bus: ide.3
>           type IDE
>         bus: ide.2
>           type IDE
>           dev: ide-cd, id ""
>             drive = "ide1-cd0"
>             logical_block_size = 512 (0x200)
>             physical_block_size = 512 (0x200)
>             min_io_size = 0 (0x0)
>             opt_io_size = 0 (0x0)
>             discard_granularity = 512 (0x200)
>             ver = "2.5+"
>             wwn = 0 (0x0)
>             serial = "QM00005"
>             model = ""
>             unit = 0 (0x0)
>         bus: ide.1
>           type IDE
>         bus: ide.0
>           type IDE
>       dev: ICH9-LPC, id ""
>         noreboot = true
>         addr = 1f.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
>         bus: isa.0
>           type ISA
>           dev: i8257, id ""
>             base = 192 (0xc0)
>             page-base = 136 (0x88)
>             pageh-base = -1 (0xffffffffffffffff)
>             dshift = 1 (0x1)
>           dev: i8257, id ""
>             base = 0 (0x0)
>             page-base = 128 (0x80)
>             pageh-base = -1 (0xffffffffffffffff)
>             dshift = 0 (0x0)
>           dev: port92, id ""
>           dev: vmmouse, id ""
>           dev: vmport, id ""
>           dev: i8042, id ""
>             isa irqs 1,12
>           dev: isa-serial, id ""
>             index = 0 (0x0)
>             iobase = 1016 (0x3f8)
>             irq = 4 (0x4)
>             chardev = "serial0"
>             wakeup = 0 (0x0)
>             isa irq 4
>           dev: isa-pcspk, id ""
>             iobase = 97 (0x61)
>           dev: kvm-pit, id ""
>             gpio-in "" 1
>             iobase = 64 (0x40)
>             lost_tick_policy = "delay"
>           dev: mc146818rtc, id ""
>             base_year = 0 (0x0)
>             lost_tick_policy = "discard"
>             isa irq 8
>           dev: kvm-i8259, id ""
>             iobase = 160 (0xa0)
>             elcr_addr = 1233 (0x4d1)
>             elcr_mask = 222 (0xde)
>             master = false
>           dev: kvm-i8259, id ""
>             iobase = 32 (0x20)
>             elcr_addr = 1232 (0x4d0)
>             elcr_mask = 248 (0xf8)
>             master = true
>       dev: mch, id ""
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
>   dev: fw_cfg_io, id ""
>     iobase = 1296 (0x510)
>     dma_iobase = 1300 (0x514)
>     dma_enabled = true
>   dev: kvmclock, id ""
>   dev: kvmvapic, id ""
> 
> 3. check the info through "lspci" in guest
> [root@localhost ~]# lspci -t -vvv
> -+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
>  +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
>  \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
>              +-01.0  Device 1234:1111
>              +-02.0  Red Hat, Inc. Device 000b
>              +-03.0  Red Hat, Inc. Device 000b
>              +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface
> Controller
>              +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
> SATA Controller [AHCI mode]
>              \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
> [root@localhost ~]# lspci
> 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
> 00:01.0 VGA compatible controller: Device 1234:1111
> 00:02.0 Host bridge: Red Hat, Inc. Device 000b
> 00:03.0 Host bridge: Red Hat, Inc. Device 000b
> 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface
> Controller (rev 02)
> 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6
> port SATA Controller [AHCI mode] (rev 02)
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev
> 02)
> 08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
> 28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device
> 
> Marcel, I think it's the expected result, could you check it?
> 
> Thanks
> Jing

The testing scenario is good, but you also need to check that the devices are now on the correct numa node.
You can check this by looking at 
cat /sys/bus/pci/devices/<device>/numa_node

And of course, I would check the device is working correctly :)
An assigned device on a pxb-pcie is also a good test.

Thanks,
Marcel
Comment 6 jingzhao 2016-05-22 22:22:05 EDT
Verfied it with fully test steps

kernel-3.10.0-402.el7.x86_64
qemu-kvm-rhev-2.6.0-2.el7.x86_64
seabios-1.9.1-4.el7.x86_64

1. Add nic to pxb bus, boot vm with following command:
/usr/libexec/qemu-kvm \
-M q35 \
-monitor stdio \
-S -vnc :20 \
-cpu SandyBridge \
-m 2G \
-vga std \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device ioh3420,bus=bridge1,id=root.0,slot=1 \
-device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
-device ioh3420,bus=bridge2,id=root.1,slot=2 \
-drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
-device virtio-blk-pci,drive=drive0,bus=root.1 \

2.Using nic with connected to pxb-pcie and install rhel7.3 guest

Test result:
1. rhel7.3 guest install successfully
2. check the info through "info qtree"
(qemu) info qtree
bus: main-system-bus
  type System
  dev: pxb-host, id ""
    bus: bridge2
      type pxb-pcie-bus
      dev: ioh3420, id "root.1"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 2 (0x2)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.1
          type PCIE
          dev: virtio-blk-pci, id ""
            class = 0 (0x0)
            ioeventfd = true
            vectors = 2 (0x2)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002)
            bar 0: i/o at 0xc000 [0xc03f]
            bar 1: mem at 0xfe600000 [0xfe600fff]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-blk-device, id ""
                drive = "drive0"
                logical_block_size = 512 (0x200)
                physical_block_size = 512 (0x200)
                min_io_size = 0 (0x0)
                opt_io_size = 0 (0x0)
                discard_granularity = 4294967295 (0xffffffff)
                cyls = 16383 (0x3fff)
                heads = 16 (0x10)
                secs = 63 (0x3f)
                serial = ""
                config-wce = true
                scsi = false
                request-merging = true
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = false
  dev: pxb-host, id ""
    bus: bridge1
      type pxb-pcie-bus
      dev: ioh3420, id "root.0"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 1 (0x1)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.0
          type PCIE
          dev: virtio-net-pci, id ""
            ioeventfd = false
            vectors = 3 (0x3)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = "pxe-virtio.rom"
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001)
            bar 0: i/o at 0xd000 [0xd01f]
            bar 1: mem at 0xfe840000 [0xfe840fff]
            bar 6: mem at 0xffffffffffffffff [0x3fffe]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-net-device, id ""
                csum = true
                guest_csum = true
                gso = true
                guest_tso4 = true
                guest_tso6 = true
                guest_ecn = true
                guest_ufo = true
                guest_announce = true
                host_tso4 = true
                host_tso6 = true
                host_ecn = true
                host_ufo = true
                mrg_rxbuf = true
                status = true
                ctrl_vq = true
                ctrl_rx = true
                ctrl_vlan = true
                ctrl_rx_extra = true
                ctrl_mac_addr = true
                ctrl_guest_offloads = true
                mq = false
                mac = "9a:6a:6b:6c:6d:6e"
                vlan = <null>
                netdev = "tap10"
                x-txtimer = 150000 (0x249f0)
                x-txburst = 256 (0x100)
                tx = ""
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = true
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 0 (0 B)
    short_root_bus = 0 (0x0)
    bus: pcie.0
      type PCIE
      dev: pxb-pcie, id "bridge2"
        bus_nr = 40 (0x28)
        numa_node = 1 (0x1)
        addr = 03.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
      dev: pxb-pcie, id "bridge1"
        bus_nr = 8 (0x8)
        numa_node = 0 (0x0)
        addr = 02.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
      dev: VGA, id ""
        vgamem_mb = 16 (0x10)
        mmio = true
        qemu-extended-regs = false
        addr = 01.0
        romfile = "vgabios-stdvga.bin"
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
        bar 0: mem at 0xfd000000 [0xfdffffff]
        bar 2: mem at 0xfea10000 [0xfea10fff]
        bar 6: mem at 0xffffffffffffffff [0xfffe]
      dev: ICH9 SMB, id ""
        addr = 1f.3
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
        bar 4: i/o at 0x700 [0x73f]
        bus: i2c
          type i2c-bus
          dev: smbus-eeprom, id ""
            address = 87 (0x57)
          dev: smbus-eeprom, id ""
            address = 86 (0x56)
          dev: smbus-eeprom, id ""
            address = 85 (0x55)
          dev: smbus-eeprom, id ""
            address = 84 (0x54)
          dev: smbus-eeprom, id ""
            address = 83 (0x53)
          dev: smbus-eeprom, id ""
            address = 82 (0x52)
          dev: smbus-eeprom, id ""
            address = 81 (0x51)
          dev: smbus-eeprom, id ""
            address = 80 (0x50)
      dev: ich9-ahci, id ""
        addr = 1f.2
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
        bar 4: i/o at 0xe040 [0xe05f]
        bar 5: mem at 0xfea11000 [0xfea11fff]
        bus: ide.5
          type IDE
        bus: ide.4
          type IDE
        bus: ide.3
          type IDE
        bus: ide.2
          type IDE
          dev: ide-cd, id ""
            drive = "ide1-cd0"
            logical_block_size = 512 (0x200)
            physical_block_size = 512 (0x200)
            min_io_size = 0 (0x0)
            opt_io_size = 0 (0x0)
            discard_granularity = 512 (0x200)
            ver = "2.5+"
            wwn = 0 (0x0)
            serial = "QM00005"
            model = ""
            unit = 0 (0x0)
        bus: ide.1
          type IDE
        bus: ide.0
          type IDE
      dev: ICH9-LPC, id ""
        noreboot = true
        addr = 1f.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
        bus: isa.0
          type ISA
          dev: i8257, id ""
            base = 192 (0xc0)
            page-base = 136 (0x88)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 1 (0x1)
          dev: i8257, id ""
            base = 0 (0x0)
            page-base = 128 (0x80)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 0 (0x0)
          dev: port92, id ""
          dev: vmmouse, id ""
          dev: vmport, id ""
          dev: i8042, id ""
            isa irqs 1,12
          dev: isa-serial, id ""
            index = 0 (0x0)
            iobase = 1016 (0x3f8)
            irq = 4 (0x4)
            chardev = "serial0"
            wakeup = 0 (0x0)
            isa irq 4
          dev: isa-pcspk, id ""
            iobase = 97 (0x61)
          dev: kvm-pit, id ""
            gpio-in "" 1
            iobase = 64 (0x40)
            lost_tick_policy = "delay"
          dev: mc146818rtc, id ""
            base_year = 0 (0x0)
            lost_tick_policy = "discard"
            isa irq 8
          dev: kvm-i8259, id ""
            iobase = 160 (0xa0)
            elcr_addr = 1233 (0x4d1)
            elcr_mask = 222 (0xde)
            master = false
          dev: kvm-i8259, id ""
            iobase = 32 (0x20)
            elcr_addr = 1232 (0x4d0)
            elcr_mask = 248 (0xf8)
            master = true
      dev: mch, id ""
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
  dev: fw_cfg_io, id ""
    iobase = 1296 (0x510)
    dma_iobase = 1300 (0x514)
    dma_enabled = true
  dev: kvmclock, id ""
  dev: kvmvapic, id ""

3. check the info through "lspci" in guest
[root@localhost ~]# lspci -t -vvv
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Device 1234:1111
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device

4. Check the devices on the correct numa node

[root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 
0
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 
1
[root@localhost ~]#
Comment 9 errata-xmlrpc 2016-11-07 15:46:18 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html

Note You need to log in before you can comment on or make changes to this bug.