1272939 – [PCIe] Add multi-root support for Q35

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1272939 - [PCIe] Add multi-root support for Q35

Summary: [PCIe] Add multi-root support for Q35

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	qemu-kvm-rhev
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Marcel Apfelbaum
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1227278 1305606 1313485
TreeView+	depends on / blocked

Reported:	2015-10-19 09:23 UTC by Marcel Apfelbaum
Modified:	2016-11-07 20:46 UTC (History)
CC List:	12 users (show)
Fixed In Version:	qemu-kvm-rhev-2.5.0-3.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-07 20:46:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:2673	0	normal	SHIPPED_LIVE	qemu-kvm-rhev bug fix and enhancement update	2016-11-08 01:06:13 UTC

Description Marcel Apfelbaum 2015-10-19 09:23:10 UTC

Add support for multiple PCIe Root Complexes as defined in PCI Express spec.

Comment 1 Marcel Apfelbaum 2015-12-22 09:38:29 UTC

A solution is posted upstream, see:
http://comments.gmane.org/gmane.comp.emulators.qemu/378868

The maintainer did not merge the code yet, since is not part of QEMU 2.5 it will be backported once merged.

Comment 2 Miroslav Rezanina 2016-03-23 09:53:59 UTC

Fix included in qemu-kvm-rhev-2.5.0-3.el7

Comment 4 jingzhao 2016-05-18 02:43:21 UTC

Tested it with following steps
kernel-3.10.0-402.el7.x86_64
qemu-kvm-rhev-2.6.0-1.el7.x86_64
seabios-1.9.1-4.el7.x86_64

test steps:
1. Add nic to pxb bus, boot vm with following command:
/usr/libexec/qemu-kvm \
-M q35 \
-monitor stdio \
-S -vnc :20 \
-cpu SandyBridge \
-m 2G \
-vga std \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device ioh3420,bus=bridge1,id=root.0,slot=1 \
-device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
-device ioh3420,bus=bridge2,id=root.1,slot=2 \
-drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
-device virtio-blk-pci,drive=drive0,bus=root.1 \

2.Using nic with connected to pxb-pcie and install rhel7.3 guest

Test result:
1. rhel7.3 guest install successfully
2. check the info through "info qtree"
(qemu) info qtree
bus: main-system-bus
  type System
  dev: pxb-host, id ""
    bus: bridge2
      type pxb-pcie-bus
      dev: ioh3420, id "root.1"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 2 (0x2)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.1
          type PCIE
          dev: virtio-blk-pci, id ""
            class = 0 (0x0)
            ioeventfd = true
            vectors = 2 (0x2)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002)
            bar 0: i/o at 0xc000 [0xc03f]
            bar 1: mem at 0xfe600000 [0xfe600fff]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-blk-device, id ""
                drive = "drive0"
                logical_block_size = 512 (0x200)
                physical_block_size = 512 (0x200)
                min_io_size = 0 (0x0)
                opt_io_size = 0 (0x0)
                discard_granularity = 4294967295 (0xffffffff)
                cyls = 16383 (0x3fff)
                heads = 16 (0x10)
                secs = 63 (0x3f)
                serial = ""
                config-wce = true
                scsi = false
                request-merging = true
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = false
  dev: pxb-host, id ""
    bus: bridge1
      type pxb-pcie-bus
      dev: ioh3420, id "root.0"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 1 (0x1)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.0
          type PCIE
          dev: virtio-net-pci, id ""
            ioeventfd = false
            vectors = 3 (0x3)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = "pxe-virtio.rom"
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001)
            bar 0: i/o at 0xd000 [0xd01f]
            bar 1: mem at 0xfe840000 [0xfe840fff]
            bar 6: mem at 0xffffffffffffffff [0x3fffe]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-net-device, id ""
                csum = true
                guest_csum = true
                gso = true
                guest_tso4 = true
                guest_tso6 = true
                guest_ecn = true
                guest_ufo = true
                guest_announce = true
                host_tso4 = true
                host_tso6 = true
                host_ecn = true
                host_ufo = true
                mrg_rxbuf = true
                status = true
                ctrl_vq = true
                ctrl_rx = true
                ctrl_vlan = true
                ctrl_rx_extra = true
                ctrl_mac_addr = true
                ctrl_guest_offloads = true
                mq = false
                mac = "9a:6a:6b:6c:6d:6e"
                vlan = <null>
                netdev = "tap10"
                x-txtimer = 150000 (0x249f0)
                x-txburst = 256 (0x100)
                tx = ""
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = true
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 0 (0 B)
    short_root_bus = 0 (0x0)
    bus: pcie.0
      type PCIE
      dev: pxb-pcie, id "bridge2"
        bus_nr = 40 (0x28)
        numa_node = 1 (0x1)
        addr = 03.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
      dev: pxb-pcie, id "bridge1"
        bus_nr = 8 (0x8)
        numa_node = 0 (0x0)
        addr = 02.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
      dev: VGA, id ""
        vgamem_mb = 16 (0x10)
        mmio = true
        qemu-extended-regs = false
        addr = 01.0
        romfile = "vgabios-stdvga.bin"
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
        bar 0: mem at 0xfd000000 [0xfdffffff]
        bar 2: mem at 0xfea10000 [0xfea10fff]
        bar 6: mem at 0xffffffffffffffff [0xfffe]
      dev: ICH9 SMB, id ""
        addr = 1f.3
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
        bar 4: i/o at 0x700 [0x73f]
        bus: i2c
          type i2c-bus
          dev: smbus-eeprom, id ""
            address = 87 (0x57)
          dev: smbus-eeprom, id ""
            address = 86 (0x56)
          dev: smbus-eeprom, id ""
            address = 85 (0x55)
          dev: smbus-eeprom, id ""
            address = 84 (0x54)
          dev: smbus-eeprom, id ""
            address = 83 (0x53)
          dev: smbus-eeprom, id ""
            address = 82 (0x52)
          dev: smbus-eeprom, id ""
            address = 81 (0x51)
          dev: smbus-eeprom, id ""
            address = 80 (0x50)
      dev: ich9-ahci, id ""
        addr = 1f.2
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
        bar 4: i/o at 0xe040 [0xe05f]
        bar 5: mem at 0xfea11000 [0xfea11fff]
        bus: ide.5
          type IDE
        bus: ide.4
          type IDE
        bus: ide.3
          type IDE
        bus: ide.2
          type IDE
          dev: ide-cd, id ""
            drive = "ide1-cd0"
            logical_block_size = 512 (0x200)
            physical_block_size = 512 (0x200)
            min_io_size = 0 (0x0)
            opt_io_size = 0 (0x0)
            discard_granularity = 512 (0x200)
            ver = "2.5+"
            wwn = 0 (0x0)
            serial = "QM00005"
            model = ""
            unit = 0 (0x0)
        bus: ide.1
          type IDE
        bus: ide.0
          type IDE
      dev: ICH9-LPC, id ""
        noreboot = true
        addr = 1f.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
        bus: isa.0
          type ISA
          dev: i8257, id ""
            base = 192 (0xc0)
            page-base = 136 (0x88)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 1 (0x1)
          dev: i8257, id ""
            base = 0 (0x0)
            page-base = 128 (0x80)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 0 (0x0)
          dev: port92, id ""
          dev: vmmouse, id ""
          dev: vmport, id ""
          dev: i8042, id ""
            isa irqs 1,12
          dev: isa-serial, id ""
            index = 0 (0x0)
            iobase = 1016 (0x3f8)
            irq = 4 (0x4)
            chardev = "serial0"
            wakeup = 0 (0x0)
            isa irq 4
          dev: isa-pcspk, id ""
            iobase = 97 (0x61)
          dev: kvm-pit, id ""
            gpio-in "" 1
            iobase = 64 (0x40)
            lost_tick_policy = "delay"
          dev: mc146818rtc, id ""
            base_year = 0 (0x0)
            lost_tick_policy = "discard"
            isa irq 8
          dev: kvm-i8259, id ""
            iobase = 160 (0xa0)
            elcr_addr = 1233 (0x4d1)
            elcr_mask = 222 (0xde)
            master = false
          dev: kvm-i8259, id ""
            iobase = 32 (0x20)
            elcr_addr = 1232 (0x4d0)
            elcr_mask = 248 (0xf8)
            master = true
      dev: mch, id ""
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
  dev: fw_cfg_io, id ""
    iobase = 1296 (0x510)
    dma_iobase = 1300 (0x514)
    dma_enabled = true
  dev: kvmclock, id ""
  dev: kvmvapic, id ""

3. check the info through "lspci" in guest
[root@localhost ~]# lspci -t -vvv
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Device 1234:1111
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device

Marcel, I think it's the expected result, could you check it?

Thanks
Jing

Comment 5 Marcel Apfelbaum 2016-05-19 19:03:07 UTC

(In reply to jingzhao from comment #4)
> Tested it with following steps
> kernel-3.10.0-402.el7.x86_64
> qemu-kvm-rhev-2.6.0-1.el7.x86_64
> seabios-1.9.1-4.el7.x86_64
> 
> test steps:
> 1. Add nic to pxb bus, boot vm with following command:
> /usr/libexec/qemu-kvm \
> -M q35 \
> -monitor stdio \
> -S -vnc :20 \
> -cpu SandyBridge \
> -m 2G \
> -vga std \
> -smp 4,sockets=4,cores=1,threads=1 \
> -object memory-backend-ram,size=1024M,id=ram-node0 \
> -numa node,nodeid=0,cpus=0,memdev=ram-node0 \
> -object memory-backend-ram,size=1024M,id=ram-node1 \
> -numa node,nodeid=1,cpus=1,memdev=ram-node1 \
> -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
> -device ioh3420,bus=bridge1,id=root.0,slot=1 \
> -device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev
> tap,id=tap10 \
> -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
> -device ioh3420,bus=bridge2,id=root.1,slot=2 \
> -drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
> -device virtio-blk-pci,drive=drive0,bus=root.1 \
> 
> 2.Using nic with connected to pxb-pcie and install rhel7.3 guest
> 
> Test result:
> 1. rhel7.3 guest install successfully
> 2. check the info through "info qtree"
> (qemu) info qtree
> bus: main-system-bus
>   type System
>   dev: pxb-host, id ""
>     bus: bridge2
>       type pxb-pcie-bus
>       dev: ioh3420, id "root.1"
>         power_controller_present = true
>         chassis = 0 (0x0)
>         slot = 2 (0x2)
>         port = 0 (0x0)
>         aer_log_max = 8 (0x8)
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
>         bus: root.1
>           type PCIE
>           dev: virtio-blk-pci, id ""
>             class = 0 (0x0)
>             ioeventfd = true
>             vectors = 2 (0x2)
>             virtio-pci-bus-master-bug-migration = true
>             disable-legacy = false
>             disable-modern = true
>             migrate-extra = true
>             modern-pio-notify = false
>             x-disable-pcie = false
>             addr = 00.0
>             romfile = ""
>             rombar = 1 (0x1)
>             multifunction = false
>             command_serr_enable = true
>             class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub
> 1af4:0002)
>             bar 0: i/o at 0xc000 [0xc03f]
>             bar 1: mem at 0xfe600000 [0xfe600fff]
>             bus: virtio-bus
>               type virtio-pci-bus
>               dev: virtio-blk-device, id ""
>                 drive = "drive0"
>                 logical_block_size = 512 (0x200)
>                 physical_block_size = 512 (0x200)
>                 min_io_size = 0 (0x0)
>                 opt_io_size = 0 (0x0)
>                 discard_granularity = 4294967295 (0xffffffff)
>                 cyls = 16383 (0x3fff)
>                 heads = 16 (0x10)
>                 secs = 63 (0x3f)
>                 serial = ""
>                 config-wce = true
>                 scsi = false
>                 request-merging = true
>                 indirect_desc = true
>                 event_idx = true
>                 notify_on_empty = true
>                 any_layout = false
>   dev: pxb-host, id ""
>     bus: bridge1
>       type pxb-pcie-bus
>       dev: ioh3420, id "root.0"
>         power_controller_present = true
>         chassis = 0 (0x0)
>         slot = 1 (0x1)
>         port = 0 (0x0)
>         aer_log_max = 8 (0x8)
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
>         bus: root.0
>           type PCIE
>           dev: virtio-net-pci, id ""
>             ioeventfd = false
>             vectors = 3 (0x3)
>             virtio-pci-bus-master-bug-migration = true
>             disable-legacy = false
>             disable-modern = true
>             migrate-extra = true
>             modern-pio-notify = false
>             x-disable-pcie = false
>             addr = 00.0
>             romfile = "pxe-virtio.rom"
>             rombar = 1 (0x1)
>             multifunction = false
>             command_serr_enable = true
>             class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub
> 1af4:0001)
>             bar 0: i/o at 0xd000 [0xd01f]
>             bar 1: mem at 0xfe840000 [0xfe840fff]
>             bar 6: mem at 0xffffffffffffffff [0x3fffe]
>             bus: virtio-bus
>               type virtio-pci-bus
>               dev: virtio-net-device, id ""
>                 csum = true
>                 guest_csum = true
>                 gso = true
>                 guest_tso4 = true
>                 guest_tso6 = true
>                 guest_ecn = true
>                 guest_ufo = true
>                 guest_announce = true
>                 host_tso4 = true
>                 host_tso6 = true
>                 host_ecn = true
>                 host_ufo = true
>                 mrg_rxbuf = true
>                 status = true
>                 ctrl_vq = true
>                 ctrl_rx = true
>                 ctrl_vlan = true
>                 ctrl_rx_extra = true
>                 ctrl_mac_addr = true
>                 ctrl_guest_offloads = true
>                 mq = false
>                 mac = "9a:6a:6b:6c:6d:6e"
>                 vlan = <null>
>                 netdev = "tap10"
>                 x-txtimer = 150000 (0x249f0)
>                 x-txburst = 256 (0x100)
>                 tx = ""
>                 indirect_desc = true
>                 event_idx = true
>                 notify_on_empty = true
>                 any_layout = true
>   dev: kvm-ioapic, id ""
>     gpio-in "" 24
>     gsi_base = 0 (0x0)
>     mmio 00000000fec00000/0000000000001000
>   dev: q35-pcihost, id ""
>     MCFG = 2952790016 (0xb0000000)
>     pci-hole64-size = 0 (0 B)
>     short_root_bus = 0 (0x0)
>     bus: pcie.0
>       type PCIE
>       dev: pxb-pcie, id "bridge2"
>         bus_nr = 40 (0x28)
>         numa_node = 1 (0x1)
>         addr = 03.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
>       dev: pxb-pcie, id "bridge1"
>         bus_nr = 8 (0x8)
>         numa_node = 0 (0x0)
>         addr = 02.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
>       dev: VGA, id ""
>         vgamem_mb = 16 (0x10)
>         mmio = true
>         qemu-extended-regs = false
>         addr = 01.0
>         romfile = "vgabios-stdvga.bin"
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
>         bar 0: mem at 0xfd000000 [0xfdffffff]
>         bar 2: mem at 0xfea10000 [0xfea10fff]
>         bar 6: mem at 0xffffffffffffffff [0xfffe]
>       dev: ICH9 SMB, id ""
>         addr = 1f.3
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
>         bar 4: i/o at 0x700 [0x73f]
>         bus: i2c
>           type i2c-bus
>           dev: smbus-eeprom, id ""
>             address = 87 (0x57)
>           dev: smbus-eeprom, id ""
>             address = 86 (0x56)
>           dev: smbus-eeprom, id ""
>             address = 85 (0x55)
>           dev: smbus-eeprom, id ""
>             address = 84 (0x54)
>           dev: smbus-eeprom, id ""
>             address = 83 (0x53)
>           dev: smbus-eeprom, id ""
>             address = 82 (0x52)
>           dev: smbus-eeprom, id ""
>             address = 81 (0x51)
>           dev: smbus-eeprom, id ""
>             address = 80 (0x50)
>       dev: ich9-ahci, id ""
>         addr = 1f.2
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
>         bar 4: i/o at 0xe040 [0xe05f]
>         bar 5: mem at 0xfea11000 [0xfea11fff]
>         bus: ide.5
>           type IDE
>         bus: ide.4
>           type IDE
>         bus: ide.3
>           type IDE
>         bus: ide.2
>           type IDE
>           dev: ide-cd, id ""
>             drive = "ide1-cd0"
>             logical_block_size = 512 (0x200)
>             physical_block_size = 512 (0x200)
>             min_io_size = 0 (0x0)
>             opt_io_size = 0 (0x0)
>             discard_granularity = 512 (0x200)
>             ver = "2.5+"
>             wwn = 0 (0x0)
>             serial = "QM00005"
>             model = ""
>             unit = 0 (0x0)
>         bus: ide.1
>           type IDE
>         bus: ide.0
>           type IDE
>       dev: ICH9-LPC, id ""
>         noreboot = true
>         addr = 1f.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = true
>         command_serr_enable = true
>         class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
>         bus: isa.0
>           type ISA
>           dev: i8257, id ""
>             base = 192 (0xc0)
>             page-base = 136 (0x88)
>             pageh-base = -1 (0xffffffffffffffff)
>             dshift = 1 (0x1)
>           dev: i8257, id ""
>             base = 0 (0x0)
>             page-base = 128 (0x80)
>             pageh-base = -1 (0xffffffffffffffff)
>             dshift = 0 (0x0)
>           dev: port92, id ""
>           dev: vmmouse, id ""
>           dev: vmport, id ""
>           dev: i8042, id ""
>             isa irqs 1,12
>           dev: isa-serial, id ""
>             index = 0 (0x0)
>             iobase = 1016 (0x3f8)
>             irq = 4 (0x4)
>             chardev = "serial0"
>             wakeup = 0 (0x0)
>             isa irq 4
>           dev: isa-pcspk, id ""
>             iobase = 97 (0x61)
>           dev: kvm-pit, id ""
>             gpio-in "" 1
>             iobase = 64 (0x40)
>             lost_tick_policy = "delay"
>           dev: mc146818rtc, id ""
>             base_year = 0 (0x0)
>             lost_tick_policy = "discard"
>             isa irq 8
>           dev: kvm-i8259, id ""
>             iobase = 160 (0xa0)
>             elcr_addr = 1233 (0x4d1)
>             elcr_mask = 222 (0xde)
>             master = false
>           dev: kvm-i8259, id ""
>             iobase = 32 (0x20)
>             elcr_addr = 1232 (0x4d0)
>             elcr_mask = 248 (0xf8)
>             master = true
>       dev: mch, id ""
>         addr = 00.0
>         romfile = ""
>         rombar = 1 (0x1)
>         multifunction = false
>         command_serr_enable = true
>         class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
>   dev: fw_cfg_io, id ""
>     iobase = 1296 (0x510)
>     dma_iobase = 1300 (0x514)
>     dma_enabled = true
>   dev: kvmclock, id ""
>   dev: kvmvapic, id ""
> 
> 3. check the info through "lspci" in guest
> [root@localhost ~]# lspci -t -vvv
> -+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
>  +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
>  \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
>              +-01.0  Device 1234:1111
>              +-02.0  Red Hat, Inc. Device 000b
>              +-03.0  Red Hat, Inc. Device 000b
>              +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface
> Controller
>              +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
> SATA Controller [AHCI mode]
>              \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
> [root@localhost ~]# lspci
> 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM
> Controller
> 00:01.0 VGA compatible controller: Device 1234:1111
> 00:02.0 Host bridge: Red Hat, Inc. Device 000b
> 00:03.0 Host bridge: Red Hat, Inc. Device 000b
> 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface
> Controller (rev 02)
> 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6
> port SATA Controller [AHCI mode] (rev 02)
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev
> 02)
> 08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
> 28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express
> Root Port 0 (rev 02)
> 29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device
> 
> Marcel, I think it's the expected result, could you check it?
> 
> Thanks
> Jing

The testing scenario is good, but you also need to check that the devices are now on the correct numa node.
You can check this by looking at 
cat /sys/bus/pci/devices/<device>/numa_node

And of course, I would check the device is working correctly :)
An assigned device on a pxb-pcie is also a good test.

Thanks,
Marcel

Comment 6 jingzhao 2016-05-23 02:22:05 UTC

Verfied it with fully test steps

kernel-3.10.0-402.el7.x86_64
qemu-kvm-rhev-2.6.0-2.el7.x86_64
seabios-1.9.1-4.el7.x86_64

1. Add nic to pxb bus, boot vm with following command:
/usr/libexec/qemu-kvm \
-M q35 \
-monitor stdio \
-S -vnc :20 \
-cpu SandyBridge \
-m 2G \
-vga std \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device ioh3420,bus=bridge1,id=root.0,slot=1 \
-device virtio-net-pci,bus=root.0,netdev=tap10,mac=9a:6a:6b:6c:6d:6e -netdev tap,id=tap10 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=40 \
-device ioh3420,bus=bridge2,id=root.1,slot=2 \
-drive if=none,id=drive0,file=/home/bug/1272939/pxb.qcow2 \
-device virtio-blk-pci,drive=drive0,bus=root.1 \

2.Using nic with connected to pxb-pcie and install rhel7.3 guest

Test result:
1. rhel7.3 guest install successfully
2. check the info through "info qtree"
(qemu) info qtree
bus: main-system-bus
  type System
  dev: pxb-host, id ""
    bus: bridge2
      type pxb-pcie-bus
      dev: ioh3420, id "root.1"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 2 (0x2)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 28:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.1
          type PCIE
          dev: virtio-blk-pci, id ""
            class = 0 (0x0)
            ioeventfd = true
            vectors = 2 (0x2)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class SCSI controller, addr 29:00.0, pci id 1af4:1001 (sub 1af4:0002)
            bar 0: i/o at 0xc000 [0xc03f]
            bar 1: mem at 0xfe600000 [0xfe600fff]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-blk-device, id ""
                drive = "drive0"
                logical_block_size = 512 (0x200)
                physical_block_size = 512 (0x200)
                min_io_size = 0 (0x0)
                opt_io_size = 0 (0x0)
                discard_granularity = 4294967295 (0xffffffff)
                cyls = 16383 (0x3fff)
                heads = 16 (0x10)
                secs = 63 (0x3f)
                serial = ""
                config-wce = true
                scsi = false
                request-merging = true
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = false
  dev: pxb-host, id ""
    bus: bridge1
      type pxb-pcie-bus
      dev: ioh3420, id "root.0"
        power_controller_present = true
        chassis = 0 (0x0)
        slot = 1 (0x1)
        port = 0 (0x0)
        aer_log_max = 8 (0x8)
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 08:00.0, pci id 8086:3420 (sub 0000:0000)
        bus: root.0
          type PCIE
          dev: virtio-net-pci, id ""
            ioeventfd = false
            vectors = 3 (0x3)
            virtio-pci-bus-master-bug-migration = true
            disable-legacy = false
            disable-modern = true
            migrate-extra = true
            modern-pio-notify = false
            x-disable-pcie = false
            addr = 00.0
            romfile = "pxe-virtio.rom"
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class Ethernet controller, addr 09:00.0, pci id 1af4:1000 (sub 1af4:0001)
            bar 0: i/o at 0xd000 [0xd01f]
            bar 1: mem at 0xfe840000 [0xfe840fff]
            bar 6: mem at 0xffffffffffffffff [0x3fffe]
            bus: virtio-bus
              type virtio-pci-bus
              dev: virtio-net-device, id ""
                csum = true
                guest_csum = true
                gso = true
                guest_tso4 = true
                guest_tso6 = true
                guest_ecn = true
                guest_ufo = true
                guest_announce = true
                host_tso4 = true
                host_tso6 = true
                host_ecn = true
                host_ufo = true
                mrg_rxbuf = true
                status = true
                ctrl_vq = true
                ctrl_rx = true
                ctrl_vlan = true
                ctrl_rx_extra = true
                ctrl_mac_addr = true
                ctrl_guest_offloads = true
                mq = false
                mac = "9a:6a:6b:6c:6d:6e"
                vlan = <null>
                netdev = "tap10"
                x-txtimer = 150000 (0x249f0)
                x-txburst = 256 (0x100)
                tx = ""
                indirect_desc = true
                event_idx = true
                notify_on_empty = true
                any_layout = true
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 0 (0 B)
    short_root_bus = 0 (0x0)
    bus: pcie.0
      type PCIE
      dev: pxb-pcie, id "bridge2"
        bus_nr = 40 (0x28)
        numa_node = 1 (0x1)
        addr = 03.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:03.0, pci id 1b36:000b (sub 1af4:1100)
      dev: pxb-pcie, id "bridge1"
        bus_nr = 8 (0x8)
        numa_node = 0 (0x0)
        addr = 02.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:02.0, pci id 1b36:000b (sub 1af4:1100)
      dev: VGA, id ""
        vgamem_mb = 16 (0x10)
        mmio = true
        qemu-extended-regs = false
        addr = 01.0
        romfile = "vgabios-stdvga.bin"
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class VGA controller, addr 00:01.0, pci id 1234:1111 (sub 1af4:1100)
        bar 0: mem at 0xfd000000 [0xfdffffff]
        bar 2: mem at 0xfea10000 [0xfea10fff]
        bar 6: mem at 0xffffffffffffffff [0xfffe]
      dev: ICH9 SMB, id ""
        addr = 1f.3
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SMBus, addr 00:1f.3, pci id 8086:2930 (sub 1af4:1100)
        bar 4: i/o at 0x700 [0x73f]
        bus: i2c
          type i2c-bus
          dev: smbus-eeprom, id ""
            address = 87 (0x57)
          dev: smbus-eeprom, id ""
            address = 86 (0x56)
          dev: smbus-eeprom, id ""
            address = 85 (0x55)
          dev: smbus-eeprom, id ""
            address = 84 (0x54)
          dev: smbus-eeprom, id ""
            address = 83 (0x53)
          dev: smbus-eeprom, id ""
            address = 82 (0x52)
          dev: smbus-eeprom, id ""
            address = 81 (0x51)
          dev: smbus-eeprom, id ""
            address = 80 (0x50)
      dev: ich9-ahci, id ""
        addr = 1f.2
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class SATA controller, addr 00:1f.2, pci id 8086:2922 (sub 1af4:1100)
        bar 4: i/o at 0xe040 [0xe05f]
        bar 5: mem at 0xfea11000 [0xfea11fff]
        bus: ide.5
          type IDE
        bus: ide.4
          type IDE
        bus: ide.3
          type IDE
        bus: ide.2
          type IDE
          dev: ide-cd, id ""
            drive = "ide1-cd0"
            logical_block_size = 512 (0x200)
            physical_block_size = 512 (0x200)
            min_io_size = 0 (0x0)
            opt_io_size = 0 (0x0)
            discard_granularity = 512 (0x200)
            ver = "2.5+"
            wwn = 0 (0x0)
            serial = "QM00005"
            model = ""
            unit = 0 (0x0)
        bus: ide.1
          type IDE
        bus: ide.0
          type IDE
      dev: ICH9-LPC, id ""
        noreboot = true
        addr = 1f.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = true
        command_serr_enable = true
        class ISA bridge, addr 00:1f.0, pci id 8086:2918 (sub 1af4:1100)
        bus: isa.0
          type ISA
          dev: i8257, id ""
            base = 192 (0xc0)
            page-base = 136 (0x88)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 1 (0x1)
          dev: i8257, id ""
            base = 0 (0x0)
            page-base = 128 (0x80)
            pageh-base = -1 (0xffffffffffffffff)
            dshift = 0 (0x0)
          dev: port92, id ""
          dev: vmmouse, id ""
          dev: vmport, id ""
          dev: i8042, id ""
            isa irqs 1,12
          dev: isa-serial, id ""
            index = 0 (0x0)
            iobase = 1016 (0x3f8)
            irq = 4 (0x4)
            chardev = "serial0"
            wakeup = 0 (0x0)
            isa irq 4
          dev: isa-pcspk, id ""
            iobase = 97 (0x61)
          dev: kvm-pit, id ""
            gpio-in "" 1
            iobase = 64 (0x40)
            lost_tick_policy = "delay"
          dev: mc146818rtc, id ""
            base_year = 0 (0x0)
            lost_tick_policy = "discard"
            isa irq 8
          dev: kvm-i8259, id ""
            iobase = 160 (0xa0)
            elcr_addr = 1233 (0x4d1)
            elcr_mask = 222 (0xde)
            master = false
          dev: kvm-i8259, id ""
            iobase = 32 (0x20)
            elcr_addr = 1232 (0x4d0)
            elcr_mask = 248 (0xf8)
            master = true
      dev: mch, id ""
        addr = 00.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class Host bridge, addr 00:00.0, pci id 8086:29c0 (sub 1af4:1100)
  dev: fw_cfg_io, id ""
    iobase = 1296 (0x510)
    dma_iobase = 1300 (0x514)
    dma_enabled = true
  dev: kvmclock, id ""
  dev: kvmvapic, id ""

3. check the info through "lspci" in guest
[root@localhost ~]# lspci -t -vvv
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio network device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Device 1234:1111
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
09:00.0 Ethernet controller: Red Hat, Inc Virtio network device
28:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02)
29:00.0 SCSI storage controller: Red Hat, Inc Virtio block device

4. Check the devices on the correct numa node

[root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 
0
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 
1
[root@localhost ~]#

Comment 9 errata-xmlrpc 2016-11-07 20:46:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html

Note You need to log in before you can comment on or make changes to this bug.