Bug 1343094

Summary: e1000e network device should be assigned an address on a PCIe controller rather than legacy PCI
Product: Red Hat Enterprise Linux 7 Reporter: Yvugenfi <yvugenfi>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Han Han <hhan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: ailan, dyuan, jasowang, laine, lmen, mst, rbalakri, xuzhang, yalzhang, yanyang
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-3.1.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:09:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1330024, 1343092, 1343429, 1361534    
Bug Blocks:    

Description Yvugenfi@redhat.com 2016-06-06 13:33:05 UTC
Description of problem:

1000e implementation was introduced into upstream QEMU. This is a next generation Intel's PCIe NIC.

It might be desirable to use e1000e instead of e1000 in Q35 chipset and in newer OSes. 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Laine Stump 2016-06-06 17:11:30 UTC
libvirt actually allows any name in the <model type='x'/> attribute, so it's possible to specify this right now - if the qemu doesn't support it, then qemu itself will return an error. Ways of making it nicer:

1) add a qemu capabilities flag for it. (this will be needed for either of the two alternatives below)

and one of the following:

2a) require explicitly choosing e1000e rather than e1000, and give an error if it's not available

*or*

2b) automatically use e1000e when someone requests e1000, if the slot it's being plugged into is a PCIe slot. (This would necessitate storing the exact driver used in some manner, probably in the same (so far undetermined) manner we will store PCI vs. PCIe for the nec-usb-xhci device to eliminate ambiguity when it's plugged into pcie.0 (since pcie.0 slots can have either PCI or PCIe))

The advantage of (2b) is that the initial <interface> definition would be the same when plugging into either a Q35 or 440fx machine, so there would be less to document, and it would be easier for management apps to just accept the default machine type (whatever it is) without needing to adjust anything else. However, as mentioned above this would require saving the result of the automatic e1000 vs. e1000e decision to eliminate ambiguity when an "e1000" device was plugged directly into pcie.0 . Also, the device ID changes from e1000 (8086:100E) to e1000e (8086:10D3) - while us keeping track of what choice was auto-made initially would prevent the device ID from changing later without manual intervention (thus validating that change), I'm unsure whether or not we think it's acceptable for the same initial config of <model type='e1000'/> to represent a different PCI ID on different machine types - is there any other precedence for this? (I can't think of any, but admit I haven't thought that hard).

So I'm undecided which is the less intrusive / most useful way to add official support.

Comment 3 Peter Krempa 2016-06-07 10:36:18 UTC
*** Bug 1343429 has been marked as a duplicate of this bug. ***

Comment 4 Laine Stump 2016-08-14 05:27:07 UTC
Since we don't automatically choose between two different devices based on machinetype/root bus type for any other device model, I scrapped idea 2b). As with any other device, if you want an e1000e, you should specify

   <model type='e1000e'/>

if you want the e1000e. I've made a patch to auto-assign e1000e devices to a PCIe slot on Q35 which will be posted upstream in a day or so. (even without this patch, you can still specify <model type='e1000e'/> and you'll get an e1000e device, it will just be auto-assigned an address on the legacy pci-bridge unless you manually specify a PCI address)

Comment 7 Laine Stump 2016-11-08 15:58:17 UTC
I've changed the description to match the actual required task - e1000e can already be used by libvirt, we just need to auto-assign it to the correct type of PCI bus.

Comment 8 Laine Stump 2017-01-26 18:14:48 UTC
This was actually resolved quite awhile ago upstream, but I forgot to update the BZ:

commit 9dfe733e99f47b605ef02639276efc514825012d
Author: Laine Stump <laine>
Date:   Mon Aug 8 05:23:57 2016 -0400

    qemu: assign e1000e network devices to PCIe slots when appropriate

Note that backporting this single patch to an earlier release will not be possible - it has *many* prerequisite patches. All of these patches are listed in Bug 1330024 (which this BZ already depends on).

Comment 10 Han Han 2017-06-09 06:20:23 UTC
Verify it on libvirt-3.2.0-9.el7.x86_64 qemu-kvm-rhev-2.9.0-9.el7.x86_64
1. Define a q35 machine with pci-bridge,pcie-root-port, pcie-switch-upstream-port, pcie-switch-downstream-por:
# virsh dumpxml e1000e
...
 <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='3'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
 <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='8' model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='9' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='9' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </controller>
...

2. Cold-plug serval e1000e NICs without PCI address to the VM.
# for i in {1..4};do virsh attach-interface e1000e network default --model e1000e --config;done
Interface attached successfully

Interface attached successfully

Interface attached successfully

Interface attached successfully

3. Check the NICs are attached to pcie controllers:
NICs:
 <interface type='network'>
      <mac address='52:54:00:9f:1c:cf'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:ad:5d:8e'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:03:4f:58'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:1d:32:bf'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
    </interface>
Controllers attached:
<controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
   <controller type='pci' index='9' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='9' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>

All NICs are attached pcie controllers.

For other functional tests on e1000e, yalzhang has finished and all pass.

Comment 11 errata-xmlrpc 2017-08-01 17:09:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 12 errata-xmlrpc 2017-08-01 23:51:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846