Bug 893738
Summary: | libvirt should log an error/fail attempted use of any SRIOV VF whose PF is offline | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Laine Stump <laine> |
Component: | libvirt | Assignee: | Laine Stump <laine> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.0 | CC: | ajia, cwei, dyuan, honzhang, laine, lmiksik, mzhan, rbalakri, sassmann, ydu |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-1.2.16-1.el7 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-11-19 05:42:01 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Laine Stump
2013-01-09 20:00:08 UTC
I've also tried a traditional <hostdev> passthrough (which doesn't set the MAC address first), and it also sets a different MAC address in the guest that what was originally in the host. Here's the libvirt XML I used for the "virsh attach-device" command: <hostdev type='pci' managed='yes'> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> Please make sure the PF is UP before running the guest. Looking at your output igbvf 0000:00:07.0: PF still in reset state, assigning new address. Is the PF interface up? igbvf 0000:00:07.0: PF still resetting suggests the PF is down. The VF MAC can only be assigned if the PF is up and running. Hope this helps. Ah yes, all the alarm for nothing :-) I previously always had NetworkManager enabled on this machine, and it wants to ifup *everything* (including all the VFs), but I disabled it awhile back because I was tired of the extra 14 interfaces listed in the output of ifconfig. Rather than close this, I'll re-assign it to libvirt - libvirt should be making sure that the PF is up (as it already does for macvtap passthrough mode). This bug was not selected to be addressed in Red Hat Enterprise Linux 6. We will look at it again within the Red Hat Enterprise Linux 7 product. Patch posted upstream: https://www.redhat.com/archives/libvir-list/2015-May/msg00745.html auto-starting the PF was considered too risky, since that would start IPv6 autoconf on an otherwise unconfiged interface, and the admin may not be prepared for that. So instead, the patch logs an error message (which advises the admin to enable the PF in the host system network config) and fails the operation. Pushed upstream, will be in 1.2.16: commit 474523fa2c67566bb61807fd413e5efc5f3510cb Author: Laine Stump <laine> Date: Tue May 5 18:27:47 2015 -0400 netdev: fail when setting up an SRIOV VF if PF is offline Hi Laine The vf still can be attached to guest when ifdown the PF.The result isn't expected. Is it right ? # rpm -q libvirt libvirt-1.2.17-6.el7.x86_64 # uname -r 3.10.0-229.el7.x86_64 # lspci|grep 82576 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) # virsh nodedev-list --tree computer | +- net_lo_00_00_00_00_00_00 +- net_virbr0_nic_52_54_00_68_5c_9a +- pci_0000_00_00_0 +- pci_0000_00_01_0 | | | +- pci_0000_03_00_0 | | | | | +- net_ens1f0_00_1b_21_39_8b_18 | | | +- pci_0000_03_00_1 | | | | | +- net_ens1f1_00_1b_21_39_8b_19 | | | +- pci_0000_03_10_0 | | | | | +- net_enp3s16_a2_2f_83_9b_89_3a | | | +- pci_0000_03_10_1 | | | | | +- net_enp3s16f1_1a_7c_dd_fd_0f_63 | | | +- pci_0000_03_10_2 | | | | | +- net_enp3s16f2_36_d8_04_90_78_f8 ..... # virsh nodedev-dumpxml pci_0000_03_00_1 <device> <name>pci_0000_03_00_1</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:00.1</path> <parent>pci_0000_00_01_0</parent> <driver> <name>igb</name> </driver> <capability type='pci'> <domain>0</domain> <bus>3</bus> <slot>0</slot> <function>1</function> <product id='0x10c9'>82576 Gigabit Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> <address domain='0x0000' bus='0x03' slot='0x10' function='0x3'/> <address domain='0x0000' bus='0x03' slot='0x10' function='0x5'/> <address domain='0x0000' bus='0x03' slot='0x10' function='0x7'/> <address domain='0x0000' bus='0x03' slot='0x11' function='0x1'/> <address domain='0x0000' bus='0x03' slot='0x11' function='0x3'/> <address domain='0x0000' bus='0x03' slot='0x11' function='0x5'/> </capability> <iommuGroup number='14'> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='2.5' width='4'/> <link validity='sta' speed='2.5' width='4'/> </pci-express> </capability> </device> # ip link show ens1f1 35: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:1b:21:39:8b:19 brd ff:ff:ff:ff:ff:ff vf 0 MAC e2:80:6a:a9:cd:c5, spoof checking on, link-state auto vf 1 MAC 4e:4e:04:5f:40:9c, spoof checking on, link-state auto vf 2 MAC 5a:b3:0a:73:93:25, spoof checking on, link-state auto vf 3 MAC 82:f3:ee:0e:84:20, spoof checking on, link-state auto vf 4 MAC 86:5b:f2:5e:14:df, spoof checking on, link-state auto vf 5 MAC 2a:0c:80:16:15:77, spoof checking on, link-state auto vf 6 MAC aa:15:44:55:16:57, spoof checking on, link-state auto # cat vf.xml <interface type='hostdev' managed='yes'> <mac address='52:54:00:3b:3e:02'/> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <model type='virtio'/> </interface> # ifdown ens1f1 Device 'ens1f1' successfully disconnected. # virsh start r7 Domain r7 started # virsh attach-device r7 vf.xml Device attached successfully # ip link show ens1f1 35: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:1b:21:39:8b:19 brd ff:ff:ff:ff:ff:ff vf 0 MAC 52:54:00:3b:3e:02, spoof checking on, link-state auto vf 1 MAC 4e:4e:04:5f:40:9c, spoof checking on, link-state auto vf 2 MAC 5a:b3:0a:73:93:25, spoof checking on, link-state auto vf 3 MAC 82:f3:ee:0e:84:20, spoof checking on, link-state auto vf 4 MAC 86:5b:f2:5e:14:df, spoof checking on, link-state auto vf 5 MAC 2a:0c:80:16:15:77, spoof checking on, link-state auto vf 6 MAC aa:15:44:55:16:57, spoof checking on, link-state auto That's not the result I get: error: Failed to start domain RHEL6 error: internal error: Unable to configure VF 0 of PF 'enp3s0f1' because the PF is not online. Please change host network config to put the PF online. Are you running NetworkManager on your host? "something" is preventing the interface from going down. NetworkManager is well known for doing that - it is essentially impossible to force an interface to the "down" state when NetworkManager is running. Please disable NetworkManager then try the test again. (Now that the rhel-7.2.0 flag has been reset from ?, I'm not sure if I can move it back to ON_QA, but I will try) (btw, I notice that you ran "ifdown ens1f1" in your test, but didn't run "ip link show ens1f1" to see if the status cahnge actually registered) Hi Laine It still can be reproduced after stopping th NetworkManager. Could you help to check it again ? # rpm -q libvirt libvirt-1.2.17-8.el7.x86_64 # uname -r 3.10.0-306.0.1.el7.x86_64 # lspci|grep 82576 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) # service NetworkManager stop Redirecting to /bin/systemctl stop NetworkManager.service # service NetworkManager status Redirecting to /bin/systemctl status NetworkManager.service ● NetworkManager.service - Network Manager Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled) Active: inactive (dead) since Tue 2015-09-15 16:22:02 CST; 5s ago ...... # virsh nodedev-list --tree computer | +- net_lo_00_00_00_00_00_00 +- net_virbr0_nic_52_54_00_38_c8_ad +- pci_0000_00_00_0 +- pci_0000_00_01_0 | | | +- pci_0000_03_00_0 | | | | | +- net_ens1f0_00_1b_21_39_8b_18 ..... | # virsh nodedev-dumpxml pci_0000_03_00_0 <device> <name>pci_0000_03_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>igb</name> </driver> <capability type='pci'> <domain>0</domain> <bus>3</bus> <slot>0</slot> <function>0</function> <product id='0x10c9'>82576 Gigabit Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/> <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </capability> <iommuGroup number='14'> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='2.5' width='4'/> <link validity='sta' speed='2.5' width='4'/> </pci-express> </capability> </device> # ifdown ens1f0 # ifconfig ens1f0 ens1f0: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether 00:1b:21:39:8b:18 txqueuelen 1000 (Ethernet) RX packets 724362 bytes 52700770 (50.2 MiB) RX errors 0 dropped 0 overruns 1186 frame 0 TX packets 20808 bytes 5858654 (5.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xf4800000-f481ffff # virsh start r7 Domain r7 started # virsh dumpxml r7|grep /hostdev -B6 <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> # cat vf.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0' bus='0x3' slot='0x10' function='0x0'/> </source> </hostdev> # virsh attach-device r7 vf.xml Device attached successfully # virsh dumpxml r7 |grep /hostdev -B7 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> Wait. Now I understand. You are assigning the device to the guest with <hostdev>, which has no concept of SRIOV PFs and VFs, so it can't possibly know that there is a PF that needs to be iff_up, or even that the device being assigned is a network device. You need to assign the VF as a *network* device, e.g.: <interface type='hostdev' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface> Sorry I didn't catch this flaw in the test before - that's what happens when I skim too fast. Please retest using <interface type='hostdev' managed='yes'> instead of <hostdev managed='yes'>. Verify it as follows. The result is expected. Move its status to VERIFIED. # rpm -q libvirt libvirt-1.2.17-9.el7.x86_64 # service NetworkManager stop Redirecting to /bin/systemctl stop NetworkManager.service # virsh nodedev-list --tree computer | ...... | | | +- pci_0000_03_00_0 | | | | | +- net_ens1f0_00_1b_21_39_8b_18 | | | +- pci_0000_03_00_1 | | | | | +- net_ens1f1_00_1b_21_39_8b_19 ...... # virsh nodedev-dumpxml pci_0000_03_00_0 <device> <name>pci_0000_03_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>igb</name> </driver> <capability type='pci'> <domain>0</domain> <bus>3</bus> <slot>0</slot> <function>0</function> <product id='0x10c9'>82576 Gigabit Network Connection</product> <vendor id='0x8086'>Intel Corporation</vendor> <capability type='virt_functions'> <address domain='0x0000' bus='0x03' slot='0x10' function='0x0'/> <address domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </capability> <iommuGroup number='14'> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='2.5' width='4'/> <link validity='sta' speed='2.5' width='4'/> </pci-express> </capability> </device> # ifdown ens1f0 # virsh edit r7 (Add the following xml to guest) Domain r7 XML configuration edited. <interface type='hostdev' managed='yes'> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x2'/> </source> </interface> # virsh start r7 error: Failed to start domain r7 error: internal error: Unable to configure VF 1 of PF 'ens1f0' because the PF is not online. Please change host network config to put the PF online. ============================================ Remove the interface xml which contains VF from guest , then start guest # virsh start r7 Domain r7 started # cat if-hostdev.xml <interface type='network'> <source network='hostdev'/> </interface> # virsh attach-device r7 if.xml error: Failed to attach device from if.xml error: internal error: Unable to configure VF 1 of PF 'ens1f0' because the PF is not online. Please change host network config to put the PF online. # ifup ens1f0 # virsh attach-device r7 if.xml Device attached successfully Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2202.html |