Bug 1234328 - [RFE] SR-IOV --> add support for Hotplug/unplug of VFs
Summary: [RFE] SR-IOV --> add support for Hotplug/unplug of VFs
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: General
Version: ---
Hardware: x86_64
OS: Linux
Target Milestone: ovirt-4.0.0-beta
: 4.0.0
Assignee: Edward Haas
QA Contact: Michael Burman
Depends On:
Blocks: 868811 1425313
TreeView+ depends on / blocked
Reported: 2015-06-22 11:41 UTC by Michael Burman
Modified: 2017-02-21 07:41 UTC (History)
11 users (show)

Fixed In Version: ovirt 4.0.0 alpha1
Doc Type: Enhancement
Doc Text:
It is now possible to hot plug and unplug SR-IOV virtual functions to running virtual machines.
Clone Of:
Last Closed: 2016-07-05 08:05:10 UTC
oVirt Team: Network
rule-engine: ovirt-4.0.0+
gklein: testing_plan_complete-
rule-engine: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+

Attachments (Terms of Use)
screenshot (152.39 KB, image/png)
2015-06-22 11:47 UTC, Michael Burman
no flags Details
screenshot_engine_log (262.06 KB, image/png)
2015-06-22 11:53 UTC, Michael Burman
no flags Details
failed qa logs (718.82 KB, application/x-gzip)
2016-04-07 12:17 UTC, Michael Burman
no flags Details

System ID Private Priority Status Summary Last Updated
oVirt gerrit 55993 0 'None' MERGED Revert "hostdev/sr-iov: don't reattach devices" 2021-02-08 05:44:34 UTC
oVirt gerrit 56013 0 'None' MERGED Revert "hostdev: don't implicitly reattach devices" 2021-02-08 05:44:34 UTC

Description Michael Burman 2015-06-22 11:41:02 UTC
Description of problem:
SR-IOV --> Hotplug/unplug VFs profiles interfaces is not supported in 3.6 like it should be.
Hotplug/unplug VFs should be supported in 3.6 as part of the SRIOV feature, but it currently not supported and such operations failing with this error messages when trying to hotplug/unplug VM's vNIC with pci-passthrough type-->

- Failed to plug Network Interface nic2 (PCI Passthrough) to VM SR_VM4_. (User: admin@internal)
- Failed to add Interface nic2 (PCI Passthrough) to VM SR_VM4_. (User: admin@internal)
- Failed to unplug Network Interface nic1 (PCI Passthrough) from VM SR_VM4_. (User: admin@internal)
- Failed to update Interface nic1 (PCI Passthrough) for VM SR_VM4_. (User: admin@internal)

- If this support will not be ready for 3.6, then, engine and vdsm(or only engine) should continue blocking from Hotplug/unplug vNICs with pci-passthrough type profiles, but error message/s on engine side need some improve, explaining why i can't Hotplug/unplug vNIC to running VM with 'pci-passthrough' type.  

' Error while executing action Add NIC to VM: Failed to activate VM Network Interface.' Is not descriptive enough.

Comment 1 Michael Burman 2015-06-22 11:47:53 UTC
Created attachment 1041748 [details]

Comment 2 Michael Burman 2015-06-22 11:53:11 UTC
Created attachment 1041751 [details]

Comment 3 Dan Kenigsberg 2015-06-23 12:50:33 UTC
This should first be implemented in Vdsm. Attaching a VF during vm.create is based on attaching of generic host device, with some network decoration (vlan id). Similarly, attaching a VF after the VM is running (=hotplugging) should be based on hot-plugging of a host device, which requires a new Vdsm verb.

Comment 4 Red Hat Bugzilla Rules Engine 2015-10-19 11:00:14 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Dan Kenigsberg 2015-11-16 08:30:28 UTC
Could you try if hot unplug magically works with ovirt-3.6.1's vdsm? To do this, please start a VM with a VF as its nic, and then use

  vdsClient -S 0 hotunplugNic <vmid> macAddr:<macaddress>

(you may need to play a bit with this command line as I have not used it for years)

Please include vdsm.log's section where the hotunplug request is passed to libvirt.

Comment 6 Ido Barkan 2015-11-29 13:39:35 UTC
surprisingly (or not) calling hot(un)plug almost works.
The missing part is automatically attach the device back to the host once it is unplugged from the vm and the other way around.

We tried to do the following step by hand:
unplug steps:
calling hostdev.reattach_detachable on vdsm
calling hotunplugNic
the device was sucssessfully removed from the vma and reattached to host

plug steps:
calling hostdev.detach_detachable on vdsm
calling hotplugNic
the device was attached to the vm and detached from host, but detach_detachable returned the following libvirt error:
2015-11-29 13:27:40.638+0000: 1731: error : virPCIDeviceReattach:1377 : internal error: Not reattaching active device 0000:05:10.1

from vdsm.log:

Thread-19191::DEBUG::2015-11-29 15:27:40,634::bindingxmlrpc::1257::vds::(wrapper) client []::call vmHotplugNic with ({'nic': {'device': 'hostdev', 'type': 'interface', 'macAddr': '00:00:00:00:00:22', 'h
ostdev': 'pci_0000_05_10_1', 'specParams': {}}, 'vmId': 'ed5a35cc-4811-4000-8af8-51a8daae03b2'},) {}
Thread-19191::INFO::2015-11-29 15:27:40,637::vm::1979::virt.vm::(hotplugNic) vmId=`ed5a35cc-4811-4000-8af8-51a8daae03b2`::Hotplug NIC xml: <interface managed="no" type="hostdev">
        <mac address="00:00:00:00:00:22"/>
                <address bus="5" domain="0" function="1" slot="16" type="pci"/>
        <driver name="vfio"/>

Thread-19191::ERROR::2015-11-29 15:27:40,639::vm::1984::virt.vm::(hotplugNic) vmId=`ed5a35cc-4811-4000-8af8-51a8daae03b2`::Hotplug failed
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 1982, in hotplugNic
  File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 530, in attachDevice
    if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self)
libvirtError: internal error: Not reattaching active device 0000:05:10.1
Thread-19191::DEBUG::2015-11-29 15:27:40,640::bindingxmlrpc::1264::vds::(wrapper) return vmHotplugNic with {'status': {'message': 'internal error: Not reattaching active device 0000:05:10.1', 'code': 49}}

so it seems that the basic missing functionality is small and there is a small libvirt issue that needs to be resolved,

Comment 8 Dan Kenigsberg 2016-01-04 07:36:28 UTC
Relevant patches have been merged to the master branch of both Engine is Vdsm.

Comment 9 Michael Burman 2016-04-07 12:09:49 UTC
Tested on 4.0.0-0.0.master.20160404161620.git4ffd5a4.el7.centos with vdsm-4.17.999-892.git435139d.el7.centos.noarch and failed QA.

We have few problems here, not sure if it's engine or vdsm or both side.


1) Enable Vfs on host
2) Run VM with 'passthorugh' vNIC(1 VF is taken)
3) Hotunpug the vNIC --> VF not released and still considered as non-FREE
4) Hotplug back the vNIC --> Another VF is taken from the host
5) For every hotunplug and hotplug back, another VF will be taken from the host
6) Shutdown the VM won't release the VFs and all of them will be considered as non-free until we will reboot the server. 

7) Question: When unplugging a 'pci-passthrouhg' vNIC, can we update the profile with another 'passthough' profile? should it be blocked or supported? 

Moving back to assigned and attaching logs.

Comment 10 Red Hat Bugzilla Rules Engine 2016-04-07 12:09:57 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 11 Michael Burman 2016-04-07 12:17:22 UTC
Created attachment 1144693 [details]
failed qa logs

Comment 12 Sandro Bonazzola 2016-05-02 10:10:38 UTC
Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.

Comment 13 Dan Kenigsberg 2016-05-18 11:08:43 UTC
VF leakage was fixed by https://gerrit.ovirt.org/56013 please recheck.

regarding your Question 7: you may edit vnic profiles while VMs are running. The changes would take effect on the next time the profile is used on unplug or startup.

Comment 14 Michael Burman 2016-05-25 08:59:53 UTC
Verified on - 4.0.0-0.0.master.20160524171531.gitfba61e9.el7.centos and 

Comment 15 Sandro Bonazzola 2016-07-05 08:05:10 UTC
oVirt 4.0.0 has been released, closing current release.

Note You need to log in before you can comment on or make changes to this bug.