Red Hat Bugzilla – Bug 1234328
[RFE] SR-IOV --> add support for Hotplug/unplug of VFs
Last modified: 2017-02-21 02:41:24 EST
Description of problem: SR-IOV --> Hotplug/unplug VFs profiles interfaces is not supported in 3.6 like it should be. Hotplug/unplug VFs should be supported in 3.6 as part of the SRIOV feature, but it currently not supported and such operations failing with this error messages when trying to hotplug/unplug VM's vNIC with pci-passthrough type--> - Failed to plug Network Interface nic2 (PCI Passthrough) to VM SR_VM4_. (User: admin@internal) - Failed to add Interface nic2 (PCI Passthrough) to VM SR_VM4_. (User: admin@internal) - Failed to unplug Network Interface nic1 (PCI Passthrough) from VM SR_VM4_. (User: admin@internal) - Failed to update Interface nic1 (PCI Passthrough) for VM SR_VM4_. (User: admin@internal) - If this support will not be ready for 3.6, then, engine and vdsm(or only engine) should continue blocking from Hotplug/unplug vNICs with pci-passthrough type profiles, but error message/s on engine side need some improve, explaining why i can't Hotplug/unplug vNIC to running VM with 'pci-passthrough' type. ' Error while executing action Add NIC to VM: Failed to activate VM Network Interface.' Is not descriptive enough.
Created attachment 1041748 [details] screenshot
Created attachment 1041751 [details] screenshot_engine_log
This should first be implemented in Vdsm. Attaching a VF during vm.create is based on attaching of generic host device, with some network decoration (vlan id). Similarly, attaching a VF after the VM is running (=hotplugging) should be based on hot-plugging of a host device, which requires a new Vdsm verb.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Could you try if hot unplug magically works with ovirt-3.6.1's vdsm? To do this, please start a VM with a VF as its nic, and then use vdsClient -S 0 hotunplugNic <vmid> macAddr:<macaddress> (you may need to play a bit with this command line as I have not used it for years) Please include vdsm.log's section where the hotunplug request is passed to libvirt.
surprisingly (or not) calling hot(un)plug almost works. The missing part is automatically attach the device back to the host once it is unplugged from the vm and the other way around. We tried to do the following step by hand: unplug steps: calling hostdev.reattach_detachable on vdsm calling hotunplugNic result: the device was sucssessfully removed from the vma and reattached to host plug steps: calling hostdev.detach_detachable on vdsm calling hotplugNic result: the device was attached to the vm and detached from host, but detach_detachable returned the following libvirt error: 2015-11-29 13:27:40.638+0000: 1731: error : virPCIDeviceReattach:1377 : internal error: Not reattaching active device 0000:05:10.1 from vdsm.log: Thread-19191::DEBUG::2015-11-29 15:27:40,634::bindingxmlrpc::1257::vds::(wrapper) client [127.0.0.1]::call vmHotplugNic with ({'nic': {'device': 'hostdev', 'type': 'interface', 'macAddr': '00:00:00:00:00:22', 'h ostdev': 'pci_0000_05_10_1', 'specParams': {}}, 'vmId': 'ed5a35cc-4811-4000-8af8-51a8daae03b2'},) {} Thread-19191::INFO::2015-11-29 15:27:40,637::vm::1979::virt.vm::(hotplugNic) vmId=`ed5a35cc-4811-4000-8af8-51a8daae03b2`::Hotplug NIC xml: <interface managed="no" type="hostdev"> <mac address="00:00:00:00:00:22"/> <source> <address bus="5" domain="0" function="1" slot="16" type="pci"/> </source> <driver name="vfio"/> </interface> Thread-19191::ERROR::2015-11-29 15:27:40,639::vm::1984::virt.vm::(hotplugNic) vmId=`ed5a35cc-4811-4000-8af8-51a8daae03b2`::Hotplug failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 1982, in hotplugNic self._dom.attachDevice(nicXml) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 530, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirtError: internal error: Not reattaching active device 0000:05:10.1 Thread-19191::DEBUG::2015-11-29 15:27:40,640::bindingxmlrpc::1264::vds::(wrapper) return vmHotplugNic with {'status': {'message': 'internal error: Not reattaching active device 0000:05:10.1', 'code': 49}} so it seems that the basic missing functionality is small and there is a small libvirt issue that needs to be resolved,
Relevant patches have been merged to the master branch of both Engine is Vdsm.
Tested on 4.0.0-0.0.master.20160404161620.git4ffd5a4.el7.centos with vdsm-4.17.999-892.git435139d.el7.centos.noarch and failed QA. We have few problems here, not sure if it's engine or vdsm or both side. Issues: 1) Enable Vfs on host 2) Run VM with 'passthorugh' vNIC(1 VF is taken) 3) Hotunpug the vNIC --> VF not released and still considered as non-FREE 4) Hotplug back the vNIC --> Another VF is taken from the host 5) For every hotunplug and hotplug back, another VF will be taken from the host 6) Shutdown the VM won't release the VFs and all of them will be considered as non-free until we will reboot the server. 7) Question: When unplugging a 'pci-passthrouhg' vNIC, can we update the profile with another 'passthough' profile? should it be blocked or supported? Moving back to assigned and attaching logs.
Created attachment 1144693 [details] failed qa logs
Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.
VF leakage was fixed by https://gerrit.ovirt.org/56013 please recheck. regarding your Question 7: you may edit vnic profiles while VMs are running. The changes would take effect on the next time the profile is used on unplug or startup.
Verified on - 4.0.0-0.0.master.20160524171531.gitfba61e9.el7.centos and vdsm-4.18.0-43.git6f7fc83.el7.centos.x86_64
oVirt 4.0.0 has been released, closing current release.