Bug 1360970 - [RFE][TestOnly] SR-IOV live migration
Summary: [RFE][TestOnly] SR-IOV live migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 7.0 (Kilo)
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: beta
: 16.0 (Train on RHEL 8.1)
Assignee: smooney
QA Contact: James Parker
URL:
Whiteboard:
: 1631723 (view as bug list)
Depends On: 1631723
Blocks: 1732913 1756916 1780417
TreeView+ depends on / blocked
 
Reported: 2016-07-28 05:13 UTC by VIKRANT
Modified: 2022-10-26 00:42 UTC (History)
21 users (show)

Fixed In Version: openstack-nova-19.1.0-0.20190607104234.d7bad34.el8ost
Doc Type: Enhancement
Doc Text:
With this enhancement, support for live migration of instances with attached SR-IOV-based neutron interfaces has been added. Neutron SR-IOV interfaces can be grouped into two categories: direct mode and indirect mode. Direct mode SR-IOV interfaces are directly attached to the guest and exposed to the guest OS. Indirect mode SR-IOV interfaces have a software interface, for example, a macvtap, between the guest and the SR-IOV device. This feature enables transparent live migration for instances with indirect mode SR-IOV devices. As there is no generic way to copy hardware state during a live migration, direct mode migration is not transparent to the guest. For direct mode interfaces, mimic the workflow already in place for suspend and resume. For example, with SR-IOV devices, detach the direct mode interfaces before migration and re-attach them after the migration. As a result, instances with direct mode SR-IOV ports lose network connectivity during a migration unless a bond with a live migratable interface is created within the guest. Previously, it was not possible to live migrate instances with SR-IOV-based network interfaces. This was problematic as live migration is frequently used for host maintenance and similar actions. Previously, the instance had to be cold migrated which involves downtime for the guest. This enhancement results in the live migration of instances with SR-IOV-based network interfaces.
Clone Of:
: 1631723 1780417 (view as bug list)
Environment:
Last Closed: 2020-02-06 14:37:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 605116 0 'None' MERGED Add spec for sriov live migration 2020-12-15 06:57:58 UTC
Red Hat Issue Tracker OSP-7724 0 None None None 2022-08-10 09:59:27 UTC
Red Hat Knowledge Base (Solution) 2474871 0 None None None 2016-07-28 05:14:32 UTC
Red Hat Product Errata RHEA-2020:0283 0 None None None 2020-02-06 14:39:51 UTC

Description VIKRANT 2016-07-28 05:13:15 UTC
Description of problem:

Openstack envionment , config sriov then configure computer HA and then create a sriov vm  which is contained a direct type vnic  ,  shutdown the host  node of the vm  , vm rebuild on another computer node , but it got failed , because the vf with same pci number has been occupied on destination compute node. 

Version-Release number of selected component (if applicable):
RHEL OSP 7

How reproducible:
Everytime for Customer. 

Steps to Reproduce:
1. Spawn an instance using VF with pci_slot id similar to the one which is used by the instance running on destination compute node. 
2. Shutdown the compute node, as instance HA is configured hence it will try to rebuild the instance on destination compute node using same pci_slot id which is already used by the instance running on that node. 
3. Instance will end up in error state with below message.
~~~
| fault                                | {"message": "Requested operation is not valid: PCI device 0000:04:07.3 is in use by driver QEMU
, domain instance-0000019d", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 357, in 
decorated_function |
~~~

Actual results:
It's not getting spawned on destination compute node if VF with same pci_slot id is used on destination compute node. 

Expected results:
It should get spawned successfully. nova fitting logic should use another VF if the vf which instance tries to choose is already used. 

Additional info:

It look like a same issue which we have seen with cpu pinned instance mentioned in BZ https://bugzilla.redhat.com/show_bug.cgi?id=1319385

Comment 12 Stephen Finucane 2018-02-07 16:58:16 UTC

*** This bug has been marked as a duplicate of bug 1222414 ***

Comment 13 Sahid Ferdjaoui 2018-02-08 10:24:32 UTC
This issue is related to live-migrate instances with SRIOV and should not be considered as the same as live-migrating with CPU pinning. Both can work independently and even is the fix handle both situations (which i have some doubts) QA will have to test them independently.

Comment 14 Stephen Finucane 2018-02-08 13:52:29 UTC
(In reply to Sahid Ferdjaoui from comment #13)
> This issue is related to live-migrate instances with SRIOV and should not be
> considered as the same as live-migrating with CPU pinning. Both can work
> independently and even is the fix handle both situations (which i have some
> doubts) QA will have to test them independently.

OK, thanks for the context. I've updated the title accordingly.

Comment 15 Stephen Gordon 2018-03-07 16:00:31 UTC
Requirements:

- Support live migration with passthrough of full PCI NIC.
- Support live migration with passthrough of PF.
- Support live migration with passthrough of VF.
- In all cases, performance of networking in general VM lifecycle should not be impacted. Performance degradation during live migration is acceptable.

Comment 16 Daniel Berrangé 2018-03-07 16:07:13 UTC
(In reply to Stephen Gordon from comment #15)
> Requirements:
> 
> - Support live migration with passthrough of full PCI NIC.
> - Support live migration with passthrough of PF.
> - Support live migration with passthrough of VF.
> - In all cases, performance of networking in general VM lifecycle should not
> be impacted. Performance degradation during live migration is acceptable.

Achieving this from a technical POV, would require a multi-nic setup in the guest with bonding/teaming. ie every guest would need to have 2 NICs, one SRIOV based and one emulated, both connected to same host network. At migration the SRIOV device would have to be hot-removed, and a new one added afterwards. 

IOW, as well as impacting guest network performance, you need to mandate special guest setup, and guest cooperation for hot-unplug at start of migration. The implication is if the guest OS is crashed, is in early boot up phase, or otherwise non-responsive, live migration still won't be possible as it won't be responding to the initial hot-unplug request. Not a showstopper though - largely a documentation / expectation setting problem.

Comment 20 Lee Yarwood 2019-01-09 16:57:42 UTC
*** Bug 1631723 has been marked as a duplicate of this bug. ***

Comment 22 smooney 2019-03-21 17:01:21 UTC
Note this feature is being targeted for OSP 16 and will not be backportable.

Comment 23 smooney 2019-05-20 16:55:32 UTC
all functional code releated to this RFE has merged upstream in master.
there is 1 minor follow up patch still pending to adress some code style nits https://review.opendev.org/#/c/659101/
and a docs only patch that needs to be written but this RFE is now feature commplete upstream
and testing can start.

Comment 30 errata-xmlrpc 2020-02-06 14:37:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283


Note You need to log in before you can comment on or make changes to this bug.