Bug 1203160

Summary:	Add compatibility code to PCIRequest NovaObject and a migration script, so that upgrades will work for RHOS6->RHOS7
Product:	Red Hat OpenStack	Reporter:	Nikola Dipanov <ndipanov>
Component:	openstack-nova	Assignee:	Nikola Dipanov <ndipanov>
Status:	CLOSED ERRATA	QA Contact:	Prasanth Anbalagan <panbalag>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	7.0 (Kilo)	CC:	berrange, dasmith, eglynn, jschluet, kchamart, ltoscano, mbooth, mlopes, ndipanov, nlevinki, ohochman, pbrady, sbauza, sclewis, sferdjao, sgordon, stoner, tvvcox, vromanso, yeylon
Target Milestone:	ga	Keywords:	Reopened
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-2015.1.0-14.el7ost	Doc Type:	Release Note
Doc Text:	After fully upgrading to Red Hat Enterprise Linux OpenStack Platform 7 from version 6 (and all nodes are running version 7 code), you should start a background migration of PCI device NUMA node information from the old location to the new location. Version 7 conductor nodes will do this automatically when necessary, but the rest of the idle data needs to be migrated in the background. This is critical to complete before the version 8 release, where support for the old location will be dropped. Use '$ nova-manage db migrate_rhos_6_pci_device_data --max-number X' to perform this transition (where X is the maximum number of devices to be migrated in one run). Note that this is relevant only for users making use of the PCI pass-through features of Compute.	Story Points:	---
Clone Of:
Clones:	1250248 (view as bug list)		Environment:
Last Closed:	2015-08-05 13:22:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1104926
Bug Blocks:	799011, 1038706, 1077198, 1250248

Description Nikola Dipanov 2015-03-18 09:51:01 UTC

As part of resolving bz#1104926 the backported patches were modified so as to avoid having to do an additional database schema migration (not viable in a .z release of RHOS)

Brief overview of RHOS 7 and RHOS 8 changes that the no-db version of this feature will require is basically:

RHOS 7.0 will need to have code in the PciDevice object that can handle:

1) forward-porting the old version objects, so that RHOS 7 code can access it as a db field like it expects (since in a cloud where we have both RHOS 6 and RHOS 7 compute nodes running, the older compute nodes will always save the nume_node data in the extra_info blob. This means that there is no real way to do a pro-active migration of data, but there is no real need either. Once all of the compute nodes are upgraded to RHOS 7, all the data will be saved in the new format. NovaObjects expose a method that is meant for such forward-porting (_obj_from_primitive) that we can override to do what we need.

Additionally we will probably need to provide for compatibility for direct DB reads as well. This is again pretty straightforward with NovaObjects.

All of the places that touch the database use objects (in RHOS6 code) which makes the above sufficient to provide full compatibility with both versions of data.

2) Backporting of RHOS 7 objects when being sent over RPC to RHOS 6 compute nodes. This is pretty straightforward and NovaObjects also have a standard hook for this (obj_make_compatible)

3) In addition, we may want to provide a data migration script for customers who may be interested in doing a one shot updgrade

After we do the above in RHOS 7 - we should be able to drop all of the custom code in RHOS 8.

Further modifications that need backporting to RHOS 6:
Should we encounter further modifications, we can leverage the .z versioning of objects to account for it which should cover all but the most contrived possibilities. Each such backport has a potential to complicate the work that needs to happen in RHOS 7.

Comment 3 Nikola Dipanov 2015-06-25 13:55:34 UTC

The 2 proposed changes elaborate a bit more on the discussion in the Description of the bug.

Once a user upgrades their RHOS 6 environment to RHOS 7 there is nothing they need to worry about immediately as the above code will make sure that the data is being migrated to the new format as the cloud is running.

Under normal operating circumstances, restarting the nova-compute service (done anyway when doing an upgrade of a compute node to the new release) should be enough to migrate all relevant data. However there are scenarios where this would not happen, so we are in addition providing a script that administrators can use to do upgrades in small batches (maybe in a cron job) so that they are not disruptive to the operation of the cloud.

We provide a new nova-manage command that can take a number of records to migrate at a time. 

(actual invocation tbd once the review process is done and the patch is merged)

As the compatibility code will be dropped in future versions of the product - it will be strongly advised that all users running RHOS 6 Nova and using PCI pass-trough support, to complete the data migration of PCI device data.

Comment 4 Nikola Dipanov 2015-06-26 13:56:06 UTC

To fully test the following feature it is necessary to upgrade an ideally multi-node RHOS 6 deployment that makes use of PCI pass-through functionality to RHOS 7, and observe that the information about the PCI device in the pci_device DB table is being moved from the extra_info column to numa_node column after service restart.

Another useful use case to test could be to disable (through the use of whitelists) some PCI devices that were enabled at one point, and do the upgrade, and then run the nova-manage command described in the doc-text and observe that it migrates all the outstanding records (for the disabled devices).

Comment 5 RHEL Program Management 2015-06-26 15:46:14 UTC

Quality Engineering Management has reviewed and declined this request.
You may appeal this decision by reopening this request.

Comment 7 Prasanth Anbalagan 2015-07-24 20:25:56 UTC

The following bug blocks the verification of this bug,

https://bugzilla.redhat.com/show_bug.cgi?id=1246665

Comment 8 Nikola Dipanov 2015-07-27 17:55:37 UTC

*** Bug 1228267 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2015-08-05 13:22:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1548