Bug 1526082

Summary:	Unable to resize nova instance after upgrade to OSP 10
Product:	Red Hat OpenStack	Reporter:	nalmond
Component:	openstack-nova	Assignee:	Artom Lifshitz <alifshit>
Status:	CLOSED ERRATA	QA Contact:	Gabriel Szasz <gszasz>
Severity:	high	Docs Contact:
Priority:	high
Version:	10.0 (Newton)	CC:	alifshit, berrange, dasmith, eglynn, gszasz, kchamart, lyarwood, mlopes, owalsh, sbauza, sferdjao, sgordon, slinaber, srevivo, vromanso
Target Milestone:	z7	Keywords:	Triaged, ZStream
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-14.0.9-4.el7ost	Doc Type:	Bug Fix
Doc Text:	Previously, the `request_spec.spec` column's TEXT type was too small, with the result that large request_specs may not have fit successfully. Consequently, running the online data migration as part of an OSP9 to OSP10 upgrade could have resulted in failure if an instance created before the upgrade had a large request_spec (for example, by having many members in its affinity/anti-affinity group). With this update, the request_spec.spec column has been increased to `MEDIUMTEXT`. As a result, online data migration should no longer fail during a OSP9 to OSP10 upgrade.	Story Points:	---
Clone Of:
Clones:	1527643 1527644 (view as bug list)		Environment:
Last Closed:	2018-02-27 16:24:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1527643, 1527644

Description nalmond 2017-12-14 18:08:07 UTC

Description of problem:
Following an upgrade to OSP 10, instances that were created before the upgrade produce an error when attempting to resize.

Resize() command failed for VM with UUID: aa8d6f33-4b62-4ea1-aa15-21266eaaf50d. Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'RequestSpecNotFound_Remote'> (HTTP 500) (Request-ID: req-c1af65e1-2072-4168-b0f4-71e0c36e6ee5)

This is due to 'nova-manage db online_data_migrations' not being run as part of the upgrade per https://bugs.launchpad.net/nova/+bug/1707934

After running this manually, an error is produced and the command does not complete:
# nova-manage db online_data_migrations
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".
Running batches of 50 until complete
Error attempting to run <function migrate_instances_add_request_spec at 0x5c13758>

After adding a raise to the exception block in /nova/cmd/manage.py a mysql error was observed:

error: (pymysql.err.DataError) (1406, u"Data too long for column 'spec' at row 1") [SQL: u'INSERT INTO request_specs

This is due to the RequestSpec.instance_group.members being massive. After manually skipping the affected instances, the nova-manage command was able to complete and the resize operation began working as expected.

Version-Release number of selected component (if applicable):
openstack-nova-common-14.0.8-5.el7ost.noarch

How reproducible:
Every time with this deployment

Steps to Reproduce:
1. Upgrade from OSP 8 -> 9 -> 10
2. Attempt to migrate an instance that was created before the upgrade

Actual results:
An error is generated, but the resize is actually successful and can be confirmed. Error is not generated after implementing workaround to allow 'nova-manage db online_data_migrations' to complete.

Expected results:
No error is produced

Additional info:
Upstream bug has been opened to address the mysql error https://bugs.launchpad.net/nova/+bug/1738094

Comment 12 errata-xmlrpc 2018-02-27 16:24:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0369