Bug 1526082 - Unable to resize nova instance after upgrade to OSP 10
Summary: Unable to resize nova instance after upgrade to OSP 10
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z7
: 10.0 (Newton)
Assignee: Artom Lifshitz
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On:
Blocks: 1527643 1527644
TreeView+ depends on / blocked
 
Reported: 2017-12-14 18:08 UTC by nalmond
Modified: 2019-09-09 15:12 UTC (History)
15 users (show)

Fixed In Version: openstack-nova-14.0.9-4.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the `request_spec.spec` column's TEXT type was too small, with the result that large request_specs may not have fit successfully. Consequently, running the online data migration as part of an OSP9 to OSP10 upgrade could have resulted in failure if an instance created before the upgrade had a large request_spec (for example, by having many members in its affinity/anti-affinity group). With this update, the request_spec.spec column has been increased to `MEDIUMTEXT`. As a result, online data migration should no longer fail during a OSP9 to OSP10 upgrade.
Clone Of:
: 1527643 1527644 (view as bug list)
Environment:
Last Closed: 2018-02-27 16:24:48 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:0369 normal SHIPPED_LIVE Moderate: openstack-nova and python-novaclient security, bug fix, and enhancement update 2018-02-27 21:24:56 UTC
Launchpad 1738094 None None None 2017-12-15 10:07:48 UTC

Description nalmond 2017-12-14 18:08:07 UTC
Description of problem:
Following an upgrade to OSP 10, instances that were created before the upgrade produce an error when attempting to resize.

Resize() command failed for VM with UUID: aa8d6f33-4b62-4ea1-aa15-21266eaaf50d. Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'RequestSpecNotFound_Remote'> (HTTP 500) (Request-ID: req-c1af65e1-2072-4168-b0f4-71e0c36e6ee5)

This is due to 'nova-manage db online_data_migrations' not being run as part of the upgrade per https://bugs.launchpad.net/nova/+bug/1707934

After running this manually, an error is produced and the command does not complete:
# nova-manage db online_data_migrations
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".
Running batches of 50 until complete
Error attempting to run <function migrate_instances_add_request_spec at 0x5c13758>

After adding a raise to the exception block in /nova/cmd/manage.py a mysql error was observed:

error: (pymysql.err.DataError) (1406, u"Data too long for column 'spec' at row 1") [SQL: u'INSERT INTO request_specs

This is due to the RequestSpec.instance_group.members being massive. After manually skipping the affected instances, the nova-manage command was able to complete and the resize operation began working as expected.

Version-Release number of selected component (if applicable):
openstack-nova-common-14.0.8-5.el7ost.noarch

How reproducible:
Every time with this deployment

Steps to Reproduce:
1. Upgrade from OSP 8 -> 9 -> 10
2. Attempt to migrate an instance that was created before the upgrade

Actual results:
An error is generated, but the resize is actually successful and can be confirmed. Error is not generated after implementing workaround to allow 'nova-manage db online_data_migrations' to complete.

Expected results:
No error is produced

Additional info:
Upstream bug has been opened to address the mysql error https://bugs.launchpad.net/nova/+bug/1738094

Comment 12 errata-xmlrpc 2018-02-27 16:24:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0369


Note You need to log in before you can comment on or make changes to this bug.