Bug 1413972

Cloning BZ 1411856 so we can assign DFG:Upgrades and track the changes we need in instack-undercloud/puppet-nova to run this during the upgrade


+++ This bug was initially created as a clone of Bug #1411856 +++

Description of problem:
-----------------------
Attempt to upgrade undercloud (rhos-10) upgraded from rhos-9 fails:

Notice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::end]: Triggered 'refresh' from 1 eventsESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: An error has occurred:ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: Traceback (most recent call last):ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1584, in mainESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     ret = fn(*fn_args, **fn_kwargs)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 783, in syncESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     return migration.db_sync(version)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_syncESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     return IMPL.db_sync(version=version, database=database, context=context)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_syncESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     repository, version)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgradeESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     return _migrate(url, repository, version, upgrade=True, err=err, **opts)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "<string>", line 2, in _migrateESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engineESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     return f(*a, **kw)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrateESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     schema.runchange(ver, change, changeset.step)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchangeESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     change.run(self.engine, step)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in runESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     script_func(engine)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migrate_repo/versions/345_require_online_migration_completion.py", line 44, in upgradeESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns:     raise exception.ValidationError(detail=msg)ESC[0m
Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: ValidationError: Migration cannot continue until all these have been migrated to the api database. Please run `nova-manage db online_migrations' on Newton code before continuing.There are still 8 unmigrated flavors. ESC[0m
Error: /usr/bin/nova-manage  db sync returned 1 instead of one of [0]ESC[0m
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: /usr/bin/nova-manage  db sync returned 1 instead of one of [0]

***
Attempt to run command `nova-manage db online_migrations` failed:

nova-manage db online_migrations                                                                                                                                             
usage: nova-manage db [-h]
                      
                      {archive_deleted_rows,null_instance_uuid_scan,online_data_migrations,sync,version}
                      ...
nova-manage db: error: argument action: invalid choice: 'online_migrations' (choose from 'archive_deleted_rows', 'null_instance_uuid_scan', 'online_data_migrations', 'sync', 'version')

***
Running next command helped to bypass error and upgrade undercloud:

nova-manage db online_data_migrations
Running batches of 50 until complete
8 rows matched query migrate_flavors, 8 migrated
8 rows matched query migrate_instance_keypairs, 8 migrated
8 rows matched query migrate_instances_add_request_spec, 0 migrated
1 rows matched query migrate_keypairs_to_api_db, 1 migrated
+---------------------------------------+--------------+-----------+
|               Migration               | Total Needed | Completed |
+---------------------------------------+--------------+-----------+
| aggregate_uuids_online_data_migration |      0       |     0     |
| migrate_aggregate_reset_autoincrement |      0       |     0     |
|           migrate_aggregates          |      0       |     0     |
|   migrate_flavor_reset_autoincrement  |      0       |     0     |
|            migrate_flavors            |      0       |     0     |
|   migrate_instance_groups_to_api_db   |      0       |     0     |
|       migrate_instance_keypairs       |      0       |     0     |
|   migrate_instances_add_request_spec  |      0       |     0     |
|       migrate_keypairs_to_api_db      |      0       |     0     |
+---------------------------------------+--------------+-----------+

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-nova-placement-api-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-console-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-scheduler-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
python-novaclient-6.0.0-1.el7ost.noarch
openstack-nova-common-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-novncproxy-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
python-nova-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-cells-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-cert-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
puppet-nova-10.1.0-0.20161216185556.4801dd0.el7ost.noarch
openstack-nova-conductor-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-network-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-api-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
python-nova-tests-15.0.0-0.20161220035916.5eb3144.el7ost.noarch
openstack-nova-compute-15.0.0-0.20161220035916.5eb3144.el7ost.noarch


openstack-tripleo-heat-templates-compat-5.1.1-0.20161219183418.6ef0417.el7ost.noarch
openstack-heat-templates-0.0.1-0.20161213042949.d10069a.el7ost.noarch
openstack-tripleo-heat-templates-6.0.0-0.20161220000655.58d711e.el7ost.noarch


Steps to Reproduce:
-------------------
1. Upgrade RHOS-9 setup(3controllers + 2computes + 3ceph) to RHOS-10
2*. During upgrade VM on overcloud was migrated between computes
3. Try to upgrade undercloud to RHOS-11

    openstack undercloud upgrade 

Actual results:
---------------
Undercloud upgrade fails

Expected results:
-----------------
Undercloud upgrade succeeds 


Additional info:
----------------
Virtual setup - 3controllers + 2computes + 3ceph

--- Additional comment from Sven Anderson on 2017-01-13 20:54:00 EET ---

undercloud upgrades are done with the same puppet files as the installs, so no special processing is applied. Therefore, after upgrading to OSP 10, no "nova-manage db online_data_migrations" is executed after the "nova-manage db sync", which is mandatory if an upgrade happened. This makes a manual "manage db online_data_migrations" necessary.

--- Additional comment from Marius Cornea on 2017-01-16 11:18:25 EET ---

(In reply to Sven Anderson from comment #1)
> undercloud upgrades are done with the same puppet files as the installs, so
> no special processing is applied. Therefore, after upgrading to OSP 10, no
> "nova-manage db online_data_migrations" is executed after the "nova-manage
> db sync", which is mandatory if an upgrade happened. This makes a manual
> "manage db online_data_migrations" necessary.

I think this should be executed automatically during the undercloud upgrade flow and the operator shouldn't be required to run the "nova-manage db" command manually.

--- Additional comment from marios on 2017-01-16 12:58:42 EET ---


hi o/ just poked at this a bit (someone must have added me to cc as it appeared in my inbox even though it isn't tagged DFG:Upgrades).

So to be clear, because we've had other nova related migration issues (cells API) that was specific to 10->11 at BZ 1409857, this bz is specific to environments starting at OSP9. I'm not clear on the details of why (i.e. if you start at 10 then you don't need to run "nova-manage db online_data_migrations"). I am also not clear if the 'migrate vms as part of the upgrade' in comment #0 is necessary to reproduce this.

All that being said, I think we can easily add this. The db syncs usually happen at https://github.com/openstack/instack-undercloud/blob/554977801aad85f3ac08e16762fcdc8550c95754/elements/puppet-stack-config/puppet-stack-config.pp#L26 but we want this to happen only on upgrade and then only on newton. In which case probably the easiest place to add it is just at https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/undercloud.py#L54 after the undercloud was upgraded to stable/newton. 

I filed the LP bug so we aren't pointing at BZ in the upstream repos at https://bugs.launchpad.net/tripleo/+bug/1656791 and it points here and then I put the onliner at https://review.openstack.org/#/c/420637/ to add the migration to the client for now. Lets see what folks think of that.

One last note... this BZ is assigned puppet-nova and is being used to fixup to nova for the s/online_migrations/online_data_migrations'  @ https://review.openstack.org/#/c/420060/. Should we be tracking the fixup during the undercloud upgrade in a different BZ (and it can also be tagged as DFG:Upgrades) - i.e. should we clone this bug for the fix I put out as per @ansiwen's comment #1 and mcornea comment #2


thanks, marios

--- Additional comment from marios on 2017-01-16 16:51:02 EET ---

after discussion on the Upgrade scrum just now I'll clone this BZ and in fact move the db-sync to the instack puppet-stack-config.pp I pointed to in comment #3. Going to remove the upgrades related changes from this BZ now.