Cloning BZ 1411856 so we can assign DFG:Upgrades and track the changes we need in instack-undercloud/puppet-nova to run this during the upgrade +++ This bug was initially created as a clone of Bug #1411856 +++ Description of problem: ----------------------- Attempt to upgrade undercloud (rhos-10) upgraded from rhos-9 fails: Notice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::end]: Triggered 'refresh' from 1 eventsESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: An error has occurred:ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: Traceback (most recent call last):ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1584, in mainESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: ret = fn(*fn_args, **fn_kwargs)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 783, in syncESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: return migration.db_sync(version)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_syncESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: return IMPL.db_sync(version=version, database=database, context=context)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_syncESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: repository, version)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgradeESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: return _migrate(url, repository, version, upgrade=True, err=err, **opts)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "<string>", line 2, in _migrateESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engineESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: return f(*a, **kw)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrateESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: schema.runchange(ver, change, changeset.step)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchangeESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change.run(self.engine, step)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in runESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: script_func(engine)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migrate_repo/versions/345_require_online_migration_completion.py", line 44, in upgradeESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: raise exception.ValidationError(detail=msg)ESC[0m Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: ValidationError: Migration cannot continue until all these have been migrated to the api database. Please run `nova-manage db online_migrations' on Newton code before continuing.There are still 8 unmigrated flavors. ESC[0m Error: /usr/bin/nova-manage db sync returned 1 instead of one of [0]ESC[0m Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: /usr/bin/nova-manage db sync returned 1 instead of one of [0] *** Attempt to run command `nova-manage db online_migrations` failed: nova-manage db online_migrations usage: nova-manage db [-h] {archive_deleted_rows,null_instance_uuid_scan,online_data_migrations,sync,version} ... nova-manage db: error: argument action: invalid choice: 'online_migrations' (choose from 'archive_deleted_rows', 'null_instance_uuid_scan', 'online_data_migrations', 'sync', 'version') *** Running next command helped to bypass error and upgrade undercloud: nova-manage db online_data_migrations Running batches of 50 until complete 8 rows matched query migrate_flavors, 8 migrated 8 rows matched query migrate_instance_keypairs, 8 migrated 8 rows matched query migrate_instances_add_request_spec, 0 migrated 1 rows matched query migrate_keypairs_to_api_db, 1 migrated +---------------------------------------+--------------+-----------+ | Migration | Total Needed | Completed | +---------------------------------------+--------------+-----------+ | aggregate_uuids_online_data_migration | 0 | 0 | | migrate_aggregate_reset_autoincrement | 0 | 0 | | migrate_aggregates | 0 | 0 | | migrate_flavor_reset_autoincrement | 0 | 0 | | migrate_flavors | 0 | 0 | | migrate_instance_groups_to_api_db | 0 | 0 | | migrate_instance_keypairs | 0 | 0 | | migrate_instances_add_request_spec | 0 | 0 | | migrate_keypairs_to_api_db | 0 | 0 | +---------------------------------------+--------------+-----------+ Version-Release number of selected component (if applicable): ------------------------------------------------------------- openstack-nova-placement-api-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-console-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-scheduler-15.0.0-0.20161220035916.5eb3144.el7ost.noarch python-novaclient-6.0.0-1.el7ost.noarch openstack-nova-common-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-novncproxy-15.0.0-0.20161220035916.5eb3144.el7ost.noarch python-nova-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-cells-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-cert-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-15.0.0-0.20161220035916.5eb3144.el7ost.noarch puppet-nova-10.1.0-0.20161216185556.4801dd0.el7ost.noarch openstack-nova-conductor-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-network-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-api-15.0.0-0.20161220035916.5eb3144.el7ost.noarch python-nova-tests-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-nova-compute-15.0.0-0.20161220035916.5eb3144.el7ost.noarch openstack-tripleo-heat-templates-compat-5.1.1-0.20161219183418.6ef0417.el7ost.noarch openstack-heat-templates-0.0.1-0.20161213042949.d10069a.el7ost.noarch openstack-tripleo-heat-templates-6.0.0-0.20161220000655.58d711e.el7ost.noarch Steps to Reproduce: ------------------- 1. Upgrade RHOS-9 setup(3controllers + 2computes + 3ceph) to RHOS-10 2*. During upgrade VM on overcloud was migrated between computes 3. Try to upgrade undercloud to RHOS-11 openstack undercloud upgrade Actual results: --------------- Undercloud upgrade fails Expected results: ----------------- Undercloud upgrade succeeds Additional info: ---------------- Virtual setup - 3controllers + 2computes + 3ceph --- Additional comment from Sven Anderson on 2017-01-13 20:54:00 EET --- undercloud upgrades are done with the same puppet files as the installs, so no special processing is applied. Therefore, after upgrading to OSP 10, no "nova-manage db online_data_migrations" is executed after the "nova-manage db sync", which is mandatory if an upgrade happened. This makes a manual "manage db online_data_migrations" necessary. --- Additional comment from Marius Cornea on 2017-01-16 11:18:25 EET --- (In reply to Sven Anderson from comment #1) > undercloud upgrades are done with the same puppet files as the installs, so > no special processing is applied. Therefore, after upgrading to OSP 10, no > "nova-manage db online_data_migrations" is executed after the "nova-manage > db sync", which is mandatory if an upgrade happened. This makes a manual > "manage db online_data_migrations" necessary. I think this should be executed automatically during the undercloud upgrade flow and the operator shouldn't be required to run the "nova-manage db" command manually. --- Additional comment from marios on 2017-01-16 12:58:42 EET --- hi o/ just poked at this a bit (someone must have added me to cc as it appeared in my inbox even though it isn't tagged DFG:Upgrades). So to be clear, because we've had other nova related migration issues (cells API) that was specific to 10->11 at BZ 1409857, this bz is specific to environments starting at OSP9. I'm not clear on the details of why (i.e. if you start at 10 then you don't need to run "nova-manage db online_data_migrations"). I am also not clear if the 'migrate vms as part of the upgrade' in comment #0 is necessary to reproduce this. All that being said, I think we can easily add this. The db syncs usually happen at https://github.com/openstack/instack-undercloud/blob/554977801aad85f3ac08e16762fcdc8550c95754/elements/puppet-stack-config/puppet-stack-config.pp#L26 but we want this to happen only on upgrade and then only on newton. In which case probably the easiest place to add it is just at https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/undercloud.py#L54 after the undercloud was upgraded to stable/newton. I filed the LP bug so we aren't pointing at BZ in the upstream repos at https://bugs.launchpad.net/tripleo/+bug/1656791 and it points here and then I put the onliner at https://review.openstack.org/#/c/420637/ to add the migration to the client for now. Lets see what folks think of that. One last note... this BZ is assigned puppet-nova and is being used to fixup to nova for the s/online_migrations/online_data_migrations' @ https://review.openstack.org/#/c/420060/. Should we be tracking the fixup during the undercloud upgrade in a different BZ (and it can also be tagged as DFG:Upgrades) - i.e. should we clone this bug for the fix I put out as per @ansiwen's comment #1 and mcornea comment #2 thanks, marios --- Additional comment from marios on 2017-01-16 16:51:02 EET --- after discussion on the Upgrade scrum just now I'll clone this BZ and in fact move the db-sync to the instack puppet-stack-config.pp I pointed to in comment #3. Going to remove the upgrades related changes from this BZ now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245