Bug 1409857

Summary: openstack undercloud upgrade fails for nova-manage api_db sync because of new cell mappings dependency
Product: Red Hat OpenStack Reporter: Marios Andreou <mandreou>
Component: instack-undercloudAssignee: Marios Andreou <mandreou>
Status: CLOSED ERRATA QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: dbellant, dclarke, jcoufal, jschluet, mandreou, mburns, mcornea, morazi, rhel-osp-director-maint, sathlang
Target Milestone: betaKeywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-17 19:54:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marios Andreou 2017-01-03 16:11:00 UTC
Description of problem:

The upgrade of the OSP10 undercloud to OSP11 fails because of a nova-api dbsync related problem:

Version-Release number of selected component (if applicable):
OSP11

How reproducible:


Steps to Reproduce:
1. setup OSP11 repos (e.g. sudo rhos-release -P 11)
2. stop services
3. openstack undercloud upgrade

Actual results:

2017-01-03 07:34:55,735 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync
2017-01-03 07:34:55,735 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     repository, version)
2017-01-03 07:34:55,735 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
2017-01-03 07:34:55,736 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2017-01-03 07:34:55,736 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "<string>", line 2, in _migrate
2017-01-03 07:34:55,736 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2017-01-03 07:34:55,736 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return f(*a, **kw)
2017-01-03 07:34:55,737 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
2017-01-03 07:34:55,737 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     schema.runchange(ver, change, changeset.step)
2017-01-03 07:34:55,737 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
2017-01-03 07:34:55,737 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     change.run(self.engine, step)
2017-01-03 07:34:55,738 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
2017-01-03 07:34:55,738 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     script_func(engine)
2017-01-03 07:34:55,738 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py", line 41, in upgrade
2017-01-03 07:34:55,738 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     raise exception.ValidationError(detail=msg)
2017-01-03 07:34:55,739 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing.

Expected results:


Additional info:
Manually running the nova-manage cell_v2 simple_cell_setup as suggested isn't working in my environment. This bug is already reported upstream and there are reviews in place from Alex Schultz (adding the bug to external trackers). The reviews are actually against puppet-nova/tripleo/openstacklib as well as tripleo-heat-templates and instack-undercloud. I have set instack-undercloud as the component here for now and we will use this BZ to track the right things land where they are needed for the OSP11 undercloud upgrade

Comment 1 Marios Andreou 2017-01-03 16:12:59 UTC
tried manually running the simple_cell_setup (both with and without the --transport-url) but I still get the error running the api_db sync:

[stack@instack ~]$ sudo nova-manage cell_v2 simple_cell_setup --transport-url "rabbit://97ae6497ff5c3436313e8dd28bc1c008ce8f187d:f21294fb551cf49ab6aff3574cb84cda66486795.24.1//"
Already setup, nothing to do.
[stack@instack ~]$ sudo /usr/bin/nova-manage -v api_db sync
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1584, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 948, in sync
    return migration.db_sync(version, database='api')
  File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync
    return IMPL.db_sync(version=version, database=database, context=context)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync
    repository, version)
  File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
    change.run(self.engine, step)
  File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
    script_func(engine)
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py", line 41, in upgrade
    raise exception.ValidationError(detail=msg)
ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing.

Comment 3 Sofer Athlan-Guyot 2017-03-21 18:28:07 UTC
Hi,

putting this to POST.  Marios I think it's safe to move it to closed, but waiting for your definitive input :)

Thanks,

Comment 4 Marios Andreou 2017-03-22 11:25:27 UTC
yeah been running for a while without any workarounds downstream OSP10 to OSP11 latest puddle should go ON_QA ... setting.

Comment 5 errata-xmlrpc 2017-05-17 19:54:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245

Comment 6 Daniel Bellantuono 2018-07-26 08:12:20 UTC
This issue happen again upgrading from OSP10z6 to OSP11 (FFU)

The workaround is to run "nova-manage cell_v2 simple_cell_setup" manually and run the upgrade again

Packages version after the update:
openstack-tripleo-heat-templates-6.2.12-2.el7ost.noarch
python-nova-15.1.0-3.el7ost.noarch
openstack-nova-common-15.1.0-3.el7ost.noarch


2018-07-26 03:20:16,799 INFO: Notice: /Stage[main]/Nova::Deps/Anchor[nova::db::end]: Triggered 'refresh' from 2 events
2018-07-26 03:20:16,800 INFO: Notice: /Stage[main]/Nova::Deps/Anchor[nova::dbsync_api::begin]: Triggered 'refresh' from 1 events
2018-07-26 03:21:41,503 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: An error has occurred:
2018-07-26 03:21:41,503 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: Traceback (most recent call last):
2018-07-26 03:21:41,504 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1678, in main
2018-07-26 03:21:41,504 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     ret = fn(*fn_args, **fn_kwargs)
2018-07-26 03:21:41,504 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 820, in sync
2018-07-26 03:21:41,504 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return migration.db_sync(version, database='api')
2018-07-26 03:21:41,505 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync
2018-07-26 03:21:41,505 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return IMPL.db_sync(version=version, database=database, context=context)
2018-07-26 03:21:41,505 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync
2018-07-26 03:21:41,505 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     repository, version)
2018-07-26 03:21:41,506 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
2018-07-26 03:21:41,506 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2018-07-26 03:21:41,506 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "<string>", line 2, in _migrate
2018-07-26 03:21:41,506 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2018-07-26 03:21:41,506 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return f(*a, **kw)
2018-07-26 03:21:41,507 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
2018-07-26 03:21:41,507 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     schema.runchange(ver, change, changeset.step)
2018-07-26 03:21:41,507 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
2018-07-26 03:21:41,507 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     change.run(self.engine, step)
2018-07-26 03:21:41,508 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
2018-07-26 03:21:41,508 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     script_func(engine)
2018-07-26 03:21:41,508 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py", line 44, in upgrade
2018-07-26 03:21:41,508 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     raise exception.ValidationError(detail=msg)
2018-07-26 03:21:41,509 INFO: Notice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing.
2018-07-26 03:21:41,509 INFO: Error: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0]
2018-07-26 03:21:41,509 INFO: Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: change from notrun to 0 failed: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0]

2018-07-26 03:23:06,041 INFO: Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Failed to call refresh: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0]
2018-07-26 03:23:06,041 INFO: Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0]
2018-07-26 03:23:06,041 INFO: Notice: /Stage[main]/Nova::Deps/Anchor[nova::dbsync_api::end]: Dependency Exec[nova-db-sync-api] has failures: true
2018-07-26 03:23:06,042 INFO: Warning: /Stage[main]/Nova::Deps/Anchor[nova::dbsync_api::end]: Skipping because of failed dependencies