Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1615125

Summary: Undercloud upgrade from 10 to 11 fails because of failed nova db sync
Product: Red Hat OpenStack Reporter: Sasha Smolyak <ssmolyak>
Component: instack-undercloudAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 11.0 (Ocata)CC: aschultz, ccamacho, cschwede, lbezdick, mburns, owalsh, sathlang, ssmolyak
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-29 08:55:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1541051, 1541053, 1541054, 1541056    
Attachments:
Description Flags
undercloud-install.log none

Description Sasha Smolyak 2018-08-12 13:10:19 UTC 
Created attachment 1475356 [details]
undercloud-install.log

Description of problem:
While FFU tried to upgrade undercloud from 10 to 11. Upgrade failed, log attached

Version-Release number of selected component (if applicable):
puddle 2018-08-04.1

How reproducible:
100%

Steps to Reproduce:
1. Install the z9 puddle - undercloud
2. Added OpsTools, deployed overcloud
3. Launched several instances
4. Started upgrade according to this document: https://gitlab.cee.redhat.com/mcornea/osp10-13-ffu/blob/master/README.md#overcloud-fast-forward-upgrade-prepare-1

Actual results:
While upgrading undercloud from 10 to 11 the openstack undercloud upgrade command failed.

Expected results: The upgrade passes


Additional info: log attached
Comment 1 Lukas Bezdicka 2018-08-20 12:52:42 UTC 
2018-08-12 08:52:48,913 INFO: [mNotice: /Stage[main]/Nova::Deps/Anchor[nova::db::end]: Triggered 'refresh' from 2 events[0m
2018-08-12 08:52:48,914 INFO: [mNotice: /Stage[main]/Nova::Deps/Anchor[nova::dbsync_api::begin]: Triggered 'refresh' from 1 events[0m
2018-08-12 08:54:10,211 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: An error has occurred:[0m
2018-08-12 08:54:10,211 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: Traceback (most recent call last):[0m
2018-08-12 08:54:10,212 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1678, in main[0m
2018-08-12 08:54:10,212 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     ret = fn(*fn_args, **fn_kwargs)[0m
2018-08-12 08:54:10,212 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 820, in sync[0m
2018-08-12 08:54:10,212 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return migration.db_sync(version, database='api')[0m
2018-08-12 08:54:10,212 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync[0m
2018-08-12 08:54:10,213 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return IMPL.db_sync(version=version, database=database, context=context)[0m
2018-08-12 08:54:10,213 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync[0m
2018-08-12 08:54:10,213 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     repository, version)[0m
2018-08-12 08:54:10,213 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return _migrate(url, repository, version, upgrade=True, err=err, **opts)[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "<string>", line 2, in _migrate[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return f(*a, **kw)[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate[0m
2018-08-12 08:54:10,214 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     schema.runchange(ver, change, changeset.step)[0m
2018-08-12 08:54:10,215 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange[0m
2018-08-12 08:54:10,215 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     change.run(self.engine, step)[0m
2018-08-12 08:54:10,215 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run[0m
2018-08-12 08:54:10,215 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     script_func(engine)[0m
2018-08-12 08:54:10,215 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py", line 44, in upgrade[0m
2018-08-12 08:54:10,216 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     raise exception.ValidationError(detail=msg)[0m
2018-08-12 08:54:10,216 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing.[0m
2018-08-12 08:54:10,216 INFO: [1;31mError: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0][0m
2018-08-12 08:54:10,216 INFO: [1;31mError: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: change from notrun to 0 failed: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0][0m
2018-08-12 08:55:31,629 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: An error has occurred:[0m
2018-08-12 08:55:31,629 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: Traceback (most recent call last):[0m
2018-08-12 08:55:31,629 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1678, in main[0m
2018-08-12 08:55:31,630 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     ret = fn(*fn_args, **fn_kwargs)[0m
2018-08-12 08:55:31,630 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 820, in sync[0m
2018-08-12 08:55:31,630 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return migration.db_sync(version, database='api')[0m
2018-08-12 08:55:31,630 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/migration.py", line 26, in db_sync[0m
2018-08-12 08:55:31,631 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return IMPL.db_sync(version=version, database=database, context=context)[0m
2018-08-12 08:55:31,631 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/migration.py", line 57, in db_sync[0m
2018-08-12 08:55:31,631 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     repository, version)[0m
2018-08-12 08:55:31,631 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade[0m
2018-08-12 08:55:31,632 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return _migrate(url, repository, version, upgrade=True, err=err, **opts)[0m
2018-08-12 08:55:31,632 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "<string>", line 2, in _migrate[0m
2018-08-12 08:55:31,632 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine[0m
2018-08-12 08:55:31,632 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     return f(*a, **kw)[0m
2018-08-12 08:55:31,633 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate[0m
2018-08-12 08:55:31,633 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     schema.runchange(ver, change, changeset.step)[0m
2018-08-12 08:55:31,633 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange[0m
2018-08-12 08:55:31,633 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     change.run(self.engine, step)[0m
2018-08-12 08:55:31,634 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run[0m
2018-08-12 08:55:31,634 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     script_func(engine)[0m
2018-08-12 08:55:31,634 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py", line 44, in upgrade[0m
2018-08-12 08:55:31,634 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns:     raise exception.ValidationError(detail=msg)[0m
2018-08-12 08:55:31,634 INFO: [mNotice: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]/returns: ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing.[0m
2018-08-12 08:55:31,635 INFO: [1;31mError: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Failed to call refresh: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0][0m
2018-08-12 08:55:31,635 INFO: [1;31mError: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: /usr/bin/nova-manage  api_db sync returned 1 instead of one of [0][0m
Comment 2 Christian Schwede (cschwede) 2018-08-22 12:09:32 UTC 
I don't see any Swift error here, but an error with the Nova DB sync command - thus moving to the Compute DFG.
Comment 3 Ollie Walsh 2018-08-23 15:18:50 UTC 
Could you include the instack-undercloud version please? Possible dup of https://bugzilla.redhat.com/show_bug.cgi?id=1594265.
Comment 4 Sasha Smolyak 2018-08-29 08:14:45 UTC 
The upgrade was from OSPz9, which was then 2018-08-04.1, I can't find this puddle anymore, probably it was killed, can't say more than that