Bug 1536010 - OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.
Summary: OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: beta
: 13.0 (Queens)
Assignee: Emilien Macchi
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-18 12:52 UTC by Marius Cornea
Modified: 2018-06-27 13:43 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.0-0.20180304031149.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:42:33 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1744056 None None None 2018-01-18 12:52:57 UTC
OpenStack gerrit 521545 'None' MERGED ffu: Introduce Cinder fast-forward upgrade tasks 2020-08-30 04:01:42 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:43:13 UTC

Description Marius Cornea 2018-01-18 12:52:57 UTC
Description of problem:

OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."

TASK [Cinder db sync] *************************************************************************************************************************************************************************************************************************
fatal: [192.168.24.9]: FAILED! => {"changed": true, "cmd": ["cinder-manage", "db", "sync"], "delta": "0:00:02.538621", "end": "2018-01-18 12:26:04.920501", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-01-18 12:26:
02.381880", "stderr": "Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"DEFAULT\".", "stderr_lines": ["Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"D
EFAULT\"."], "stdout": "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.", "stdout_lines": ["Error during database migration: There
 are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."]}

Note: one cinder volume was created before starting the ffu upgrade procedure

Comment 2 Marius Cornea 2018-01-18 13:04:51 UTC
Note for test: the existing volume was in an ERROR state.

Comment 3 Marios Andreou 2018-01-25 12:24:23 UTC
o/ mcornea to clarify I think you are testing with the review at https://review.openstack.org/#/c/521545/15/docker/services/cinder-api.yaml@253 .. which is where the "cinder-manage db sync" comes from. The only other place I saw is https://github.com/openstack/tripleo-heat-templates/blob/5926cbaa90ca0c608af00ec8d6d5b7be90f5af32/docker/services/cinder-api.yaml#L138 but that is on docker_config so not relevant for the fast_forward_upgrade_playbook

I'm adding /#/c/521545/ to trackers even though that is what introdocues/has the bug since I think that is the right place to fix it. Marking triaged and assigning to jfrancoa since he's looking after that review these days I believe.

Comment 4 Gorka Eguileor 2018-01-31 12:26:35 UTC
The error is an internal check that we have in Cinder Pike to verify that online migrations from Ocata have been completed.

Just like with Nova you need to run online migration using cinder-manage since you won't be running the service during a FF upgrade.

We don't seem to have any online migrations in Pike, but calling the command will be a NOOP, so it's better to just run this command in Ocata and later releases after the DB sync, except on the final destination version, where we should leave them to be done online.

Comment 5 Jose Luis Franco 2018-02-02 14:10:55 UTC
Added extra step before the db sync, which executes "cinder-manage db online_data_migrations" only in Ocata.

Comment 6 Sofer Athlan-Guyot 2018-03-19 14:40:22 UTC
The review is in upstream queens.

Comment 15 errata-xmlrpc 2018-06-27 13:42:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.