Bug 1536010

Summary: OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: dbecker, geguileo, jschluet, mandreou, mbracho, mbultel, mburns, morazi, rhel-osp-director-maint, sathlang
Target Milestone: betaKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.0-0.20180304031149.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:42:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Marius Cornea 2018-01-18 12:52:57 UTC
Description of problem:

OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."

TASK [Cinder db sync] *************************************************************************************************************************************************************************************************************************
fatal: [192.168.24.9]: FAILED! => {"changed": true, "cmd": ["cinder-manage", "db", "sync"], "delta": "0:00:02.538621", "end": "2018-01-18 12:26:04.920501", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-01-18 12:26:
02.381880", "stderr": "Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"DEFAULT\".", "stderr_lines": ["Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"D
EFAULT\"."], "stdout": "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.", "stdout_lines": ["Error during database migration: There
 are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."]}

Note: one cinder volume was created before starting the ffu upgrade procedure

Comment 2 Marius Cornea 2018-01-18 13:04:51 UTC
Note for test: the existing volume was in an ERROR state.

Comment 3 Marios Andreou 2018-01-25 12:24:23 UTC
o/ mcornea to clarify I think you are testing with the review at https://review.openstack.org/#/c/521545/15/docker/services/cinder-api.yaml@253 .. which is where the "cinder-manage db sync" comes from. The only other place I saw is https://github.com/openstack/tripleo-heat-templates/blob/5926cbaa90ca0c608af00ec8d6d5b7be90f5af32/docker/services/cinder-api.yaml#L138 but that is on docker_config so not relevant for the fast_forward_upgrade_playbook

I'm adding /#/c/521545/ to trackers even though that is what introdocues/has the bug since I think that is the right place to fix it. Marking triaged and assigning to jfrancoa since he's looking after that review these days I believe.

Comment 4 Gorka Eguileor 2018-01-31 12:26:35 UTC
The error is an internal check that we have in Cinder Pike to verify that online migrations from Ocata have been completed.

Just like with Nova you need to run online migration using cinder-manage since you won't be running the service during a FF upgrade.

We don't seem to have any online migrations in Pike, but calling the command will be a NOOP, so it's better to just run this command in Ocata and later releases after the DB sync, except on the final destination version, where we should leave them to be done online.

Comment 5 Jose Luis Franco 2018-02-02 14:10:55 UTC
Added extra step before the db sync, which executes "cinder-manage db online_data_migrations" only in Ocata.

Comment 6 Sofer Athlan-Guyot 2018-03-19 14:40:22 UTC
The review is in upstream queens.

Comment 15 errata-xmlrpc 2018-06-27 13:42:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086