Description of problem: OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated." TASK [Cinder db sync] ************************************************************************************************************************************************************************************************************************* fatal: [192.168.24.9]: FAILED! => {"changed": true, "cmd": ["cinder-manage", "db", "sync"], "delta": "0:00:02.538621", "end": "2018-01-18 12:26:04.920501", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-01-18 12:26: 02.381880", "stderr": "Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"DEFAULT\".", "stderr_lines": ["Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"D EFAULT\"."], "stdout": "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.", "stdout_lines": ["Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."]} Note: one cinder volume was created before starting the ffu upgrade procedure
Note for test: the existing volume was in an ERROR state.
o/ mcornea to clarify I think you are testing with the review at https://review.openstack.org/#/c/521545/15/docker/services/cinder-api.yaml@253 .. which is where the "cinder-manage db sync" comes from. The only other place I saw is https://github.com/openstack/tripleo-heat-templates/blob/5926cbaa90ca0c608af00ec8d6d5b7be90f5af32/docker/services/cinder-api.yaml#L138 but that is on docker_config so not relevant for the fast_forward_upgrade_playbook I'm adding /#/c/521545/ to trackers even though that is what introdocues/has the bug since I think that is the right place to fix it. Marking triaged and assigning to jfrancoa since he's looking after that review these days I believe.
The error is an internal check that we have in Cinder Pike to verify that online migrations from Ocata have been completed. Just like with Nova you need to run online migration using cinder-manage since you won't be running the service during a FF upgrade. We don't seem to have any online migrations in Pike, but calling the command will be a NOOP, so it's better to just run this command in Ocata and later releases after the DB sync, except on the final destination version, where we should leave them to be done online.
Added extra step before the db sync, which executes "cinder-manage db online_data_migrations" only in Ocata.
The review is in upstream queens.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086