Bug 1536010 – OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1536010 - OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.

Summary:	OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates (Show other bugs)
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified Unspecified

Priority	high Severity urgent
Target Milestone:	beta
Target Release:	13.0 (Queens)
Assigned To:	Emilien Macchi
QA Contact:	Marius Cornea
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-01-18 07:52 EST by Marius Cornea
Modified:	2018-06-27 09:43 EDT (History)
CC List:	10 users (show)

See Also:
Fixed In Version:	openstack-tripleo-heat-templates-8.0.0-0.20180304031149.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-06-27 09:42:33 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1744056	None	None	None	2018-01-18 07:52 EST
OpenStack gerrit	521545	None	master: MERGED	tripleo-heat-templates: ffu: Introduce Cinder fast-forward upgrade tasks (I172c3a1868a8b7a94b282cbe5c2f6b323f7ca101)	2018-03-22 12:55 EDT
Red Hat Product Errata	RHEA-2018:2086	None	None	None	2018-06-27 09:43 EDT

Groups: None (edit)

Description Marius Cornea 2018-01-18 07:52:57 EST

Description of problem:

OSP10 -> OSP13 FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."

TASK [Cinder db sync] *************************************************************************************************************************************************************************************************************************
fatal: [192.168.24.9]: FAILED! => {"changed": true, "cmd": ["cinder-manage", "db", "sync"], "delta": "0:00:02.538621", "end": "2018-01-18 12:26:04.920501", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-01-18 12:26:
02.381880", "stderr": "Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"DEFAULT\".", "stderr_lines": ["Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"D
EFAULT\"."], "stdout": "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.", "stdout_lines": ["Error during database migration: There
 are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."]}

Note: one cinder volume was created before starting the ffu upgrade procedure

Comment 2 Marius Cornea 2018-01-18 08:04:51 EST

Note for test: the existing volume was in an ERROR state.

Comment 3 Marios Andreou 2018-01-25 07:24:23 EST

o/ mcornea to clarify I think you are testing with the review at https://review.openstack.org/#/c/521545/15/docker/services/cinder-api.yaml@253 .. which is where the "cinder-manage db sync" comes from. The only other place I saw is https://github.com/openstack/tripleo-heat-templates/blob/5926cbaa90ca0c608af00ec8d6d5b7be90f5af32/docker/services/cinder-api.yaml#L138 but that is on docker_config so not relevant for the fast_forward_upgrade_playbook

I'm adding /#/c/521545/ to trackers even though that is what introdocues/has the bug since I think that is the right place to fix it. Marking triaged and assigning to jfrancoa since he's looking after that review these days I believe.

Comment 4 Gorka Eguileor 2018-01-31 07:26:35 EST

The error is an internal check that we have in Cinder Pike to verify that online migrations from Ocata have been completed.

Just like with Nova you need to run online migration using cinder-manage since you won't be running the service during a FF upgrade.

We don't seem to have any online migrations in Pike, but calling the command will be a NOOP, so it's better to just run this command in Ocata and later releases after the DB sync, except on the final destination version, where we should leave them to be done online.

Comment 5 Jose Luis Franco 2018-02-02 09:10:55 EST

Added extra step before the db sync, which executes "cinder-manage db online_data_migrations" only in Ocata.

Comment 6 Sofer Athlan-Guyot 2018-03-19 10:40:22 EDT

The review is in upstream queens.

Comment 15 errata-xmlrpc 2018-06-27 09:42:33 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086

Note You need to log in before you can comment on or make changes to this bug.