Bug 1638448

Summary: Online data migrations should be run outside the controller upgrade window
Product: Red Hat OpenStack Reporter: Jiri Stransky <jstransk>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: jstransk, mburns
Target Milestone: betaKeywords: Regression, Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060865.ffbe879.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:53:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jiri Stransky 2018-10-11 14:59:31 UTC
We have a solution in testing for OSP 12/13, we should implement one for OSP 14 too, it's going to be slightly different due to the differences in upgrade framework since 14+. 

Currently online data migrations in 14 still happen during controller upgrade.

Comment 17 Yurii Prokulevych 2018-12-17 13:28:50 UTC
Verified with openstack-tripleo-heat-templates-9.0.1-0.20181013060906.el7ost.noarch

Ran next command before upgrade converge.

openstack overcloud external-upgrade run \
    --stack qe-Cloud-0 \
    --tags online_upgrade 2>&1
...2018-12-17 08:21:25 |  u'TASK [Online data migration for Cinder] ****************************************',
2018-12-17 08:21:25 |  u'Monday 17 December 2018  08:21:17 -0500 (0:00:00.098)       0:00:32.496 ******* ',
2018-12-17 08:21:25 |  u'changed: [undercloud -> 192.168.24.8] => {"changed": true, "cmd": ["docker", "exec", "cinder_api", "cinder-manage", "db", "online_data_migrations"], "delta": "0:00:02.313713", "end": "2018-12-17 13:21:19.709889", "rc": 0, "start": "2018-12-17 13:21:17.396176", "stderr": "Deprecated: Option \\"logdir\\" from group \\"DEFAULT\\" is deprecated. Use option \\"log-dir\\" from group \\"DEFAULT\\".", "stderr_lines": ["Deprecated: Option \\"logdir\\" from group \\"DEFAULT\\" is deprecated. Use option \\"log-dir\\" from group \\"DEFAULT\\"."], "stdout": "Running batches of 50 until complete.\\n+--------------------------------------------+--------------+-----------+\\n|                 Migration                  | Total Needed | Completed |\\n+--------------------------------------------+--------------+-----------+\\n|   attachment_specs_online_data_migration   |      0       |     0     |\\n|      backup_service_online_migration       |      0       |     0     |\\n|    service_uuids_online_data_migration     |      0       |     0     |\\n|    shared_targets_online_data_migration    |      0       |     0     |\\n| volume_service_uuids_online_data_migration |      0       |     0     |\\n+--------------------------------------------+--------------+-----------+", "stdout_lines": ["Running batches of 50 until complete.", "+--------------------------------------------+--------------+-----------+", "|                 Migration                  | Total Needed | Completed |", "+--------------------------------------------+--------------+-----------+", "|   attachment_specs_online_data_migration   |      0       |     0     |", "|      backup_service_online_migration       |      0       |     0     |", "|    service_uuids_online_data_migration     |      0       |     0     |", "|    shared_targets_online_data_migration    |      0       |     0     |", "| volume_service_uuids_online_data_migration |      0       |     0     |", "+--------------------------------------------+--------------+-----------+"]}',
2018-12-17 08:21:25 |  u'',
2018-12-17 08:21:25 |  u'TASK [Online data migration for Nova] ******************************************',
2018-12-17 08:21:25 |  u'Monday 17 December 2018  08:21:19 -0500 (0:00:02.767)       0:00:35.263 ******* ',
2018-12-17 08:21:26 |  u'changed: [undercloud -> 192.168.24.8] => {"changed": true, "cmd": ["docker", "exec", "nova_api", "nova-manage", "db", "online_data_migrations"], "delta": "0:00:04.313646", "end": "2018-12-17 13:21:24.344519", "rc": 0, "start": "2018-12-17 13:21:20.030873", "stderr": "", "stderr_lines": [], "stdout": "Running batches of 50 until complete\\n+---------------------------------------------+--------------+-----------+\\n|                  Migration                  | Total Needed | Completed |\\n+---------------------------------------------+--------------+-----------+\\n|         create_incomplete_consumers         |      0       |     0     |\\n| delete_build_requests_with_no_instance_uuid |      0       |     0     |\\n|      migrate_instances_add_request_spec     |      0       |     0     |\\n|          migrate_keypairs_to_api_db         |      0       |     0     |\\n|       migrate_quota_classes_to_api_db       |      0       |     0     |\\n|        migrate_quota_limits_to_api_db       |      0       |     0     |\\n|          migration_migrate_to_uuid          |      0       |     0     |\\n|     populate_missing_availability_zones     |      0       |     0     |\\n|          populate_queued_for_delete         |      0       |     0     |\\n|                populate_uuids               |      0       |     0     |\\n|     service_uuids_online_data_migration     |      0       |     0     |\\n+---------------------------------------------+--------------+-----------+", "stdout_lines": ["Running batches of 50 until complete", "+---------------------------------------------+--------------+-----------+", "|                  Migration                  | Total Needed | Completed |", "+---------------------------------------------+--------------+-----------+", "|         create_incomplete_consumers         |      0       |     0     |", "| delete_build_requests_with_no_instance_uuid |      0       |     0     |", "|      migrate_instances_add_request_spec     |      0       |     0     |", "|          migrate_keypairs_to_api_db         |      0       |     0     |", "|       migrate_quota_classes_to_api_db       |      0       |     0     |", "|        migrate_quota_limits_to_api_db       |      0       |     0     |", "|          migration_migrate_to_uuid          |      0       |     0     |", "|     populate_missing_availability_zones     |      0       |     0     |", "|          populate_queued_for_delete         |      0       |     0     |", "|                populate_uuids               |      0       |     0     |", "|     service_uuids_online_data_migration     |      0       |     0     |", "+---------------------------------------------+--------------+-----------+"]}',
2018-12-17 08:21:26 |  u'',
...

Comment 19 errata-xmlrpc 2019-01-11 11:53:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045