Bug 1584951 - FFU: openstack overcloud ffwd-upgrade prepare fails with: u'message': u'Init Minor update failed'
Summary: FFU: openstack overcloud ffwd-upgrade prepare fails with: u'message': u'Init...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: All
OS: All
urgent
urgent
Target Milestone: rc
: 13.0 (Queens)
Assignee: mathieu bultel
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-01 02:57 UTC by Marius Cornea
Modified: 2018-06-27 13:58 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-common-8.6.1-19.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:57:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
ffwd prepare log (308.68 KB, text/plain)
2018-06-01 02:58 UTC, Marius Cornea
no flags Details
sosreport (17.60 MB, application/x-xz)
2018-06-01 03:03 UTC, Marius Cornea
no flags Details
stack.tar.gz containing all environment files (580.63 KB, application/x-gzip)
2018-06-01 03:05 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 571765 0 'None' MERGED Get deployment_stack_id via deployment_ref for config download 2020-07-13 14:34:26 UTC
OpenStack gerrit 571983 0 'None' MERGED Get deployment_stack_id via deployment_ref for config download 2020-07-13 14:34:26 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:58:15 UTC

Description Marius Cornea 2018-06-01 02:57:22 UTC
Description of problem:

FFU: openstack overcloud ffwd-upgrade prepare fails with:  u'message': u'Init Minor update failed':

2018-06-01 02:47:45Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerPrePuppet]: DELETE_IN_PROGRESS  state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-ControllerPrePuppet-6chp5nf3gnro]: DELETE_IN_PROGRESS  Stack DELETE started
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-ControllerPrePuppet-6chp5nf3gnro.ControllerPrePuppetMaintenanceModeDeployment]: DELETE_IN_PROGRESS  state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.CephStorageConfig]: DELETE_IN_PROGRESS  state changed
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-CephStorageConfig-6anashbrrrau]: DELETE_IN_PROGRESS  Stack DELETE started
2018-06-01 02:47:46Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya-CephStorageConfig-6anashbrrrau.CephStoragePuppetConfigImpl]: DELETE_IN_PROGRESS  state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerConfig]: DELETE_COMPLETE  state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ComputeConfig]: DELETE_COMPLETE  state changed
2018-06-01 02:47:47Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.CephStorageConfig]: DELETE_COMPLETE  state changed
2018-06-01 02:47:51Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya.ControllerPrePuppet]: DELETE_COMPLETE  state changed
2018-06-01 02:47:51Z [overcloud-AllNodesDeploySteps-gu5fxv3wdfya]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-06-01 02:47:52Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2018-06-01 02:47:55Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: e0327a73-d609-4182-9b81-894aa4759c55
Waiting for messages on queue 'tripleo' with no timeout.
{u'execution': {u'created_at': u'2018-06-01 02:47:58',
                u'id': u'e0327a73-d609-4182-9b81-894aa4759c55',
                u'input': {u'container': u'overcloud',
                           u'queue_name': u'tripleo'},
                u'name': u'tripleo.package_update.v1.get_config',
                u'params': {u'namespace': u''},
                u'spec': {u'input': [u'container',
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'get_config',
                          u'tags': [u'tripleo-common-managed'],
                          u'tasks': {u'get_config': {u'action': u'tripleo.config.get_overcloud_config container=<% $.container %>',
                                                     u'name': u'get_config',
                                                     u'on-complete': u'send_message',
                                                     u'publish': {u'message': u'<% task().result %>',
                                                                  u'status': u'SUCCESS'},
                                                     u'publish-on-error': {u'message': u'Init Minor update failed',
                                                                           u'status': u'FAILED'},
                                                     u'type': u'direct',
                                                     u'version': u'2.0'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.package_update.v1.package_update_plan'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
                                                       u'type': u'direct',
                                                       u'version': u'2.0'}},
                          u'version': u'2.0'},
                u'updated_at': u'2018-06-01 02:47:58'},
 u'message': u'Init Minor update failed',
 u'status': u'FAILED'}


Version-Release number of selected component (if applicable):
python-tripleoclient-9.2.1-12.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 with 3 controller + 2 compute nodes + 3 ceph OSD nodes
2. Upgrade undercloud to OSP11/12/13
3. Convert deprecated parameters to their new correspondents according to https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/fast_forward_upgrades/#preparing_for_deprecated_parameters

Parameters converted: OvercloudControlFlavor, NovaComputeExtraConfig, NovaComputeSchedulerHints, NovaComputeIPs
4. Run  openstack overcloud ffwd-upgrade prepare:

openstack overcloud ffwd-upgrade prepare --templates /usr/share/openstack-tripleo-heat-templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-management.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/enable-cpu-pinning.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/scheduler_hints_env.yaml \
-e ~/openstack_deployment/environments/ips-from-pool-all.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e ~/openstack_deployment/environments/custom_hiera.yaml \
-e $HOME/ffu_repos.yaml \
--container-registry-file /home/stack/virt/docker-images.yaml \

Actual results:
Fails.

Expected results:
Doesn't fail.

Additional info:
Attaching sosreport and ffwd prepare output.

Comment 1 Marius Cornea 2018-06-01 02:58:21 UTC
Created attachment 1446498 [details]
ffwd prepare log

Comment 2 Marius Cornea 2018-06-01 03:03:27 UTC
Created attachment 1446500 [details]
sosreport

Comment 3 Marius Cornea 2018-06-01 03:05:05 UTC
Created attachment 1446501 [details]
stack.tar.gz containing all environment files

Comment 4 Marius Cornea 2018-06-01 03:12:41 UTC
Error from /var/log/mistral/executor.log:


 ERROR: The Stack (ef48bd3a-8db4-40b3-b22a-c4595df81ab5) could not be found.: HTTPNotFound: ERROR: The Stack (ef48bd3a-8db4-40b3-b22a-c4595df81ab5) could not be found.
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor Traceback (most recent call last):
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/config.py", line 50, in run
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     config_path = config.download_config(self.container, self.config_dir)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/tripleo_common/utils/config.py", line 243, in download_config
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     deployment_stack_id, resolve_outputs=False)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 280, in get
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     resp = self.client.get('/stacks/%s' % stack_id, **kwargs)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 286, in get
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     return self.client_request("GET", url, **kwargs)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 279, in client_request
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     resp, body = self.json_request(method, url, **kwargs)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 268, in json_request
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     resp = self._http_request(url, method, **kwargs)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor   File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 231, in _http_request
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor     raise exc.from_response(resp)
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor HTTPNotFound: ERROR: The Stack (ef48bd3a-8db4-40b3-b22a-c4595df81ab5) could not be found.
2018-05-31 22:48:43.923 1475 ERROR mistral.executors.default_executor

Comment 5 Marius Cornea 2018-06-01 16:36:44 UTC
I applied https://review.openstack.org/#/c/571765/1 and wasn't able to reproduce the issue reported initially.

Comment 7 mathieu bultel 2018-06-05 06:58:47 UTC
Hi Maria, can you PM_ack this one, I need to build the package for this blocker.
Thank you very much.

Mathieu

Comment 8 Carlos Camacho 2018-06-05 10:30:35 UTC
For creating the package we need the blocker flag accepted

Comment 15 Marius Cornea 2018-06-07 18:50:11 UTC
openstack-tripleo-common-8.6.1-19.el7ost.noarch

Comment 17 errata-xmlrpc 2018-06-27 13:57:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.