Bug 1646817

Summary: 'openstack overcloud status --plan overcloud' gets stuck and exits with 'The read operation timed out'
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: agurenko, apetrich, jcoufal, jjoyce, jschluet, jslagle, mburns, mlammon, sasha, slinaber, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-9.4.1-0.20181012010871.67bab16.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:54:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2018-11-06 04:35:39 UTC
'openstack overcloud status --plan overcloud' gets stuck and exits with 'The read operation timed out'


Environment:
instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch
python2-mistralclient-3.7.0-0.20180810140142.f0ee48f.el7ost.noarch
puppet-mistral-13.3.1-0.20181013122848.082509f.el7ost.noarch
openstack-tripleo-heat-templates-9.0.1-0.20181013060867.ffbe879.el7ost.noarch
python2-mistral-lib-1.0.0-0.20180821152751.d1ccfd0.el7ost.noarch


Steps to reproduce:
Deploy OC.
Run 'openstack overcloud status --plan overcloud'

Result:
gets stuck and exits with 'The read operation timed out'

The issue reproduces.

Comment 2 James Slagle 2018-11-06 15:28:16 UTC
this appears to be the error:

2018-11-05 23:27:48.132 1 DEBUG mistral.expressions.yaql_expression [req-62253e40-d568-4af8-8561-220f5e99dc77 abffc8424b844cdf87ff7eaa44ed6cb6 42f9e3e9b4904886a147af5a6ed4415e - default default] Start to evaluate YAQL expression. [expression='<% $.status_update = null and $.deployment_status != null %>', context={}] evaluate /usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py:149
2018-11-05 23:27:48.150 1 INFO workflow_trace [req-62253e40-d568-4af8-8561-220f5e99dc77 abffc8424b844cdf87ff7eaa44ed6cb6 42f9e3e9b4904886a147af5a6ed4415e - default default] Workflow 'tripleo.deployment.v1.get_deployment_status' [RUNNING -> ERROR, msg=Failed to run task [error=Failed to find action [action_name=tripleo.deployment.get_deployment_status], wf=tripleo.deployment.v1.get_deployment_status, task=get_deployment_status]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 63, in run_task
    task.run()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 390, in run
    self._run_new()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper
    result = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 419, in _run_new
    self._schedule_actions()
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 486, in _schedule_actions
    action = self._build_action()
  File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 564, in _build_action
    self.wf_spec.get_name()
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 635, in resolve_action_definition
    "Failed to find action [action_name=%s]" % action_spec_name
InvalidActionException: Failed to find action [action_name=tripleo.deployment.get_deployment_status]

It looks like perhaps the undercloud was not updated to pull new mistral containers.

what version of mistral containers do you have?
what version of openstack-tripleo-common is in each mistral container?

Comment 3 Alexander Chuzhoy 2018-11-06 15:33:13 UTC
 [root@undercloud-0 ~]# for i in `docker ps|awk '/mistral/ {print $NF}'`; do echo $i; docker exec -u root -it $i rpm -q openstack-tripleo-common; done
mistral_api
openstack-tripleo-common-9.4.1-0.20181012010869.67bab16.el7ost.noarch
mistral_engine
openstack-tripleo-common-9.4.1-0.20181012010869.67bab16.el7ost.noarch
mistral_event_engine
openstack-tripleo-common-9.4.1-0.20181012010869.67bab16.el7ost.noarch
mistral_executor
openstack-tripleo-common-9.4.1-0.20181012010869.67bab16.el7ost.noarch



 [root@undercloud-0 ~]# docker images|grep mistral
192.168.24.1:8787/rhosp14/openstack-mistral-api                  2018-10-29.1        44f103c8b07a        7 days ago          946 MB
192.168.24.1:8787/rhosp14/openstack-mistral-engine               2018-10-29.1        e790ead4ffc6        7 days ago          908 MB
192.168.24.1:8787/rhosp14/openstack-mistral-executor             2018-10-29.1        e0ab00a72888        7 days ago          1.12 GB
192.168.24.1:8787/rhosp14/openstack-mistral-event-engine         2018-10-29.1        d1656ee738a0        7 days ago          908 MB

Comment 7 Gurenko Alex 2018-11-15 10:58:37 UTC
Verified on puddle 2018-11-13.1

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status --plan overcloud
None
+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2018-11-15 09:16:39 | 2018-11-15 09:16:39 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

(undercloud) [stack@undercloud-0 ~]$ rpm -q openstack-tripleo-common
openstack-tripleo-common-9.4.1-0.20181012010875.67bab16.el7ost.noarch

Comment 10 errata-xmlrpc 2019-01-11 11:54:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045