Bug 1785773

Summary:	Scale Down Stage Hangs in Jenkins Regression Jobs
Product:	Red Hat OpenStack	Reporter:	David Rosenfeld <drosenfe>
Component:	openstack-tripleo-common	Assignee:	Adriano Petrich <apetrich>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	16.0 (Train)	CC:	apetrich, aschultz, mburns, ramishra, scohen, shrjoshi, slinaber
Target Milestone:	rc	Keywords:	AutomationBlocker, Triaged
Target Release:	16.0 (Train on RHEL 8.1)
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-common-11.3.2-0.20191220123322.1e24c7f.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-06 14:44:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Rosenfeld 2019-12-20 22:05:46 UTC

Description of problem: There are 12 OSP 16 regression jobs that hang in Scale Down stage when executed.  Patch  https://review.opendev.org/#/c/699007 needs to be included in puddle.


Version-Release number of selected component (if applicable): RHOS_TRUNK-16.0-RHEL-8-20191217.n.1


How reproducible: Every time one of jobs is executed ex. DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown


Steps to Reproduce:
1. Execute split stack job DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown in Jenkins
2.
3.

Actual results: Job hangs and aborts after six hours


Expected results: Job completes successfully


Additional info:

Comment 1 David Rosenfeld 2019-12-20 22:08:10 UTC

Logs to a failing test:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/splitstack/job/DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown/3/

Comment 2 Alex Schultz 2019-12-20 23:26:49 UTC

2019-12-19 17:12:12.006 7 WARNING mistral.executors.default_executor [req-bda9084c-fc2a-40fe-ab5b-4a1ec02874e0 a32461c4f16343b198bc11f5ece0d32d 2ad5231cbb0040c099b828471e88faa1 - default default] The action raised
 an exception [action_ex_id=dbd59d99-ab0c-4899-a366-1edc463ad68f, msg='IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)', action_cls='<class 'mistral.actions.action_factory.Iron
icAction'>', attributes='{'client_method_name': 'node.get_by_instance_uuid'}', params='{'instance_uuid': '28a06013-dcc4-4427-8227-57639c8323e7', 'fields': ['instance_info']}']: mistral.exceptions.ActionException: 
IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 117, in run
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     result = method(**self._kwargs_for_run)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/ironicclient/v1/node.py", line 354, in get_by_instance_uuid
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     raise exc.NotFound()
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor ironicclient.common.apiclient.exceptions.NotFound: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor During handling of the above exception, another exception occurred:
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 133, in _do_run_action
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 130, in run
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     (self.__class__.__name__, self.client_method_name, str(e))
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor mistral.exceptions.ActionException: IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.031 7 DEBUG mistral.executors.executor_server [req-bda9084c-fc2a-40fe-ab5b-4a1ec02874e0 a32461c4f16343b198bc11f5ece0d32d 2ad5231cbb0040c099b828471e88faa1 - default default] Sending action result to engine [action_ex_id=dbd59d99-ab0c-4899-a366-1edc463ad68f, action_cls=mistral.actions.openstack.actions.IronicAction] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:114

Comment 7 David Rosenfeld 2020-01-06 17:56:45 UTC

Pass log:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/rfe/job/DFG-df-rfe-16-virsh-3cont_3comp_3ceph-blacklist-1compute-1control-compute-replacement/9/

Comment 10 errata-xmlrpc 2020-02-06 14:44:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283