1785773 – Scale Down Stage Hangs in Jenkins Regression Jobs

Bug 1785773 - Scale Down Stage Hangs in Jenkins Regression Jobs

Summary: Scale Down Stage Hangs in Jenkins Regression Jobs

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-common
Sub Component:
Version:	16.0 (Train)
Hardware:	Unspecified
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	16.0 (Train on RHEL 8.1)
Assignee:	Adriano Petrich
QA Contact:	Alexander Chuzhoy
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-12-20 22:05 UTC by David Rosenfeld
Modified:	2020-02-06 14:44 UTC (History)
CC List:	7 users (show)
Fixed In Version:	openstack-tripleo-common-11.3.2-0.20191220123322.1e24c7f.el8ost
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-02-06 14:44:12 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2020:0283	0	None	None	None	2020-02-06 14:44:39 UTC

Description David Rosenfeld 2019-12-20 22:05:46 UTC

Description of problem: There are 12 OSP 16 regression jobs that hang in Scale Down stage when executed.  Patch  https://review.opendev.org/#/c/699007 needs to be included in puddle.


Version-Release number of selected component (if applicable): RHOS_TRUNK-16.0-RHEL-8-20191217.n.1


How reproducible: Every time one of jobs is executed ex. DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown


Steps to Reproduce:
1. Execute split stack job DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown in Jenkins
2.
3.

Actual results: Job hangs and aborts after six hours


Expected results: Job completes successfully


Additional info:

Comment 1 David Rosenfeld 2019-12-20 22:08:10 UTC

Logs to a failing test:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/splitstack/job/DFG-df-splitstack-16-virsh-3cont_2comp_3ceph-scaledown/3/

Comment 2 Alex Schultz 2019-12-20 23:26:49 UTC

2019-12-19 17:12:12.006 7 WARNING mistral.executors.default_executor [req-bda9084c-fc2a-40fe-ab5b-4a1ec02874e0 a32461c4f16343b198bc11f5ece0d32d 2ad5231cbb0040c099b828471e88faa1 - default default] The action raised
 an exception [action_ex_id=dbd59d99-ab0c-4899-a366-1edc463ad68f, msg='IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)', action_cls='<class 'mistral.actions.action_factory.Iron
icAction'>', attributes='{'client_method_name': 'node.get_by_instance_uuid'}', params='{'instance_uuid': '28a06013-dcc4-4427-8227-57639c8323e7', 'fields': ['instance_info']}']: mistral.exceptions.ActionException: 
IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 117, in run
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     result = method(**self._kwargs_for_run)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/ironicclient/v1/node.py", line 354, in get_by_instance_uuid
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     raise exc.NotFound()
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor ironicclient.common.apiclient.exceptions.NotFound: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor During handling of the above exception, another exception occurred:
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 133, in _do_run_action
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 130, in run
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor     (self.__class__.__name__, self.client_method_name, str(e))
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor mistral.exceptions.ActionException: IronicAction.node.get_by_instance_uuid failed: Not Found (HTTP HTTPStatus.NOT_FOUND)
2019-12-19 17:12:12.006 7 ERROR mistral.executors.default_executor 
2019-12-19 17:12:12.031 7 DEBUG mistral.executors.executor_server [req-bda9084c-fc2a-40fe-ab5b-4a1ec02874e0 a32461c4f16343b198bc11f5ece0d32d 2ad5231cbb0040c099b828471e88faa1 - default default] Sending action result to engine [action_ex_id=dbd59d99-ab0c-4899-a366-1edc463ad68f, action_cls=mistral.actions.openstack.actions.IronicAction] run_action /usr/lib/python3.6/site-packages/mistral/executors/executor_server.py:114

Comment 7 David Rosenfeld 2020-01-06 17:56:45 UTC

Pass log:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/rfe/job/DFG-df-rfe-16-virsh-3cont_3comp_3ceph-blacklist-1compute-1control-compute-replacement/9/

Comment 10 errata-xmlrpc 2020-02-06 14:44:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283

Note You need to log in before you can comment on or make changes to this bug.