Bug 1430106

Summary: Many worker processes in stopping state and are not being killed
Product: Red Hat CloudForms Management Engine Reporter: myoder
Component: ApplianceAssignee: Gregg Tanzillo <gtanzill>
Status: CLOSED DUPLICATE QA Contact: Dave Johnson <dajohnso>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.6.0CC: abellott, jhardy, jrafanie, obarenbo
Target Milestone: GA   
Target Release: cfme-future   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-07 21:47:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:

Description myoder 2017-03-07 21:27:11 UTC 
Description of problem:

Seeing the SmartProxy and VimBroker workers appearing in the stopping state but are not being killed.  Only thing to fix is to restart the evmserverd.

Below are 8 SmartProxy workers in the stopping state and 2 in the started state.    One of the SmartProxy workers has a last heartbeat of 08:59 UTC and the status command was run at 13:30 PST, which is about 12.5 hours.


[root@mprdcfmeas306 ~]# evmserver.sh status
Checking EVM status...
 Zone                  | Server Name   | Status  |              ID |   PID | SPID | URL                     | Started On           | Last Heartbeat       | Active Roles
-----------------------+---------------+---------+-----------------+-------+------+-------------------------+----------------------+----------------------+---------------------------------------------------------------------------
 QDC-C PreProd Workers | mprdcfmeas306 | started | 509000000000053 | 23026 | 4820 | druby://127.0.0.1:34568 | 2017-03-05T16:57:26Z | 2017-03-06T21:29:16Z | automate:ems_operations:smartproxy:smartstate:user_interface:web_services

 Worker Type         | Status   |              ID |   PID | SPID  | Queue Name / URL        | Started On           | Last Heartbeat
---------------------+----------+-----------------+-------+-------+-------------------------+----------------------+----------------------
 MiqGenericWorker    | started  | 509000001568704 | 19379 | 20564 | generic                 | 2017-03-06T14:05:06Z | 2017-03-06T21:29:37Z
 MiqGenericWorker    | started  | 509000001568599 | 18097 | 18442 | generic                 | 2017-03-06T13:53:29Z | 2017-03-06T21:29:36Z
 MiqPriorityWorker   | stopping | 509000001566595 | 23278 | 5956  | generic                 | 2017-03-05T16:57:30Z | 2017-03-05T17:01:15Z
 MiqPriorityWorker   | started  | 509000001566618 | 28252 | 19436 | generic                 | 2017-03-05T17:02:07Z | 2017-03-06T21:29:38Z
 MiqPriorityWorker   | started  | 509000001566594 | 23270 | 5955  | generic                 | 2017-03-05T16:57:30Z | 2017-03-06T21:29:37Z
 MiqScheduleWorker   | started  | 509000001566596 | 23286 | 5957  |                         | 2017-03-05T16:57:30Z | 2017-03-06T21:29:28Z
 MiqSmartProxyWorker | stopping | 509000001567815 | 21268 | 8746  | smartproxy              | 2017-03-06T06:59:33Z | 2017-03-06T08:59:32Z
 MiqSmartProxyWorker | stopping | 509000001567814 | 21260 | 8745  | smartproxy              | 2017-03-06T06:59:32Z | 2017-03-06T08:59:32Z
 MiqSmartProxyWorker | started  | 509000001569196 | 18524 | 6559  | smartproxy              | 2017-03-06T21:00:55Z | 2017-03-06T21:29:36Z
 MiqSmartProxyWorker | started  | 509000001569197 | 18532 | 6560  | smartproxy              | 2017-03-06T21:00:55Z | 2017-03-06T21:29:36Z
 MiqSmartProxyWorker | stopping | 509000001568754 | 23006 | 20294 | smartproxy              | 2017-03-06T15:00:21Z | 2017-03-06T17:00:15Z
 MiqSmartProxyWorker | stopping | 509000001568755 | 23014 | 20295 | smartproxy              | 2017-03-06T15:00:21Z | 2017-03-06T17:00:15Z
 MiqSmartProxyWorker | stopping | 509000001569049 |  9659 | 11450 | smartproxy              | 2017-03-06T19:00:46Z | 2017-03-06T21:00:46Z
 MiqSmartProxyWorker | stopping | 509000001569048 |  9651 | 11449 | smartproxy              | 2017-03-06T19:00:46Z | 2017-03-06T21:00:46Z
 MiqSmartProxyWorker | stopping | 509000001568900 | 31125 | 15924 | smartproxy              | 2017-03-06T17:00:32Z | 2017-03-06T19:00:29Z
 MiqSmartProxyWorker | stopping | 509000001568901 | 31133 | 15925 | smartproxy              | 2017-03-06T17:00:32Z | 2017-03-06T19:00:29Z
 MiqUiWorker         | started  | 509000001566602 | 23360 |       | http://127.0.0.1:3000   | 2017-03-05T16:58:05Z | 2017-03-06T21:29:29Z
 MiqVimBrokerWorker  | started  | 509000001566603 | 23374 | 7602  | druby://127.0.0.1:43655 | 2017-03-05T16:58:05Z | 2017-03-06T21:29:33Z
 MiqWebServiceWorker | started  | 509000001566604 | 23382 |       | http://127.0.0.1:4000   | 2017-03-05T16:58:05Z | 2017-03-06T21:29:29Z

[root@mprdcfmeas306 ~]# date
Mon Mar  6 13:30:31 PST 2017

Version-Release number of selected component (if applicable):
5.6.6.3

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 myoder 2017-03-07 21:28:04 UTC 
Created attachment 1260982 [details]
evm log
Comment 3 Joe Rafaniello 2017-03-07 21:47:00 UTC 
This should be resolved by the fix done for bug 1395736 found in:
https://github.com/ManageIQ/manageiq/pull/13805
https://github.com/ManageIQ/manageiq/pull/13919

This is marked for darga and euwe backport.

If this does not resolve the issue, please re-open.

thanks!

*** This bug has been marked as a duplicate of bug 1395736 ***