Description of problem: Seeing the SmartProxy and VimBroker workers appearing in the stopping state but are not being killed. Only thing to fix is to restart the evmserverd. Below are 8 SmartProxy workers in the stopping state and 2 in the started state. One of the SmartProxy workers has a last heartbeat of 08:59 UTC and the status command was run at 13:30 PST, which is about 12.5 hours. [root@mprdcfmeas306 ~]# evmserver.sh status Checking EVM status... Zone | Server Name | Status | ID | PID | SPID | URL | Started On | Last Heartbeat | Active Roles -----------------------+---------------+---------+-----------------+-------+------+-------------------------+----------------------+----------------------+--------------------------------------------------------------------------- QDC-C PreProd Workers | mprdcfmeas306 | started | 509000000000053 | 23026 | 4820 | druby://127.0.0.1:34568 | 2017-03-05T16:57:26Z | 2017-03-06T21:29:16Z | automate:ems_operations:smartproxy:smartstate:user_interface:web_services Worker Type | Status | ID | PID | SPID | Queue Name / URL | Started On | Last Heartbeat ---------------------+----------+-----------------+-------+-------+-------------------------+----------------------+---------------------- MiqGenericWorker | started | 509000001568704 | 19379 | 20564 | generic | 2017-03-06T14:05:06Z | 2017-03-06T21:29:37Z MiqGenericWorker | started | 509000001568599 | 18097 | 18442 | generic | 2017-03-06T13:53:29Z | 2017-03-06T21:29:36Z MiqPriorityWorker | stopping | 509000001566595 | 23278 | 5956 | generic | 2017-03-05T16:57:30Z | 2017-03-05T17:01:15Z MiqPriorityWorker | started | 509000001566618 | 28252 | 19436 | generic | 2017-03-05T17:02:07Z | 2017-03-06T21:29:38Z MiqPriorityWorker | started | 509000001566594 | 23270 | 5955 | generic | 2017-03-05T16:57:30Z | 2017-03-06T21:29:37Z MiqScheduleWorker | started | 509000001566596 | 23286 | 5957 | | 2017-03-05T16:57:30Z | 2017-03-06T21:29:28Z MiqSmartProxyWorker | stopping | 509000001567815 | 21268 | 8746 | smartproxy | 2017-03-06T06:59:33Z | 2017-03-06T08:59:32Z MiqSmartProxyWorker | stopping | 509000001567814 | 21260 | 8745 | smartproxy | 2017-03-06T06:59:32Z | 2017-03-06T08:59:32Z MiqSmartProxyWorker | started | 509000001569196 | 18524 | 6559 | smartproxy | 2017-03-06T21:00:55Z | 2017-03-06T21:29:36Z MiqSmartProxyWorker | started | 509000001569197 | 18532 | 6560 | smartproxy | 2017-03-06T21:00:55Z | 2017-03-06T21:29:36Z MiqSmartProxyWorker | stopping | 509000001568754 | 23006 | 20294 | smartproxy | 2017-03-06T15:00:21Z | 2017-03-06T17:00:15Z MiqSmartProxyWorker | stopping | 509000001568755 | 23014 | 20295 | smartproxy | 2017-03-06T15:00:21Z | 2017-03-06T17:00:15Z MiqSmartProxyWorker | stopping | 509000001569049 | 9659 | 11450 | smartproxy | 2017-03-06T19:00:46Z | 2017-03-06T21:00:46Z MiqSmartProxyWorker | stopping | 509000001569048 | 9651 | 11449 | smartproxy | 2017-03-06T19:00:46Z | 2017-03-06T21:00:46Z MiqSmartProxyWorker | stopping | 509000001568900 | 31125 | 15924 | smartproxy | 2017-03-06T17:00:32Z | 2017-03-06T19:00:29Z MiqSmartProxyWorker | stopping | 509000001568901 | 31133 | 15925 | smartproxy | 2017-03-06T17:00:32Z | 2017-03-06T19:00:29Z MiqUiWorker | started | 509000001566602 | 23360 | | http://127.0.0.1:3000 | 2017-03-05T16:58:05Z | 2017-03-06T21:29:29Z MiqVimBrokerWorker | started | 509000001566603 | 23374 | 7602 | druby://127.0.0.1:43655 | 2017-03-05T16:58:05Z | 2017-03-06T21:29:33Z MiqWebServiceWorker | started | 509000001566604 | 23382 | | http://127.0.0.1:4000 | 2017-03-05T16:58:05Z | 2017-03-06T21:29:29Z [root@mprdcfmeas306 ~]# date Mon Mar 6 13:30:31 PST 2017 Version-Release number of selected component (if applicable): 5.6.6.3 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1260982 [details] evm log
This should be resolved by the fix done for bug 1395736 found in: https://github.com/ManageIQ/manageiq/pull/13805 https://github.com/ManageIQ/manageiq/pull/13919 This is marked for darga and euwe backport. If this does not resolve the issue, please re-open. thanks! *** This bug has been marked as a duplicate of bug 1395736 ***