After looking at the top_output.log, we found that there were two MIQ Server processes running on the system with their own set of workers running below them. It appears that manually running evmserver.sh start, outside of using systemctl evmserverd, may have complicated the issue and could be a reason for the two server processes on the same machine. Nevertheless, the machines were rebooted, and the request was to always use systemd through systemctl to start and stop evmserverd. PID PPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19502 1 root 20 0 666920 295924 6752 S 5.4 1.8 39:43.70 MIQ Server 5899 1 root 20 0 652524 299860 6752 S 3.7 1.8 5672:04 MIQ Server The multiple servers would cause the workers to constantly try to stop workers and possibly prevent them from successfully exiting.
https://github.com/ManageIQ/manageiq/pull/15491
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/9090b3e20967bf2f9fe25047260dce7aa5a0de41 commit 9090b3e20967bf2f9fe25047260dce7aa5a0de41 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Jun 30 16:51:56 2017 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Jun 30 17:54:33 2017 -0400 Only remove my process' pidfile. https://bugzilla.redhat.com/show_bug.cgi?id=1464698 at_exit handlers created in the server process are inherited when the server forks workers. As soon as one of these workers exits, this at_exit was being run and removing the server's pidfile. This allowed a subsequent rake evm:start or systemctl start evmserverd to fail to detect an existing server process, allowing a second MIQ Server to start. We can workaround this by only removing a pidfile that matches my Process.pid. lib/pid_file.rb | 2 +- spec/lib/pid_file_spec.rb | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
Verified in 5.9.0.2