Bug 1572615

Summary:	apache keeps restarting
Product:	Red Hat CloudForms Management Engine	Reporter:	Niladri Roy <niroy>
Component:	Appliance	Assignee:	Joe Rafaniello <jrafanie>
Status:	CLOSED NOTABUG	QA Contact:	Dave Johnson <dajohnso>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.7.0	CC:	abellott, cpelland, jrafanie, niroy, obarenbo
Target Milestone:	GA
Target Release:	cfme-future
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-04 15:00:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 4 Joe Rafaniello 2018-04-27 21:41:32 UTC

I don't think it's causing problems but I opened a BZ to fix the frequent warnings seen above:
AH01574: module ssl_module is already loaded, skipping
https://bugzilla.redhat.com/show_bug.cgi?id=1572793

Comment 5 Joe Rafaniello 2018-04-27 21:56:47 UTC

It looks like the user_interface, web_services, websocket roles (along with other roles) are frequently being removed, if this happens, it's likely we're stopping apache if all 3 of those above roles are no longer active and presumably are restarting it when they are added back.... so it might be a problem with roles going down and back up.

$ zgrep -E "log_role_changes.+Roles removed.+(user_interface|web_services|websocket)" log/evm.log*
log/evm.log:[----] I, [2018-04-26T08:15:45.081816 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log:[----] I, [2018-04-26T09:31:55.901642 #22855:1003114]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T07:06:37.947538 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T10:03:32.396084 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T10:53:44.024588 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T16:03:22.523571 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T00:03:10.453688 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T00:08:54.805438 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T01:03:22.924609 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-13T11:04:32.097365 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-13T16:03:50.340595 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-14T03:03:16.128949 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T05:03:18.767163 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T06:22:06.554445 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T10:15:45.217082 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T14:27:36.534127 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T17:06:28.475454 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T17:40:22.690665 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T20:37:10.462000 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-15T01:05:56.213786 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-15T07:13:19.430001 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-15T17:40:58.613919 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T00:24:49.176572 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T00:29:10.419286 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T02:03:31.608100 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T04:15:27.977464 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T05:06:11.243015 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T10:34:45.369918 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T15:07:27.730737 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T18:04:51.352175 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T23:06:32.298454 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T10:05:27.529263 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T10:11:29.310354 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:07:03.320680 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:39:24.695533 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:49:28.365164 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T18:06:59.389195 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T19:14:03.891355 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T22:05:36.043867 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T22:09:47.130236 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T23:06:34.023440 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-18T02:06:47.189212 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-18T02:55:35.919773 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T06:06:30.823953 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T11:13:57.974350 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T17:14:41.510703 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T19:05:53.091337 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-19T00:07:18.627222 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T04:07:05.010415 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T07:03:39.864901 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:07:22.722948 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:35:23.529910 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:41:30.568776 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T17:07:00.854557 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T17:30:10.004357 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:07:02.491231 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:19:23.113992 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:51:24.248398 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T22:07:55.117517 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-20T00:29:31.462352 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-20T00:34:48.435116 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T04:47:55.316806 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T05:06:55.801442 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T07:07:02.336026 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T08:18:19.278976 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T11:05:56.245746 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T13:04:14.939428 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T20:07:59.518502 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T20:13:01.493215 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T22:07:18.756687 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-21T00:06:59.645577 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T06:12:06.152813 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T14:07:26.102897 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T21:15:04.201769 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T05:07:02.862123 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T05:12:07.323844 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T07:03:48.755312 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T08:03:26.230827 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T10:03:48.882080 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T14:05:41.787696 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-23T02:04:14.664285 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T08:06:24.428734 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T08:16:42.520593 #3746:f9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T18:04:28.997683 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T05:30:22.393354 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T14:06:28.030038 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T18:05:04.865990 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T22:06:09.931356 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-25T02:03:13.031663 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T10:15:27.290712 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T15:07:17.911718 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T15:24:25.059555 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T22:03:55.798580 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-26T01:03:26.714943 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]

Comment 7 Joe Rafaniello 2018-04-27 22:08:13 UTC

Niladri,

Can you check the appliances to make sure their time is in sync?  From just the one server's logs, it looks like each server is mark each other as not responding.  When we deactivate the ALL of the roles that need apache, (user_interface, web_services, websocket), we stop apache.  If we then "come back to life" and are responding and start these roles, we'd restart apache.

So, the role problem is why are we frequently marking servers as not responding.

It might be worthwhile checking other server's logs, looking for the mark_as_not_responding message I have grepped above.  Then, they should verify the time on each appliance is correct.

Comment 8 Joe Rafaniello 2018-04-27 22:09:05 UTC

Note, this is the code where we stop and start apache based on role changes:

https://github.com/ManageIQ/manageiq/blob/f8e4a47560df6af862e0b12a97f6caf364dfd09b/app/models/miq_server/worker_management/monitor.rb#L147-L148

Comment 9 Joe Rafaniello 2018-05-03 18:16:12 UTC

Hi Niladri,

Do you have an update based on my comment 7?

Comment 14 Joe Rafaniello 2018-05-08 21:01:10 UTC

So, the prior comment shows we're marking other servers as not responding still. 
If you look at just the times, they all appear to be happening at 3-5 minutes past each hour:

2018-05-07T07:03:10
2018-05-07T07:03:51
2018-05-07T08:03:02
2018-05-07T08:03:36
2018-05-07T08:04:22
2018-05-07T09:03:07
2018-05-07T10:03:38
2018-05-07T10:04:11
2018-05-07T10:04:48
2018-05-07T10:05:21
2018-05-07T11:03:19
2018-05-07T11:03:55
2018-05-07T11:04:33
2018-05-07T11:05:12
2018-05-07T11:05:37
2018-05-07T11:05:50
2018-05-07T12:03:11
2018-05-07T12:03:58

Do they have something running each hour that's possibly changing the time?

Comment 15 Joe Rafaniello 2018-05-14 15:35:59 UTC

Niladri,

Do you have an update on this?  Please see prior comments.

Comment 19 Joe Rafaniello 2018-05-15 16:06:26 UTC

Niladri,

The ems_metrics_processor role for one of the zones above has over 4,000,000+ rows in the miq_queue table. ems_metrics_collector has 37,000+.

At this point, the metrics collecting and processing is so backed up that they'll need to remove these rows and try to figure out why it's backing up after the systems have recovered.

Perhaps there aren't enough metrics worker processes.  We won't know until the table is cleaned up and we're back to normal operations.

Comment 20 Joe Rafaniello 2018-06-01 15:18:14 UTC

Hi Niladri, can you please post an update on this BZ?  Can it be closed?

Comment 21 Niladri Roy 2018-06-04 10:30:17 UTC

Hi Joe,

Cu has cleared out the miq_queue table for roles ems_metrics_collector and ems_metrics_processor, they also performed a full vacuum on the miq_queue table.

They have monitored there environment for over a week
 The httpd service is not restarted anymore on the UI appliance and Role removals does not show up in logs anymore.

Thanks for your help, you may please close this BZ.

Comment 22 Joe Rafaniello 2018-06-04 15:00:58 UTC

Closing since clearing the queue of metrics collecting/processing has prevented the apache service from continually restarting.