Bug 1572615 - apache keeps restarting
Summary: apache keeps restarting
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: cfme-future
Assignee: Joe Rafaniello
QA Contact: Dave Johnson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-27 12:22 UTC by Niladri Roy
Modified: 2018-06-05 13:58 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-04 15:00:58 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)

Comment 4 Joe Rafaniello 2018-04-27 21:41:32 UTC
I don't think it's causing problems but I opened a BZ to fix the frequent warnings seen above:
AH01574: module ssl_module is already loaded, skipping
https://bugzilla.redhat.com/show_bug.cgi?id=1572793

Comment 5 Joe Rafaniello 2018-04-27 21:56:47 UTC
It looks like the user_interface, web_services, websocket roles (along with other roles) are frequently being removed, if this happens, it's likely we're stopping apache if all 3 of those above roles are no longer active and presumably are restarting it when they are added back.... so it might be a problem with roles going down and back up.

$ zgrep -E "log_role_changes.+Roles removed.+(user_interface|web_services|websocket)" log/evm.log*
log/evm.log:[----] I, [2018-04-26T08:15:45.081816 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log:[----] I, [2018-04-26T09:31:55.901642 #22855:1003114]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T07:06:37.947538 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T10:03:32.396084 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T10:53:44.024588 #2525:ed312c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-12T16:03:22.523571 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T00:03:10.453688 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T00:08:54.805438 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180413.gz:[----] I, [2018-04-13T01:03:22.924609 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-13T11:04:32.097365 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-13T16:03:50.340595 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180414.gz:[----] I, [2018-04-14T03:03:16.128949 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T05:03:18.767163 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T06:22:06.554445 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T10:15:45.217082 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T14:27:36.534127 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T17:06:28.475454 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T17:40:22.690665 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-14T20:37:10.462000 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180415.gz:[----] I, [2018-04-15T01:05:56.213786 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-15T07:13:19.430001 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-15T17:40:58.613919 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T00:24:49.176572 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T00:29:10.419286 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180416.gz:[----] I, [2018-04-16T02:03:31.608100 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T04:15:27.977464 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T05:06:11.243015 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T10:34:45.369918 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T15:07:27.730737 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T18:04:51.352175 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180417.gz:[----] I, [2018-04-16T23:06:32.298454 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T10:05:27.529263 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T10:11:29.310354 #2526:11b113c]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:07:03.320680 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:39:24.695533 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T17:49:28.365164 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T18:06:59.389195 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T19:14:03.891355 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T22:05:36.043867 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T22:09:47.130236 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-17T23:06:34.023440 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-18T02:06:47.189212 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180418.gz:[----] I, [2018-04-18T02:55:35.919773 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T06:06:30.823953 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T11:13:57.974350 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T17:14:41.510703 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-18T19:05:53.091337 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180419.gz:[----] I, [2018-04-19T00:07:18.627222 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T04:07:05.010415 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T07:03:39.864901 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:07:22.722948 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:35:23.529910 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T15:41:30.568776 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T17:07:00.854557 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T17:30:10.004357 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:07:02.491231 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:19:23.113992 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T21:51:24.248398 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-19T22:07:55.117517 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-20T00:29:31.462352 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180420.gz:[----] I, [2018-04-20T00:34:48.435116 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T04:47:55.316806 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T05:06:55.801442 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T07:07:02.336026 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T08:18:19.278976 #16991:4fd134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T11:05:56.245746 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T13:04:14.939428 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T20:07:59.518502 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T20:13:01.493215 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-20T22:07:18.756687 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180421.gz:[----] I, [2018-04-21T00:06:59.645577 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T06:12:06.152813 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T14:07:26.102897 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180422.gz:[----] I, [2018-04-21T21:15:04.201769 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T05:07:02.862123 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T05:12:07.323844 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T07:03:48.755312 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T08:03:26.230827 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T10:03:48.882080 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-22T14:05:41.787696 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180423.gz:[----] I, [2018-04-23T02:04:14.664285 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T08:06:24.428734 #25814:a9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T08:16:42.520593 #3746:f9b140]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180424.gz:[----] I, [2018-04-23T18:04:28.997683 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T05:30:22.393354 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T14:06:28.030038 #2564:647134]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_metrics_collector", "ems_metrics_processor", "ems_operations", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T18:05:04.865990 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-24T22:06:09.931356 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180425.gz:[----] I, [2018-04-25T02:03:13.031663 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T10:15:27.290712 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T15:07:17.911718 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T15:24:25.059555 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-25T22:03:55.798580 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]
log/evm.log-20180426.gz:[----] I, [2018-04-26T01:03:26.714943 #22693:835108]  INFO -- : MIQ(MiqServer#log_role_changes)   Roles removed:   ["automate", "database_operations", "ems_inventory", "ems_metrics_collector", "ems_metrics_coordinator", "ems_metrics_processor", "ems_operations", "event", "reporting", "user_interface", "web_services", "websocket"]

Comment 7 Joe Rafaniello 2018-04-27 22:08:13 UTC
Niladri,

Can you check the appliances to make sure their time is in sync?  From just the one server's logs, it looks like each server is mark each other as not responding.  When we deactivate the ALL of the roles that need apache, (user_interface, web_services, websocket), we stop apache.  If we then "come back to life" and are responding and start these roles, we'd restart apache.

So, the role problem is why are we frequently marking servers as not responding.

It might be worthwhile checking other server's logs, looking for the mark_as_not_responding message I have grepped above.  Then, they should verify the time on each appliance is correct.

Comment 8 Joe Rafaniello 2018-04-27 22:09:05 UTC
Note, this is the code where we stop and start apache based on role changes:

https://github.com/ManageIQ/manageiq/blob/f8e4a47560df6af862e0b12a97f6caf364dfd09b/app/models/miq_server/worker_management/monitor.rb#L147-L148

Comment 9 Joe Rafaniello 2018-05-03 18:16:12 UTC
Hi Niladri,

Do you have an update based on my comment 7?

Comment 14 Joe Rafaniello 2018-05-08 21:01:10 UTC
So, the prior comment shows we're marking other servers as not responding still. 
If you look at just the times, they all appear to be happening at 3-5 minutes past each hour:

2018-05-07T07:03:10
2018-05-07T07:03:51
2018-05-07T08:03:02
2018-05-07T08:03:36
2018-05-07T08:04:22
2018-05-07T09:03:07
2018-05-07T10:03:38
2018-05-07T10:04:11
2018-05-07T10:04:48
2018-05-07T10:05:21
2018-05-07T11:03:19
2018-05-07T11:03:55
2018-05-07T11:04:33
2018-05-07T11:05:12
2018-05-07T11:05:37
2018-05-07T11:05:50
2018-05-07T12:03:11
2018-05-07T12:03:58

Do they have something running each hour that's possibly changing the time?

Comment 15 Joe Rafaniello 2018-05-14 15:35:59 UTC
Niladri,

Do you have an update on this?  Please see prior comments.

Comment 19 Joe Rafaniello 2018-05-15 16:06:26 UTC
Niladri,

The ems_metrics_processor role for one of the zones above has over 4,000,000+ rows in the miq_queue table. ems_metrics_collector has 37,000+.

At this point, the metrics collecting and processing is so backed up that they'll need to remove these rows and try to figure out why it's backing up after the systems have recovered.

Perhaps there aren't enough metrics worker processes.  We won't know until the table is cleaned up and we're back to normal operations.

Comment 20 Joe Rafaniello 2018-06-01 15:18:14 UTC
Hi Niladri, can you please post an update on this BZ?  Can it be closed?

Comment 21 Niladri Roy 2018-06-04 10:30:17 UTC
Hi Joe,

Cu has cleared out the miq_queue table for roles ems_metrics_collector and ems_metrics_processor, they also performed a full vacuum on the miq_queue table.

They have monitored there environment for over a week
 The httpd service is not restarted anymore on the UI appliance and Role removals does not show up in logs anymore.

Thanks for your help, you may please close this BZ.

Comment 22 Joe Rafaniello 2018-06-04 15:00:58 UTC
Closing since clearing the queue of metrics collecting/processing has prevented the apache service from continually restarting.


Note You need to log in before you can comment on or make changes to this bug.