Bug 1652302

Summary:	Embedded Ansible Service Stops Running
Product:	Red Hat CloudForms Management Engine	Reporter:	Tuan <tuado>
Component:	Appliance	Assignee:	Gregg Tanzillo <gtanzill>
Status:	CLOSED DUPLICATE	QA Contact:	Satyajit Bulage <sbulage>
Severity:	urgent	Docs Contact:	Red Hat CloudForms Documentation <cloudforms-docs>
Priority:	urgent
Version:	5.9.3	CC:	abellott, akarol, dmetzger, ncarboni, obarenbo, rspagnol
Target Milestone:	GA
Target Release:	5.9.7
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-27 20:25:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	Bug
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:

Comment 5 Nick Carboni 2018-11-27 20:22:36 UTC

It turns out that this issue was caused by too many entries in the event_streams table. This was causing some queries to take ~12 hours to run. The backend connection to the database would remain open throughout the runtime of the query even though the worker which issued the query was killed.

When the workers issuing these queries were killed they would be restarted and then attempt to run the same query again. These connections associated with long running queries were accumulating until we hit the limit of 1000.

To solve this issue we truncated the event_streams table which allowed the workers to run and the number of connections stabilized.

As a fix, we should allow blacklisting events for openshift (which accounted for the majority of the 28 million rows in the table). But I'm not sure if we should attach that fix to this bug as the description here is not directly related to the fix.

Comment 6 Nick Carboni 2018-11-27 20:25:10 UTC


*** This bug has been marked as a duplicate of bug 1653871 ***