Bug 1370248

Summary:	CFME 5.6.0.13 stops processing Service provisoning requests and seems to be not dequeuing work from miq_queue table
Product:	Red Hat CloudForms Management Engine	Reporter:	Thomas Hennessy <thenness>
Component:	Appliance	Assignee:	Gregg Tanzillo <gtanzill>
Status:	CLOSED NOTABUG	QA Contact:	Dave Johnson <dajohnso>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	5.6.0	CC:	abellott, jhardy, obarenbo
Target Milestone:	GA
Target Release:	5.7.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-31 13:48:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Thomas Hennessy 2016-08-25 17:46:22 UTC

Description of problem:Service Template provisioning was working in POC environment and then stopped working.  Also noticing that all sorts of work that should be happening is not: appliance startup updates every 5 minutes, updates to message counts by queue and status... log_status... very odd.


Version-Release number of selected component (if applicable):


How reproducible:unknown



Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Gregg Tanzillo 2016-08-31 13:48:09 UTC

We worked through this issue on a session with the customer. We think that an event storm occurred as a result of the updates that were made to the customer's vSphere env.

We cleared the queue of ~170k "automate" and "notifications" messages which accounted for almost all of the messages in the queue. 

After doing that, the customer was able to successfully provision.