Bug 861979

Summary:	when alert notification fails, it appears alert doesn't even get saved in history
Product:	[JBoss] JBoss Operations Network	Reporter:	John Mazzitelli <mazz>
Component:	Monitoring - Alerts	Assignee:	RHQ Project Maintainer <rhq-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Mike Foley <mfoley>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	JON 3.1.0	CC:	hrupp, lkrejci, myarboro
Target Milestone:	---
Target Release:	JON 3.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-02-05 11:14:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Mazzitelli 2012-10-01 13:47:36 UTC

Set up an alert with an alert notification (I was on the AS7 branch with a CLI script notification, which isn't working yet, so I expected the notification to fail)

Here's what I think is happening, in a general sense:

1)  an alert's conditions go true - RHQ wants to send an alert
2) the alert definition has a notification definition 
3) RHQ tries to send the notification, but "some bad" error occurs (I don't know of a good way to replicate this - in my case, I was in AS7 branch with CLI script notification that is known to fail because that stuff doesn't yet work in the AS7 branch)
4) it appears RHQ, on failure of notification, fails to then even add a row to RHQ_ALERT (so you don't even see the alert in the Alert history subtab)

4 is a bug, if its true that this is what happened

Comment 1 Charles Crouch 2012-10-01 15:25:06 UTC

Targeting this for investigation in JON312

Comment 2 Lukas Krejci 2013-02-05 11:14:20 UTC

I could not reproduce this in RHQ 4.6.0-SNAPSHOT. CLI alerts seem to work OK now that the port to AS7 is complete.

Also, looking at the code, the firing of notifications is guarded by a catch of Throwable (which is logged and saved as the notification result). Also a mere failure to save a notification result is just logged and doesn't cause the overall process of alert firing to fail so I am sceptical that the CLI notifications in and of themselves were the sole cause of the perceived failure.

I am closing this as "works for me". If someone ever finds repro steps, I am all ears to hear them as this would be quite serious.