Bug 861979

Summary: when alert notification fails, it appears alert doesn't even get saved in history
Product: [JBoss] JBoss Operations Network Reporter: John Mazzitelli <mazz>
Component: Monitoring - AlertsAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED WORKSFORME QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: JON 3.1.0CC: hrupp, lkrejci, myarboro
Target Milestone: ---   
Target Release: JON 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-05 06:14:20 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description John Mazzitelli 2012-10-01 09:47:36 EDT
Set up an alert with an alert notification (I was on the AS7 branch with a CLI script notification, which isn't working yet, so I expected the notification to fail)

Here's what I think is happening, in a general sense:

1)  an alert's conditions go true - RHQ wants to send an alert
2) the alert definition has a notification definition 
3) RHQ tries to send the notification, but "some bad" error occurs (I don't know of a good way to replicate this - in my case, I was in AS7 branch with CLI script notification that is known to fail because that stuff doesn't yet work in the AS7 branch)
4) it appears RHQ, on failure of notification, fails to then even add a row to RHQ_ALERT (so you don't even see the alert in the Alert history subtab)

4 is a bug, if its true that this is what happened
Comment 1 Charles Crouch 2012-10-01 11:25:06 EDT
Targeting this for investigation in JON312
Comment 2 Lukas Krejci 2013-02-05 06:14:20 EST
I could not reproduce this in RHQ 4.6.0-SNAPSHOT. CLI alerts seem to work OK now that the port to AS7 is complete.

Also, looking at the code, the firing of notifications is guarded by a catch of Throwable (which is logged and saved as the notification result). Also a mere failure to save a notification result is just logged and doesn't cause the overall process of alert firing to fail so I am sceptical that the CLI notifications in and of themselves were the sole cause of the perceived failure.

I am closing this as "works for me". If someone ever finds repro steps, I am all ears to hear them as this would be quite serious.