Bug 1028487

Summary:	Recovery alert cache refresh needs to happen prior to alert notification processing
Product:	[Other] RHQ Project	Reporter:	Jay Shaughnessy <jshaughn>
Component:	Alerts	Assignee:	Jay Shaughnessy <jshaughn>
Status:	ON_QA ---	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.9	CC:	hrupp
Target Milestone:	GA
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1030111 (view as bug list)		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1030111

Description Jay Shaughnessy 2013-11-08 15:00:54 UTC

Currently alert notifications are processed in the same transaction as the alert creation, recovery alert activation, etc.  There are a few issues related to the notif processing taking place in this transaction:

1) It simply extends the length of an already complicated transaction, potentially holding locks, delaying the alert commit, delaying the global cache refresh flags being committed (such that HA servers can pick up on it), etc...

2) Notifications can actually initiate recovery actions such as executing resource operations, invoking CLI scripts, etc.  This should not happen prior to our ability to update the global alert condition cache, which must happen to begin condition matching on activated recovery alert definitions.  Otherwise we risk an actual recovery happening prior to the recovery alert being ready.


The alert notification processing should happen outside of the alert creation transaction and after cache refresh.

Comment 1 Jay Shaughnessy 2013-11-08 15:57:06 UTC

master commit 28becd282f8cd3ed4327a56ea0c08f8431845dba
Author: Jay Shaughnessy <jshaughn>
Date:   Fri Nov 8 10:42:46 2013 -0500

Restructure SLSB methods (locals only, no remote changes) to process alert
notifications later in the workflow, after the alert is committed and after
we have a chance to update the condition caches (for more reliable recovery
alerting).  Needed to be able to pass back the new alert through the call chain.