Bug 613081

Tested with one more platform alert, the alert details are as given below:

If Condition: System Load > -1.0% 
Dampening Rule: Each time condition set is true 
Action Filters: Disable alert until re-enabled manually or by recovery alert  : true

Rajan
Can you reproduce this on a non-aix environment?

This issue only found in aix environment.

It would be helpful to have debug server logs prior to the issue occurring. 
Can you restart the server, turn on debug logging and then trigger the send email failure and attach a new set of logs.
Thanks

Retest same issue on AIX box with clean JON installation (send mail is running on AIX box).

Set RHQ_SERVER_DEBUG=true and modified jboss-log4j.xml file to set debug mode.

Crated 3 platform alerts (email notification):
1) Name:  	Rajan_EmailAlert_test_platform_sysload
If Condition:  	 System Load > -1.0%
Dampening Rule: 	Each time condition set is true
Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
Sender Info:  	1 notifications exist for this alert definition

2) Name:  	Rajan_EmailAlert_test_platform_total_mem
If Condition:  	 Total Memory > 0.1B
Dampening Rule: 	Each time condition set is true
Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
Sender Info:  	1 notifications exist for this alert definition

3) Name:  	Rajan_EmailAlert_test_platform_mem
If Condition:  	 Free Memory > 0.1B
Dampening Rule: 	Each time condition set is true
Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
Sender Info: 	1 notifications exist for this alert definition


Exception in server log:
2010-07-15 00:56:34,601 WARN  [org.hibernate.hql.ast.QueryTranslatorImpl] firstResult/maxResults specified with collection fetch; applying in memory!
2010-07-15 00:56:34,646 INFO  [STDOUT] SerialUtility.prepare [SubjectManager.findSubjectsByCriteria] Detached in: 16ms, Size is: 7999
2010-07-15 00:56:35,380 INFO  [STDOUT] SerialUtility.prepare [AlertService.findAlertsByCriteria] Detached in: 0ms, Size is: 400
2010-07-15 00:56:35,456 INFO  [STDOUT] SerialUtility.prepare [findBundlesByCriteria] Detached in: 0ms, Size is: 400
2010-07-15 00:56:35,720 INFO  [STDOUT] SerialUtility.prepare [findBundlesWithLatestVersionCompositesByCriteria] Detached in: 0ms, Size is: 476
2010-07-15 00:57:35,460 INFO  [STDOUT] SerialUtility.prepare [AlertService.findAlertsByCriteria] Detached in: 1ms, Size is: 400
2010-07-15 00:57:37,174 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id a106048:b846:4c3eb8e1:2ae7 invoked while multiple threads active within it.
2010-07-15 00:57:37,178 INFO  [org.rhq.enterprise.server.common.TransactionInterruptInterceptor] Transaction [a106048:b846:4c3eb8e1:2ae7] is aborting with active thread [WorkManager(2)-4]. interrupting=[false]
java.lang.Throwable: STACK TRACE OF ACTIVE THREAD IN TERMINATING TRANSACTION
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:140)
	at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:97)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:248)
	at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
	at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:1440)
	at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1022)
	at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
	at javax.mail.Transport.send0(Transport.java:169)
	at javax.mail.Transport.send(Transport.java:120)
	at org.rhq.enterprise.server.core.EmailManagerBean.sendEmail(EmailManagerBean.java:121)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       ...

Created attachment 432068 [details]
server log

Rajan, can you confirm if you associate multiple notifications with an alert (where one notification is email) that the others (say operations or snmp) still work when the email one fails? What we're testing here is that a failure of one notification doesn't impact any of the others.

Retested alert on JON 2.4 (build #73) for AXI platform 
version: 2.4.0.GA_QA build number:10868:d9790b0

The alert notification like operations, system roles, system users are working fine.

The observation is alert email notification is not working and gives following error in server log.

2010-07-22 03:58:09,916 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id a106048:d79a:4c480989:9518 invoked while multiple threads active within it.
2010-07-22 03:58:09,917 INFO  [org.rhq.enterprise.server.common.TransactionInterruptInterceptor] Transaction [a106048:d79a:4c480989:9518] is aborting with active thread [WorkManager(2)-19]. interrupting=[false]
java.lang.Throwable: STACK TRACE OF ACTIVE THREAD IN TERMINATING TRANSACTION
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:173)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:758)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:789)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1125)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:818)
	at org.rhq.enterprise.server.util.concurrent.AlertSerializer.lock(AlertSerializer.java:59)
	at org.rhq.enterprise.server.alert.engine.jms.AlertConditionConsumerBean.onMessage(AlertConditionConsumerBean.java:87)
     ...

Triaged 24-Sept

out of date