Back to bug 1009981
| Who | When | What | Removed | Added |
|---|---|---|---|---|
| Brian Stansberry | 2013-09-27 15:34:16 UTC | Link ID | JBoss Issue Tracker JBTM-1943 | |
| Depends On | 1012986 | |||
| CC | brian.stansberry | |||
| Brian Stansberry | 2013-10-07 15:31:40 UTC | Status | NEW | POST |
| Target Milestone | --- | Pending | ||
| CC | brian.stansberry | |||
| Brian Stansberry | 2013-10-08 00:23:25 UTC | Status | POST | MODIFIED |
| Target Release | --- | EAP 6.2.0 | ||
| CC | brian.stansberry | |||
| Target Milestone | Pending | ER6 | ||
| Paul Gier | 2013-10-17 00:12:03 UTC | Status | MODIFIED | ON_QA |
| Brian Stansberry | 2013-10-17 01:58:46 UTC | CC | brian.stansberry | |
| Ondrej Chaloupka | 2013-10-29 10:21:47 UTC | Doc Text | Description: When machine is under bigger load of the receiving messages there could happen the server log would contain XAException from Periodic Recovery of rollbacking a transaction. We hit this for HornetQ but this is supposed to happen for any messaging system used under EAP. The transaction is in fact not rollbacked at all. The commit is done but meanwhile the Periodic Recovery process comes into play. The scenario is described by Tom in the comment: https://bugzilla.redhat.com/show_bug.cgi?id=1009981#c6 Cause: The transaction handling and periodic recovery are two independent threads which access a same transaction record. The RAR informs periodic recovery about not commited transaction and tries to recover it but meanwhile the RAR procceed with the commit. Consequence: Log could contains XAException which informs about fail to rollback a transaction by Periodic Recovery. But in fact there was no rollback and all messages were correctly received and commited. Fix: There isn't any generic fix possible being done in Narayana code. When the problem occurs it's recommended to increase interval between the two phases of the recovery process. Use property com.arjuna.ats.jta.orphanSafetyInterval at the start of the EAP server and set it to the bigger number than the 10000 milliseconds which is default value. Result: Use com.arjuna.ats.jta.orphanSafetyInterval property when problem occurs. |
|
| Ondrej Chaloupka | 2013-10-29 14:33:10 UTC | Status | ON_QA | VERIFIED |
| Stephan Vollmer | 2013-10-30 09:02:20 UTC | CC | stephan.vollmer | |
| Russell Dickenson | 2013-11-26 04:33:15 UTC | Doc Text | Description: When machine is under bigger load of the receiving messages there could happen the server log would contain XAException from Periodic Recovery of rollbacking a transaction. We hit this for HornetQ but this is supposed to happen for any messaging system used under EAP. The transaction is in fact not rollbacked at all. The commit is done but meanwhile the Periodic Recovery process comes into play. The scenario is described by Tom in the comment: https://bugzilla.redhat.com/show_bug.cgi?id=1009981#c6 Cause: The transaction handling and periodic recovery are two independent threads which access a same transaction record. The RAR informs periodic recovery about not commited transaction and tries to recover it but meanwhile the RAR procceed with the commit. Consequence: Log could contains XAException which informs about fail to rollback a transaction by Periodic Recovery. But in fact there was no rollback and all messages were correctly received and commited. Fix: There isn't any generic fix possible being done in Narayana code. When the problem occurs it's recommended to increase interval between the two phases of the recovery process. Use property com.arjuna.ats.jta.orphanSafetyInterval at the start of the EAP server and set it to the bigger number than the 10000 milliseconds which is default value. Result: Use com.arjuna.ats.jta.orphanSafetyInterval property when problem occurs. | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continue to completion. At the time the periodic recovery attempts and fails the rollback, it logs the rollback failure in the log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery processby setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. |
| Doc Type | Bug Fix | Known Issue | ||
| Ondrej Chaloupka | 2013-11-26 13:35:46 UTC | Doc Text | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continue to completion. At the time the periodic recovery attempts and fails the rollback, it logs the rollback failure in the log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery processby setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continue to completion. At the time the periodic recovery attempts and fails the rollback, it logs the rollback failure in the log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery process by setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. Please note that this does not solved the issue just decrease probability that it will be hit and that the warning message will be shown in the log. |
| Russell Dickenson | 2013-11-26 21:37:17 UTC | Doc Text | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continue to completion. At the time the periodic recovery attempts and fails the rollback, it logs the rollback failure in the log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery process by setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. Please note that this does not solved the issue just decrease probability that it will be hit and that the warning message will be shown in the log. | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continues to completion. At the time the periodic recovery attempts but fails the rollback, it records the rollback failure in the server log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery process by setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. Please note that this does not solve the issue, instead it decreases the probability that it will occur and that the warning message will be shown in the log. |
| Russell Dickenson | 2013-11-28 02:12:40 UTC | Doc Text | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---------- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ------------- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continues to completion. At the time the periodic recovery attempts but fails the rollback, it records the rollback failure in the server log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery process by setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. Please note that this does not solve the issue, instead it decreases the probability that it will occur and that the warning message will be shown in the log. | When the server is under an excessive load, the server's log may contain messages like those below, indicating that a transaction rollback had occurred. ---- 17:20:00,441 ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1692) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:494) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.rollback(HornetQXAResourceWrapper.java:126) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:738) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:644) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:417) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) 17:20:00,441 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) ---- Under heavy load, the processing time taken by a transaction can overlap with the timing of the periodic recovery process's activity. The periodic recovery process detects the transaction still in progress and attempts to initiate a rollback but in fact the transaction continues to completion. At the time the periodic recovery attempts but fails the rollback, it records the rollback failure in the server log. The underlying cause of this issue will be addressed in a future release, but in the meantime a workaround is available. Increase the interval between the two phases of the recovery process by setting the property `com.arjuna.ats.jta.orphanSafetyInterval` to higher than the default of 10000 milliseconds. A value of 40000 milliseconds is recommended. Please note that this does not solve the issue, instead it decreases the probability that it will occur and that the warning message will be shown in the log. |
| Tom Ross | 2013-12-11 14:59:22 UTC | Blocks | 1040521 | |
| mark yarborough | 2013-12-15 16:19:41 UTC | Status | VERIFIED | CLOSED |
| Resolution | --- | CURRENTRELEASE | ||
| Last Closed | 2013-12-15 11:19:41 UTC | |||
| Miroslav Novak | 2014-11-14 09:46:41 UTC | See Also | https://bugzilla.redhat.com/show_bug.cgi?id=1164184 | |
| Russell Dickenson | 2017-10-10 00:11:53 UTC | Link ID | JBoss Issue Tracker JBTM-2583 | |
| Docs Contact | rdickens |
Back to bug 1009981