Bug 1026836

Summary: Job scheduled does not recover when postgres connection is terminated for a while
Product: [JBoss] JBoss Operations Network Reporter: Libor Zoubek <lzoubek>
Component: Core ServerAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: low Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: ahovsepy, jshaughn, theute
Target Milestone: ER02   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 14:00:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Libor Zoubek 2013-11-05 14:05:50 UTC
Description of problem:

When JON server looses connection ton postgres database for a while (just by restarting it) it goes to state when everything seems to be working, but it cannot schedule quartz job.

This can be observed for example when you deploy a bundle. Having server in such broken state means that bundle overall deployment status will stay IN_PROGRESS forever, because JON was unable to schedule job fixing this state based on agent's bundle deployment results.


Version-Release number of selected component (if applicable):
JON 3.2.ER4

How reproducible: always


Steps to Reproduce:
1. run JON server
2. restart database backend


Actual results: quartz does not recover from closed db connection and keeps throwing (when there is an attempt to schedule someting):

releaseTriggerRetryLoop: RuntimeException null: java.lang.reflect.UndeclaredThrowableException
        at $Proxy149.rollback(Unknown Source)   at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3601) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3764) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3722) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.releaseAcquiredTrigger(JobStoreSupport.java:2805) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.core.QuartzSchedulerThread.releaseTriggerRetryLoop(QuartzSchedulerThread.java:534) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:354) [quartz-1.6.5.jar:1.6.5]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71) [quartz-1.6.5.jar:1.6.5]
        ... 7 more
Caused by: org.postgresql.util.PSQLException: This connection has been closed.



Expected results: JON server should be fully-working when DB connection is restored.


Additional info:

Comment 2 Jay Shaughnessy 2014-09-04 20:43:35 UTC
This looks like a duplicate of Bug 1095016.  I'm setting to ON_QA in case you think it needs validation but you can likely just close as a duplicate.