Description of problem:

When JON server looses connection ton postgres database for a while (just by restarting it) it goes to state when everything seems to be working, but it cannot schedule quartz job.

This can be observed for example when you deploy a bundle. Having server in such broken state means that bundle overall deployment status will stay IN_PROGRESS forever, because JON was unable to schedule job fixing this state based on agent's bundle deployment results.


Version-Release number of selected component (if applicable):
JON 3.2.ER4

How reproducible: always


Steps to Reproduce:
1. run JON server
2. restart database backend


Actual results: quartz does not recover from closed db connection and keeps throwing (when there is an attempt to schedule someting):

releaseTriggerRetryLoop: RuntimeException null: java.lang.reflect.UndeclaredThrowableException
        at $Proxy149.rollback(Unknown Source)   at org.quartz.impl.jdbcjobstore.JobStoreSupport.rollbackConnection(JobStoreSupport.java:3601) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3764) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3722) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.releaseAcquiredTrigger(JobStoreSupport.java:2805) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.core.QuartzSchedulerThread.releaseTriggerRetryLoop(QuartzSchedulerThread.java:534) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:354) [quartz-1.6.5.jar:1.6.5]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source) [:1.7.0_09-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea]
        at org.quartz.impl.jdbcjobstore.AttributeRestoringConnectionInvocationHandler.invoke(AttributeRestoringConnectionInvocationHandler.java:71) [quartz-1.6.5.jar:1.6.5]
        ... 7 more
Caused by: org.postgresql.util.PSQLException: This connection has been closed.



Expected results: JON server should be fully-working when DB connection is restored.


Additional info: