Bug 1293350

Summary: Data Calc Job fails to complete when JBoss ON Server is set to DEBUG mode
Product: [JBoss] JBoss Operations Network Reporter: bkramer <bkramer>
Component: Core ServerAssignee: Michael Burman <miburman>
Status: CLOSED ERRATA QA Contact: Sunil Kondkar <skondkar>
Severity: urgent Docs Contact:
Priority: high    
Version: JON 3.3.4CC: fbrychta, loleary, mfoley, skondkar, spinder
Target Milestone: ER01Keywords: Triaged
Target Release: JON 3.3.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-03 15:04:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1293352    
Bug Blocks:    

Description bkramer 2015-12-21 14:06:59 UTC
Description of problem:

Every hour, when JBoss ON server is set in DEBUG mode, the following is logged in the server.log file:
*************************************************************
18:00:08,613 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-1) JBAS014134: EJB Invocation failed on component MeasurementOOBManagerBean for method public abstract void org.rhq.enterprise.server.measurement.MeasurementOOBManagerLocal.computeOOBsForLastHour(org.rhq.core.domain.auth.Subject,java.lang.Iterable): javax.ejb.EJBException: java.lang.RuntimeException: javax.ejb.EJBException: org.hibernate.LazyInitializationException: could not initialize proxy - no Session
...
	at org.rhq.enterprise.server.measurement.MeasurementOOBManagerLocal$$$view160.computeOOBsForLastHour(Unknown Source) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
	at org.rhq.enterprise.server.scheduler.jobs.DataCalcJob.calculateOOBs(DataCalcJob.java:150) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
	at org.rhq.enterprise.server.scheduler.jobs.DataCalcJob.executeJobCode(DataCalcJob.java:85) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
	at org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob.execute(AbstractStatefulJob.java:48) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.5.jar:1.6.5]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) [quartz-1.6.5.jar:1.6.5]
Caused by: java.lang.RuntimeException: javax.ejb.EJBException: org.hibernate.LazyInitializationException: could not initialize proxy - no Session
	at org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean.computeOOBsForLastHour(MeasurementOOBManagerBean.java:253) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
...
	... 29 more
Caused by: javax.ejb.EJBException: org.hibernate.LazyInitializationException: could not initialize proxy - no Session

...

18:00:08,621 ERROR [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-1) Data Calc Job FAILED TO COMPLETE. Cause: javax.ejb.EJBException: java.lang.RuntimeException: javax.ejb.EJBException: org.hibernate.LazyInitializationException: could not initialize proxy - no Session
18:00:08,621 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-1) Data Calc Job FINISHED [8591]ms
*************************************************************


Version-Release number of selected component (if applicable):
JBoss ON 3.3.4

How reproducible:
Always

Steps to Reproduce:
1. Set JBoss ON Server into DEBUG mode
2.
3.

Actual results:
Data Calc Job fails to complete with message "could not initialize proxy - no Session"

Expected results:
Data Calc Job finishes with success and no error message is logged.

Additional info:

Jay Shaughnessy already had a look and here is his comment:
***********************************************************
It's a failure of the DataCalcJob which, as you probably know, is a scheduled job that runs to generate baselines and OOBs and stuff like that.  Having said that, this is not a good failure.  I did some research and it looks to me like this is found in both of the logs because both logs were likely performing debug logging.  The exception is actually generated in a debug logging statement (ugh).

I've fixed this issue in the master branch, I would suggest we cherry-pick this to the 3.3.x branch as well, it will benefit future 3.x releases.

Master commit 772b6cdb80f20a9e6e3a7552f2ea185a352c8218
 Author: Jay Shaughnessy <jshaughn>
 Date:   Fri Dec 18 10:19:41 2015 -0500
 
     Never use a lazy loaded variable in toString() impl. This blew up in
     DEBUG mode if the MeasurementBaseline was not a managed entity.
***********************************************************

Comment 1 Simeon Pinder 2016-01-08 21:50:41 UTC
Moving to MODIFIED. Committed to release/jon3.3.x with the following commit:

branch: release/jon3.3.x
commit: 00935c1fae2aae147b57364d2960963c68fdc50f
Author:     Jay Shaughnessy <jshaughn>
AuthorDate: Fri Dec 18 10:19:41 2015 -0500
Commit:     Simeon Pinder <spinder>
CommitDate: Thu Jan 7 09:38:38 2016 -0500

    Never use a lazy loaded variable in toString() impl. This blew up in
    DEBUG mode if the MeasurementBaseline was not a managed entity.

Comment 2 Simeon Pinder 2016-01-16 08:17:36 UTC
Moving to ON_QA for testing with the following build:

https://brewweb.devel.redhat.com//buildinfo?buildID=474795

http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/75/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip
 *Note: jon-server-patch-3.3.0.GA.zip maps to ER01 build of
 jon-server-3.3.0.GA-update-05.zip.

Comment 3 Sunil Kondkar 2016-01-22 08:17:12 UTC
Verified on Version : 3.3.0.GA Update 05 Build Number :	4242930:1c9f74b

Verified that DataCalcJob does not fail and there are no error messages when baselines exist and server is running in debug mode.

Comment 5 errata-xmlrpc 2016-02-03 15:04:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-0118.html