Red Hat Bugzilla – Bug 752110
RHQ server OOM error
Last modified: 2015-02-05 20:19:29 EST
Description of problem:
Left RHQ server 4.2 running over night. Unresponsive in the AM, logs showed the server was out of heap space.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
2.Imported agent running on same machine
3.Configured Apache, Postgres
4. Created drift template, made a change to monitored directory
5. Went away for a long time.
Created attachment 532312 [details]
latest server log
Created attachment 532313 [details]
I have heap dump if helpful, but it's 1.3 GB
I note that you are running on H2. I presume you haven't seen this on PG or
ORA? From the server logs there appear to be a lot of DB related errors prior
to the memory issues, e.g. trying to redeliver a message to the JMS dead letter
queue (dlq) nearly 900,000 times. Also exceptions such as "Caused by:
java.lang.ClassCastException: org.h2.jdbc.JdbcBlob cannot be cast to
org.jboss.mq.SpyMessage" that I'm very suspicious of.
Given the secondary level support we have for H2, below PG and Oracle, I don't
see this a release blocker. Mike confirmed that he frequently runs drift
overnight using Oracle as the DB and has not observed similar issues. I will
make this issue block the drift tracking bug, but set it as medium priority.
Once we're done with major drift related issues for the release, we can see if
we can reproduce this.
that's fair. I'm only this build/database on the laptop that's acting like a desktop. I'll swap out the database to pg and see if it reoccurs.
I guess - secondary to this bz - is what caused the missing queue. Is it possible to simulate a similar JMS dead letter failure using another database?
We used to have huge issues when jms was on hsqldb (the predecessor of h2). While they say h2 is much much better than hsqldb, it may be the same here in the end.