Created attachment 916892 [details]
/var/log/activemq/wrapper.log for the failed activemq
Description of problem:
I don't know if this is worth investigating, just going to note what happened. I have an OSE 2.1 installation on our internal OpenStack instance. I left it idle for a few days and when I returned, the activemq instance was no longer running.
Probably not very
Steps to Reproduce:
1. Install OSE 2.1 (I have separate broker + node)
2. Keep it very busy for a few days with creating/removing gears
3. Leave it idle for a few days
INFO | jvm 2 | 2014/06/30 03:11:17 | Java Runtime: Sun Microsystems Inc. 1.6.0_30 /usr/lib/jvm/java-1.6.0-openjdk-184.108.40.206.x86_64/jre
INFO | jvm 2 | 2014/06/30 03:11:17 | Heap sizes: current=58816k free=54792k max=932096k
INFO | jvm 2 | 2014/06/30 03:11:17 | JVM args: -Dactivemq.home=/usr/share/activemq [...]
INFO | jvm 2 | 2014/06/30 03:11:17 | Extensions classpath:
INFO | jvm 2 | 2014/06/30 03:11:17 | [/usr/share/activemq/lib,/usr/share/activemq/lib/camel,[...]
INFO | jvm 2 | 2014/06/30 03:11:17 | ACTIVEMQ_HOME: /usr/share/activemq
INFO | jvm 2 | 2014/06/30 03:11:17 | ACTIVEMQ_BASE: /usr/share/activemq
INFO | jvm 2 | 2014/06/30 03:11:17 | ACTIVEMQ_CONF: /usr/share/activemq/conf
INFO | jvm 2 | 2014/06/30 03:11:17 | ACTIVEMQ_DATA: /usr/share/activemq/data
INFO | jvm 2 | 2014/06/30 03:11:17 | Loading message broker from: xbean:activemq.xml
ERROR | wrapper | 2014/07/07 11:11:22 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2014/07/07 11:11:22 | JVM did not exit on request, terminated
STATUS | wrapper | 2014/07/07 11:11:25 | JVM exited in response to signal SIGKILL (9).
ERROR | wrapper | 2014/07/07 11:11:25 | Unable to start a JVM
STATUS | wrapper | 2014/07/07 11:11:25 | <-- Wrapper Stopped
2014-07-03 14:58:38,927 | INFO | mcollective.reply.broker.hosts.ose211.example.com_3128 Inactive for longer than 300000 ms - removing ... | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[broker.hosts.ose211.example.com] Scheduler
2014-07-07 11:08:35,233 | WARN | Transport Connection to: tcp://[...]:20147 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30500) long: tcp://172.16.4.139:20147 | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker
2014-07-07 11:09:34,675 | WARN | Transport Connection to: tcp://[...]:24464 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30500) long: tcp://172.16.4.137:24464 | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker
(There's nothing more of interest - just showing that the last activity was on 07/03, then an exception on 07/07, and that's the end of the log. It's representative of the rest of the log.)
ActiveMQ should not go down inexplicably. Even if it does, the wrapper should be able to revive it.
I'll attach the wrapper.log which has some other interesting error messages in it. It seems this actually not uncommon (perhaps our OpenStack deployment does not provide consistent access to resources, perhaps the mostly-idle instances are getting swapped out some way, resulting in unexpectedly long delays in responses to monitoring signals?), and what's unusual here is that the wrapper wasn't able to recover.
Particularly note -
INFO | jvm 1 | 2014/06/16 13:21:02 | WARNING - Unable to load the Wrapper's native library [...]
INFO | jvm 1 | 2014/06/16 13:21:02 | System signals will not be handled correctly.
Starting ActiveMQ manually after discovering this - worked fine, naturally.
OpenShift Enterprise v2 has officially reached EoL. This product is no longer supported and bugs will be closed.
Please look into the replacement enterprise-grade container option, OpenShift Container Platform v3. https://www.openshift.com/container-platform/
More information can be found here: https://access.redhat.com/support/policy/updates/openshift/