Description of problem: I am not able to install latest RHQ 4.8 master build on following environment: java -version java version "1.7.0" Java(TM) SE Runtime Environment (build pxi3270sr4ifix-20130305_01(SR4+IV37419)) IBM J9 VM (build 2.6, JRE 1.7.0 Linux x86-32 20130205_137358 (JIT enabled, AOT enabled) J9VM - R26_Java726_SR4_20130205_1656_B137358 JIT - r11.b03_20130131_32403 GC - R26_Java726_SR4_20130205_1656_B137358 J9CL - 20130205_137358) JCL - 20130303_01 based on Oracle 7u13-b08 uname -a Linux lzoubek1.bc.jonqe.lab.eng.bos.redhat.com 2.6.32-358.el6.i686 #1 SMP Tue Jan 29 11:48:01 EST 2013 i686 i686 i386 GNU/Linux How reproducible:always Actual results: 04:41:44,402 ERROR [org.rhq.storage.installer.StorageInstaller] The storage node reported the following errors while trying to start: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:613) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:236) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.<clinit>(SnappyCompressor.java:37) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:236) at org.apache.cassandra.config.CFMetaData.<clinit>(CFMetaData.java:82) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:236) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:81) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:471) at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:123) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:236) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:192) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:411) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:454) Caused by: java.lang.UnsatisfiedLinkError: snappyjava (Not found in java.library.path) at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1197) at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1161) at java.lang.System.loadLibrary(System.java:527) at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52) ... 25 more 04:41:44,411 ERROR [org.rhq.storage.installer.StorageInstaller] Please review your configuration for possible sources of errors such as port conflicts or invalid arguments/options passed to the java executable.
same issue happens on : java -version java version "1.6.0" Java(TM) SE Runtime Environment (build pxi3260sr13ifix-20130303_02(SR13+IV37419)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr13-20130114_134867 (JIT enabled, AOT enabled) J9VM - 20130114_134867 JIT - r9_20130108_31100 GC - 20121212_AA) JCL - 20130303_02
When Cassandra starts up it tries to load the native library for snappy-java. If snappy-java cannot load the native library, the snappy loader class (I forget the exact name) calls e.printStackTrace() in the catch block. I believe that is what you are seeing there. I will update the storage installer to suppress this output and instead just log a warning message.
You can view the code that produces that exception here - https://gist.github.com/jsanda/5735367. At start up Cassandra will try to load the native snappy-java library. If the library cannot be loaded, SnappyNativeLoader winds up making the e.printStackTrace() call. The storage installer is already configured to redirect stderr from Cassandra, and I want to add some logic in the installer that checks the stderr stream for snappy errors so that we can possibly log something useful for the user and still return a successful exit value from the installer. The problem is that the redirection is not always capturing the full stderr stream which makes it difficult to check for the error. The snappy exception causes the installer to exit with an error code which will cause rhqctl to abort execution. We do not want that to happen because Cassandra is still starting up fine. An alternative might be to check the storage node log file, assuming there are no other errors that prevent start up. If the node starts up and does not support compression, it will log the following, WARN [main] 2013-06-07 22:11:16,217 SnappyCompressor.java (line 38) Cannot initialize native Snappy library. Compression on new tables will be disabled.
master c8221f54f catches this case and allows the install to continue. We still need a better fix though, as rhqctl start ( --storage ) also prints that message (but does not abort)
We no include the snappy native library with our Cassandra bits. I am moving this to ON_QA for verification.