Bug 971772 - Cassandra node Installation fails on IBM JRE 1.7 i686 on snappy
Summary: Cassandra node Installation fails on IBM JRE 1.7 i686 on snappy
Status: ON_QA
Alias: None
Product: RHQ Project
Classification: Other
Component: Database, Installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Nobody
QA Contact:
Depends On:
Blocks: 951619
TreeView+ depends on / blocked
Reported: 2013-06-07 08:59 UTC by Libor Zoubek
Modified: 2022-03-31 04:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed:

Attachments (Terms of Use)

Description Libor Zoubek 2013-06-07 08:59:48 UTC
Description of problem: I am not able to install latest RHQ 4.8 master build on following environment:

java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxi3270sr4ifix-20130305_01(SR4+IV37419))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux x86-32 20130205_137358 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR4_20130205_1656_B137358
JIT  - r11.b03_20130131_32403
GC   - R26_Java726_SR4_20130205_1656_B137358
J9CL - 20130205_137358)
JCL - 20130303_01 based on Oracle 7u13-b08

uname -a
Linux lzoubek1.bc.jonqe.lab.eng.bos.redhat.com 2.6.32-358.el6.i686 #1 SMP Tue Jan 29 11:48:01 EST 2013 i686 i686 i386 GNU/Linux

How reproducible:always

Actual results:
04:41:44,402 ERROR [org.rhq.storage.installer.StorageInstaller] The storage node reported the following errors while trying to start:

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:613)
        at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
        at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
        at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45)
        at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55)
        at org.apache.cassandra.io.compress.SnappyCompressor.<clinit>(SnappyCompressor.java:37)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.config.CFMetaData.<clinit>(CFMetaData.java:82)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:81)
        at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:471)
        at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:123)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:192)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:411)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:454)
Caused by: java.lang.UnsatisfiedLinkError: snappyjava (Not found in java.library.path)
        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1197)
        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1161)
        at java.lang.System.loadLibrary(System.java:527)
        at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
        ... 25 more

04:41:44,411 ERROR [org.rhq.storage.installer.StorageInstaller] Please review your configuration for possible sources of errors such as port conflicts or invalid arguments/options passed to the java executable.

Comment 1 Libor Zoubek 2013-06-07 12:37:44 UTC
same issue happens on :

java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxi3260sr13ifix-20130303_02(SR13+IV37419))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr13-20130114_134867 (JIT enabled, AOT enabled)
J9VM - 20130114_134867
JIT  - r9_20130108_31100
GC   - 20121212_AA)
JCL  - 20130303_02

Comment 2 John Sanda 2013-06-07 15:59:18 UTC
When Cassandra starts up it tries to load the native library for snappy-java. If snappy-java cannot load the native library, the snappy loader class (I forget the exact name) calls e.printStackTrace() in the catch block. I believe that is what you are seeing there. I will update the storage installer to suppress this output and instead just log a warning message.

Comment 3 John Sanda 2013-06-08 15:00:12 UTC
You can view the code that produces that exception here - https://gist.github.com/jsanda/5735367. At start up Cassandra will try to load the native snappy-java library. If the library cannot be loaded, SnappyNativeLoader winds up making the e.printStackTrace() call.

The storage installer is already configured to redirect stderr from Cassandra, and I want to add some logic in the installer that checks the stderr stream for snappy errors so that we can possibly log something useful for the user and still return a successful exit value from the installer. The problem is that the redirection is not always capturing the full stderr stream which makes it difficult to check for the error.

The snappy exception causes the installer to exit with an error code which will cause rhqctl to abort execution. We do not want that to happen because Cassandra is still starting up fine.

An alternative might be to check the storage node log file, assuming there are no other errors that prevent start up. If the node starts up and does not support compression, it will log the following,

WARN [main] 2013-06-07 22:11:16,217 SnappyCompressor.java (line 38) Cannot initialize native Snappy library. Compression on new tables will be disabled.

Comment 4 Heiko W. Rupp 2013-06-24 15:37:25 UTC
master c8221f54f catches this case and allows the install to continue.

We still need a better fix though, as 

rhqctl start ( --storage ) also prints that message (but does not abort)

Comment 5 John Sanda 2013-08-23 01:42:58 UTC
We no include the snappy native library with our Cassandra bits. I am moving this to ON_QA for verification.

Note You need to log in before you can comment on or make changes to this bug.