Bug 971772 - Cassandra node Installation fails on IBM JRE 1.7 i686 on snappy
Cassandra node Installation fails on IBM JRE 1.7 i686 on snappy
Status: ON_QA
Product: RHQ Project
Classification: Other
Component: Database, Installer (Show other bugs)
4.8
Unspecified Unspecified
unspecified Severity medium (vote)
: ---
: ---
Assigned To: John Sanda
Mike Foley
:
Depends On:
Blocks: 951619
  Show dependency treegraph
 
Reported: 2013-06-07 04:59 EDT by Libor Zoubek
Modified: 2015-11-01 19:45 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Libor Zoubek 2013-06-07 04:59:48 EDT
Description of problem: I am not able to install latest RHQ 4.8 master build on following environment:

java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxi3270sr4ifix-20130305_01(SR4+IV37419))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux x86-32 20130205_137358 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR4_20130205_1656_B137358
JIT  - r11.b03_20130131_32403
GC   - R26_Java726_SR4_20130205_1656_B137358
J9CL - 20130205_137358)
JCL - 20130303_01 based on Oracle 7u13-b08


uname -a
Linux lzoubek1.bc.jonqe.lab.eng.bos.redhat.com 2.6.32-358.el6.i686 #1 SMP Tue Jan 29 11:48:01 EST 2013 i686 i686 i386 GNU/Linux


How reproducible:always


Actual results:
04:41:44,402 ERROR [org.rhq.storage.installer.StorageInstaller] The storage node reported the following errors while trying to start:

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:613)
        at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
        at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
        at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45)
        at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55)
        at org.apache.cassandra.io.compress.SnappyCompressor.<clinit>(SnappyCompressor.java:37)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.config.CFMetaData.<clinit>(CFMetaData.java:82)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:81)
        at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:471)
        at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:123)
        at java.lang.J9VMInternals.initializeImpl(Native Method)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:192)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:411)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:454)
Caused by: java.lang.UnsatisfiedLinkError: snappyjava (Not found in java.library.path)
        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1197)
        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1161)
        at java.lang.System.loadLibrary(System.java:527)
        at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
        ... 25 more



04:41:44,411 ERROR [org.rhq.storage.installer.StorageInstaller] Please review your configuration for possible sources of errors such as port conflicts or invalid arguments/options passed to the java executable.
Comment 1 Libor Zoubek 2013-06-07 08:37:44 EDT
same issue happens on :

java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxi3260sr13ifix-20130303_02(SR13+IV37419))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr13-20130114_134867 (JIT enabled, AOT enabled)
J9VM - 20130114_134867
JIT  - r9_20130108_31100
GC   - 20121212_AA)
JCL  - 20130303_02
Comment 2 John Sanda 2013-06-07 11:59:18 EDT
When Cassandra starts up it tries to load the native library for snappy-java. If snappy-java cannot load the native library, the snappy loader class (I forget the exact name) calls e.printStackTrace() in the catch block. I believe that is what you are seeing there. I will update the storage installer to suppress this output and instead just log a warning message.
Comment 3 John Sanda 2013-06-08 11:00:12 EDT
You can view the code that produces that exception here - https://gist.github.com/jsanda/5735367. At start up Cassandra will try to load the native snappy-java library. If the library cannot be loaded, SnappyNativeLoader winds up making the e.printStackTrace() call.

The storage installer is already configured to redirect stderr from Cassandra, and I want to add some logic in the installer that checks the stderr stream for snappy errors so that we can possibly log something useful for the user and still return a successful exit value from the installer. The problem is that the redirection is not always capturing the full stderr stream which makes it difficult to check for the error.

The snappy exception causes the installer to exit with an error code which will cause rhqctl to abort execution. We do not want that to happen because Cassandra is still starting up fine.

An alternative might be to check the storage node log file, assuming there are no other errors that prevent start up. If the node starts up and does not support compression, it will log the following,

WARN [main] 2013-06-07 22:11:16,217 SnappyCompressor.java (line 38) Cannot initialize native Snappy library. Compression on new tables will be disabled.
Comment 4 Heiko W. Rupp 2013-06-24 11:37:25 EDT
master c8221f54f catches this case and allows the install to continue.

We still need a better fix though, as 

rhqctl start ( --storage ) also prints that message (but does not abort)
Comment 5 John Sanda 2013-08-22 21:42:58 EDT
We no include the snappy native library with our Cassandra bits. I am moving this to ON_QA for verification.

Note You need to log in before you can comment on or make changes to this bug.