Bug 1213366

Summary:	Could not complete storage cluster schema installation due to NoClassDefFoundError
Product:	[JBoss] JBoss Operations Network	Reporter:	Filip Brychta <fbrychta>
Component:	Installer, Storage Node	Assignee:	Libor Zoubek <lzoubek>
Status:	CLOSED ERRATA	QA Contact:	Filip Brychta <fbrychta>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	JON 3.3.0	CC:	lzoubek, miburman, theute
Target Milestone:	ER01	Keywords:	Triaged
Target Release:	JON 3.3.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-07-30 16:42:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1213782
Bug Blocks:

Description Filip Brychta 2015-04-20 11:37:16 UTC

Description of problem:
This issue happened only once and I'm not able to reproduce it so any hint what could be causing it would be helpful.

Version-Release number of selected component (if applicable):
JON 3.3.2.ER01

How reproducible:
Only once

Steps to Reproduce:
1. JON 3.2.0.GA is installed and running (1 server, 1 storage node, 1 co-located agent)
2. stop everything
3. unzip JON3.3.0.GA.zip
4. ./rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA/
5. apply the patch - jon-server-3.3.0.GA-update-02/apply-updates.sh jon-server-3.3.0.GA
6. start storage node
7. ./rhqctl upgrade --storage-schema

Actual results:
[hudson@fbr-upg1 bin]$ ./rhqctl upgrade --storage-schema
08:01:54,554 INFO  [org.jboss.modules] JBoss Modules version 1.3.5.Final-redhat-1
08:01:54,706 INFO  [org.rhq.server.control.command.Upgrade] Updating RHQ Storage Cluster schema
08:01:54,716 INFO  [org.rhq.server.control.command.Upgrade] The RHQ Storage Cluster schema update is running
08:01:55,300 INFO  [org.jboss.modules] JBoss Modules version 1.3.5.Final-redhat-1
08:01:55,577 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to check storage schema compatibility.
08:01:56,857 WARN  [org.rhq.cassandra.schema.VersionManager] Storage cluster schema version:1. Required schema version: 7. Please update storage cluster schema version.
08:01:56,857 INFO  [org.rhq.cassandra.schema.VersionManager] Completed storage schema compatibility check.
08:01:56,858 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Storage cluster Schema out of date. Applying Storage Cluster schema updates.
08:01:56,859 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to install storage schema
08:01:56,860 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
08:01:57,530 INFO  [org.rhq.cassandra.schema.VersionManager] Installed storage schema version is 1
08:01:57,530 INFO  [org.rhq.cassandra.schema.VersionManager] Required storage schema version is 7
08:01:57,530 INFO  [org.rhq.cassandra.schema.VersionManager] Storage schema requires udpates. Updating from version 1 to version 7.
08:01:57,531 INFO  [org.rhq.cassandra.schema.AbstractManager] Applying update file: schema/update/0002.xml
08:01:57,640 ERROR [org.rhq.enterprise.server.installer.InstallerServiceImpl] Could not complete storage cluster schema installation: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception: com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception
	at com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:175) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:147) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.SessionManager.execute(SessionManager.java:79) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.SessionManager.execute(SessionManager.java:75) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at org.rhq.cassandra.schema.CQLStep.execute(CQLStep.java:84) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.cassandra.schema.AbstractManager.execute(AbstractManager.java:283) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.cassandra.schema.VersionManager.update(VersionManager.java:181) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.cassandra.schema.VersionManager.install(VersionManager.java:91) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.cassandra.schema.SchemaManager.install(SchemaManager.java:123) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareStorageSchema(InstallerServiceImpl.java:755) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.updateStorageSchema(InstallerServiceImpl.java:710) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:122) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:59) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65]
	at org.jboss.modules.Module.run(Module.java:312) [jboss-modules.jar:1.3.5.Final-redhat-1]
	at org.jboss.modules.Main.main(Main.java:460) [jboss-modules.jar:1.3.5.Final-redhat-1]
Caused by: com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception
	at com.datastax.driver.core.DefaultResultSetFuture.convertException(DefaultResultSetFuture.java:190) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:110) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:215) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.RequestHandler.onSet(RequestHandler.java:336) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at com.datastax.driver.core.Connection$Dispatcher.messageReceived(Connection.java:557) [cassandra-driver-core-1.0.8-jboss-1.jar:]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:68) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.1.Final.jar:]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.1.Final.jar:]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]

08:01:57,668 ERROR [org.rhq.enterprise.server.installer.Installer] java.lang.Exception:Could not complete storage cluster schema installation: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception -> com.datastax.driver.core.exceptions.DriverInternalError:An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception -> com.datastax.driver.core.exceptions.DriverInternalError:An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception


Expected results:
No errors

Additional info:
This scenario is used for verification of bz1185375

Comment 1 Libor Zoubek 2015-06-15 14:16:27 UTC

This has been fixed within Bug 1213782 with commit https://github.com/rhq-project/rhq/commit/2d173cd61

Fixed means, that exception is caught and user is told to rerun schma migration.

Comment 2 Michael Burman 2015-06-24 17:45:24 UTC

Cherry-picked to release/jon3.3.x:

commit b5b3684f4d65cd86a20e14c0cde847c9b3d1fd41
Author: Libor Zoubek <lzoubek>
Date:   Mon Jun 15 12:19:33 2015 +0200

    Bug 1213782 - Could not complete storage cluster schema installation:
    java.lang.NullPointerException
    
    Correctly handle errors which might occur in KeyScanner constructor
    
    (cherry picked from commit 2d173cd61f7d0e9eb3eaf8e86f9ed72eb3f6272a)

Comment 3 Simeon Pinder 2015-07-10 18:55:38 UTC

Available for test with 3.3.3 ER01 build: 
https://brewweb.devel.redhat.com/buildinfo?buildID=446732
 *Note: jon-server-patch-3.3.0.GA.zip maps to ER01 build of
 jon-server-3.3.0.GA-update-03.zip.

Comment 6 errata-xmlrpc 2015-07-30 16:42:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1525.html