Description of problem: This issue happened only once and I'm not able to reproduce it so any hint what could be causing it would be helpful. Version-Release number of selected component (if applicable): JON 3.3.2.ER01 How reproducible: Only once Steps to Reproduce: 1. JON 3.2.0.GA is installed and running (1 server, 1 storage node, 1 co-located agent) 2. stop everything 3. unzip JON3.3.0.GA.zip 4. ./rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA/ 5. apply the patch - jon-server-3.3.0.GA-update-02/apply-updates.sh jon-server-3.3.0.GA 6. start storage node 7. ./rhqctl upgrade --storage-schema Actual results: [hudson@fbr-upg1 bin]$ ./rhqctl upgrade --storage-schema 08:01:54,554 INFO [org.jboss.modules] JBoss Modules version 1.3.5.Final-redhat-1 08:01:54,706 INFO [org.rhq.server.control.command.Upgrade] Updating RHQ Storage Cluster schema 08:01:54,716 INFO [org.rhq.server.control.command.Upgrade] The RHQ Storage Cluster schema update is running 08:01:55,300 INFO [org.jboss.modules] JBoss Modules version 1.3.5.Final-redhat-1 08:01:55,577 INFO [org.rhq.cassandra.schema.VersionManager] Preparing to check storage schema compatibility. 08:01:56,857 WARN [org.rhq.cassandra.schema.VersionManager] Storage cluster schema version:1. Required schema version: 7. Please update storage cluster schema version. 08:01:56,857 INFO [org.rhq.cassandra.schema.VersionManager] Completed storage schema compatibility check. 08:01:56,858 INFO [org.rhq.enterprise.server.installer.InstallerServiceImpl] Storage cluster Schema out of date. Applying Storage Cluster schema updates. 08:01:56,859 INFO [org.rhq.cassandra.schema.VersionManager] Preparing to install storage schema 08:01:56,860 INFO [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections 08:01:57,530 INFO [org.rhq.cassandra.schema.VersionManager] Installed storage schema version is 1 08:01:57,530 INFO [org.rhq.cassandra.schema.VersionManager] Required storage schema version is 7 08:01:57,530 INFO [org.rhq.cassandra.schema.VersionManager] Storage schema requires udpates. Updating from version 1 to version 7. 08:01:57,531 INFO [org.rhq.cassandra.schema.AbstractManager] Applying update file: schema/update/0002.xml 08:01:57,640 ERROR [org.rhq.enterprise.server.installer.InstallerServiceImpl] Could not complete storage cluster schema installation: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception: com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception at com.datastax.driver.core.exceptions.DriverInternalError.copy(DriverInternalError.java:42) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:175) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:147) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.SessionManager.execute(SessionManager.java:79) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.SessionManager.execute(SessionManager.java:75) [cassandra-driver-core-1.0.8-jboss-1.jar:] at org.rhq.cassandra.schema.CQLStep.execute(CQLStep.java:84) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.cassandra.schema.AbstractManager.execute(AbstractManager.java:283) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.cassandra.schema.VersionManager.update(VersionManager.java:181) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.cassandra.schema.VersionManager.install(VersionManager.java:91) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.cassandra.schema.SchemaManager.install(SchemaManager.java:123) [rhq-cassandra-schema-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareStorageSchema(InstallerServiceImpl.java:755) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.enterprise.server.installer.InstallerServiceImpl.updateStorageSchema(InstallerServiceImpl.java:710) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:122) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at org.rhq.enterprise.server.installer.Installer.main(Installer.java:59) [rhq-installer-util-4.12.0.JON330GA-redhat-1.jar:4.12.0.JON330GA-redhat-1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_65] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_65] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_65] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_65] at org.jboss.modules.Module.run(Module.java:312) [jboss-modules.jar:1.3.5.Final-redhat-1] at org.jboss.modules.Main.main(Main.java:460) [jboss-modules.jar:1.3.5.Final-redhat-1] Caused by: com.datastax.driver.core.exceptions.DriverInternalError: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception at com.datastax.driver.core.DefaultResultSetFuture.convertException(DefaultResultSetFuture.java:190) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:110) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:215) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.RequestHandler.onSet(RequestHandler.java:336) [cassandra-driver-core-1.0.8-jboss-1.jar:] at com.datastax.driver.core.Connection$Dispatcher.messageReceived(Connection.java:557) [cassandra-driver-core-1.0.8-jboss-1.jar:] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.1.Final.jar:] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:68) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.9.1.Final.jar:] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.9.1.Final.jar:] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.9.1.Final.jar:] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.1.Final.jar:] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.1.Final.jar:] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.1.Final.jar:] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.1.Final.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65] 08:01:57,668 ERROR [org.rhq.enterprise.server.installer.Installer] java.lang.Exception:Could not complete storage cluster schema installation: An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception -> com.datastax.driver.core.exceptions.DriverInternalError:An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception -> com.datastax.driver.core.exceptions.DriverInternalError:An unexpected error occured server side: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception Expected results: No errors Additional info: This scenario is used for verification of bz1185375
This has been fixed within Bug 1213782 with commit https://github.com/rhq-project/rhq/commit/2d173cd61 Fixed means, that exception is caught and user is told to rerun schma migration.
Cherry-picked to release/jon3.3.x: commit b5b3684f4d65cd86a20e14c0cde847c9b3d1fd41 Author: Libor Zoubek <lzoubek> Date: Mon Jun 15 12:19:33 2015 +0200 Bug 1213782 - Could not complete storage cluster schema installation: java.lang.NullPointerException Correctly handle errors which might occur in KeyScanner constructor (cherry picked from commit 2d173cd61f7d0e9eb3eaf8e86f9ed72eb3f6272a)
Available for test with 3.3.3 ER01 build: https://brewweb.devel.redhat.com/buildinfo?buildID=446732 *Note: jon-server-patch-3.3.0.GA.zip maps to ER01 build of jon-server-3.3.0.GA-update-03.zip.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1525.html