Bug 1127870 - Running "Force Major Column Family Compaction" operation fails with NullPointerException
Summary: Running "Force Major Column Family Compaction" operation fails with NullPoint...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Storage Node
Version: JON 3.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ER01
: JON 3.3.0
Assignee: John Sanda
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On: 1118063
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-07 18:02 UTC by John Mazzitelli
Modified: 2018-12-06 17:41 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1118063
Environment:
Last Closed: 2014-12-11 14:01:09 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1125593 0 None None None Never

Description John Mazzitelli 2014-08-07 18:02:43 UTC
+++ This bug was initially created as a clone of Bug #1118063 +++

Description of problem:
Attempting to invoke the "Force Major Column Family Compaction" operation on a ColumnFamily fails and the operation error message is as follows:

    org.mc4j.ems.connection.EmsInvocationException: Exception on invocation of [forceTableCompaction]javax.management.RuntimeMBeanException: java.lang.NullPointerException
        at org.mc4j.ems.impl.jmx.connection.bean.operation.DOperation.invoke(DOperation.java:126)
        at org.rhq.plugins.cassandra.util.KeyspaceService.compact(KeyspaceService.java:96)
        at org.rhq.plugins.cassandra.KeyspaceComponent.compactKeyspace(KeyspaceComponent.java:168)
        at org.rhq.plugins.cassandra.ColumnFamilyComponent.invokeOperation(ColumnFamilyComponent.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
    Caused by: javax.management.RuntimeMBeanException: java.lang.NullPointerException
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:821)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
        at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1029)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59)
        at com.sun.proxy.$Proxy54.invoke(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.bean.operation.DOperation.invoke(DOperation.java:110)
        ... 12 more
    Caused by: java.lang.NullPointerException
        at org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2292)
        at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2166)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        ... 3 more
 
There is no real message written to the agent log other then:

    INFO  [ResourceContainer.invoker.nonDaemon-40] (org.rhq.plugins.cassandra.KeyspaceComponent)- Executing compaction on  keyspace [rhq]

Version-Release number of selected component (if applicable):
3.2.0.GA

How reproducible:
Always

Steps to Reproduce:
1. Install and start JBoss ON 3.2 system
2. After storage node is in inventory and reported as available, invoke the _Force Major Column Family Compaction_ operation on RHQ Storage Nodes / RHQ Storage Node(...) / Keyspaces / rhq / ColomnFamilies / raw_metrics

Actual results:
Compaction does not execute and operation status reports FAILURE with the exception and stack mentioned above.

Expected results:
Compaction operation runs as normal and operation status reports SUCCESS. Compaction is actually performed and rhq-storage.log reports something similar to:

    INFO [CompactionExecutor:76] 2014-07-09 18:33:17,065 CompactionTask.java (line 262) Compacted 2 sstables to [../../../rhq-data/data/rhq/raw_metrics/rhq-raw_metrics-ic-8,].  61,562,262 bytes to 51,079,953 (~82% of original) in 9,727ms = 5.008085MB/s.  5,943 total rows, 2,984 unique.  Row merge counts were {1:25, 2:2959, }

--- Additional comment from Michael Burman on 2014-07-15 04:42:56 EDT ---

This is happening, because ColumnFamilyDiscoveryComponent is not setting SimpleValue for "name"-property which the operation on the other hand is trying to use.

The correct parameter name is available in resourceKey also. For some reason only "takeSnapshot" is using that as columnFamilyname, while repair and compact are not.

Fixed in master:

commit b4d8b31b36ccc9a845e1140d79566bbba61889e2
Author: burmanm <miburman>
Date:   Tue Jul 15 11:42:23 2014 +0300

    [BZ 1118063] Use resourceKey as columnFamilyname on operations and set SimpleValue name when discovering columnFamily resources

Comment 1 Michael Burman 2014-08-11 09:18:52 UTC
Already in jon3.3.x branch.

Comment 2 Simeon Pinder 2014-08-19 23:50:44 UTC
Moving to ON_QA as available to test in the following brew build:

https://brewweb.devel.redhat.com//buildinfo?buildID=379025


Note You need to log in before you can comment on or make changes to this bug.