Bug 976656 - NPE during stress-test with LevelDB as cache store
NPE during stress-test with LevelDB as cache store
Status: NEW
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.3.2
Unspecified Unspecified
medium Severity medium
: ---
: One-off release
Assigned To: Tristan Tarrant
Martin Gencur
:
Depends On:
Blocks: 975081
  Show dependency treegraph
 
Reported: 2013-06-21 02:30 EDT by Radim Vansa
Modified: 2015-03-30 15:47 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Radim Vansa 2013-06-21 02:30:08 EDT
Description of problem:
In stress test with distributed async cache, eviction enabled (passivation disabled) and write-behind to LevelDB cache store a series of identical NPEs occurred.
The server was under heavy load in this time and response time for HotRod requests took long (~100 ms in average). It seems that shortly before this exception the cluster got broken - two JVMs crashed (will be investigated as separate issue).

Version-Release number of selected component (if applicable):
6.1.0.GA-redhat-2 (one-off patch)

16:35:37,123 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-150,shared=udp) ISPN000094: Received new cluster view: [node02/default|3] [node02/default, node04/default]
16:35:37,240 WARN  [org.infinispan.statetransfer.StateProviderImpl] (OOB-480,shared=udp) ISPN000212: Segments were requested by node node04/default with topology 8, older than the local topology (9)
16:35:37,240 WARN  [org.infinispan.statetransfer.StateProviderImpl] (OOB-479,shared=udp) ISPN000212: Segments were requested by node node04/default with topology 6, older than the local topology (7)
16:35:37,604 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-45) ISPN000136: Execution error: java.lang.NullPointerException
	at org.infinispan.container.InternalEntryFactoryImpl.create(InternalEntryFactoryImpl.java:50) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.CacheStoreInterceptor.getStoredEntry(CacheStoreInterceptor.java:402) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPutKeyValueCommand(DistCacheStoreInterceptor.java:102) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:103) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:166) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:71) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:125) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1186) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.CacheImpl.put(CacheImpl.java:777) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.DecoratedCache.put(DecoratedCache.java:320) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:308) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:201) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.server.core.AbstractProtocolDecoder.decodeValue(AbstractProtocolDecoder.scala:156) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:73) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]

(identical exceptions continue)
Comment 2 Shay Matasaro 2015-03-30 15:18:16 EDT
customer reports running into this issue once with version 6.3.2 and levelDB

Note You need to log in before you can comment on or make changes to this bug.