Description of problem: In stress test with distributed async cache, eviction enabled (passivation disabled) and write-behind to LevelDB cache store a series of identical NPEs occurred. The server was under heavy load in this time and response time for HotRod requests took long (~100 ms in average). It seems that shortly before this exception the cluster got broken - two JVMs crashed (will be investigated as separate issue). Version-Release number of selected component (if applicable): 6.1.0.GA-redhat-2 (one-off patch) 16:35:37,123 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-150,shared=udp) ISPN000094: Received new cluster view: [node02/default|3] [node02/default, node04/default] 16:35:37,240 WARN [org.infinispan.statetransfer.StateProviderImpl] (OOB-480,shared=udp) ISPN000212: Segments were requested by node node04/default with topology 8, older than the local topology (9) 16:35:37,240 WARN [org.infinispan.statetransfer.StateProviderImpl] (OOB-479,shared=udp) ISPN000212: Segments were requested by node node04/default with topology 6, older than the local topology (7) 16:35:37,604 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-45) ISPN000136: Execution error: java.lang.NullPointerException at org.infinispan.container.InternalEntryFactoryImpl.create(InternalEntryFactoryImpl.java:50) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.CacheStoreInterceptor.getStoredEntry(CacheStoreInterceptor.java:402) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.DistCacheStoreInterceptor.visitPutKeyValueCommand(DistCacheStoreInterceptor.java:102) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:103) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:166) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:71) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:125) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1186) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.CacheImpl.putInternal(CacheImpl.java:783) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.CacheImpl.put(CacheImpl.java:777) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.DecoratedCache.put(DecoratedCache.java:320) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:308) [infinispan-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.server.core.AbstractProtocolDecoder.put(AbstractProtocolDecoder.scala:201) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.server.core.AbstractProtocolDecoder.decodeValue(AbstractProtocolDecoder.scala:156) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:73) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.4.Final-redhat-3.jar:5.2.4.Final-redhat-3] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17] (identical exceptions continue)
customer reports running into this issue once with version 6.3.2 and levelDB