Bug 994484 - Exception while saving bucket Bucket: FileNotFoundException at server shutdown
Exception while saving bucket Bucket: FileNotFoundException at server shutdown
Status: NEW
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering (Show other bugs)
6.1.1
Unspecified Unspecified
low Severity medium
: ---
: EAP 6.4.0
Assigned To: Paul Ferraro
Michal Vinkler
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-07 07:23 EDT by Jitka Kozana
Modified: 2017-10-09 20:25 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-12 11:01:37 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jitka Kozana 2013-08-07 07:23:37 EDT
At the end of server shutdown this exception was once seen:

:53:20,980 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped default-host/clusterbench cache from web container
01:53:20,981 DEBUG [org.infinispan.xsite.BackupReceiverRepositoryImpl] (MSC service thread 1-1) Processing cache stop: EventImpl{type=CACHE_STOPPED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}. Cache name: 'default-host/clusterbench'
01:53:20,963 ERROR [org.infinispan.loaders.file.FileCacheStore] (MSC service thread 1-2) ISPN000063: Exception while saving bucket Bucket{entries={vmg36/web=ImmortalCacheEntry{key=vmg36/web, value=ImmortalCacheValue {value=vmg36.mw.lab.eng.bos.redhat.com=null}}}, bucketId='764581888'}: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory)
	at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45]
	at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
	at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213)
	at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:604)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96)
	at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

01:53:21,062 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 63) Stopping cache org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB on vmg34/ejb
01:53:21,062 INFO  [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 63) ISPN000029: Passivating all entries to disk
01:53:21,064 ERROR [org.infinispan.loaders.file.FileCacheStore] (ServerService Thread Pool -- 63) ISPN000063: Exception while saving bucket Bucket{entries={MarshalledValue{instance={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=385622366}@45106993=ImmortalCacheEntry{key=MarshalledValue{instance={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=385622366}@45106993, value=ImmortalCacheValue {value={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}}}}, bucketId='385622016'}: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory)
	at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45]
	at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
	at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213)
	at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:604)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:118)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319)
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321)
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118)
	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:331)
	at org.jboss.as.ee.component.ComponentStartService$2.run(ComponentStartService.java:72) [jboss-as-ee-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_45]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

01:53:21,040 WARN  [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-2) ISPN000189: While stopping a cache or cache manager, one of its components failed to stop: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:604)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96)
	at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
Caused by: org.infinispan.loaders.CacheLoaderException: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:339)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
	at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213)
	at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	... 14 more
Caused by: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory)
	at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45]
	at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336)
	... 24 more

01:53:21,122 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-2) Wait for on-going transactions to finish for 30 seconds.
01:53:21,123 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-2) All transactions terminated
01:53:21,123 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-2) Node vmg34/web leaving cache repl
01:53:21,123 DEBUG [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-2) Updating cluster-wide consistent hash for cache repl, topology = CacheTopology{id=13, currentCH=ReplicatedConsistentHash{members=[vmg36/web]}, pendingCH=null}
01:53:21,099 WARN  [org.infinispan.factories.ComponentRegistry] (ServerService Thread Pool -- 63) ISPN000189: While stopping a cache or cache manager, one of its components failed to stop: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:604)
	at org.infinispan.CacheImpl.stop(CacheImpl.java:599)
	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:118)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319)
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321)
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118)
	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:331)
	at org.jboss.as.ee.component.ComponentStartService$2.run(ComponentStartService.java:72) [jboss-as-ee-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_45]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.infinispan.loaders.CacheLoaderException: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:339)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)
	at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)
	at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213)
	at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	... 20 more
Caused by: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory)
	at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45]
	at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45]
	at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654)
	at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336)
	... 30 more

This was seen at a virtual machine, on pure ipv6 environment, but it does not seem to be connected to the ipv6 setup. 

The shutdown of the rest of the cluster went smoothly every time. 

Link to server log: 
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-ipv6/19/artifact/report/config/jboss-vmg34.mw.lab.eng.bos.redhat.com/server.log
Comment 2 Jitka Kozana 2013-12-09 07:14:34 EST
Update after EAP 6.2.0. testing cycle: this issue was not seen during this testing cycle. If this issue was fixed, than please change the status of this BZ accordingly.
Comment 3 Ladislav Thon 2014-03-19 07:44:24 EDT
Seen this happening with EAP 6.3.0.DR4: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-dualstack-ipv6/23/console-dev89/

Not sure if reopen is the right thing to do here.
Comment 5 Radoslav Husar 2014-06-09 07:51:12 EDT
This does look like an environmental issue since it has been only seen on vmg-s. I have inspected the machines, the /tmp from where the test are run is quite small (4 GB) with stale files from non-hudson users which could have possibly conflicted and resulted in "No such file or directory".

Could you reproduce this in clean environment such as perf*?


Let me note, that even though the previously linked job is using JVM kill as failure type, the exception happens when the cluster is gracefully shut down.
Comment 6 Ladislav Thon 2014-07-08 09:22:23 EDT
Still an issue (though very rare and it was suggested that it's possibly environmental, see comment 5), moving to 6.4 (for now, needs investigation).
Comment 7 Paul Ferraro 2014-09-02 09:52:31 EDT
Lowering priority, given comment #6.
Comment 8 Paul Ferraro 2014-09-03 21:53:02 EDT
Deferring clean shutdown issues to EAP7.

Note You need to log in before you can comment on or make changes to this bug.