Bug 994484
| Summary: | Exception while saving bucket Bucket: FileNotFoundException at server shutdown | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Jitka Kozana <jkudrnac> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED EOL | QA Contact: | Michal Vinkler <mvinkler> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 6.1.1 | CC: | jkudrnac, lthon, myarboro, rhusar, rjanik |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | EAP 6.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-19 12:48:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Update after EAP 6.2.0. testing cycle: this issue was not seen during this testing cycle. If this issue was fixed, than please change the status of this BZ accordingly. Seen this happening with EAP 6.3.0.DR4: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-dualstack-ipv6/23/console-dev89/ Not sure if reopen is the right thing to do here. I've seen this again with EAP 6.3.0.ER4: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-jvmkill-repl-sync-modcluster-ipv6/56/console-vmg34/ Reopening. This does look like an environmental issue since it has been only seen on vmg-s. I have inspected the machines, the /tmp from where the test are run is quite small (4 GB) with stale files from non-hudson users which could have possibly conflicted and resulted in "No such file or directory". Could you reproduce this in clean environment such as perf*? Let me note, that even though the previously linked job is using JVM kill as failure type, the exception happens when the cluster is gracefully shut down. Still an issue (though very rare and it was suggested that it's possibly environmental, see comment 5), moving to 6.4 (for now, needs investigation). Lowering priority, given comment #6. Deferring clean shutdown issues to EAP7. |
At the end of server shutdown this exception was once seen: :53:20,980 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped default-host/clusterbench cache from web container 01:53:20,981 DEBUG [org.infinispan.xsite.BackupReceiverRepositoryImpl] (MSC service thread 1-1) Processing cache stop: EventImpl{type=CACHE_STOPPED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}. Cache name: 'default-host/clusterbench' 01:53:20,963 ERROR [org.infinispan.loaders.file.FileCacheStore] (MSC service thread 1-2) ISPN000063: Exception while saving bucket Bucket{entries={vmg36/web=ImmortalCacheEntry{key=vmg36/web, value=ImmortalCacheValue {value=vmg36.mw.lab.eng.bos.redhat.com=null}}}, bucketId='764581888'}: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory) at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45] at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336) at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242) at org.infinispan.CacheImpl.stop(CacheImpl.java:604) at org.infinispan.CacheImpl.stop(CacheImpl.java:599) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] 01:53:21,062 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 63) Stopping cache org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB on vmg34/ejb 01:53:21,062 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 63) ISPN000029: Passivating all entries to disk 01:53:21,064 ERROR [org.infinispan.loaders.file.FileCacheStore] (ServerService Thread Pool -- 63) ISPN000063: Exception while saving bucket Bucket{entries={MarshalledValue{instance={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=385622366}@45106993=ImmortalCacheEntry{key=MarshalledValue{instance={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}, serialized=ByteArray{size=27, array=0x0301fe03ff810200..}, cachedHashCode=385622366}@45106993, value=ImmortalCacheValue {value={[-19, 96, -9, 119, -122, 35, 76, 56, -112, -61, 43, 47, -118, -55, -105, 66]}}}}, bucketId='385622016'}: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory) at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45] at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336) at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242) at org.infinispan.CacheImpl.stop(CacheImpl.java:604) at org.infinispan.CacheImpl.stop(CacheImpl.java:599) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:118) at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319) at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321) at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118) at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:331) at org.jboss.as.ee.component.ComponentStartService$2.run(ComponentStartService.java:72) [jboss-as-ee-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] at org.jboss.threads.JBossThread.run(JBossThread.java:122) 01:53:21,040 WARN [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-2) ISPN000189: While stopping a cache or cache manager, one of its components failed to stop: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242) at org.infinispan.CacheImpl.stop(CacheImpl.java:604) at org.infinispan.CacheImpl.stop(CacheImpl.java:599) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.infinispan.subsystem.CacheService.stop(CacheService.java:96) at org.jboss.as.clustering.msc.AsynchronousService.stop(AsynchronousService.java:114) [jboss-as-clustering-common-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] Caused by: org.infinispan.loaders.CacheLoaderException: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:339) at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) ... 14 more Caused by: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/web/repl/764581888 (No such file or directory) at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45] at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336) ... 24 more 01:53:21,122 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-2) Wait for on-going transactions to finish for 30 seconds. 01:53:21,123 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-2) All transactions terminated 01:53:21,123 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-2) Node vmg34/web leaving cache repl 01:53:21,123 DEBUG [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-2) Updating cluster-wide consistent hash for cache repl, topology = CacheTopology{id=13, currentCH=ReplicatedConsistentHash{members=[vmg36/web]}, pendingCH=null} 01:53:21,099 WARN [org.infinispan.factories.ComponentRegistry] (ServerService Thread Pool -- 63) ISPN000189: While stopping a cache or cache manager, one of its components failed to stop: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:242) at org.infinispan.CacheImpl.stop(CacheImpl.java:604) at org.infinispan.CacheImpl.stop(CacheImpl.java:599) at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:118) at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319) at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321) at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118) at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:331) at org.jboss.as.ee.component.ComponentStartService$2.run(ComponentStartService.java:72) [jboss-as-ee-7.2.1.Final-redhat-5.jar:7.2.1.Final-redhat-5] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: org.infinispan.loaders.CacheLoaderException: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:339) at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:213) at org.infinispan.eviction.PassivationManagerImpl.passivateAll(PassivationManagerImpl.java:110) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) ... 20 more Caused by: java.io.FileNotFoundException: /tmp/hudson/jboss-eap-6.1/standalone/data/infinispan/ejb/org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB/385622016 (No such file or directory) at java.io.FileOutputStream.open(Native Method) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:194) [rt.jar:1.6.0_45] at java.io.FileOutputStream.<init>(FileOutputStream.java:145) [rt.jar:1.6.0_45] at org.infinispan.loaders.file.FileCacheStore$PerWriteFileSync.write(FileCacheStore.java:654) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336) ... 30 more This was seen at a virtual machine, on pure ipv6 environment, but it does not seem to be connected to the ipv6 setup. The shutdown of the rest of the cluster went smoothly every time. Link to server log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-sync-modcluster-ipv6/19/artifact/report/config/jboss-vmg34.mw.lab.eng.bos.redhat.com/server.log