1430659 – [GSS]AlreadyClosedException Underlying file changed by an external force

Bug 1430659 - [GSS]AlreadyClosedException Underlying file changed by an external force

Summary: [GSS]AlreadyClosedException Underlying file changed by an external force

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterfs
Sub Component:
Version:	ocs-3.11
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Susant Kumar Palai
QA Contact:	Byreddy
Docs Contact:
URL:
Whiteboard:	ES-on-gluster-fusemount
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-03-09 09:03 UTC by Ruben Romero Montes
Modified:	2023-03-24 13:46 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-04-06 07:54:50 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Ruben Romero Montes 2017-03-09 09:03:31 UTC

Description of problem:
Setting up the Aggregated Logging using Elasticsearch with a Fuse GlusterFS storage we are receiving the following error:

[2017-03-02 16:33:54,964][WARN ][index.engine             ] [Firestar] [project.logging.d3b9dd85-ff33-11e6-998d-0050569413a6.2017.03.02][0] failed engine [lucene commit failed]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-03-02T15:33:54.836414Z, (lock=NativeFSLock(path=/elasticsearch/persistent/logging-es/data/logging-es/nodes/0/indices/project.logging.d3b9dd85-ff33-11e6-998d-0050569413a6.2017.03.02/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],ctime=2017-03-02T15:33:54.836414Z))
	at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
	at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:516)
	at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:809)
	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4418)
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2860)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2963)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2930)
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1256)
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1264)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:214)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
	at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
	at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[2017-03-02 16:33:54,975][WARN ][indices.cluster          ] [Firestar] [[project.logging.d3b9dd85-ff33-11e6-998d-0050569413a6.2017.03.02][0]] marking and sending shard failed due to [engine failure, reason [lucene commit failed]]


Version-Release number of selected component (if applicable):
glusterfs-fuse-3.7.9-12.el7.x86_64
glusterfs-client-xlators-3.7.9-12.el7.x86_64
glusterfs-libs-3.7.9-12.el7.x86_64
glusterfs-3.7.9-12.el7.x86_64

How reproducible:
Always using this setup.

Steps to Reproduce:
1. Configure 1 elasticsearch instance with a PVC provisioned by FUSE GlusterFS
2. Deploy the logging-es pod
3. The error appears on the logs

Actual results:
Elasticsearch not working due to:
AlreadyClosedException: Underlying file changed by an external force at 2017-03-02T15:33:54.836414Z

Expected results:
Elasticsearch to work normally using a dynamically provisioned PVC

Comment 2 Sayan Saha 2017-03-10 14:51:00 UTC

Elasticsearch recommends using local storage with SSDs (use hostPath in kubernetes) for it's back-end storage. We expect to mitigate this in a later release of RHGS this year, until then RHGS should not be used for backing Elasticsearch.

Comment 3 Raghavendra G 2017-03-13 06:07:41 UTC

Closing this based on Comment 2 above. Please reopen if you've concerns.

Comment 4 Niki Dokovski 2017-08-10 16:36:25 UTC

Hi

Can we please reopen the ticket?

the same error is observed with SOLR instances. What causes that behaviour on glusterFS? FileLocking mechanism at the end is translated to the underlying native FS implementation.

Thanks,
Niki

Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-08-05T11:07:08Z, (lock=NativeFSLock(path=/import/data/sitescope/data/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],ctime=2017-08-05T11:07:08Z))
        at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179)
        at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
        at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
        at ....

Comment 5 Donatas Saulys 2017-11-17 08:51:21 UTC

We're also facing AlreadyClosedException issue with ES_6.0.0/LUCENE_7.0.1 on glusterFS.
Sayan Saha, do you have any news with regards to the mentioned potential mitigation?



net usable_space [9.9gb], net total_space [9.9gb], types [fuse.glusterfs]

[2017-11-16T16:11:04,825][WARN ][o.e.c.a.s.ShardStateAction] [07wReUC] [payments][2] received shard failed for shard id [[payments][2]], allocation id [L7lfY1CETg-ejAXdJupw3A], primary term [0], message [shard failure, reason [lucene commit failed]], failure [AlreadyClosedException[Underlying file changed by an external force at 2017-11-16T16:05:51.128853Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/nodes/0/indices/qi1qlBgjRZuXq1_XaQ0BMQ/2/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2017-11-16T16:05:51.128366Z))]]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-11-16T16:05:51.128853Z, (lock=NativeFSLock(path=/usr/share/elasticsearch/data/nodes/0/indices/qi1qlBgjRZuXq1_XaQ0BMQ/2/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2017-11-16T16:05:51.128366Z))
	at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:179) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:450) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:753) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4742) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3281) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3413) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3378) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1970) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.engine.InternalEngine.commitIndexWriter(InternalEngine.java:1963) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.engine.InternalEngine.syncFlush(InternalEngine.java:1291) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.syncFlush(IndexShard.java:984) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.performSyncedFlush(SyncedFlushService.java:423) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.indices.flush.SyncedFlushService.access$1100(SyncedFlushService.java:70) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$SyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:704) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.indices.flush.SyncedFlushService$SyncedFlushTransportHandler.messageReceived(SyncedFlushService.java:700) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:659) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.0.0.jar:6.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Comment 6 fabio.martinelli 2018-05-18 13:27:15 UTC

Same issue here with ES 6.2.4 and GlusterFS as on OpenShift 3.7.2:

[2018-05-18T13:24:21,132][WARN ][o.e.c.a.s.ShardStateAction] [2vD3p_e] [.watcher-history-7-2018.05.18][0] received shard failed for shard id [[.watcher-history-7-2018.05.18][0]], allocation id [OFiTF5akRgqDVxQStTrdaA], primary term [0], message [shard failure, reason [already closed by tragic event on the index writer]], failure [AlreadyClosedException[Underlying file changed by an external force at 2018-05-18T12:54:54.515332Z, (lock=NativeFSLock(path=/var/lib/elasticsearch/nodes/0/indices/WtkYIOY2Sc6Xl7_Ra6uKGA/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-05-18T12:54:54.515789Z))]]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2018-05-18T12:54:54.515332Z, (lock=NativeFSLock(path=/var/lib/elasticsearch/nodes/0/indices/WtkYIOY2Sc6Xl7_Ra6uKGA/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-05-18T12:54:54.515789Z))
        at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.codecs.lucene70.Lucene70NormsConsumer.<init>(Lucene70NormsConsumer.java:45) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.codecs.lucene70.Lucene70NormsFormat.normsConsumer(Lucene70NormsFormat.java:85) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:288) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:125) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:556) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:672) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:300) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:275) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
        at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1402) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1384) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:861) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:698) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:902) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:812) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-05-18T13:24:25,100][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [elasticsearch[2vD3p_e][generic][T#4]]
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
        at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:60) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.xpack.watcher.WatcherService.loadWatches(WatcherService.java:223) ~[?:?]
        at org.elasticsearch.xpack.watcher.WatcherService.reload(WatcherService.java:188) ~[?:?]
        at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.lambda$checkAndSetAllocationIds$9(WatcherLifeCycleService.java:263) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.4.jar:6.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
        at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:235) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:69) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:66) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 8 more

Comment 7 devops008 2018-06-27 08:46:53 UTC

Any update for this issue. I met the same error with elasticsearch 6.0.0. Glusterfs is used to storage elasticsearch data.

errors:
[2018-06-27T05:48:31,798][WARN ][o.e.c.r.a.AllocationService] [sBJEWnS] failing shard [failed shard, shard [logstash-2018.06.27][4], node[sBJEWnS1R1qL7blJL9Wl0Q], [P], recovery_source[existing recovery], s[INITIALIZING], a[id=CSql7_M8QnmfnylNrN-LFQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-06-27T05:48:30.059Z], failed_attempts[1], delayed=false, details[failed shard on node [sBJEWnS1R1qL7blJL9Wl0Q]: shard failure, reason [already closed by tragic event on the index writer], failure AlreadyClosedException[Underlying file changed by an external force at 2018-06-27T05:47:51.755Z, (lock=NativeFSLock(path=/data/nodes/0/indices/H9SdQlUeQluw6slOsBG7AQ/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-06-27T05:47:23.576Z))]], allocation_status[no_valid_shard_copy]], message [shard failure, reason [lucene commit failed]], failure [AlreadyClosedException[Underlying file changed by an external force at 2018-06-27T05:47:51.755Z, (lock=NativeFSLock(path=/data/nodes/0/indices/H9SdQlUeQluw6slOsBG7AQ/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-06-27T05:47:23.576Z))]], markAsStale [true]]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2018-06-27T05:47:51.755Z, (lock=NativeFSLock(path=/data/nodes/0/indices/H9SdQlUeQluw6slOsBG7AQ/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-06-27T05:47:23.576Z))

Comment 8 roberto 2020-01-16 11:41:28 UTC

Same here with glusterfs 6.0 and OpenShift 3.11

Note You need to log in before you can comment on or make changes to this bug.