1609210 – While moving multiple temporary files to the same destination concurrently, renaming the destination file to a new name throws ESTALE and ENOENT

Bug 1609210 - While moving multiple temporary files to the same destination concurrently, renaming the destination file to a new name throws ESTALE and ENOENT

Summary: While moving multiple temporary files to the same destination concurrently, r...

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	distribute
Sub Component:
Version:	rhgs-3.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Susant Kumar Palai
QA Contact:	Prasad Desala
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-27 09:37 UTC by Prasad Desala
Modified:	2020-06-10 12:12 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-03-30 12:00:16 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Prasad Desala 2018-07-27 09:37:11 UTC

Description of problem:
=======================
While moving multiple temporary files to the same destination concurrently, renaming the destination file to a new name throws ESTALE and ENOENT.

Version-Release number of selected component (if applicable):
3.12.2-14.el7rhgs.x86_64

How reproducible:
always

Steps to Reproduce:
====================
1) Create a distributed-replicated volume and start it.
2) FUSE mount it on multiple clients.
3) From few clients execute,

"while true; do uuid="`uuidgen`"; echo "some data" > "test$uuid"; mv "test$uuid" "test" -f; done"
From other clients, keep sending lookups.
4) With step-3 in-progress, rename the destination file "test".
mv test 1

Actual results:
===============
Rename fails with ESTALE and ENOENT.

[root@dhcp37-109 fuse]# mv test 1
mv: cannot move ‘test’ to ‘1’: Stale file handle
[root@dhcp37-109 fuse]# mv test 2
mv: cannot move ‘test’ to ‘2’: No such file or directory
[root@dhcp37-109 fuse]# mv test 3
mv: cannot move ‘test’ to ‘3’: Stale file handle

Expected results:
=================
Rename should be successful without any errors.

Additional info:
================
Will be sharing the location of sos reports and gluster-health-check reports.

Comment 4 Raghavendra G 2018-07-28 04:54:38 UTC

Can you collect fusedump of the mount where you rename the destination file -  'mv test 1'

Comment 5 Nithya Balachandran 2018-07-30 04:02:15 UTC

What happens if you run the same test on a local filesystem?

Comment 6 Prasad Desala 2018-07-30 06:29:49 UTC

(In reply to Nithya Balachandran from comment #5)
> What happens if you run the same test on a local filesystem?

I tried the same tests on local fs before filing these rename BZs. We are not seeing any failures.

Comment 8 Raghavendra G 2018-07-30 13:27:15 UTC

From the fusedumps,

2018-07-30T16:00:52.635872226+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:16 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test1 
2018-07-30T16:00:52.65068223+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:16} "" 
2018-07-30T16:00:52.650744473+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:17 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test 
2018-07-30T16:00:52.662787606+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:17} {Nodeid:140181895929088 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:13172882735121334008 Size:10
 Blocks:1 Atime:1532946652 Mtime:1532946652 Ctime:1532946652 Atimensec:560575965 Mtimensec:569575917 Ctimensec:625575617 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:00:52.662857294+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:18 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test 
2018-07-30T16:00:52.662909772+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:18} {Nodeid:140181895929088 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:13172882735121334008 Size:10
 Blocks:1 Atime:1532946652 Mtime:1532946652 Ctime:1532946652 Atimensec:560575965 Mtimensec:569575917 Ctimensec:625575617 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:00:52.662927935+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:19 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181895929088 Nlookup:1} {Nodeid:140181760209056 Nlook
up:1} 
2018-07-30T16:00:52.662956074+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:20 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test1 
2018-07-30T16:00:52.681461396+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:20} "" 
2018-07-30T16:00:52.681579107+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:21 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test1 
2018-07-30T16:00:52.694980328+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:21} "" 
2018-07-30T16:00:52.695017218+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:22 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} {Newdir:1} test test1 
2018-07-30T16:00:52.704459499+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:22} 

> Above is the first instance of RENAME failure. Though not present in the dump, RENAME passed Nodeid:140181895929088 for file 1/test. Note the errno and its ESTALE, as a response to which kernel VFS is going to do the lookup again on 1/test and do the retry of RENAME.

2018-07-30T16:00:52.704516291+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:23 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test 
2018-07-30T16:00:52.713520539+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:23} {Nodeid:140181759878592 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:13717043528460281685 Size:10
 Blocks:1 Atime:1532946652 Mtime:1532946652 Ctime:1532946652 Atimensec:592968226 Mtimensec:598968194 Ctimensec:673967791 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 

> As expected, VFS is resolving 1/test again and as can be seen above Nodeid of 1/test has changed to Nodeid:140181759878592 (rename from another mount overwrote the previous nodeid)

2018-07-30T16:00:52.713539965+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:24 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test 
2018-07-30T16:00:52.713594738+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:24} {Nodeid:140181759878592 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:13717043528460281685 Size:10 Blocks:1 Atime:1532946652 Mtime:1532946652 Ctime:1532946652 Atimensec:592968226 Mtimensec:598968194 Ctimensec:673967791 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:00:52.713612214+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:25 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} test1 
2018-07-30T16:00:52.713785491+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:26 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181759878592 Nlookup:1} {Nodeid:140181895929088 Nlookup:1} 
2018-07-30T16:00:52.724725842+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:25} "" 
2018-07-30T16:00:52.724764285+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:27 Nodeid:1 Uid:0 Gid:0 Pid:2683 Padding:0} {Newdir:1} test test1 
2018-07-30T16:00:52.734845235+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:27} 

> Above is the retry RENAME and it failed too with ESTALE. Since VFS retries exactly once, it gave up after retry too failed with ESTALE and passed back the error to application.

Similar pattern can be observed for mv test test2
2018-07-30T16:01:05.165418668+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:31 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test 
2018-07-30T16:01:05.165456985+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:31} {Nodeid:140181895602064 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:10099565877933900856 Size:10 Blocks:1 Atime:1532946665 Mtime:1532946665 Ctime:1532946665 Atimensec:57191683 Mtimensec:65191640 Ctimensec:127191305 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:05.165474718+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:32 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181895602064 Nlookup:1} {Nodeid:140181759878592 Nlookup:1} 
2018-07-30T16:01:05.165543559+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:33 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test2 
2018-07-30T16:01:05.179461017+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:33} "" 
2018-07-30T16:01:05.179528218+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:34 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test2 
2018-07-30T16:01:05.191778871+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:34} "" 
2018-07-30T16:01:05.191810693+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:35 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} {Newdir:1} test test2 
2018-07-30T16:01:05.201087833+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:35} 
2018-07-30T16:01:05.201120916+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:36 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test 
2018-07-30T16:01:05.212540662+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:36} {Nodeid:140181760354464 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:12163492823232947228 Size:10 Blocks:1 Atime:1532946665 Mtime:1532946665 Ctime:1532946665 Atimensec:92191494 Mtimensec:102191440 Ctimensec:182191008 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:05.21258224+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:37 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test 
2018-07-30T16:01:05.212616687+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:37} {Nodeid:140181760354464 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:12163492823232947228 Size:10 Blocks:1 Atime:1532946665 Mtime:1532946665 Ctime:1532946665 Atimensec:92191494 Mtimensec:102191440 Ctimensec:182191008 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:05.212633483+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:38 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181760354464 Nlookup:1} {Nodeid:140181895602064 Nlookup:1} 
2018-07-30T16:01:05.21266811+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:39 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} test2 
2018-07-30T16:01:05.22305075+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:39} "" 
2018-07-30T16:01:05.223071429+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:40 Nodeid:1 Uid:0 Gid:0 Pid:2707 Padding:0} {Newdir:1} test test2 
2018-07-30T16:01:05.232207638+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:40} 


For mv test test3
2018-07-30T16:01:06.431714911+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:43 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test 
2018-07-30T16:01:06.439429628+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:43} {Nodeid:140181895702288 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:10082141774705206832 Size:10 Blocks:1 Atime:1532946666 Mtime:1532946666 Ctime:1532946666 Atimensec:364502059 Mtimensec:370502027 Ctimensec:421501754 Mode:33188 Nlink:2 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:06.439456323+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:44 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test 
2018-07-30T16:01:06.439495009+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:44} {Nodeid:140181895702288 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:10082141774705206832 Size:10 Blocks:1 Atime:1532946666 Mtime:1532946666 Ctime:1532946666 Atimensec:364502059 Mtimensec:370502027 Ctimensec:421501754 Mode:33188 Nlink:2 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:06.43953806+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:45 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181895702288 Nlookup:1} {Nodeid:140181760354464 Nlookup:1} 
2018-07-30T16:01:06.439588534+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:46 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test3 
2018-07-30T16:01:06.447598734+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:46} "" 
2018-07-30T16:01:06.447658852+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:47 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test3 
2018-07-30T16:01:06.459657235+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:47} "" 
2018-07-30T16:01:06.459685245+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:48 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} {Newdir:1} test test3 
2018-07-30T16:01:06.471081265+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:48} 
2018-07-30T16:01:06.471106686+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:49 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test 
2018-07-30T16:01:06.482078957+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:49} {Nodeid:140181760267696 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:10430532470831280699 Size:10 Blocks:1 Atime:1532946666 Mtime:1532946666 Ctime:1532946666 Atimensec:396501887 Mtimensec:404501845 Ctimensec:457501561 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:06.482110321+05:30 "GLUSTER\xf5" LOOKUP {Len:45 Opcode:1 Unique:50 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test 
2018-07-30T16:01:06.48214975+05:30 "GLUSTER\xf5" {Len:144 Error:0 Unique:50} {Nodeid:140181760267696 Generation:0 EntryValid:1 AttrValid:1 EntryValidNsec:0 AttrValidNsec:0 Attr:{Ino:10430532470831280699 Size:10 Blocks:1 Atime:1532946666 Mtime:1532946666 Ctime:1532946666 Atimensec:396501887 Mtimensec:404501845 Ctimensec:457501561 Mode:33188 Nlink:1 Uid:0 Gid:0 Rdev:0 Blksize:131072 Padding:0}} 
2018-07-30T16:01:06.482167429+05:30 "GLUSTER\xf5" BATCH_FORGET {Len:80 Opcode:42 Unique:51 Nodeid:0 Uid:0 Gid:0 Pid:0 Padding:0} {Count:2 Dummy:0} {Nodeid:140181760267696 Nlookup:1} {Nodeid:140181895702288 Nlookup:1} 
2018-07-30T16:01:06.48221715+05:30 "GLUSTER\xf5" LOOKUP {Len:46 Opcode:1 Unique:52 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} test3 
2018-07-30T16:01:06.495917537+05:30 "GLUSTER\xf5" {Len:16 Error:-2 Unique:52} "" 
2018-07-30T16:01:06.495935771+05:30 "GLUSTER\xf5" RENAME {Len:59 Opcode:12 Unique:53 Nodeid:1 Uid:0 Gid:0 Pid:2709 Padding:0} {Newdir:1} test test3 
2018-07-30T16:01:06.505442738+05:30 "GLUSTER\xf5" {Len:16 Error:-116 Unique:53} 


The retry count of 1 is not sufficient to maintain the atomicity of rename.

Comment 9 Raghavendra G 2018-07-30 13:33:45 UTC

Jeff,

As can be seen from comment #8, retry count of 1 on seeing ESTALE errors for path based operations is not sufficient (at least for Glusterfs). Any thoughts on fixing this issue are welcome.

regards,
Raghavendra

Comment 10 Jeff Layton 2018-07-30 14:16:30 UTC

Is not sufficient for what workload?

Again, we can always manufacture a workload on any filesystem (NFS included) where a single retry is not enough. The question is whether there are legitimate workloads where we need to do more. Testcases that thrash the dcache excessively don't actually count here.

Comment 13 Raghavendra G 2018-08-21 17:26:26 UTC

Hit open failure while initializing pgbench:

# pgbench -i -s 100 newdb7

9900000 of 10000000 tuples (99%) done (elapsed 116.10 s, remaining 1.17 s)
10000000 of 10000000 tuples (100%) done (elapsed 117.13 s, remaining 0.00 s)
vacuum...
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle
WARNING:  could not open statistics file "pg_stat_tmp/global.stat": Stale file handle

Note the traffic between kernel and glusterfs:
[2018-08-21 16:48:07.634269] T [fuse-bridge.c:1943:fuse_rename_resume] 0-glusterfs-fuse: 3479647: RENAME `<gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.tmp ()' -> `<gfid:4e1297f3-bae6-4f25-843b-
0efa51954f1d>/pg_stat_tmp/global.stat ()'
[2018-08-21 16:48:07.635928] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479648: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.657579] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479648: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.657644] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479649: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.657669] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479649: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.657704] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479650: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.657717] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479650: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.657744] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479651: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.660329] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479651: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.660546] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479653: WRITE (0x7f28d800ad90, size=86, offset=3558)
[2018-08-21 16:48:07.660646] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479653: WRITE => 86/86,3558/0
[2018-08-21 16:48:07.670581] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479654: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.672981] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479654: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.673040] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479655: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.673063] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479655: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.673098] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479656: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.673112] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479656: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.673140] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479657: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.677894] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479657: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.678102] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479659: WRITE (0x7f28d800ad90, size=86, offset=3644)
[2018-08-21 16:48:07.678171] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479659: WRITE => 86/86,3644/0
[2018-08-21 16:48:07.688485] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479660: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.692245] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479660: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.692409] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479661: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.692478] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479661: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.692565] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479662: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.692596] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479662: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.692641] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479663: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.695427] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479663: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.695602] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479665: WRITE (0x7f28d800ad90, size=86, offset=3730)
[2018-08-21 16:48:07.695657] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479665: WRITE => 86/86,3730/0
[2018-08-21 16:48:07.705989] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479666: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.710344] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479666: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.710389] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479667: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.710405] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479667: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.710434] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479668: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.710442] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479668: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.710457] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479669: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.711552] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479669: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.711640] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479671: WRITE (0x7f28d800ad90, size=86, offset=3816)
[2018-08-21 16:48:07.711684] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479671: WRITE => 86/86,3816/0
[2018-08-21 16:48:07.721818] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479672: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.724030] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479672: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.724086] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479673: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.724108] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479673: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.724140] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479674: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.724152] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479674: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.724176] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479675: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.725691] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479677: WRITE (0x7f28d800ad90, size=86, offset=3902)
[2018-08-21 16:48:07.725729] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479677: WRITE => 86/86,3902/0
[2018-08-21 16:48:07.735975] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479678: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.738108] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479678: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.738180] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479679: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.738200] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479679: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.738228] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479680: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.738237] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479680: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.738259] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479681: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.740610] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479681: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.740820] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479683: WRITE (0x7f28d800ad90, size=86, offset=3988)
[2018-08-21 16:48:07.740875] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479683: WRITE => 86/86,3988/0
[2018-08-21 16:48:07.750825] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479684: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.754680] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479684: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.754753] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479685: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.754789] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479685: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.754820] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479686: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.754829] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479686: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.754852] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479687: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.763108] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479687: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.763697] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479689: WRITE (0x7f28d800ad90, size=86, offset=4074)
[2018-08-21 16:48:07.764047] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479689: WRITE => 86/86,4074/0
[2018-08-21 16:48:07.773575] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479690: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.774978] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479690: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.775019] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479691: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.775038] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479691: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.775062] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479692: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.775070] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479692: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.775087] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479693: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.776263] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479693: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.776350] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479695: WRITE (0x7f28d800ad90, size=86, offset=4160)
[2018-08-21 16:48:07.776391] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479695: WRITE => 86/86,4160/0
[2018-08-21 16:48:07.786486] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479696: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.790359] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479696: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.790421] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479697: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.790453] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479697: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.790501] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479698: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.790519] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479698: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.790575] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479699: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.793257] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479699: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.794009] T [fuse-bridge.c:2451:fuse_write_resume] 0-glusterfs-fuse: 3479701: WRITE (0x7f28d800ad90, size=86, offset=4246)
[2018-08-21 16:48:07.794465] T [fuse-bridge.c:2407:fuse_writev_cbk] 0-glusterfs-fuse: 3479701: WRITE => 86/86,4246/0
[2018-08-21 16:48:07.803813] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479702: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat
[2018-08-21 16:48:07.807297] W [fuse-bridge.c:1094:fuse_fd_cbk] 0-glusterfs-fuse: 3479702: OPEN() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => -1 (Stale file handle)
[2018-08-21 16:48:07.807343] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479703: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp(b15007b2-9105-495a-ac10-3a9e7a1b971e)
[2018-08-21 16:48:07.807379] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479703: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp => 12398474226478847774
[2018-08-21 16:48:07.807404] T [fuse-bridge.c:651:fuse_lookup_resume] 0-glusterfs-fuse: 3479704: LOOKUP <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat(cda253b0-c19e-47ae-ae83-d34a8661f5c6)
[2018-08-21 16:48:07.807412] T [fuse-bridge.c:493:fuse_entry_cbk] 0-glusterfs-fuse: 3479704: LOOKUP() <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat => 12575126901584164294
[2018-08-21 16:48:07.807430] T [fuse-bridge.c:2272:fuse_open_resume] 0-glusterfs-fuse: 3479705: OPEN <gfid:4e1297f3-bae6-4f25-843b-0efa51954f1d>/pg_stat_tmp/global.stat

Note sure why there is series of open failures though there is only one rename.

Comment 14 Raghavendra G 2018-12-18 12:16:01 UTC

(In reply to Raghavendra G from comment #13)
> 
> Note sure why there is series of open failures though there is only one
> rename.

This might be due to stale dentry in fuse inode table as seen in bz https://bugzilla.redhat.com/show_bug.cgi?id=1600923. If lookups resolve to older inode whose md-cache is still warm, a lookup after open failure (with ESTALE) still returns older gfid, which causes next open to fail too.

Comment 15 Sahina Bose 2019-11-25 07:23:01 UTC

(In reply to Raghavendra G from comment #14)
> (In reply to Raghavendra G from comment #13)
> > 
> > Note sure why there is series of open failures though there is only one
> > rename.
> 
> This might be due to stale dentry in fuse inode table as seen in bz
> https://bugzilla.redhat.com/show_bug.cgi?id=1600923. If lookups resolve to
> older inode whose md-cache is still warm, a lookup after open failure (with
> ESTALE) still returns older gfid, which causes next open to fail too.

Should we close as duplicate of bug 1600923 ?

Comment 16 Raghavendra G 2019-11-25 09:48:33 UTC

(In reply to Sahina Bose from comment #15)
> (In reply to Raghavendra G from comment #14)
> > (In reply to Raghavendra G from comment #13)
> > > 
> > > Note sure why there is series of open failures though there is only one
> > > rename.
> > 
> > This might be due to stale dentry in fuse inode table as seen in bz
> > https://bugzilla.redhat.com/show_bug.cgi?id=1600923. If lookups resolve to
> > older inode whose md-cache is still warm, a lookup after open failure (with
> > ESTALE) still returns older gfid, which causes next open to fail too.
> 
> Should we close as duplicate of bug 1600923 ?

Current bug is not a duplicate of bz 1600923. The logs observed were due bz 1600923.

This bug is more on atomicity of rename and hence should be fixed. This bug is similar to bz 1609224 and bz 1610258

Note You need to log in before you can comment on or make changes to this bug.