Bug 1452865

Summary: fuse mount dies.
Product: [Community] GlusterFS Reporter: Thomas Davis <tdavis>
Component: fuseAssignee: Csaba Henk <csaba>
Status: CLOSED EOL QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.10CC: bugs, jahernan, nbalacha
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-20 18:29:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Thomas Davis 2017-05-19 18:56:32 UTC 
Description of problem:

fuse mount dies when a certain operation is performed.

Version-Release number of selected component (if applicable):


How reproducible:

everytime.

Steps to Reproduce:

create a disperse volume
mount disperse volume
cd into volume
git clone https://github.com/chaos/powerman.git

./setup.sh
./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking installation directory variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... rm: cannot remove 'conftest.dir': Transport endpoint is not connected
./configure: line 4: echo: write error: Transport endpoint is not connected
./configure: line 8: printf: write error: Transport endpoint is not connected
./configure: line 9: echo: write error: Transport endpoint is not connected
sort: fflush failed: standard output: Transport endpoint is not connected
sort: write error: Transport endpoint is not connected
./configure: line 40: echo: write error: Transport endpoint is not connected
./configure: line 44: printf: write error: Transport endpoint is not connected
./configure: line 45: echo: write error: Transport endpoint is not connected
sort: fflush failed: standard output: Transport endpoint is not connected
sort: write error: Transport endpoint is not connected
./configure: line 54: echo: write error: Transport endpoint is not connected
./configure: line 75: printf: write error: Transport endpoint is not connected
./configure: line 76: echo: write error: Transport endpoint is not connected
cat: standard output: Transport endpoint is not connected
cat: write error: Transport endpoint is not connected
./configure: line 78: echo: write error: Transport endpoint is not connected
./configure: line 82: printf: write error: Transport endpoint is not connected
rm: cannot remove 'core': Transport endpoint is not connected
rm: cannot remove '*.core': Transport endpoint is not connected
rm: cannot remove 'core.conftest.*': Transport endpoint is not connected


1.
2.
3.


Expected results:

mount should not fail.


Additional info:

from /var/log/gluster on the FUSE client:

[2017-05-19 18:30:22.119570] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.1 (args: /usr/sbin/glusterfs --volfile-server=d8-r13-c1-n8 --volfile-id=/home /home)
[2017-05-19 18:30:22.130974] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-05-19 18:30:22.135171] I [MSGID: 122067] [ec-code.c:1046:ec_code_detect] 0-home-disperse-0: Using 'sse' CPU extensions
[2017-05-19 18:30:22.135199] E [MSGID: 122074] [ec-code.c:425:ec_code_space_create] 0-home-disperse-0: Unable to create a temporary file for the ec dynamic code [No such file or directory]
[2017-05-19 18:30:22.135891] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-05-19 18:30:22.136242] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-home-readdir-ahead: option 'parallel-readdir' is not recognized
[2017-05-19 18:30:22.136327] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-0: parent translators are ready, attempting connect on transport
[2017-05-19 18:30:22.139095] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-1: parent translators are ready, attempting connect on transport
[2017-05-19 18:30:22.139718] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-0: changing port to 49153 (from 0)
[2017-05-19 18:30:22.142853] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-2: parent translators are ready, attempting connect on transport
[2017-05-19 18:30:22.144597] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-1: changing port to 49157 (from 0)
[2017-05-19 18:30:22.146807] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-3: parent translators are ready, attempting connect on transport
[2017-05-19 18:30:22.149918] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-19 18:30:22.150199] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-19 18:30:22.150297] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-2: changing port to 49153 (from 0)
[2017-05-19 18:30:22.153824] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-4: parent translators are ready, attempting connect on transport
[2017-05-19 18:30:22.154162] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-0: Connected to home-client-0, attached to remote volume '/data/home'.
[2017-05-19 18:30:22.154176] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-19 18:30:22.154240] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-1: Connected to home-client-1, attached to remote volume '/data/home'.
[2017-05-19 18:30:22.154250] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-19 18:30:22.154350] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-0: Server lk version = 1
[2017-05-19 18:30:22.154492] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-19 18:30:22.154565] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-1: Server lk version = 1
[2017-05-19 18:30:22.154717] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-3: changing port to 49155 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume home-client-0
  2:     type protocol/client
  3:     option opversion 31000
  4:     option clnt-lk-version 1
  5:     option volfile-checksum 0
  6:     option volfile-key /home
  7:     option client-version 3.10.1
  8:     option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-0-0-0
  9:     option fops-version 1298437
 10:     option ping-timeout 42
 11:     option remote-host d8-r13-c1-n8
 12:     option remote-subvolume /data/home
 13:     option transport-type socket
 14:     option transport.address-family inet
 15:     option filter-O_DIRECT enable
 16:     option send-gids true
 17: end-volume
 18:  
 19: volume home-client-1
 20:     type protocol/client
 21:     option opversion 31000
 22:     option clnt-lk-version 1
 23:     option volfile-checksum 0
 24:     option volfile-key /home
 25:     option client-version 3.10.1
 26:     option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-1-0-0
 27:     option fops-version 1298437
 28:     option ping-timeout 42
 29:     option remote-host d8-r13-c2-n8
 30:     option remote-subvolume /data/home
 31:     option transport-type socket
 32:     option transport.address-family inet
 33:     option filter-O_DIRECT enable
 34:     option send-gids true
 35: end-volume
 36:  
 37: volume home-client-2
 38:     type protocol/client
 39:     option opversion 31000
 40:     option clnt-lk-version 1
 41:     option volfile-checksum 0
 42:     option volfile-key /home
 43:     option client-version 3.10.1
 44:     option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-2-0-0
 45:     option fops-version 1298437
 46:     option ping-timeout 42
 47:     option remote-host d8-r13-c3-n8
 48:     option remote-subvolume /data/home
 49:     option transport-type socket
 50:     option transport.address-family inet
 51:     option filter-O_DIRECT enable
 52:     option send-gids true
 53: end-volume
 54:  
 55: volume home-client-3
 56:     type protocol/client
 57:     option ping-timeout 42
 58:     option remote-host d8-r13-c4-n8
 59:     option remote-subvolume /data/home
 60:     option transport-type socket
 61:     option transport.address-family inet
 62:     option filter-O_DIRECT enable
 63:     option send-gids true
 64: end-volume
 65:  
 66: volume home-client-4
 67:     type protocol/client
 68:     option ping-timeout 42
 69:     option remote-host d8-r13-c5-n8
 70:     option remote-subvolume /data/home
 71:     option transport-type socket
 72:     option transport.address-family inet
 73:     option filter-O_DIRECT enable
 74:     option send-gids true
 75: end-volume
 76:  
 77: volume home-disperse-0
 78:     type cluster/disperse
 79:     option redundancy 1
 80:     option self-heal-daemon enable
 81:     subvolumes home-client-0 home-client-1 home-client-2 home-client-3 home-client-4
 82: end-volume
 83:  
 84: volume home-dht
 85:     type cluster/distribute
 86:     option lock-migration off
 87:     subvolumes home-disperse-0
 88: end-volume
 89:  
 90: volume home-write-behind
 91:     type performance/write-behind
 92:     option cache-size 4MB
 93:     subvolumes home-dht
 94: end-volume
 95:  
 96: volume home-readdir-ahead
 97:     type performance/readdir-ahead
 98:     option parallel-readdir off
 99:     option rda-request-size 131072
100:     option rda-cache-limit 10MB
101:     subvolumes home-write-behind
102: end-volume
103:  
104: volume home-io-cache
105:     type performance/io-cache
106:     option cache-size 1GB
107:     subvolumes home-readdir-ahead
108: end-volume
109:  
110: volume home-open-behind
111:     type performance/open-behind
112:     subvolumes home-io-cache
113: end-volume
114:  
115: volume home-io-threads
116:     type performance/io-threads
117:     option thread-count 64
118:     option low-prio-threads 32
119:     subvolumes home-open-behind
120: end-volume
121:  
122: volume home
123:     type debug/io-stats
124:     option log-level INFO
125:     option latency-measurement off
126:     option count-fop-hits off
127:     subvolumes home-io-threads
128: end-volume
129:  
130: volume meta-autoload
131:     type meta
132:     subvolumes home
133: end-volume
134:  
+------------------------------------------------------------------------------+
[2017-05-19 18:30:22.157094] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-2: Connected to home-client-2, attached to remote volume '/data/home'.
[2017-05-19 18:30:22.157107] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-19 18:30:22.157264] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-2: Server lk version = 1
[2017-05-19 18:30:22.157396] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-4: changing port to 49153 (from 0)
[2017-05-19 18:30:22.161615] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-19 18:30:22.162110] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-05-19 18:30:22.162273] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-3: Connected to home-client-3, attached to remote volume '/data/home'.
[2017-05-19 18:30:22.162284] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-19 18:30:22.162464] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-3: Server lk version = 1
[2017-05-19 18:30:22.162755] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-4: Connected to home-client-4, attached to remote volume '/data/home'.
[2017-05-19 18:30:22.162768] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2017-05-19 18:30:22.162803] I [MSGID: 122061] [ec.c:340:ec_up] 0-home-disperse-0: Going UP
[2017-05-19 18:30:22.163862] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-4: Server lk version = 1
[2017-05-19 18:30:22.164292] I [fuse-bridge.c:4146:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26
[2017-05-19 18:30:22.164307] I [fuse-bridge.c:4831:fuse_graph_sync] 0-fuse: switched to graph 0
[2017-05-19 18:30:44.102817] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-home-dht: renaming /tdavis/git/powerman/conftest.dir/sub/conftest.TPo (hash=home-disperse-0/cache=home-disperse-0) => /tdavis/git/powerman/conftest.dir/sub/conftest.Po (hash=home-disperse-0/cache=<nul>)
[2017-05-19 18:30:44.169707] W [MSGID: 114031] [client-rpc-fops.c:2203:client3_3_seek_cbk] 0-home-client-4: remote operation failed [No such device or address]
pending frames:
frame : type(1) op(SEEK)
frame : type(1) op(SEEK)
frame : type(1) op(FLUSH)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(FSTAT)
frame : type(1) op(STAT)
frame : type(1) op(FSTAT)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(FLUSH)
frame : type(1) op(FLUSH)
frame : type(1) op(OPEN)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2017-05-19 18:30:44
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.10.1
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f1cad237450]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f1cad240d54]
/lib64/libc.so.6(+0x35250)[0x7f1cab91b250]
/usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x28b14)[0x7f1c9f310b14]
/usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x1142b)[0x7f1c9f2f942b]
/usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x11608)[0x7f1c9f2f9608]
/usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x1172f)[0x7f1c9f2f972f]
/usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x25c3b)[0x7f1c9f30dc3b]
/usr/lib64/glusterfs/3.10.1/xlator/protocol/client.so(+0x20402)[0x7f1c9f587402]
/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f1cad001880]
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1e7)[0x7f1cad001b67]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f1cacffda23]
/usr/lib64/glusterfs/3.10.1/rpc-transport/socket.so(+0x7384)[0x7f1ca1c79384]
/usr/lib64/glusterfs/3.10.1/rpc-transport/socket.so(+0x9825)[0x7f1ca1c7b825]
/lib64/libglusterfs.so.0(+0x84fb0)[0x7f1cad291fb0]
/lib64/libpthread.so.0(+0x7dc5)[0x7f1cac098dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f1cab9dd73d]
---------
on one of the 5 disperse gluster servers:

in /var/log/gluster/glustershd.log:

[2017-05-19 18:28:05.004264] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory]
[2017-05-19 18:28:05.004279] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory]
[2017-05-19 18:28:05.004327] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory]
[2017-05-19 18:28:05.004328] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory]
[2017-05-19 18:28:05.004408] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory]
[2017-05-19 18:28:05.005094] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory]
[2017-05-19 18:28:05.005131] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory]
[2017-05-19 18:28:05.005164] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory]
[2017-05-19 18:28:05.005189] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory]
[2017-05-19 18:28:05.005200] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory]
[2017-05-19 18:28:05.007103] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory]
[2017-05-19 18:28:05.007105] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory]
[2017-05-19 18:28:05.007144] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory]
[2017-05-19 18:28:05.007163] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory]
[2017-05-19 18:28:05.007217] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory]
[2017-05-19 18:28:05.008134] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory]
[2017-05-19 18:28:05.008165] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory]
[2017-05-19 18:28:05.008179] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory]
[2017-05-19 18:28:05.008297] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory]
[2017-05-19 18:28:05.008417] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory]
[2017-05-19 18:28:05.009098] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory]
[2017-05-19 18:28:05.009151] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory]
[2017-05-19 18:28:05.009176] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory]
[2017-05-19 18:28:05.009195] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory]
[2017-05-19 18:28:05.009332] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory]
[2017-05-19 18:28:05.010052] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> (d54dbecf-d566-4938-b4db-d554091e821c) [No such file or directory]
[2017-05-19 18:28:05.010081] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> (d54dbecf-d566-4938-b4db-d554091e821c) [No such file or directory]
[2017-05-19 18:28:05.010079] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> 

(keeps repeating itself)
Comment 1 Thomas Davis 2017-05-19 19:26:07 UTC 
I've been using the kernel-ml-4.11.1 from el-repo and switched back to the latest Centos7 kernel (3.10.x), rebooted, and now it all works.

So it's some sort of problem with the newer kernels.
Comment 2 Xavi Hernandez 2017-05-22 07:09:12 UTC 
Could you check if the problem is solved by the patch on bug https://bugzilla.redhat.com/show_bug.cgi?id=1438813
Comment 3 Shyamsundar 2018-06-20 18:29:08 UTC 
This bug reported is against a version of Gluster that is no longer maintained
(or has been EOL'd). See https://www.gluster.org/release-schedule/ for the
versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline
gluster repository, request that it be reopened and the Version field be marked
appropriately.