Description of problem: fuse mount dies when a certain operation is performed. Version-Release number of selected component (if applicable): How reproducible: everytime. Steps to Reproduce: create a disperse volume mount disperse volume cd into volume git clone https://github.com/chaos/powerman.git ./setup.sh ./configure checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking installation directory variables... yes checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking whether to enable maintainer-specific portions of Makefiles... no checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... rm: cannot remove 'conftest.dir': Transport endpoint is not connected ./configure: line 4: echo: write error: Transport endpoint is not connected ./configure: line 8: printf: write error: Transport endpoint is not connected ./configure: line 9: echo: write error: Transport endpoint is not connected sort: fflush failed: standard output: Transport endpoint is not connected sort: write error: Transport endpoint is not connected ./configure: line 40: echo: write error: Transport endpoint is not connected ./configure: line 44: printf: write error: Transport endpoint is not connected ./configure: line 45: echo: write error: Transport endpoint is not connected sort: fflush failed: standard output: Transport endpoint is not connected sort: write error: Transport endpoint is not connected ./configure: line 54: echo: write error: Transport endpoint is not connected ./configure: line 75: printf: write error: Transport endpoint is not connected ./configure: line 76: echo: write error: Transport endpoint is not connected cat: standard output: Transport endpoint is not connected cat: write error: Transport endpoint is not connected ./configure: line 78: echo: write error: Transport endpoint is not connected ./configure: line 82: printf: write error: Transport endpoint is not connected rm: cannot remove 'core': Transport endpoint is not connected rm: cannot remove '*.core': Transport endpoint is not connected rm: cannot remove 'core.conftest.*': Transport endpoint is not connected 1. 2. 3. Expected results: mount should not fail. Additional info: from /var/log/gluster on the FUSE client: [2017-05-19 18:30:22.119570] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.1 (args: /usr/sbin/glusterfs --volfile-server=d8-r13-c1-n8 --volfile-id=/home /home) [2017-05-19 18:30:22.130974] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-19 18:30:22.135171] I [MSGID: 122067] [ec-code.c:1046:ec_code_detect] 0-home-disperse-0: Using 'sse' CPU extensions [2017-05-19 18:30:22.135199] E [MSGID: 122074] [ec-code.c:425:ec_code_space_create] 0-home-disperse-0: Unable to create a temporary file for the ec dynamic code [No such file or directory] [2017-05-19 18:30:22.135891] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2017-05-19 18:30:22.136242] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-home-readdir-ahead: option 'parallel-readdir' is not recognized [2017-05-19 18:30:22.136327] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-0: parent translators are ready, attempting connect on transport [2017-05-19 18:30:22.139095] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-1: parent translators are ready, attempting connect on transport [2017-05-19 18:30:22.139718] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-0: changing port to 49153 (from 0) [2017-05-19 18:30:22.142853] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-2: parent translators are ready, attempting connect on transport [2017-05-19 18:30:22.144597] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-1: changing port to 49157 (from 0) [2017-05-19 18:30:22.146807] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-3: parent translators are ready, attempting connect on transport [2017-05-19 18:30:22.149918] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-19 18:30:22.150199] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-19 18:30:22.150297] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-2: changing port to 49153 (from 0) [2017-05-19 18:30:22.153824] I [MSGID: 114020] [client.c:2352:notify] 0-home-client-4: parent translators are ready, attempting connect on transport [2017-05-19 18:30:22.154162] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-0: Connected to home-client-0, attached to remote volume '/data/home'. [2017-05-19 18:30:22.154176] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-05-19 18:30:22.154240] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-1: Connected to home-client-1, attached to remote volume '/data/home'. [2017-05-19 18:30:22.154250] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-1: Server and Client lk-version numbers are not same, reopening the fds [2017-05-19 18:30:22.154350] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-0: Server lk version = 1 [2017-05-19 18:30:22.154492] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-19 18:30:22.154565] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-1: Server lk version = 1 [2017-05-19 18:30:22.154717] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-3: changing port to 49155 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume home-client-0 2: type protocol/client 3: option opversion 31000 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key /home 7: option client-version 3.10.1 8: option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-0-0-0 9: option fops-version 1298437 10: option ping-timeout 42 11: option remote-host d8-r13-c1-n8 12: option remote-subvolume /data/home 13: option transport-type socket 14: option transport.address-family inet 15: option filter-O_DIRECT enable 16: option send-gids true 17: end-volume 18: 19: volume home-client-1 20: type protocol/client 21: option opversion 31000 22: option clnt-lk-version 1 23: option volfile-checksum 0 24: option volfile-key /home 25: option client-version 3.10.1 26: option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-1-0-0 27: option fops-version 1298437 28: option ping-timeout 42 29: option remote-host d8-r13-c2-n8 30: option remote-subvolume /data/home 31: option transport-type socket 32: option transport.address-family inet 33: option filter-O_DIRECT enable 34: option send-gids true 35: end-volume 36: 37: volume home-client-2 38: type protocol/client 39: option opversion 31000 40: option clnt-lk-version 1 41: option volfile-checksum 0 42: option volfile-key /home 43: option client-version 3.10.1 44: option process-uuid build7-3116-2017/05/19-18:30:22:118808-home-client-2-0-0 45: option fops-version 1298437 46: option ping-timeout 42 47: option remote-host d8-r13-c3-n8 48: option remote-subvolume /data/home 49: option transport-type socket 50: option transport.address-family inet 51: option filter-O_DIRECT enable 52: option send-gids true 53: end-volume 54: 55: volume home-client-3 56: type protocol/client 57: option ping-timeout 42 58: option remote-host d8-r13-c4-n8 59: option remote-subvolume /data/home 60: option transport-type socket 61: option transport.address-family inet 62: option filter-O_DIRECT enable 63: option send-gids true 64: end-volume 65: 66: volume home-client-4 67: type protocol/client 68: option ping-timeout 42 69: option remote-host d8-r13-c5-n8 70: option remote-subvolume /data/home 71: option transport-type socket 72: option transport.address-family inet 73: option filter-O_DIRECT enable 74: option send-gids true 75: end-volume 76: 77: volume home-disperse-0 78: type cluster/disperse 79: option redundancy 1 80: option self-heal-daemon enable 81: subvolumes home-client-0 home-client-1 home-client-2 home-client-3 home-client-4 82: end-volume 83: 84: volume home-dht 85: type cluster/distribute 86: option lock-migration off 87: subvolumes home-disperse-0 88: end-volume 89: 90: volume home-write-behind 91: type performance/write-behind 92: option cache-size 4MB 93: subvolumes home-dht 94: end-volume 95: 96: volume home-readdir-ahead 97: type performance/readdir-ahead 98: option parallel-readdir off 99: option rda-request-size 131072 100: option rda-cache-limit 10MB 101: subvolumes home-write-behind 102: end-volume 103: 104: volume home-io-cache 105: type performance/io-cache 106: option cache-size 1GB 107: subvolumes home-readdir-ahead 108: end-volume 109: 110: volume home-open-behind 111: type performance/open-behind 112: subvolumes home-io-cache 113: end-volume 114: 115: volume home-io-threads 116: type performance/io-threads 117: option thread-count 64 118: option low-prio-threads 32 119: subvolumes home-open-behind 120: end-volume 121: 122: volume home 123: type debug/io-stats 124: option log-level INFO 125: option latency-measurement off 126: option count-fop-hits off 127: subvolumes home-io-threads 128: end-volume 129: 130: volume meta-autoload 131: type meta 132: subvolumes home 133: end-volume 134: +------------------------------------------------------------------------------+ [2017-05-19 18:30:22.157094] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-2: Connected to home-client-2, attached to remote volume '/data/home'. [2017-05-19 18:30:22.157107] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-2: Server and Client lk-version numbers are not same, reopening the fds [2017-05-19 18:30:22.157264] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-2: Server lk version = 1 [2017-05-19 18:30:22.157396] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-home-client-4: changing port to 49153 (from 0) [2017-05-19 18:30:22.161615] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-19 18:30:22.162110] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-home-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2017-05-19 18:30:22.162273] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-3: Connected to home-client-3, attached to remote volume '/data/home'. [2017-05-19 18:30:22.162284] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-3: Server and Client lk-version numbers are not same, reopening the fds [2017-05-19 18:30:22.162464] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-3: Server lk version = 1 [2017-05-19 18:30:22.162755] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-home-client-4: Connected to home-client-4, attached to remote volume '/data/home'. [2017-05-19 18:30:22.162768] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-home-client-4: Server and Client lk-version numbers are not same, reopening the fds [2017-05-19 18:30:22.162803] I [MSGID: 122061] [ec.c:340:ec_up] 0-home-disperse-0: Going UP [2017-05-19 18:30:22.163862] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-home-client-4: Server lk version = 1 [2017-05-19 18:30:22.164292] I [fuse-bridge.c:4146:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.26 [2017-05-19 18:30:22.164307] I [fuse-bridge.c:4831:fuse_graph_sync] 0-fuse: switched to graph 0 [2017-05-19 18:30:44.102817] I [MSGID: 109066] [dht-rename.c:1608:dht_rename] 0-home-dht: renaming /tdavis/git/powerman/conftest.dir/sub/conftest.TPo (hash=home-disperse-0/cache=home-disperse-0) => /tdavis/git/powerman/conftest.dir/sub/conftest.Po (hash=home-disperse-0/cache=<nul>) [2017-05-19 18:30:44.169707] W [MSGID: 114031] [client-rpc-fops.c:2203:client3_3_seek_cbk] 0-home-client-4: remote operation failed [No such device or address] pending frames: frame : type(1) op(SEEK) frame : type(1) op(SEEK) frame : type(1) op(FLUSH) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FSTAT) frame : type(1) op(STAT) frame : type(1) op(FSTAT) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(OPEN) frame : type(0) op(0) patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 2017-05-19 18:30:44 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.10.1 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f1cad237450] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f1cad240d54] /lib64/libc.so.6(+0x35250)[0x7f1cab91b250] /usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x28b14)[0x7f1c9f310b14] /usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x1142b)[0x7f1c9f2f942b] /usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x11608)[0x7f1c9f2f9608] /usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x1172f)[0x7f1c9f2f972f] /usr/lib64/glusterfs/3.10.1/xlator/cluster/disperse.so(+0x25c3b)[0x7f1c9f30dc3b] /usr/lib64/glusterfs/3.10.1/xlator/protocol/client.so(+0x20402)[0x7f1c9f587402] /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f1cad001880] /lib64/libgfrpc.so.0(rpc_clnt_notify+0x1e7)[0x7f1cad001b67] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f1cacffda23] /usr/lib64/glusterfs/3.10.1/rpc-transport/socket.so(+0x7384)[0x7f1ca1c79384] /usr/lib64/glusterfs/3.10.1/rpc-transport/socket.so(+0x9825)[0x7f1ca1c7b825] /lib64/libglusterfs.so.0(+0x84fb0)[0x7f1cad291fb0] /lib64/libpthread.so.0(+0x7dc5)[0x7f1cac098dc5] /lib64/libc.so.6(clone+0x6d)[0x7f1cab9dd73d] --------- on one of the 5 disperse gluster servers: in /var/log/gluster/glustershd.log: [2017-05-19 18:28:05.004264] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory] [2017-05-19 18:28:05.004279] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory] [2017-05-19 18:28:05.004327] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory] [2017-05-19 18:28:05.004328] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory] [2017-05-19 18:28:05.004408] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:3e12ef71-809b-44de-90df-203a901e532e> (3e12ef71-809b-44de-90df-203a901e532e) [No such file or directory] [2017-05-19 18:28:05.005094] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory] [2017-05-19 18:28:05.005131] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory] [2017-05-19 18:28:05.005164] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory] [2017-05-19 18:28:05.005189] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory] [2017-05-19 18:28:05.005200] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:1e27f609-3e83-4a16-b93c-9335efc05559> (1e27f609-3e83-4a16-b93c-9335efc05559) [No such file or directory] [2017-05-19 18:28:05.007103] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory] [2017-05-19 18:28:05.007105] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory] [2017-05-19 18:28:05.007144] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory] [2017-05-19 18:28:05.007163] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory] [2017-05-19 18:28:05.007217] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:1cd06237-d8ea-435e-a535-98bc947af054> (1cd06237-d8ea-435e-a535-98bc947af054) [No such file or directory] [2017-05-19 18:28:05.008134] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory] [2017-05-19 18:28:05.008165] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory] [2017-05-19 18:28:05.008179] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory] [2017-05-19 18:28:05.008297] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory] [2017-05-19 18:28:05.008417] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:d7643a21-4964-43a4-b8e0-c90e61f1eaa4> (d7643a21-4964-43a4-b8e0-c90e61f1eaa4) [No such file or directory] [2017-05-19 18:28:05.009098] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory] [2017-05-19 18:28:05.009151] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-1: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory] [2017-05-19 18:28:05.009176] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory] [2017-05-19 18:28:05.009195] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory] [2017-05-19 18:28:05.009332] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-2: remote operation failed. Path: <gfid:b3f395b5-32bc-4fca-bb7a-31d631e2a64d> (b3f395b5-32bc-4fca-bb7a-31d631e2a64d) [No such file or directory] [2017-05-19 18:28:05.010052] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-4: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> (d54dbecf-d566-4938-b4db-d554091e821c) [No such file or directory] [2017-05-19 18:28:05.010081] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-3: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> (d54dbecf-d566-4938-b4db-d554091e821c) [No such file or directory] [2017-05-19 18:28:05.010079] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 0-home-client-0: remote operation failed. Path: <gfid:d54dbecf-d566-4938-b4db-d554091e821c> (keeps repeating itself)
I've been using the kernel-ml-4.11.1 from el-repo and switched back to the latest Centos7 kernel (3.10.x), rebooted, and now it all works. So it's some sort of problem with the newer kernels.
Could you check if the problem is solved by the patch on bug https://bugzilla.redhat.com/show_bug.cgi?id=1438813
This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained. As a result this bug is being closed. If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.