On my systemic setup, I am seeing lot of error messages on my clients as below [2016-11-14 02:43:50.274000] E [snapview-client.c:283:gf_svc_lookup_cbk] 0-sysvol-snapview-client: Lookup failed on normal graph with error Transport endpoint is not connected [2016-11-14 02:43:50.275390] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/replicate.so(+0x5d75c) [0x7f2a4ee4175c] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/distribute.so(+0x4623c) [0x7f2a4eba023c] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/distribute.so(+0x99b0) [0x7f2a4eb639b0] ) 0-sysvol-dht: invalid argument: inode [Invalid argument] I see these message repeating about every 20 min in bulk --- Additional comment from Nithya Balachandran on 2016-11-15 09:40:06 EST --- Steps to see the issue: 1. Create a pure dist volume with 2 bricks. 2. Fuse mount the volume. 3. Create a directory dir1 and cd into it. 4. Kill one of the brick processes (kill -9) 5. Try to create directories inside dir1 The mount log has the messages: [2016-11-15 14:06:39.738234] E [dht-helper.c:1666:dht_inode_ctx_time_update] (-->/usr/local/lib/glusterfs/3.10dev/xlator/protocol/c lient.so(+0x2f750) [0x7f1fa58a8750] -->/usr/local/lib/glusterfs/3.10dev/xlator/cluster/distribute.so(+0x39ffe) [0x7f1fa5604ffe] --> /usr/local/lib/glusterfs/3.10dev/xlator/cluster/distribute.so(+0xde27) [0x7f1fa55d8e27] ) 0-time-dht: invalid argument: inode [Invalid argument] RCA: The function dht_lookup_dir_cbk () does not check if the lookup succeeded on at least one subvolume before attempting to set the inode ctx. this_call_cnt = dht_frame_return (frame); if (is_last_call (this_call_cnt)) { if (local->need_selfheal) { local->need_selfheal = 0; dht_lookup_everywhere (frame, this, &local->loc); return 0; } if (local->op_ret == 0) { ret = dht_layout_normalize (this, &local->loc, layout); if (ret != 0) { gf_msg_debug (this->name, 0, "fixing assignment on %s", local->loc.path); goto selfheal; } dht_layout_set (this, local->inode, layout); } dht_inode_ctx_time_update (local->inode, this, &local->stbuf, 1); <-- local->inode is NULL here as the directory was not found on any brick. --- Additional comment from Worker Ant on 2016-11-15 10:15:55 EST --- REVIEW: http://review.gluster.org/15847 (cluster/dht: Check for null inode) posted (#1) for review on master by N Balachandran (nbalacha) --- Additional comment from Worker Ant on 2016-11-15 22:42:37 EST --- COMMIT: http://review.gluster.org/15847 committed in master by Atin Mukherjee (amukherj) ------ commit 8313d53accaa22feb14d284fb91245be0a32e16e Author: N Balachandran <nbalacha> Date: Tue Nov 15 20:40:08 2016 +0530 cluster/dht: Check for null inode Check for NULL inode before attempting to set dht inode ctx. Change-Id: I7693c18445f138221d8417df5e95b118cedb818a BUG: 1395261 Signed-off-by: N Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/15847 Smoke: Gluster Build System <jenkins.org> Reviewed-by: Shyamsundar Ranganathan <srangana> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Atin Mukherjee <amukherj>
REVIEW: http://review.gluster.org/15851 (cluster/dht: Check for null inode) posted (#1) for review on release-3.9 by N Balachandran (nbalacha)
COMMIT: http://review.gluster.org/15851 committed in release-3.9 by Pranith Kumar Karampuri (pkarampu) ------ commit 2173267f4a646ad2edef20a7791719168a9039c7 Author: N Balachandran <nbalacha> Date: Wed Nov 16 10:33:30 2016 +0530 cluster/dht: Check for null inode Check for NULL inode before attempting to set dht inode ctx. > Change-Id: I7693c18445f138221d8417df5e95b118cedb818a > BUG: 1395261 > Signed-off-by: N Balachandran <nbalacha> > Reviewed-on: http://review.gluster.org/15847 > Smoke: Gluster Build System <jenkins.org> > Reviewed-by: Shyamsundar Ranganathan <srangana> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Atin Mukherjee <amukherj> (cherry picked from commit 8313d53accaa22feb14d284fb91245be0a32e16e) Change-Id: Id8c7bfe181bb40a02cd49b0f5fc3b45cabf5afa6 BUG: 1395517 Signed-off-by: N Balachandran <nbalacha> Reviewed-on: http://review.gluster.org/15851 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Shyamsundar Ranganathan <srangana> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.1, please open a new bug report. glusterfs-3.9.1 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-January/029725.html [2] https://www.gluster.org/pipermail/gluster-users/