+++ This bug was initially created as a clone of Bug #1370350 +++ Description of problem: ----------------------- After replacing the defunct brick of replica 3 sharded volume, the hosted engine VM running with its image on that volume went to paused state. Fuse mount logs showed EIO Version-Release number of selected component (if applicable): -------------------------------------------------------------- RHGS 3.1.3 RHV 4.0.2 How reproducible: ----------------- 1/1 Steps to Reproduce: ------------------- 1. Create a replica 3 sharded volume optimized for VM store 2. Create a hosted engine VM with this volume as a 'data domain' 3. After hosted-engine up and operational, kill one of the bricks of the volume 4. Add a new node to the cluster 5. Replace the old-brick with the new-brick from the newly added node to the cluster Actual results: --------------- hosted-engine vm went to paused state, fuse mount showed EIO with error 'Lookup on shard 3 failed.' Expected results: ----------------- There shouldn't be any error messages, after performing replace brick operation --- Additional comment from SATHEESARAN on 2016-08-25 22:38:05 EDT --- 1. Setup details ---------------- 4 Nodes in the cluster, all running RHGS 3.1.3 ( glusterfs-3.7.9.10.el7rhgs ) Hosts are : cambridge.lab.eng.blr.redhat.com ( 10.70.37.73 ) zod.lab.eng.blr.redhat.com ( 10.70.37.76 ) tettnang.lab.eng.blr.redhat.com ( 10.70.37.77 ) <-- This host is defunct for more than 12 hours ** Here is the newly added host ** : yarrow.lab.eng.blr.redhat.com ( 10.70.37.78 ) 2. Peer status -------------- [root@cambridge ~]# gluster pe s Number of Peers: 3 Hostname: tettnang-nic2.lab.eng.blr.redhat.com Uuid: d9cd9f98-6dc3-436f-b6ca-aab0b059fc41 State: Peer in Cluster (Disconnected) Other names: 10.70.36.77 Hostname: 10.70.36.76 Uuid: 7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a State: Peer in Cluster (Connected) Hostname: yarrow.lab.eng.blr.redhat.com Uuid: 3d98430d-50f3-4aa8-9389-ac133a58c9b3 State: Peer in Cluster (Connected) ---------- [root@cambridge ~]# gluster pool list UUID Hostname State d9cd9f98-6dc3-436f-b6ca-aab0b059fc41 tettnang-nic2.lab.eng.blr.redhat.com Disconnected 7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a 10.70.36.76 Connected 3d98430d-50f3-4aa8-9389-ac133a58c9b3 yarrow.lab.eng.blr.redhat.com Connected 3c3ad2a9-e8f8-496b-bdde-1c0777552dee localhost Connected 3. Gluster volume details ------------------------- [root@cambridge ~]# gluster volume info enginevol Volume Name: enginevol Type: Replicate Volume ID: 46a261df-f527-479c-9776-4bdb21fe19b1 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.36.73:/rhgs/engine/enginebrick Brick2: 10.70.36.76:/rhgs/engine/enginebrick Brick3: yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick <-- new brick Options Reconfigured: performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: off cluster.quorum-type: auto cluster.server-quorum-type: server storage.owner-uid: 36 storage.owner-gid: 36 features.shard: on features.shard-block-size: 512MB performance.low-prio-threads: 32 cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 performance.strict-o-direct: on network.ping-timeout: 30 user.cifs: off nfs.disable: on Note: the old brick was - 10.70.36.77:/rhgs/engine/enginebrick 4. volume status ---------------- [root@cambridge ~]# gluster volume status enginevol Status of volume: enginevol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.36.73:/rhgs/engine/enginebrick 49152 0 Y 18194 Brick 10.70.36.76:/rhgs/engine/enginebrick 49152 0 Y 15595 Brick yarrow.lab.eng.blr.redhat.com:/rhgs/e ngine/enginebrick 49152 0 Y 19457 Self-heal Daemon on localhost N/A N/A Y 12863 Self-heal Daemon on yarrow.lab.eng.blr.redh at.com N/A N/A Y 19462 Self-heal Daemon on 10.70.36.76 N/A N/A Y 32641 Task Status of Volume enginevol ------------------------------------------------------------------------------ There are no active volume tasks --- Additional comment from SATHEESARAN on 2016-08-25 22:39:22 EDT --- self-heal and split-brain info [root@cambridge ~]# gluster volume heal enginevol info Brick 10.70.36.73:/rhgs/engine/enginebrick /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 Status: Connected Number of entries: 31 Brick 10.70.36.76:/rhgs/engine/enginebrick /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 Status: Connected Number of entries: 31 Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 /.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 Status: Connected Number of entries: 2 [root@cambridge ~]# gluster volume heal enginevol info split-brain Brick 10.70.36.73:/rhgs/engine/enginebrick Status: Connected Number of entries in split-brain: 0 Brick 10.70.36.76:/rhgs/engine/enginebrick Status: Connected Number of entries in split-brain: 0 Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick Status: Connected Number of entries in split-brain: 0 --- Additional comment from SATHEESARAN on 2016-08-25 22:45:40 EDT --- [2016-08-25 08:16:44.373211] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:16:44.373283] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:16:44.373343] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:16:44.374685] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db [Input/output error] [2016-08-25 08:16:44.374734] W [fuse-bridge.c:2224:fuse_readv_cbk] 0-glusterfs-fuse: 3986689: READ => -1 (Input/output error) [2016-08-25 08:37:22.183269] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:37:22.183355] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:37:22.183414] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-08-25 08:37:22.184586] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db [Input/output error] [2016-08-25 08:37:22.184625] W [fuse-bridge.c:2224:fuse_readv_cbk] 0-glusterfs-fuse: 4023987: READ => -1 (Input/output error) --- Additional comment from SATHEESARAN on 2016-08-25 22:46:06 EDT --- comment3 is the snip from fuse mount.log --- Additional comment from SATHEESARAN on 2016-08-30 23:05:51 EDT --- I couldn't hit back for the second time, when Krutika asked for debug enabled logs. Later, Krutika also confirmed that the community user too seeing this problem --- Additional comment from Krutika Dhananjay on 2016-09-13 09:58:51 EDT --- (In reply to SATHEESARAN from comment #5) > I couldn't hit back for the second time, when Krutika asked for debug > enabled logs. > > Later, Krutika also confirmed that the community user too seeing this problem I stand corrected. I figured later that it was with granular-entry-heal enabled and a case where the same brick was wiped off and healed. The issue there was a combination of dated documentation and the lack of reset-brick functionality. I did see logs of the kind you have pasted in comment #3, of lookups failing with EINVAL but no input/output errors. -Krutika --- Additional comment from Krutika Dhananjay on 2016-09-13 10:06:52 EDT --- Sas, Do you have the logs from this run, of the bricks, shds and the clients? -Krutika --- Additional comment from SATHEESARAN on 2016-10-14 06:55:56 EDT --- (In reply to Krutika Dhananjay from comment #7) > Sas, > > Do you have the logs from this run, of the bricks, shds and the clients? > > -Krutika Hi Krutika, I have missed the logs. I will try to reproduce this issue. But could you guess any problems with the error messages in comment3 ? --- Additional comment from Krutika Dhananjay on 2016-10-24 03:08:04 EDT --- Not quite, Sas. The EINVAL seems to be getting propagated by the brick(s) since protocol/client which is the lowest layer in the client stack is receiving EINVAL from over the network. -Krutika --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-11-07 05:23:09 EST --- This bug is automatically being provided 'pm_ack+' for the release flag 'rhgs‑3.2.0', the current release of Red Hat Gluster Storage 3 under active development, having been appropriately marked for the release, and having been provided ACK from Development and QE If the 'blocker' flag had been proposed/set on this BZ, it has now been unset, since the 'blocker' flag is not valid for the current phase of RHGS 3.2.0 development
REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too for named lookups on individual shards) posted (#1) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too for named lookups on individual shards) posted (#2) for review on master by Krutika Dhananjay (kdhananj)
COMMIT: http://review.gluster.org/15788 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit e9023083b3a165390a8cc8fc77253f354744e81a Author: Krutika Dhananjay <kdhananj> Date: Mon Nov 7 16:06:56 2016 +0530 features/shard: Fill loc.pargfid too for named lookups on individual shards On a sharded volume when a brick is replaced while IO is going on, named lookup on individual shards as part of read/write was failing with ENOENT on the replaced brick, and as a result AFR initiated name heal in lookup callback. But since pargfid was empty (which is what this patch attempts to fix), the resolution of the shards by protocol/server used to fail and the following pattern of logs was seen: Brick-logs: [2016-11-08 07:41:49.387127] W [MSGID: 115009] [server-resolve.c:566:server_resolve] 0-rep-server: no resolution type for (null) (LOOKUP) [2016-11-08 07:41:49.387157] E [MSGID: 115050] [server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null) (00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82) ==> (Invalid argument) [Invalid argument] Client-logs: [2016-11-08 07:41:27.497687] W [MSGID: 114031] [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-11-08 07:41:27.497755] W [MSGID: 114031] [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-11-08 07:41:27.498500] W [MSGID: 114031] [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument] [2016-11-08 07:41:27.499680] E [MSGID: 133010] Also, this patch makes AFR by itself choose a non-NULL pargfid even if its ancestors fail to initialize all pargfid placeholders. Change-Id: I5f85b303ede135baaf92e87ec8e09941f5ded6c1 BUG: 1392445 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: http://review.gluster.org/15788 CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Ravishankar N <ravishankar> Reviewed-by: Pranith Kumar Karampuri <pkarampu> Smoke: Gluster Build System <jenkins.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/