+++ This bug was initially created as a clone of Bug #1392445 +++

+++ This bug was initially created as a clone of Bug #1370350 +++

Description of problem:
-----------------------
After replacing the defunct brick of replica 3 sharded volume, the hosted engine VM running with its image on that volume went to paused state.

Fuse mount logs showed EIO

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHGS 3.1.3
RHV 4.0.2

How reproducible:
-----------------
1/1

Steps to Reproduce:
-------------------
1. Create a replica 3 sharded volume optimized for VM store
2. Create a hosted engine VM with this volume as a 'data domain'
3. After hosted-engine up and operational, kill one of the bricks of the volume
4. Add a new node to the cluster
5. Replace the old-brick with the new-brick from the newly added node to the cluster

Actual results:
---------------
hosted-engine vm went to paused state, fuse mount showed EIO with error 'Lookup on shard 3 failed.'

Expected results:
-----------------
There shouldn't be any error messages, after performing replace brick operation

--- Additional comment from SATHEESARAN on 2016-08-25 22:38:05 EDT ---


1. Setup details
----------------
   4 Nodes in the cluster, all running RHGS 3.1.3 ( glusterfs-3.7.9.10.el7rhgs )
   Hosts are :
   cambridge.lab.eng.blr.redhat.com ( 10.70.37.73 )
   zod.lab.eng.blr.redhat.com ( 10.70.37.76 )
   tettnang.lab.eng.blr.redhat.com ( 10.70.37.77 ) <-- This host is defunct for more than 12 hours
   
   ** Here is the newly added host ** :
   yarrow.lab.eng.blr.redhat.com ( 10.70.37.78 )


2. Peer status
--------------

[root@cambridge ~]# gluster pe s
Number of Peers: 3

Hostname: tettnang-nic2.lab.eng.blr.redhat.com
Uuid: d9cd9f98-6dc3-436f-b6ca-aab0b059fc41
State: Peer in Cluster (Disconnected)
Other names:
10.70.36.77

Hostname: 10.70.36.76
Uuid: 7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a
State: Peer in Cluster (Connected)

Hostname: yarrow.lab.eng.blr.redhat.com
Uuid: 3d98430d-50f3-4aa8-9389-ac133a58c9b3
State: Peer in Cluster (Connected)

----------

[root@cambridge ~]# gluster pool list
UUID					Hostname                            	State
d9cd9f98-6dc3-436f-b6ca-aab0b059fc41	tettnang-nic2.lab.eng.blr.redhat.com	Disconnected 
7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a	10.70.36.76                         	Connected 
3d98430d-50f3-4aa8-9389-ac133a58c9b3	yarrow.lab.eng.blr.redhat.com       	Connected 
3c3ad2a9-e8f8-496b-bdde-1c0777552dee	localhost                           	Connected 

3. Gluster volume details
-------------------------
[root@cambridge ~]# gluster volume info enginevol
 
Volume Name: enginevol
Type: Replicate
Volume ID: 46a261df-f527-479c-9776-4bdb21fe19b1
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.36.73:/rhgs/engine/enginebrick
Brick2: 10.70.36.76:/rhgs/engine/enginebrick
Brick3: yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick <-- new brick
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
performance.strict-o-direct: on
network.ping-timeout: 30
user.cifs: off
nfs.disable: on

Note: the old brick was - 10.70.36.77:/rhgs/engine/enginebrick

4. volume status
----------------
[root@cambridge ~]# gluster volume status enginevol
Status of volume: enginevol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.36.73:/rhgs/engine/enginebrick  49152     0          Y       18194
Brick 10.70.36.76:/rhgs/engine/enginebrick  49152     0          Y       15595
Brick yarrow.lab.eng.blr.redhat.com:/rhgs/e
ngine/enginebrick                           49152     0          Y       19457
Self-heal Daemon on localhost               N/A       N/A        Y       12863
Self-heal Daemon on yarrow.lab.eng.blr.redh
at.com                                      N/A       N/A        Y       19462
Self-heal Daemon on 10.70.36.76             N/A       N/A        Y       32641
 
Task Status of Volume enginevol
------------------------------------------------------------------------------
There are no active volume tasks

--- Additional comment from SATHEESARAN on 2016-08-25 22:39:22 EDT ---

self-heal and split-brain info

[root@cambridge ~]# gluster volume heal enginevol info
Brick 10.70.36.73:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 
Status: Connected
Number of entries: 31

Brick 10.70.36.76:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 
Status: Connected
Number of entries: 31

Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
Status: Connected
Number of entries: 2

[root@cambridge ~]# gluster volume heal enginevol info split-brain
Brick 10.70.36.73:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

Brick 10.70.36.76:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

--- Additional comment from SATHEESARAN on 2016-08-25 22:45:40 EDT ---

[2016-08-25 08:16:44.373211] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:16:44.373283] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:16:44.373343] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:16:44.374685] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db [Input/output error]
[2016-08-25 08:16:44.374734] W [fuse-bridge.c:2224:fuse_readv_cbk] 0-glusterfs-fuse: 3986689: READ => -1 (Input/output error)
[2016-08-25 08:37:22.183269] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:37:22.183355] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:37:22.183414] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-08-25 08:37:22.184586] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db [Input/output error]
[2016-08-25 08:37:22.184625] W [fuse-bridge.c:2224:fuse_readv_cbk] 0-glusterfs-fuse: 4023987: READ => -1 (Input/output error)

--- Additional comment from SATHEESARAN on 2016-08-25 22:46:06 EDT ---

comment3 is the snip from fuse mount.log

--- Additional comment from SATHEESARAN on 2016-08-30 23:05:51 EDT ---

I couldn't hit back for the second time, when Krutika asked for debug enabled logs.

Later, Krutika also confirmed that the community user too seeing this problem

--- Additional comment from Krutika Dhananjay on 2016-09-13 09:58:51 EDT ---

(In reply to SATHEESARAN from comment #5)
> I couldn't hit back for the second time, when Krutika asked for debug
> enabled logs.
> 
> Later, Krutika also confirmed that the community user too seeing this problem

I stand corrected. I figured later that it was with granular-entry-heal enabled and a case where the same brick was wiped off and healed. The issue there was a combination of dated documentation and the lack of reset-brick functionality.

I did see logs of the kind you have pasted in comment #3, of lookups failing with EINVAL but no input/output errors.

-Krutika

--- Additional comment from Krutika Dhananjay on 2016-09-13 10:06:52 EDT ---

Sas,

Do you have the logs from this run, of the bricks, shds and the clients?

-Krutika

--- Additional comment from SATHEESARAN on 2016-10-14 06:55:56 EDT ---

(In reply to Krutika Dhananjay from comment #7)
> Sas,
> 
> Do you have the logs from this run, of the bricks, shds and the clients?
> 
> -Krutika

Hi Krutika,

I have missed the logs.
I will try to reproduce this issue.

But could you guess any problems with the error messages in comment3 ?

--- Additional comment from Krutika Dhananjay on 2016-10-24 03:08:04 EDT ---

Not quite, Sas. The EINVAL seems to be getting propagated by the brick(s) since protocol/client which is the lowest layer in the client stack is receiving EINVAL from over the network.

-Krutika

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-11-07 05:23:09 EST ---

This bug is automatically being provided 'pm_ack+' for the release flag 'rhgs‑3.2.0', the current release of Red Hat Gluster Storage 3 under active development, having been appropriately marked for the release, and having been provided ACK from Development and QE

If the 'blocker' flag had been proposed/set on this BZ, it has now been unset, since the 'blocker' flag is not valid for the current phase of RHGS 3.2.0 development

--- Additional comment from Worker Ant on 2016-11-07 09:17:21 EST ---

REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too for named lookups on individual shards) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Worker Ant on 2016-11-08 03:45:10 EST ---

REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too for named lookups on individual shards) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Worker Ant on 2016-11-08 06:05:31 EST ---

COMMIT: http://review.gluster.org/15788 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit e9023083b3a165390a8cc8fc77253f354744e81a
Author: Krutika Dhananjay <kdhananj>
Date:   Mon Nov 7 16:06:56 2016 +0530

    features/shard: Fill loc.pargfid too for named lookups on individual shards
    
    On a sharded volume when a brick is replaced while IO is going on, named
    lookup on individual shards as part of read/write was failing with
    ENOENT on the replaced brick, and as a result AFR initiated name heal in
    lookup callback. But since pargfid was empty (which is what this patch
    attempts to fix), the resolution of the shards by protocol/server used
    to fail and the following pattern of logs was seen:
    
    Brick-logs:
    
    [2016-11-08 07:41:49.387127] W [MSGID: 115009]
    [server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
    for (null) (LOOKUP)
    [2016-11-08 07:41:49.387157] E [MSGID: 115050]
    [server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
    (00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
    ==> (Invalid argument) [Invalid argument]
    
    Client-logs:
    [2016-11-08 07:41:27.497687] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.497755] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.498500] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.499680] E [MSGID: 133010]
    
    Also, this patch makes AFR by itself choose a non-NULL pargfid even if
    its ancestors fail to initialize all pargfid placeholders.
    
    Change-Id: I5f85b303ede135baaf92e87ec8e09941f5ded6c1
    BUG: 1392445
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/15788
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Smoke: Gluster Build System <jenkins.org>