+++ This bug was initially created as a clone of Bug #1297311 +++

+++ This bug was initially created as a clone of Bug #1296048 +++

Description of problem:
=======================

Created a 2x(4+2) EC volume and nfs mounted on the client with quota and uss enabled. Started IO (linux untar, mkdir's, dd (parallel - 1000's). Tried attaching the tier (2x2 dist-rep) and seeing invalid argument errors continuously. The same errors were seen during detach-tier but IO resumed after some time. In this case, complete IO fails with the error messages. 

If quota and uss are turned off, below errors are seen for some time and then the IO resumes.

tar: linux-4.1.1/Documentation/devicetree/bindings/input/touchscreen/zforce_ts.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt: Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt: Cannot open: File exists


Version-Release number of selected component (if applicable):
===============================================================
3.7.5-14

How reproducible:
=================
100%

Steps to Reproduce:
1. Create a disperse volume 2x(4+2)
2. NFS mount on the client.
3. Start IO (linux untar - 2 instances, mkdir (1000 in parallel), dd (1000 in parallel)
4. Attach tier (2x2 dist-rep) volume


Actual results:
===============
Invalid argument errors

Expected results:
=================
No errors to be seen and IO should be smooth.

Additional info:
================
sosrepots in rhsqe.

--- Additional comment from RHEL Product and Program Management on 2016-01-06 05:10:03 EST ---

This request has been proposed as a blocker, but a release flag has
not been requested. Please set a release flag to ? to ensure we may
track this bug against the appropriate upcoming release, and reset
the blocker flag to ?.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-01-06 09:28:10 EST ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Mohammed Rafi KC on 2016-01-11 04:02:31 EST ---

RCA:


After add-brick, NFS server will be restarted to load new graph. ie NFS server inode table will be fresh after restarting the process. So as part of the fop, resolver will send a lookup on an entry if inode is not lookedup before. During the lookup if healing requires for the entry from DHT (when directories are not present on all of the subvol), we will initiate a healing to create the directories on all of the subvolume. As part of the healing, we are doing a series of named lookup on all the parents starting from root if the inodes are not present, so for a successful lookup we will link the inode to inode table also. This lookup will be initiated from dht, so inode ctx will be created only for the xlators which are beneath of dht.

Since we already linked the inode, ie  resolver will not do a lookup for next fop. So xlator which are above dht will not have inode ctx.

Here in this case, svc_access was complaining about missing inode_ctx.

Possible solutions:

1)  Move dht healing code to interface layer, if healing is required then dht should let the interface layer about healing, and need to give a path to heal. So that each interface layer should do a healing which include fuse, nfs, gfapi.

2) Do not link the inode from any of the xlators other than master xlators, ie do not link from dht. This will cause a huge performance degradation in healing code path, and we might need to do some hack to heal without a linked inode.

3) During resolving of an entry, currently resolving will be successful if there is an inode in the inode table. Make an extra check to see if the inode_ctx is present or not, if inode_ctx is not present for a linked inode, then resolver should consider as an invalid inode and need to do a lookup with the same inode.

--- Additional comment from Vijay Bellur on 2016-01-12 01:50:20 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 02:37:09 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 04:36:11 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 04:40:38 EST ---

REVIEW: http://review.gluster.org/13227 (snapview-client: remove check for parent inode type) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 07:30:19 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 07:30:28 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 07:34:32 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 08:52:19 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-12 09:04:10 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 01:07:04 EST ---

COMMIT: http://review.gluster.org/13227 committed in master by Rajesh Joseph (rjoseph) 
------
commit e9d20f0e1a17e0ef2f148a9b49d1cf9596cbe609
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue Jan 12 12:13:15 2016 +0530

    snapview-client: remove check for parent inode type
    
    In lookup call back path, we are checking for parent inode
    type to make sure parent inode was set properly, so that
    next op on the inode can be successfuly completed.
    
    More info can be found at bugid 1297311
    
    Change-Id: Ifb1b17f472f855f9b12088c3dd8328389d895e77
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/13227
    Reviewed-by: Avra Sengupta <asengupt>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Rajesh Joseph <rjoseph>

--- Additional comment from Vijay Bellur on 2016-01-13 01:52:14 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 01:52:22 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 01:59:13 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 04:03:18 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 04:03:23 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 05:54:40 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 05:58:55 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#6) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 06:33:53 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#7) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 08:14:27 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#6) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 08:14:32 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Vijay Bellur on 2016-01-13 20:35:26 EST ---

COMMIT: http://review.gluster.org/13224 committed in master by Dan Lambright (dlambrig) 
------
commit 14f925f5262ecabb2faf8142267c37103413e189
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue Jan 12 12:04:59 2016 +0530

    nfs: send lookup if inode_ctx is not set
    
    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup.
    
    This patch also make sure that inode_ctx will be created
    after every inode_link.
    
    Change-Id: I137a7e2510635ff4ea6d007b671961341f89c949
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/13224
    Reviewed-by: soumya k <skoduri>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>

--- Additional comment from Vijay Bellur on 2016-01-13 22:20:32 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#6) for review on master by Dan Lambright (dlambrig)

--- Additional comment from Vijay Bellur on 2016-01-13 22:26:46 EST ---

COMMIT: http://review.gluster.org/13225 committed in master by Dan Lambright (dlambrig) 
------
commit ad8c65612d6d7e32c06fd3aed11b7a9edd514607
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue Jan 12 12:54:42 2016 +0530

    fuse: send lookup if inode_ctx is not set
    
    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup.
    
    This patch also make sure that inode_ctx will be created
    after every inode_link
    
    Change-Id: I4211533ca96a51b89d9f010fc57133470e52dc11
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/13225
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>

--- Additional comment from Vijay Bellur on 2016-01-13 22:27:49 EST ---

COMMIT: http://review.gluster.org/13226 committed in master by Dan Lambright (dlambrig) 
------
commit 252fe523e2932e83a98f203f71678f391eee6b22
Author: Mohammed Rafi KC <rkavunga>
Date:   Tue Jan 12 15:04:46 2016 +0530

    gfapi: send lookup if inode_ctx is not set
    
    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup to pupulate inode
    ctx for every xlators
    
    This patch also make sure that inode_ctx will be created
    after every inode_link. We will store inode_ctx value as
    LOOKUP_NEEDED if the inode is liked via readdirp, in all
    other case we will store inode_ctx value as LOOKUP_NOT_NEEDED.
    
    Change-Id: I3a10c298944200fa3862127187ae8988e582d352
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: http://review.gluster.org/13226
    Reviewed-by: Poornima G <pgurusid>
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Dan Lambright <dlambrig>
    Tested-by: Dan Lambright <dlambrig>