+++ This bug was initially created as a clone of Bug #1296048 +++ Description of problem: ======================= Created a 2x(4+2) EC volume and nfs mounted on the client with quota and uss enabled. Started IO (linux untar, mkdir's, dd (parallel - 1000's). Tried attaching the tier (2x2 dist-rep) and seeing invalid argument errors continuously. The same errors were seen during detach-tier but IO resumed after some time. In this case, complete IO fails with the error messages. If quota and uss are turned off, below errors are seen for some time and then the IO resumes. tar: linux-4.1.1/Documentation/devicetree/bindings/input/touchscreen/zforce_ts.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt tar: linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/ linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt: Cannot open: File exists linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt tar: linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt: Cannot open: File exists Version-Release number of selected component (if applicable): =============================================================== 3.7.5-14 How reproducible: ================= 100% Steps to Reproduce: 1. Create a disperse volume 2x(4+2) 2. NFS mount on the client. 3. Start IO (linux untar - 2 instances, mkdir (1000 in parallel), dd (1000 in parallel) 4. Attach tier (2x2 dist-rep) volume Actual results: =============== Invalid argument errors Expected results: ================= No errors to be seen and IO should be smooth. Additional info: ================ sosrepots in rhsqe. --- Additional comment from RHEL Product and Program Management on 2016-01-06 05:10:03 EST --- This request has been proposed as a blocker, but a release flag has not been requested. Please set a release flag to ? to ensure we may track this bug against the appropriate upcoming release, and reset the blocker flag to ?. --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-01-06 09:28:10 EST --- This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag.
RCA: After add-brick, NFS server will be restarted to load new graph. ie NFS server inode table will be fresh after restarting the process. So as part of the fop, resolver will send a lookup on an entry if inode is not lookedup before. During the lookup if healing requires for the entry from DHT (when directories are not present on all of the subvol), we will initiate a healing to create the directories on all of the subvolume. As part of the healing, we are doing a series of named lookup on all the parents starting from root if the inodes are not present, so for a successful lookup we will link the inode to inode table also. This lookup will be initiated from dht, so inode ctx will be created only for the xlators which are beneath of dht. Since we already linked the inode, ie resolver will not do a lookup for next fop. So xlator which are above dht will not have inode ctx. Here in this case, svc_access was complaining about missing inode_ctx. Possible solutions: 1) Move dht healing code to interface layer, if healing is required then dht should let the interface layer about healing, and need to give a path to heal. So that each interface layer should do a healing which include fuse, nfs, gfapi. 2) Do not link the inode from any of the xlators other than master xlators, ie do not link from dht. This will cause a huge performance degradation in healing code path, and we might need to do some hack to heal without a linked inode. 3) During resolving of an entry, currently resolving will be successful if there is an inode in the inode table. Make an extra check to see if the inode_ctx is present or not, if inode_ctx is not present for a linked inode, then resolver should consider as an invalid inode and need to do a lookup with the same inode.
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#1) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13227 (snapview-client: remove check for parent inode type) posted (#1) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#2) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi kc (rkavunga)
COMMIT: http://review.gluster.org/13227 committed in master by Rajesh Joseph (rjoseph) ------ commit e9d20f0e1a17e0ef2f148a9b49d1cf9596cbe609 Author: Mohammed Rafi KC <rkavunga> Date: Tue Jan 12 12:13:15 2016 +0530 snapview-client: remove check for parent inode type In lookup call back path, we are checking for parent inode type to make sure parent inode was set properly, so that next op on the inode can be successfuly completed. More info can be found at bugid 1297311 Change-Id: Ifb1b17f472f855f9b12088c3dd8328389d895e77 BUG: 1297311 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/13227 Reviewed-by: Avra Sengupta <asengupt> Tested-by: NetBSD Build System <jenkins.org> Reviewed-by: Rajesh Joseph <rjoseph>
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#3) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#4) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#6) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not set) posted (#7) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not set) posted (#6) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#5) for review on master by mohammed rafi kc (rkavunga)
COMMIT: http://review.gluster.org/13224 committed in master by Dan Lambright (dlambrig) ------ commit 14f925f5262ecabb2faf8142267c37103413e189 Author: Mohammed Rafi KC <rkavunga> Date: Tue Jan 12 12:04:59 2016 +0530 nfs: send lookup if inode_ctx is not set During resolving of an entry or inode, if inode ctx was not set, we will send a lookup. This patch also make sure that inode_ctx will be created after every inode_link. Change-Id: I137a7e2510635ff4ea6d007b671961341f89c949 BUG: 1297311 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/13224 Reviewed-by: soumya k <skoduri> Tested-by: NetBSD Build System <jenkins.org> Reviewed-by: Dan Lambright <dlambrig> Tested-by: Dan Lambright <dlambrig>
REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not set) posted (#6) for review on master by Dan Lambright (dlambrig)
COMMIT: http://review.gluster.org/13225 committed in master by Dan Lambright (dlambrig) ------ commit ad8c65612d6d7e32c06fd3aed11b7a9edd514607 Author: Mohammed Rafi KC <rkavunga> Date: Tue Jan 12 12:54:42 2016 +0530 fuse: send lookup if inode_ctx is not set During resolving of an entry or inode, if inode ctx was not set, we will send a lookup. This patch also make sure that inode_ctx will be created after every inode_link Change-Id: I4211533ca96a51b89d9f010fc57133470e52dc11 BUG: 1297311 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/13225 Reviewed-by: Dan Lambright <dlambrig> Tested-by: Dan Lambright <dlambrig>
COMMIT: http://review.gluster.org/13226 committed in master by Dan Lambright (dlambrig) ------ commit 252fe523e2932e83a98f203f71678f391eee6b22 Author: Mohammed Rafi KC <rkavunga> Date: Tue Jan 12 15:04:46 2016 +0530 gfapi: send lookup if inode_ctx is not set During resolving of an entry or inode, if inode ctx was not set, we will send a lookup to pupulate inode ctx for every xlators This patch also make sure that inode_ctx will be created after every inode_link. We will store inode_ctx value as LOOKUP_NEEDED if the inode is liked via readdirp, in all other case we will store inode_ctx value as LOOKUP_NOT_NEEDED. Change-Id: I3a10c298944200fa3862127187ae8988e582d352 BUG: 1297311 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/13226 Reviewed-by: Poornima G <pgurusid> Tested-by: NetBSD Build System <jenkins.org> Reviewed-by: Shyamsundar Ranganathan <srangana> Reviewed-by: Dan Lambright <dlambrig> Tested-by: Dan Lambright <dlambrig>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user