+++ This bug was initially created as a clone of Bug #1318204 +++ Description of problem: When doing a chmod -R on an NFS mounted volume, I get an Input / Ouput error on each file processed. After speaking on IRC about it, it seems we are two encountering that bug at least, on Debian 8 with the same version. No problem on previous versions though. Version-Release number of selected component (if applicable): glusterfs 3.7.8 built on Feb 11 2016 05:58:55 How reproducible: ? Steps to Reproduce: 1. Install Debian 8 2. Install the latest glusterFS from gluster repos 3. Create a volume 4. Mount it with NFS 5. chmod -R some files on it Actual results: Input / Output error on each file Expected results: No errors Additional info: Doesn't seem to happen with the same setup but older versions of glusterfs. We spoke about it on IRC with jiffin on 15/03/2016 around 2 pm UTC, who asked for us to file a bug. --- Additional comment from Jules on 2016-03-16 05:52:27 EDT --- Having exactly same beavior on Debian 8 with lastest GlusterFS. nfs.log showing me: [2016-03-15 14:07:43.776687] W [MSGID: 112199] [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: <gfid:a6ee3547-fd39-40ca-b381-041be04bc732>/StaticRecordCollection.php => (XID: d6165f74, SETATTR: NFS: 5(I/O error), POSIX: 5(Input/output error)) on all files that produce Output "Input / Output error on each file" while doing chmod -R or chown -R If i recursively change file permissions on directories with just a few files the error doesn't occur. So it seems to be an issue with the mass of files that passing a permission change same time. --- Additional comment from Soumya Koduri on 2016-03-22 08:36:14 EDT --- Please provide nfs logs and brick logs. Also it will be helpful if you could collect packet trace (while the test is being executed) on the node where gluster-nfs server is running using below command - tcpdump -i any -s 0 -w /var/tmp/nfs-hang.pcap tcp and not port 22 If possible, please collect the packet trace in the older version servers as well. --- Additional comment from Jules on 2016-04-13 08:17:52 EDT --- glusterfs-server version 3.7.10-1 seems to have fixed this issue but brought a new one. All my Glusterfs setups seem to loose their Quorum without any notice in log files at irregular days. I only can identify this after all client mounts went to "readonly filesystem" state and following log lines appear in nfs.log: [2016-04-13 06:19:12.797704] W [MSGID: 112199] [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: /www/CENSORED/shared/usersettings/1/settings.conf => (XID: 3e2a6261, SETATTR: NFS: 30(Read-only file system), POSIX: 30(Read-only file system)) [Invalid argument] for me it looks like it was an attempt to fix the previous reported bug and brought another one. --- Additional comment from Vijay Bellur on 2016-06-06 08:52:39 EDT --- REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#1) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-07 02:41:39 EDT --- REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#2) for review on master by jiffin tony Thottan (jthottan) --- Additional comment from Vijay Bellur on 2016-06-07 03:08:33 EDT --- REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#3) for review on master by jiffin tony Thottan (jthottan)
REVIEW: http://review.gluster.org/14659 (nfs : store sattr properly in nfs3_setattr() call) posted (#1) for review on release-3.7 by Manikandan Selvaganesh (mselvaga)
REVIEW: http://review.gluster.org/14659 (nfs : store sattr properly in nfs3_setattr() call) posted (#2) for review on release-3.7 by Manikandan Selvaganesh (mselvaga)
COMMIT: http://review.gluster.org/14659 committed in release-3.7 by Kaushal M (kaushal) ------ commit c04ee47d1e9847b50c459734a42681450005ee60 Author: Jiffin Tony Thottan <jthottan> Date: Mon Jun 6 18:10:09 2016 +0530 nfs : store sattr properly in nfs3_setattr() call nfs3_setattr stores the input arguments in cs->stbuf. However, inode/entry resolution code overwrites cs->stbuf after a successful resolution, thereby overwriting the input arguments with iatt values stored on backend. Hence operations like chmod/chown turns out to be a NOP. Specifically following are the functions that overwrite cs->stbuf: nfs3_fh_resolve_inode_lookup_cbk nfs3_fh_resolve_entry_lookup_cbk Since we resort to inode resolution only when inode is not found in inode table and lru limit guards the number of inodes in itable, we run into this issue only when the data set is bigger than lru limit of itable. Fix is to store input arguments in a member other than cs->stbuf. Thanks Du for suggesting the fix Backport of http://review.gluster.org/#/c/14657/ > Change-Id: I7caef48839d4f177c3557d7823fc1d35c8294939 > BUG: 1318204 > Signed-off-by: Jiffin Tony Thottan <jthottan> > Signed-off-by: Manikandan Selvaganesh <mselvaga> Change-Id: I7caef48839d4f177c3557d7823fc1d35c8294939 BUG: 1343362 Signed-off-by: Manikandan Selvaganesh <mselvaga> Reviewed-on: http://review.gluster.org/14659 Reviewed-by: jiffin tony Thottan <jthottan> Reviewed-by: Niels de Vos <ndevos> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> Smoke: Gluster Build System <jenkins.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.12, please open a new bug report. glusterfs-3.7.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://www.gluster.org/pipermail/gluster-devel/2016-June/049918.html [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user