Bug 1343368

+++ This bug was initially created as a clone of Bug #1343362 +++

+++ This bug was initially created as a clone of Bug #1318204 +++

Description of problem:
When doing a chmod -R on an NFS mounted volume, I get an Input / Ouput error on each file processed. After speaking on IRC about it, it seems we are two encountering that bug at least, on Debian 8 with the same version.
No problem on previous versions though.

Version-Release number of selected component (if applicable):
glusterfs 3.7.8 built on Feb 11 2016 05:58:55

How reproducible:
?

Steps to Reproduce:
1. Install Debian 8
2. Install the latest glusterFS from gluster repos
3. Create a volume
4. Mount it with NFS
5. chmod -R some files on it

Actual results:
Input / Output error on each file

Expected results:
No errors

Additional info:
Doesn't seem to happen with the same setup but older versions of glusterfs.
We spoke about it on IRC with jiffin on 15/03/2016 around 2 pm UTC, who asked for us to file a bug.

--- Additional comment from Jules on 2016-03-16 05:52:27 EDT ---

Having exactly same beavior on Debian 8 with lastest GlusterFS.

nfs.log showing me: [2016-03-15 14:07:43.776687] W [MSGID: 112199] [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: <gfid:a6ee3547-fd39-40ca-b381-041be04bc732>/StaticRecordCollection.php => (XID: d6165f74, SETATTR: NFS: 5(I/O error), POSIX: 5(Input/output error))
on all files that produce Output "Input / Output error on each file" while doing chmod -R or chown -R

If i recursively change file permissions on directories with just a few files the error doesn't occur. So it seems to be an issue with the mass of files that passing a permission change same time.

--- Additional comment from Soumya Koduri on 2016-03-22 08:36:14 EDT ---

Please provide nfs logs and brick logs. Also it will be helpful if you could collect packet trace  (while the test is being executed) on the node where gluster-nfs server is running using below command -

 tcpdump -i any -s 0 -w /var/tmp/nfs-hang.pcap tcp and not port 22

If possible, please collect the packet trace in the older version servers as well.

--- Additional comment from Jules on 2016-04-13 08:17:52 EDT ---

glusterfs-server version 3.7.10-1 seems to have fixed this issue but brought a new one.

All my Glusterfs setups seem to loose their Quorum without any notice in log files at irregular days. 
I only can identify this after all client mounts went to "readonly filesystem" state and following log lines appear in nfs.log:

[2016-04-13 06:19:12.797704] W [MSGID: 112199] [nfs3-helpers.c:3418:nfs3_log_common_res] 0-nfs-nfsv3: /www/CENSORED/shared/usersettings/1/settings.conf => (XID: 3e2a6261, SETATTR: NFS: 30(Read-only file system), POSIX: 30(Read-only file system)) [Invalid argument]

for me it looks like it was an attempt to fix the previous reported bug and brought another one.

--- Additional comment from Vijay Bellur on 2016-06-06 08:52:39 EDT ---

REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-07 02:41:39 EDT ---

REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-07 03:08:33 EDT ---

REVIEW: http://review.gluster.org/14657 (nfs : store sattr properly in nfs3_setattr() call) posted (#3) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-06-07 04:35:14 EDT ---

REVIEW: http://review.gluster.org/14659 (nfs : store sattr properly in nfs3_setattr() call) posted (#1) for review on release-3.7 by Manikandan Selvaganesh (mselvaga)