The problem:

internal fops or management fops: fops that are not issued by the user application, for eg, the fops needed for self-heal.

These internal fops should be differentiated with the fops issued by the user in terms of the uid,gid that is used to issues these fops. Till now, we have been performing all fops on the bricks, specifically, in posix, as root user while depending on FUSE to perform the access control checks. This approach does not work for NFS since it caches file/dir attributes on the client. Here is how it fails:

On NFS client system:
1. Create a file with (uid,gid)=(65534,65531) with permissions as 644.
2. Change user using su to user with (uid,gid)=(65532,65532). For the purpose
of operating on the above file, this user will be considered as "Other" in the
file mode bits.
3. Since posix functions as root, any change to the file above by this user is allowed since posix is not performing a setfs[ug]id, i.e. the access checks pass as the modifying fop is being done as root.
4. Enabling the setfs[ug]id calls in posix makes this work correctly, due to obvious reasons. However, this change, I've been told will break translators like replicate which try to issue a self-heal using the (uid,gid) of the user that caused a stat fop on the file/dir. BUT, this user might not have the permission for a stat on the file/dir leading to a failure of the self-heal operation.

I'd request every translator maintainer to please change any internal/management fop in your translator to used root uid/gid by default while letting the user-generated fops to be run with the uid,gid of the user.