Description of problem: Have a 2 x 2 = 4 Gluster fs volume. All 4 servers also have this volume mounted using glusterfs fuse. One out of 3 servers started to have more and more intermittent "Permission Denied" errors while accessing random files. Brick log contains lots of these errors: [2016-04-15 00:01:01.534700] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51047: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied] [2016-04-15 00:01:01.537632] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51049: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied] [2016-04-15 00:01:01.541006] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51050: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied] [2016-04-15 00:01:01.543561] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51053: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied] Mount log has these corresponding log entries: [2016-04-15 02:40:01.740260] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (7b917079-4e50-4f85-b50e-a152e8f39769) [Permission denied] [2016-04-15 02:40:01.743183] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied] Version-Release number of selected component (if applicable): Gluster fs Version 3.7.9 running on OpenSuSE 13.2 How reproducible: Hard to reproduce. Some random files generate permission denied while accessing them. Additional info: None of those files are new files, so it's not the same as the other bug. Also quota is disabled. This only happens on one server, other 3 servers do not exhibit the same issue. I have recreated the entire volume after upgrading to 3.7.9 from 3.7.4 due to other issues. This server had the same problem in the past.
Hi, Can you check selinux settings on that specific server?
The SeLinux is completely disabled on all servers. $ls -Z / ? bin ? boot ? dev ? etc ? export ? home ? lib ? lib64 ? mnt ? opt ? proc ? root ? run ? sbin ? selinux ? srv ? sys ? tmp ? usr ? var $ls -la /selinux/ total 0 drwxr-xr-x 2 root root 6 Sep 25 2014 . drwxr-xr-x 23 root root 270 Mar 19 17:40 ..
Removing Triaged since it was not really looked at. I do not have SE Linux enabled.
I can observe this bug is similar to Bug#1229422. Fixes are posted. Although the bug#1229422 says quota is enabled (and yours say quota is disabled), this may be due to other reasons also. Please check http://review.gluster.org/#/c/13862/ commit message.
I've upgraded GlusterFS on the server with problems. Made sure that no processes using old version. Re-mounted the volume. Other 3 are still running 3.7.9. The problem still remains: ==> /var/log/glusterfs/bricks/export.log <== [2016-05-01 15:18:06.447703] E [MSGID: 115050] [server-rpc-fops.c:179:server_lookup_cbk] 0-gv1-server: 111: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/CMTSchecker) ==> (Permission denied) [Permission denied] ==> /var/log/glusterfs/mnt-data.log <== [2016-05-01 15:18:06.447926] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (f7e7dfd2-86c9-4590-b6ce-83e07f1de327) [Permission denied] Other servers only have warnings in /var/log/glusterfs/mnt-data.log: [2016-05-01 15:20:01.702505] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (7b917079-4e50-4f85-b50e-a152e8f39769) [Permission denied] [2016-05-01 15:20:01.707055] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]
Any news? Still seeing lots of errors on that one host. Other 3 hosts are running fine. But would like to put it back into rotation. Bumping version, since problem exists with last released version as well.
Can you please attach brick logs? Moving back to NEW as we've not identified the component yet.
Created attachment 1154213 [details] Brik log Here you go. It's more of the same thing.
Any news? I am still unable to use this one host.
Hi Raghavendra, Any clues for further debugging?
Hi, any updates? Any more information you need from me to troubleshoot this issue?
Think I might found the problem. Oh this server the mounted file system mount point was set 0750 permissions. After changing that to 755 errors went away. Any ideas why this might cause those permission denied errors? Gulsterfsd daemon is running as root.
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.