Bug 1327380

Summary: Random permission denied errors accessing files
Product: [Community] GlusterFS Reporter: Vitaliy Margolen <vitaliy>
Component: unclassifiedAssignee: bugs <bugs>
Status: CLOSED EOL QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.11CC: bugs, rgowdapp, sarumuga, skoduri
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: ykaul: needinfo+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-08 10:52:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Brik log none

Description Vitaliy Margolen 2016-04-15 03:12:14 UTC
Description of problem:
Have a 2 x 2 = 4 Gluster fs volume. All 4 servers also have this volume mounted using glusterfs fuse. One out of 3 servers started to have more and more intermittent "Permission Denied" errors while accessing random files. Brick log contains lots of these errors:
[2016-04-15 00:01:01.534700] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51047: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied]
[2016-04-15 00:01:01.537632] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51049: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied]
[2016-04-15 00:01:01.541006] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51050: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied]
[2016-04-15 00:01:01.543561] E [MSGID: 115050] [server-rpc-fops.c:178:server_lookup_cbk] 0-gv1-server: 51053: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/natcm) ==> (Permission denied) [Permission denied]

Mount log has these corresponding log entries:
[2016-04-15 02:40:01.740260] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (7b917079-4e50-4f85-b50e-a152e8f39769) [Permission denied]
[2016-04-15 02:40:01.743183] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]


Version-Release number of selected component (if applicable):
Gluster fs Version 3.7.9 running on OpenSuSE 13.2

How reproducible:
Hard to reproduce. Some random files generate permission denied while accessing them.

Additional info:
None of those files are new files, so it's not the same as the other bug. Also quota is disabled. This only happens on one server, other 3 servers do not exhibit the same issue.

I have recreated the entire volume after upgrading to 3.7.9 from 3.7.4 due to other issues. This server had the same problem in the past.

Comment 1 Saravanakumar 2016-04-19 12:32:33 UTC
Hi,

Can you check selinux settings on that specific server?

Comment 2 Vitaliy Margolen 2016-04-19 13:35:55 UTC
The SeLinux is completely disabled on all servers.
$ls -Z /
? bin  ? boot  ? dev  ? etc  ? export  ? home  ? lib  ? lib64  ? mnt  ? opt  ? proc  ? root  ? run  ? sbin  ? selinux  ? srv  ? sys  ? tmp  ? usr  ? var
$ls -la /selinux/
total 0
drwxr-xr-x  2 root root   6 Sep 25  2014 .
drwxr-xr-x 23 root root 270 Mar 19 17:40 ..

Comment 3 Vitaliy Margolen 2016-04-27 01:04:50 UTC
Removing Triaged since it was not really looked at. I do not have SE Linux enabled.

Comment 4 Saravanakumar 2016-04-27 06:32:28 UTC
I can observe this bug is similar to Bug#1229422. 
Fixes are posted. 

Although the bug#1229422 says quota is enabled (and yours say quota is disabled), this may be due to other reasons also. 
Please check http://review.gluster.org/#/c/13862/ commit message.

Comment 5 Vitaliy Margolen 2016-05-01 15:28:15 UTC
I've upgraded GlusterFS on the server with problems. Made sure that no processes using old version. Re-mounted the volume. Other 3 are still running 3.7.9.
The problem still remains:

==> /var/log/glusterfs/bricks/export.log <==
[2016-05-01 15:18:06.447703] E [MSGID: 115050] [server-rpc-fops.c:179:server_lookup_cbk] 0-gv1-server: 111: LOOKUP /natcm (00000000-0000-0000-0000-000000000001/CMTSchecker) ==> (Permission denied) [Permission denied]

==> /var/log/glusterfs/mnt-data.log <==
[2016-05-01 15:18:06.447926] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (f7e7dfd2-86c9-4590-b6ce-83e07f1de327) [Permission denied]


Other servers only have warnings in /var/log/glusterfs/mnt-data.log:
[2016-05-01 15:20:01.702505] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: /natcm (7b917079-4e50-4f85-b50e-a152e8f39769) [Permission denied]
[2016-05-01 15:20:01.707055] W [MSGID: 114031] [client-rpc-fops.c:2974:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Permission denied]

Comment 6 Vitaliy Margolen 2016-05-04 13:42:48 UTC
Any news? Still seeing lots of errors on that one host. Other 3 hosts are running fine. But would like to put it back into rotation.
Bumping version, since problem exists with last released version as well.

Comment 7 Raghavendra G 2016-05-05 08:46:56 UTC
Can you please attach brick logs? Moving back to NEW as we've not identified the component yet.

Comment 8 Vitaliy Margolen 2016-05-05 13:42:19 UTC
Created attachment 1154213 [details]
Brik log

Here you go. It's more of the same thing.

Comment 9 Vitaliy Margolen 2016-05-11 15:19:30 UTC
Any news? I am still unable to use this one host.

Comment 10 Soumya Koduri 2016-05-17 12:26:57 UTC
Hi Raghavendra,
Any clues for further debugging?

Comment 11 Vitaliy Margolen 2016-05-26 12:52:53 UTC
Hi, any updates? Any more information you need from me to troubleshoot this issue?

Comment 12 Vitaliy Margolen 2016-06-06 12:46:18 UTC
Think I might found the problem. Oh this server the mounted file system  mount point was set 0750 permissions. After changing that to 755 errors went away.

Any ideas why this might cause those permission denied errors? Gulsterfsd daemon is running as root.

Comment 13 Kaushal 2017-03-08 10:52:04 UTC
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.