Bug 1320887

+++ This bug was initially created as a clone of Bug #1320818 +++

On a dist-rep over EC tiered volume, I tried to mounted the volume on my desktop using nfs and copied some mp3 files into it.
Now using vlc player, i tried to play the files, in a suffle mode (about 30 songs)

Intially about 10 mp3 files completed playing and then we start getting permission denied error as below:



VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah (2013) ~320Kbps/04 - Banthi Poola Janaki [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah%20%282013%29%20~320Kbps/04%20-%20Banthi%20Poola%20Janaki%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:





gluster version:3.7.5-17

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-01-27 10:20:41 EST ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from nchilaka on 2016-01-28 00:20:54 EST ---

gluster v info:
[root@dhcp37-202 ~]# gluster v info nagvol
 
Volume Name: nagvol
Type: Tier
Volume ID: 5972ca44-130a-4543-8cc0-abf76a133a34
Status: Started
Number of Bricks: 36
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 6 x 2 = 12
Brick1: 10.70.37.120:/rhs/brick7/nagvol_hot
Brick2: 10.70.37.60:/rhs/brick7/nagvol_hot
Brick3: 10.70.37.69:/rhs/brick7/nagvol_hot
Brick4: 10.70.37.101:/rhs/brick7/nagvol_hot
Brick5: 10.70.35.163:/rhs/brick7/nagvol_hot
Brick6: 10.70.35.173:/rhs/brick7/nagvol_hot
Brick7: 10.70.35.232:/rhs/brick7/nagvol_hot
Brick8: 10.70.35.176:/rhs/brick7/nagvol_hot
Brick9: 10.70.35.222:/rhs/brick7/nagvol_hot
Brick10: 10.70.35.155:/rhs/brick7/nagvol_hot
Brick11: 10.70.37.195:/rhs/brick7/nagvol_hot
Brick12: 10.70.37.202:/rhs/brick7/nagvol_hot
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (8 + 4) = 24
Brick13: 10.70.37.202:/rhs/brick1/nagvol
Brick14: 10.70.37.195:/rhs/brick1/nagvol
Brick15: 10.70.35.155:/rhs/brick1/nagvol
Brick16: 10.70.35.222:/rhs/brick1/nagvol
Brick17: 10.70.35.108:/rhs/brick1/nagvol
Brick18: 10.70.35.44:/rhs/brick1/nagvol
Brick19: 10.70.35.89:/rhs/brick1/nagvol
Brick20: 10.70.35.231:/rhs/brick1/nagvol
Brick21: 10.70.35.176:/rhs/brick1/nagvol
Brick22: 10.70.35.232:/rhs/brick1/nagvol
Brick23: 10.70.35.173:/rhs/brick1/nagvol
Brick24: 10.70.35.163:/rhs/brick1/nagvol
Brick25: 10.70.37.101:/rhs/brick1/nagvol
Brick26: 10.70.37.69:/rhs/brick1/nagvol
Brick27: 10.70.37.60:/rhs/brick1/nagvol
Brick28: 10.70.37.120:/rhs/brick1/nagvol
Brick29: 10.70.37.202:/rhs/brick2/nagvol
Brick30: 10.70.37.195:/rhs/brick2/nagvol
Brick31: 10.70.35.155:/rhs/brick2/nagvol
Brick32: 10.70.35.222:/rhs/brick2/nagvol
Brick33: 10.70.35.108:/rhs/brick2/nagvol
Brick34: 10.70.35.44:/rhs/brick2/nagvol
Brick35: 10.70.35.89:/rhs/brick2/nagvol
Brick36: 10.70.35.231:/rhs/brick2/nagvol
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
features.quota-deem-statfs: off
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on



[root@dhcp37-202 ~]# gluster v status nagvol
Status of volume: nagvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.37.120:/rhs/brick7/nagvol_hot   49156     0          Y       32513
Brick 10.70.37.60:/rhs/brick7/nagvol_hot    49156     0          Y       4060 
Brick 10.70.37.69:/rhs/brick7/nagvol_hot    49156     0          Y       32442
Brick 10.70.37.101:/rhs/brick7/nagvol_hot   49156     0          Y       4199 
Brick 10.70.35.163:/rhs/brick7/nagvol_hot   49156     0          Y       617  
Brick 10.70.35.173:/rhs/brick7/nagvol_hot   49156     0          Y       32751
Brick 10.70.35.232:/rhs/brick7/nagvol_hot   49156     0          Y       32361
Brick 10.70.35.176:/rhs/brick7/nagvol_hot   49156     0          Y       32383
Brick 10.70.35.222:/rhs/brick7/nagvol_hot   49155     0          Y       22713
Brick 10.70.35.155:/rhs/brick7/nagvol_hot   49155     0          Y       22505
Brick 10.70.37.195:/rhs/brick7/nagvol_hot   49156     0          Y       25832
Brick 10.70.37.202:/rhs/brick7/nagvol_hot   49156     0          Y       26275
Cold Bricks:
Brick 10.70.37.202:/rhs/brick1/nagvol       49152     0          Y       16950
Brick 10.70.37.195:/rhs/brick1/nagvol       49152     0          Y       16702
Brick 10.70.35.155:/rhs/brick1/nagvol       49152     0          Y       13578
Brick 10.70.35.222:/rhs/brick1/nagvol       49152     0          Y       13546
Brick 10.70.35.108:/rhs/brick1/nagvol       49152     0          Y       4675 
Brick 10.70.35.44:/rhs/brick1/nagvol        49152     0          Y       12288
Brick 10.70.35.89:/rhs/brick1/nagvol        49152     0          Y       12261
Brick 10.70.35.231:/rhs/brick1/nagvol       49152     0          Y       22810
Brick 10.70.35.176:/rhs/brick1/nagvol       49152     0          Y       22781
Brick 10.70.35.232:/rhs/brick1/nagvol       49152     0          Y       22783
Brick 10.70.35.173:/rhs/brick1/nagvol       49152     0          Y       22795
Brick 10.70.35.163:/rhs/brick1/nagvol       49152     0          Y       22805
Brick 10.70.37.101:/rhs/brick1/nagvol       49152     0          Y       22847
Brick 10.70.37.69:/rhs/brick1/nagvol        49152     0          Y       22847
Brick 10.70.37.60:/rhs/brick1/nagvol        49152     0          Y       22895
Brick 10.70.37.120:/rhs/brick1/nagvol       49152     0          Y       22916
Brick 10.70.37.202:/rhs/brick2/nagvol       49153     0          Y       16969
Brick 10.70.37.195:/rhs/brick2/nagvol       49153     0          Y       16721
Brick 10.70.35.155:/rhs/brick2/nagvol       49153     0          Y       13597
Brick 10.70.35.222:/rhs/brick2/nagvol       49153     0          Y       13565
Brick 10.70.35.108:/rhs/brick2/nagvol       49153     0          Y       4694 
Brick 10.70.35.44:/rhs/brick2/nagvol        49153     0          Y       12307
Brick 10.70.35.89:/rhs/brick2/nagvol        49153     0          Y       12280
Brick 10.70.35.231:/rhs/brick2/nagvol       49153     0          Y       22829
NFS Server on localhost                     2049      0          Y       26295
Self-heal Daemon on localhost               N/A       N/A        Y       26303
Quota Daemon on localhost                   N/A       N/A        Y       26311
NFS Server on 10.70.37.101                  2049      0          Y       4219 
Self-heal Daemon on 10.70.37.101            N/A       N/A        Y       4227 
Quota Daemon on 10.70.37.101                N/A       N/A        Y       4235 
NFS Server on 10.70.37.69                   2049      0          Y       32462
Self-heal Daemon on 10.70.37.69             N/A       N/A        Y       32470
Quota Daemon on 10.70.37.69                 N/A       N/A        Y       32478
NFS Server on 10.70.37.195                  2049      0          Y       25852
Self-heal Daemon on 10.70.37.195            N/A       N/A        Y       25860
Quota Daemon on 10.70.37.195                N/A       N/A        Y       25868
NFS Server on 10.70.37.60                   2049      0          Y       4080 
Self-heal Daemon on 10.70.37.60             N/A       N/A        Y       4088 
Quota Daemon on 10.70.37.60                 N/A       N/A        Y       4096 
NFS Server on 10.70.37.120                  2049      0          Y       32533
Self-heal Daemon on 10.70.37.120            N/A       N/A        Y       32541
Quota Daemon on 10.70.37.120                N/A       N/A        Y       32549
NFS Server on 10.70.35.173                  2049      0          Y       303  
Self-heal Daemon on 10.70.35.173            N/A       N/A        Y       311  
Quota Daemon on 10.70.35.173                N/A       N/A        Y       319  
NFS Server on 10.70.35.232                  2049      0          Y       32381
Self-heal Daemon on 10.70.35.232            N/A       N/A        Y       32389
Quota Daemon on 10.70.35.232                N/A       N/A        Y       32397
NFS Server on 10.70.35.176                  2049      0          Y       32403
Self-heal Daemon on 10.70.35.176            N/A       N/A        Y       32411
Quota Daemon on 10.70.35.176                N/A       N/A        Y       32419
NFS Server on 10.70.35.231                  2049      0          Y       32446
Self-heal Daemon on 10.70.35.231            N/A       N/A        Y       32455
Quota Daemon on 10.70.35.231                N/A       N/A        Y       32463
NFS Server on 10.70.35.163                  2049      0          Y       637  
Self-heal Daemon on 10.70.35.163            N/A       N/A        Y       645  
Quota Daemon on 10.70.35.163                N/A       N/A        Y       653  
NFS Server on 10.70.35.222                  2049      0          Y       22733
Self-heal Daemon on 10.70.35.222            N/A       N/A        Y       22742
Quota Daemon on 10.70.35.222                N/A       N/A        Y       22750
NFS Server on 10.70.35.108                  2049      0          Y       13877
Self-heal Daemon on 10.70.35.108            N/A       N/A        Y       13885
Quota Daemon on 10.70.35.108                N/A       N/A        Y       13893
NFS Server on 10.70.35.155                  2049      0          Y       22525
Self-heal Daemon on 10.70.35.155            N/A       N/A        Y       22533
Quota Daemon on 10.70.35.155                N/A       N/A        Y       22541
NFS Server on 10.70.35.44                   2049      0          Y       21479
Self-heal Daemon on 10.70.35.44             N/A       N/A        Y       21487
Quota Daemon on 10.70.35.44                 N/A       N/A        Y       21495
NFS Server on 10.70.35.89                   2049      0          Y       20671
Self-heal Daemon on 10.70.35.89             N/A       N/A        Y       20679
Quota Daemon on 10.70.35.89                 N/A       N/A        Y       20687
 
Task Status of Volume nagvol
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 0870550a-70ba-4cd1-98da-b456059bd6cc
Status               : in progress         
 






Later changed some performance settings as per Rafi's instruction to debug the issue:
[root@dhcp37-202 glusterfs]# gluster v info nagvol
 
Volume Name: nagvol
Type: Tier
Volume ID: 5972ca44-130a-4543-8cc0-abf76a133a34
Status: Started
Number of Bricks: 36
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 6 x 2 = 12
Brick1: 10.70.37.120:/rhs/brick7/nagvol_hot
Brick2: 10.70.37.60:/rhs/brick7/nagvol_hot
Brick3: 10.70.37.69:/rhs/brick7/nagvol_hot
Brick4: 10.70.37.101:/rhs/brick7/nagvol_hot
Brick5: 10.70.35.163:/rhs/brick7/nagvol_hot
Brick6: 10.70.35.173:/rhs/brick7/nagvol_hot
Brick7: 10.70.35.232:/rhs/brick7/nagvol_hot
Brick8: 10.70.35.176:/rhs/brick7/nagvol_hot
Brick9: 10.70.35.222:/rhs/brick7/nagvol_hot
Brick10: 10.70.35.155:/rhs/brick7/nagvol_hot
Brick11: 10.70.37.195:/rhs/brick7/nagvol_hot
Brick12: 10.70.37.202:/rhs/brick7/nagvol_hot
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (8 + 4) = 24
Brick13: 10.70.37.202:/rhs/brick1/nagvol
Brick14: 10.70.37.195:/rhs/brick1/nagvol
Brick15: 10.70.35.155:/rhs/brick1/nagvol
Brick16: 10.70.35.222:/rhs/brick1/nagvol
Brick17: 10.70.35.108:/rhs/brick1/nagvol
Brick18: 10.70.35.44:/rhs/brick1/nagvol
Brick19: 10.70.35.89:/rhs/brick1/nagvol
Brick20: 10.70.35.231:/rhs/brick1/nagvol
Brick21: 10.70.35.176:/rhs/brick1/nagvol
Brick22: 10.70.35.232:/rhs/brick1/nagvol
Brick23: 10.70.35.173:/rhs/brick1/nagvol
Brick24: 10.70.35.163:/rhs/brick1/nagvol
Brick25: 10.70.37.101:/rhs/brick1/nagvol
Brick26: 10.70.37.69:/rhs/brick1/nagvol
Brick27: 10.70.37.60:/rhs/brick1/nagvol
Brick28: 10.70.37.120:/rhs/brick1/nagvol
Brick29: 10.70.37.202:/rhs/brick2/nagvol
Brick30: 10.70.37.195:/rhs/brick2/nagvol
Brick31: 10.70.35.155:/rhs/brick2/nagvol
Brick32: 10.70.35.222:/rhs/brick2/nagvol
Brick33: 10.70.35.108:/rhs/brick2/nagvol
Brick34: 10.70.35.44:/rhs/brick2/nagvol
Brick35: 10.70.35.89:/rhs/brick2/nagvol
Brick36: 10.70.35.231:/rhs/brick2/nagvol
Options Reconfigured:
cluster.watermark-hi: 50
cluster.watermark-low: 30
cluster.tier-mode: cache
features.ctr-enabled: on
features.quota-deem-statfs: off
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
[root@dhcp37-202 glusterfs]# gluster v status nagvol
Status of volume: nagvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.37.120:/rhs/brick7/nagvol_hot   49156     0          Y       32513
Brick 10.70.37.60:/rhs/brick7/nagvol_hot    49156     0          Y       4060 
Brick 10.70.37.69:/rhs/brick7/nagvol_hot    49156     0          Y       32442
Brick 10.70.37.101:/rhs/brick7/nagvol_hot   49156     0          Y       4199 
Brick 10.70.35.163:/rhs/brick7/nagvol_hot   49156     0          Y       617  
Brick 10.70.35.173:/rhs/brick7/nagvol_hot   49156     0          Y       32751
Brick 10.70.35.232:/rhs/brick7/nagvol_hot   49156     0          Y       32361
Brick 10.70.35.176:/rhs/brick7/nagvol_hot   49156     0          Y       32383
Brick 10.70.35.222:/rhs/brick7/nagvol_hot   49155     0          Y       22713
Brick 10.70.35.155:/rhs/brick7/nagvol_hot   49155     0          Y       22505
Brick 10.70.37.195:/rhs/brick7/nagvol_hot   49156     0          Y       25832
Brick 10.70.37.202:/rhs/brick7/nagvol_hot   49156     0          Y       26275
Cold Bricks:
Brick 10.70.37.202:/rhs/brick1/nagvol       49152     0          Y       16950
Brick 10.70.37.195:/rhs/brick1/nagvol       49152     0          Y       16702
Brick 10.70.35.155:/rhs/brick1/nagvol       49152     0          Y       13578
Brick 10.70.35.222:/rhs/brick1/nagvol       49152     0          Y       13546
Brick 10.70.35.108:/rhs/brick1/nagvol       49152     0          Y       4675 
Brick 10.70.35.44:/rhs/brick1/nagvol        49152     0          Y       12288
Brick 10.70.35.89:/rhs/brick1/nagvol        49152     0          Y       2668 
Brick 10.70.35.231:/rhs/brick1/nagvol       49152     0          Y       22810
Brick 10.70.35.176:/rhs/brick1/nagvol       49152     0          Y       22781
Brick 10.70.35.232:/rhs/brick1/nagvol       49152     0          Y       22783
Brick 10.70.35.173:/rhs/brick1/nagvol       49152     0          Y       22795
Brick 10.70.35.163:/rhs/brick1/nagvol       49152     0          Y       22805
Brick 10.70.37.101:/rhs/brick1/nagvol       49152     0          Y       22847
Brick 10.70.37.69:/rhs/brick1/nagvol        49152     0          Y       22847
Brick 10.70.37.60:/rhs/brick1/nagvol        49152     0          Y       22895
Brick 10.70.37.120:/rhs/brick1/nagvol       49152     0          Y       22916
Brick 10.70.37.202:/rhs/brick2/nagvol       49153     0          Y       16969
Brick 10.70.37.195:/rhs/brick2/nagvol       49153     0          Y       16721
Brick 10.70.35.155:/rhs/brick2/nagvol       49153     0          Y       13597
Brick 10.70.35.222:/rhs/brick2/nagvol       49153     0          Y       13565
Brick 10.70.35.108:/rhs/brick2/nagvol       49153     0          Y       4694 
Brick 10.70.35.44:/rhs/brick2/nagvol        49153     0          Y       12307
Brick 10.70.35.89:/rhs/brick2/nagvol        49153     0          Y       2683 
Brick 10.70.35.231:/rhs/brick2/nagvol       49153     0          Y       22829
NFS Server on localhost                     2049      0          Y       3356 
Self-heal Daemon on localhost               N/A       N/A        Y       3364 
Quota Daemon on localhost                   N/A       N/A        Y       3372 
NFS Server on 10.70.37.195                  2049      0          Y       2354 
Self-heal Daemon on 10.70.37.195            N/A       N/A        Y       2362 
Quota Daemon on 10.70.37.195                N/A       N/A        Y       2370 
NFS Server on 10.70.37.120                  2049      0          Y       9573 
Self-heal Daemon on 10.70.37.120            N/A       N/A        Y       9581 
Quota Daemon on 10.70.37.120                N/A       N/A        Y       9589 
NFS Server on 10.70.37.101                  2049      0          Y       13331
Self-heal Daemon on 10.70.37.101            N/A       N/A        Y       13339
Quota Daemon on 10.70.37.101                N/A       N/A        Y       13347
NFS Server on 10.70.37.60                   2049      0          Y       13071
Self-heal Daemon on 10.70.37.60             N/A       N/A        Y       13079
Quota Daemon on 10.70.37.60                 N/A       N/A        Y       13087
NFS Server on 10.70.37.69                   2049      0          Y       9368 
Self-heal Daemon on 10.70.37.69             N/A       N/A        Y       9376 
Quota Daemon on 10.70.37.69                 N/A       N/A        Y       9384 
NFS Server on 10.70.35.176                  2049      0          Y       9438 
Self-heal Daemon on 10.70.35.176            N/A       N/A        Y       9446 
Quota Daemon on 10.70.35.176                N/A       N/A        Y       9454 
NFS Server on 10.70.35.155                  2049      0          Y       31698
Self-heal Daemon on 10.70.35.155            N/A       N/A        Y       31706
Quota Daemon on 10.70.35.155                N/A       N/A        Y       31714
NFS Server on 10.70.35.232                  2049      0          Y       9301 
Self-heal Daemon on 10.70.35.232            N/A       N/A        Y       9309 
Quota Daemon on 10.70.35.232                N/A       N/A        Y       9317 
NFS Server on 10.70.35.163                  2049      0          Y       9935 
Self-heal Daemon on 10.70.35.163            N/A       N/A        Y       9943 
Quota Daemon on 10.70.35.163                N/A       N/A        Y       9951 
NFS Server on 10.70.35.89                   2049      0          Y       2483 
Self-heal Daemon on 10.70.35.89             N/A       N/A        Y       2560 
Quota Daemon on 10.70.35.89                 N/A       N/A        Y       2597 
NFS Server on 10.70.35.222                  2049      0          Y       32079
Self-heal Daemon on 10.70.35.222            N/A       N/A        Y       32087
Quota Daemon on 10.70.35.222                N/A       N/A        Y       32095
NFS Server on 10.70.35.173                  2049      0          Y       9724 
Self-heal Daemon on 10.70.35.173            N/A       N/A        Y       9732 
Quota Daemon on 10.70.35.173                N/A       N/A        Y       9740 
NFS Server on 10.70.35.231                  2049      0          Y       9171 
Self-heal Daemon on 10.70.35.231            N/A       N/A        Y       9179 
Quota Daemon on 10.70.35.231                N/A       N/A        Y       9187 
NFS Server on 10.70.35.44                   2049      0          Y       30600
Self-heal Daemon on 10.70.35.44             N/A       N/A        Y       30608
Quota Daemon on 10.70.35.44                 N/A       N/A        Y       30616
NFS Server on 10.70.35.108                  2049      0          Y       23000
Self-heal Daemon on 10.70.35.108            N/A       N/A        Y       23008
Quota Daemon on 10.70.35.108                N/A       N/A        Y       23016
 
Task Status of Volume nagvol
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 0870550a-70ba-4cd1-98da-b456059bd6cc
Status               : in progress         
 
[root@dhcp37-202 glusterfs]# 
[root@dhcp37-202 glusterfs]# 




[root@dhcp37-202 ~]# rpm -qa|grep gluster
glusterfs-client-xlators-3.7.5-17.el7rhgs.x86_64
glusterfs-server-3.7.5-17.el7rhgs.x86_64
gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64
vdsm-gluster-4.16.30-1.3.el7rhgs.noarch
glusterfs-3.7.5-17.el7rhgs.x86_64
glusterfs-api-3.7.5-17.el7rhgs.x86_64
glusterfs-cli-3.7.5-17.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-17.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-17.el7rhgs.x86_64
gluster-nagios-common-0.2.3-1.el7rhgs.noarch
python-gluster-3.7.5-16.el7rhgs.noarch
glusterfs-libs-3.7.5-17.el7rhgs.x86_64
glusterfs-fuse-3.7.5-17.el7rhgs.x86_64
glusterfs-rdma-3.7.5-17.el7rhgs.x86_64




-->Mounted the volume on 4 different clients(all NFS mounts): 3 were RHEL and 1 was the fedora personal laptop machine, where i had seen the problem 
--->total 16 gluster nodes 





================= Error thrown by VLC application================
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu (2013)~128Kbps/02 - Yaevaindho [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu%20(2013)~128Kbps/02%20-%20Yaevaindho%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/Laahiri Laahiri Laahiri Lo/OHOHO_CHILAKAMMA.mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/Laahiri%20Laahiri%20Laahiri%20Lo/OHOHO_CHILAKAMMA.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah (2013) ~320Kbps/04 - Banthi Poola Janaki [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah%20%282013%29%20~320Kbps/04%20-%20Banthi%20Poola%20Janaki%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu (2013)~128Kbps/03 - Lucky Lucky Rai [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu%20(2013)~128Kbps/03%20-%20Lucky%20Lucky%20Rai%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah (2013) ~320Kbps/02 - Diamond Girl [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Baadshah%20%282013%29%20~320Kbps/02%20-%20Diamond%20Girl%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu (2012) ~320Kbps/02.Jaga Jaga Jagadeka Veera [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu%20(2012)%20~320Kbps/02.Jaga%20Jaga%20Jagadeka%20Veera%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu (2012) ~320Kbps/01.Made For Each Other [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu%20(2012)%20~320Kbps/01.Made%20For%20Each%20Other%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu (2013) ~320 Kbps/06 - Pimple Dimple [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu%20(2013)%20~320%20Kbps/06%20-%20Pimple%20Dimple%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_sa rajkumar/Yavvana Veena.mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_sa%20rajkumar/Yavvana%20Veena.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu (2013)~128Kbps/04 - Padipoyaanila [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Balupu%20(2013)~128Kbps/04%20-%20Padipoyaanila%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu (2013) ~320 Kbps/05 - Oye Oye [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu%20(2013)%20~320%20Kbps/05%20-%20Oye%20Oye%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu (2012) ~320Kbps/05.Kaatuka Kallu [www.AtoZmp3.Net].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Sarocharu%20(2012)%20~320Kbps/05.Kaatuka%20Kallu%20%5Bwww.AtoZmp3.Net%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/DADDY/03 VANA VANA.MP3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/DADDY/03%20VANA%20VANA.MP3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/Deviputhrudu/03___OKATA_RENDA.MP3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/Deviputhrudu/03___OKATA_RENDA.MP3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_sa rajkumar/Panchadara_Chilaka-Anukunnana.mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_sa%20rajkumar/Panchadara_Chilaka-Anukunnana.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/DADDY/01 LKKI.MP3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/DADDY/01%20LKKI.MP3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/001_2015/Beeruva (2014) ~320Kbps/04 - Pisthol Bava [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/001_2015/Beeruva%20(2014)%20~320Kbps/04%20-%20Pisthol%20Bava%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/001_2015/Beeruva (2014) ~320Kbps/02 - Chinnadana Chinnadana [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/001_2015/Beeruva%20(2014)%20~320Kbps/02%20-%20Chinnadana%20Chinnadana%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.
File reading failed:
VLC could not open the file "/mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu (2013) ~320 Kbps/02 - Nee Jathaga  [www.AtoZmp3.in].mp3" (Permission denied).
Your input can't be opened:
VLC is unable to open the MRL 'file:///mnt/nagvol/my_lappy/01_Telugu/01_2013/Yevadu%20(2013)%20~320%20Kbps/02%20-%20Nee%20Jathaga%20%20%5Bwww.AtoZmp3.in%5D.mp3'. Check the log for details.


============================================================================
How reproducible:
Reproduced it atleast  3 times

--- Additional comment from nchilaka on 2016-01-28 00:37:44 EST ---

sosreports:
[nchilaka@rhsqe-repo nchilaka]$ /home/repo/sosreports/nchilaka/bug.1302355

[nchilaka@rhsqe-repo nchilaka]$ hostname
rhsqe-repo.lab.eng.blr.redhat.com

--- Additional comment from Nithya Balachandran on 2016-01-28 03:42:03 EST ---

Which NFS server did you use to mount the volume?

--- Additional comment from nchilaka on 2016-01-29 05:21:49 EST ---

I saw the problem with atleast 2 different servers at different times.
10.70.37.202 and 10.70.37.120

--- Additional comment from Soumya Koduri on 2016-02-02 05:01:21 EST ---

Here is our analysis so far:

NFS clients send ACCESS fop to get permission of any file. GlusterFS server encapsulates the file permissions in the op_errno before sending to gluster-NFS server. In the pkt trace collected, we have seen few ACCESS fops sent with op_errno set to zero (that too for root gfid/inode). That means brick processes have been sending NULL permissions. We have seen this issue hit with cold-tier bricks.


When checked brick processes, we found out that, posix-acl xlator is checking against NULL perms at times. posix-acl xlator (when no acl set) stores permissions of the inodes in its 'ctx->perm' structure and uses those bits to decide access permission for any user. This 'ctx->perm' is updated as part of posix_ctx_update() which gets called as part of variety of fops (like lookup, stat, readdir etc).


gluster-nfs server:
(gdb)
1453                    gf_msg (this->name, GF_LOG_WARNING,
(gdb) c
Continuing.

Breakpoint 1, client3_3_access (frame=0x7f9885331570, this=0x7f9874031150, data=0x7f986bffe7a0)
    at client-rpc-fops.c:3520
3520    {
(gdb) p this->name
$10 = 0x7f9874030ae0 "nagvol-client-20"
(gdb)

volfile:
volume nagvol-client-20
    type protocol/client
    option send-gids true
    option password 492e0b7f-255f-469d-8b9c-6982079dbcd1
    option username 727545cc-cd4f-4d92-9fed-81763b6d3d29
    option transport-type tcp
    option remote-subvolume /rhs/brick2/nagvol
    option remote-host 10.70.35.108
    option ping-timeout 42
end-volume 


Brick process:

Breakpoint 3, posix_acl_ctx_update (inode=0x7f7b1a1b306c, this=this@entry=0x7f7b340106b0,
    buf=buf@entry=0x7f7acc4d9148) at posix-acl.c:734
734    {
(gdb) p inode>gfid
No symbol "gfid" in current context.
(gdb) p inode->gfid
$23 = '\000' <repeats 15 times>, "\001"
(gdb) n
743            ctx = posix_acl_ctx_get (inode, this);
(gdb)
744            if (!ctx) {
(gdb)
743            ctx = posix_acl_ctx_get (inode, this);
(gdb)
744            if (!ctx) {
(gdb)
749            LOCK(&inode->lock);
(gdb)
751                    ctx->uid   = buf->ia_uid;
(gdb)
753                    ctx->perm  = st_mode_from_ia (buf->ia_prot, buf->ia_type);
(gdb)
751                    ctx->uid   = buf->ia_uid;
(gdb)
752                    ctx->gid   = buf->ia_gid;
(gdb)
753                    ctx->perm  = st_mode_from_ia (buf->ia_prot, buf->ia_type);
(gdb)
753                    ctx->perm  = st_mode_from_ia (buf->ia_prot, buf->ia_type);
(gdb)
755            acl = ctx->acl_access;
(gdb) p/x ctx->perm
$24 = 0x4000
(gdb) p buf->ia_prot
$25 = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000',
    exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000',
    write = 0 '\000', exec = 0 '\000'}} 
(gdb) bt
#0  posix_acl_ctx_update (inode=0x7f7b1a1b306c, this=this@entry=0x7f7b340106b0, buf=buf@entry=0x7f7acc4d9148)
    at posix-acl.c:755
#1  0x00007f7b39932af5 in posix_acl_readdirp_cbk (frame=0x7f7b46619a14, cookie=<optimized out>,
    this=0x7f7b340106b0, op_ret=7, op_errno=0, entries=0x7f7ab008da50, xdata=0x0) at posix-acl.c:1625
#2  0x00007f7b39b46bef in br_stub_readdirp_cbk (frame=0x7f7b46627674, cookie=<optimized out>, this=0x7f7b3400f270,
    op_ret=7, op_errno=0, entries=0x7f7ab008da50, dict=0x0) at bit-rot-stub.c:2546
#3  0x00007f7b3b0af163 in posix_readdirp (frame=0x7f7b46618aa0, this=<optimized out>, fd=<optimized out>,
    size=<optimized out>, off=<optimized out>, dict=<optimized out>) at posix.c:6022
#4  0x00007f7b48b10535 in default_readdirp (frame=0x7f7b46618aa0, this=0x7f7b34009240, fd=0x7f7b2c005408, size=0,
    off=0, xdata=0x7f7b48ddd5b8) at defaults.c:2101
#5  0x00007f7b48b10535 in default_readdirp (frame=0x7f7b46618aa0, this=0x7f7b3400a880, fd=0x7f7b2c005408, size=0,
    off=0, xdata=0x7f7b48ddd5b8) at defaults.c:2101
#6  0x00007f7b48b10535 in default_readdirp (frame=0x7f7b46618aa0, this=0x7f7b3400d2e0, fd=0x7f7b2c005408, size=0,
    off=0, xdata=0x7f7b48ddd5b8) at defaults.c:2101
#7  0x00007f7b39b404db in br_stub_readdirp (frame=0x7f7b46627674, this=0x7f7b3400f270, fd=0x7f7b2c005408, size=0,
    offset=0, dict=0x7f7b48ddd5b8) at bit-rot-stub.c:2581
#8  0x00007f7b39930949 in posix_acl_readdirp (frame=0x7f7b46619a14, this=0x7f7b340106b0, fd=0x7f7b2c005408, size=0,
    offset=0, dict=0x7f7b48ddd5b8) at posix-acl.c:1674
#9  0x00007f7b39718817 in pl_readdirp (frame=0x7f7b4661b0ec, this=0x7f7b34011ac0, fd=0x7f7b2c005408, size=0,
    offset=0, xdata=0x7f7b48ddd5b8) at posix.c:2213
#10 0x00007f7b39506c85 in up_readdirp (frame=0x7f7b46631504, this=0x7f7b34012e60, fd=0x7f7b2c005408, size=0, off=0,
    dict=0x7f7b48ddd5b8) at upcall.c:1342
#11 0x00007f7b48b1e19d in default_readdirp_resume (frame=0x7f7b4662b4f0, this=0x7f7b340142d0, fd=0x7f7b2c005408,
    size=0, off=0, xdata=0x7f7b48ddd5b8) at defaults.c:1657
#12 0x00007f7b48b3b17d in call_resume (stub=0x7f7b46113684) at call-stub.c:2576
#13 0x00007f7b392f6363 in iot_worker (data=0x7f7b340546a0) at io-threads.c:215
#14 0x00007f7b47973dc5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f7b472ba21d in clone () from /lib64/libc.so.6
(gdb) f 1
#1  0x00007f7b39932af5 in posix_acl_readdirp_cbk (frame=0x7f7b46619a14, cookie=<optimized out>,
    this=0x7f7b340106b0, op_ret=7, op_errno=0, entries=0x7f7ab008da50, xdata=0x0) at posix-acl.c:1625
1625                    posix_acl_ctx_update (entry->inode, this, &entry->d_stat);
(gdb) p entry
$26 = (gf_dirent_t *) 0x7f7acc4d9120
(gdb) p entry->d_stat
$27 = {ia_ino = 0, ia_gfid = '\000' <repeats 15 times>, "\001", ia_dev = 0, ia_type = IA_IFDIR, ia_prot = {
    suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000',
      exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000',
      write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0,
  ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0,
  ia_ctime_nsec = 0} 


As we can see above, as part of readdirp, we ended up with root inode entry which hasn't got update with right stat (entry->d_stat). When looked at the code,

int
posix_make_ancestryfromgfid (xlator_t *this, char *path, int pathsize,
                             gf_dirent_t *head, int type, uuid_t gfid,
                             const size_t handle_size,
                             const char *priv_base_path, inode_table_t *itable,
                             inode_t **parent, dict_t *xdata, int32_t *op_errno)
{
        char        *linkname   = NULL; /* "../../<gfid[0]>/<gfid[1]/"
                                         "<gfidstr>/<NAME_MAX>" */ 

...........
...........
        if (__is_root_gfid (gfid)) {
                if (parent) {
                        if (*parent) {
                                inode_unref (*parent);
                        }

                        *parent = inode_ref (itable->root);
                }

                inode = itable->root;

                memset (&iabuf, 0, sizeof (iabuf));
                gf_uuid_copy (iabuf.ia_gfid, inode->gfid);
                iabuf.ia_type = inode->ia_type;

                ret = posix_make_ancestral_node (priv_base_path, path, pathsize,
                                                 head, "/", &iabuf, inode, type,
                                                 xdata);
                if (ret < 0)
                        *op_errno = ENOMEM;
                return ret;
        } 
.............

}

For root inode entry, we do not seem to fetching stat (for other entries 'posix_resolve()' call is made which updates stat). So we suspect this could have resulted in root entry with NULL perms which in turn got updated in posix_acl xlator 'ctx->perm' resulting in EPERM error for ACCESS fop. But its not yet sure why this issue hasn't been hit till now.

--- Additional comment from Vijay Bellur on 2016-03-24 00:04:58 EDT ---

REVIEW: http://review.gluster.org/13730 (storage/posix: send proper iatt attributes for the root inode) posted (#5) for review on master by Raghavendra Bhat (raghavendra)

--- Additional comment from Vijay Bellur on 2016-03-24 03:26:02 EDT ---

REVIEW: http://review.gluster.org/13730 (storage/posix: send proper iatt attributes for the root inode) posted (#6) for review on master by Vijaikumar Mallikarjuna (vmallika)

--- Additional comment from Vijaikumar Mallikarjuna on 2016-03-24 05:27:52 EDT ---

upstream patch: http://review.gluster.org/#/c/13730/