Description of problem: While copying files from snapshot to the mount point, can't copy all the files. For few files its gives error " Permission denied". Version-Release number of selected component (if applicable): [root@rhs001 glusterfs]# rpm -qa | grep glusterfs glusterfs-3.7.5-19.el7rhgs.x86_64 glusterfs-cli-3.7.5-19.el7rhgs.x86_64 samba-vfs-glusterfs-4.2.4-12.el7rhgs.x86_64 glusterfs-libs-3.7.5-19.el7rhgs.x86_64 glusterfs-client-xlators-3.7.5-19.el7rhgs.x86_64 glusterfs-debuginfo-3.7.5-17.el7rhgs.x86_64 glusterfs-api-3.7.5-19.el7rhgs.x86_64 glusterfs-server-3.7.5-19.el7rhgs.x86_64 glusterfs-fuse-3.7.5-19.el7rhgs.x86_64 glusterfs-geo-replication-3.7.5-19.el7rhgs.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create 2*2 distribute replicated volume 2. Enable quata and set limit usage 3. Enable uss 4. Do CIFS mount and create some data files 5. Create Snapshot and activate 6. Delete all the files from mount point 7. Copy files from snapshot Actual results: Some files are not copied. Error "Permission denied" Expected results: All files should get copied Additional info: [root@rhs001 glusterfs]# gluster v info Volume Name: testvol Type: Distributed-Replicate Volume ID: 07eed041-3948-44b6-bf36-bcda269fd08f Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.70.47.143:/rhs/brick1/b1 Brick2: 10.70.47.145:/rhs/brick1/b2 Brick3: 10.70.47.2:/rhs/brick1/b3 Brick4: 10.70.46.99:/rhs/brick1/b4 Options Reconfigured: features.uss: enable features.barrier: disable features.quota-deem-statfs: on features.inode-quota: on features.quota: on server.allow-insecure: on features.show-snapshot-directory: on storage.batch-fsync-delay-usec: 0 performance.stat-prefetch: off performance.readdir-ahead: on
sosreports uploaded @ rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1305474/
Please provide more details about the bug. Following details are needed for us to start investigation: 1) Which server/machine is used as cifs mount? 2) What is the quota limit set? 3) How many files (both in terms of count and total size) were copied to volume? 4) Which files are giving permission denied errors? - Any special modes or permission present on the file? - Are they large files? - What is special about these files? Or is it just random? 5) Is the bug scene with Quota disabled?
Set quota limit-limit to 10 GB on / of the volume. Create 1000 regular data files on 1MB each. Around 50 files didn't get copied from USS to mount point. Error "Permission Denied". Will try with quota disable and update.
this bug is reproducible when quota is disabled.
Looking into this. I have managed to reproduce the issue in simple. Now onto debugging. Will update soon with more data.
How to reproduce this issue consistently. 1. create a file on normal graph using cifs 2. take snapshot of volume 3. delete the file on normal graph from windows 4. perform a stat on the file on cifs mount. expected result: stat failed with no such file or directory actual result: stat failed with permission denied
backtrace for cbk path for lstat #0 dht_lookup_cbk (frame=frame@entry=0x7fc971cdb31c, cookie=0x7fc971cdb9d4, this=0x7fc96400dd60, op_ret=-1, op_errno=2, inode=0x7fc962c7f378, stbuf=stbuf@entry=0x7fc98d74fc30, xattr=0x0, postparent=0x7fc98d74fca0) at dht-common.c:2011 #1 0x00007fc96b39e27a in afr_lookup_done ( frame=frame@entry=0x7fc971cdb9d4, this=this@entry=0x7fc96400bbc0) at afr-common.c:1796 #2 0x00007fc96b39eb54 in afr_lookup_metadata_heal_check ( frame=frame@entry=0x7fc971cdb9d4, this=0x7fc96400bbc0, this@entry=0x73a44551f21dc100) at afr-common.c:2043 #3 0x00007fc96b39f29f in afr_lookup_entry_heal ( frame=frame@entry=0x7fc971cdb9d4, this=0x73a44551f21dc100, this@entry=0x7fc96400bbc0) at afr-common.c:2132 #4 0x00007fc96b39f5cd in afr_lookup_cbk (frame=0x7fc971cdb9d4, cookie=<optimized out>, this=0x7fc96400bbc0, op_ret=<optimized out>, op_errno=<optimized out>, inode=<optimized out>, buf=0x7fc969e5a870, xdata=0x0, postparent=0x7fc969e5a8e0) at afr-common.c:2180 #5 0x00007fc96b5e7477 in client3_3_lookup_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7fc971cdba80) at client-rpc-fops.c:2981 #6 0x00007fc9740e3b20 in rpc_clnt_handle_reply ( clnt=clnt@entry=0x7fc96410a370, pollin=pollin@entry=0x7fc95c01ffa0) at rpc-clnt.c:766 #7 0x00007fc9740e3ddf in rpc_clnt_notify (trans=<optimized out>, mydata=0x7fc96410a3a0, event=<optimized out>, data=0x7fc95c01ffa0) at rpc-clnt.c:907 #8 0x00007fc9740df913 in rpc_transport_notify ( this=this@entry=0x7fc964119f00, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data=data@entry=0x7fc95c01ffa0) at rpc-transport.c:545
Removing this from 3.1.3. Will be fixing it later.