Bug 1305474

Summary: [USS-CIFS] Unable to copy all the files from USS to mount point , gives error permission denied
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Anil Shah <ashah>
Component: snapshotAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED WONTFIX QA Contact: storage-qa-internal <storage-qa-internal>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: rhs-bugs, rtalur, sankarshan
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-16 16:04:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Anil Shah 2016-02-08 10:51:34 UTC
Description of problem:

While copying files from snapshot to the mount point, can't copy all the files. For few files its gives error " Permission denied".

Version-Release number of selected component (if applicable):

[root@rhs001 glusterfs]# rpm -qa | grep glusterfs
glusterfs-3.7.5-19.el7rhgs.x86_64
glusterfs-cli-3.7.5-19.el7rhgs.x86_64
samba-vfs-glusterfs-4.2.4-12.el7rhgs.x86_64
glusterfs-libs-3.7.5-19.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-19.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-17.el7rhgs.x86_64
glusterfs-api-3.7.5-19.el7rhgs.x86_64
glusterfs-server-3.7.5-19.el7rhgs.x86_64
glusterfs-fuse-3.7.5-19.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-19.el7rhgs.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create 2*2 distribute replicated volume
2. Enable quata and set limit usage
3. Enable uss 
4. Do CIFS mount and create some data files
5. Create Snapshot and activate 
6. Delete all the  files from mount point 
7. Copy files from snapshot 



Actual results:

Some files are not copied. Error "Permission denied"

Expected results:

All files should get copied

Additional info:

[root@rhs001 glusterfs]# gluster v info
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 07eed041-3948-44b6-bf36-bcda269fd08f
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.47.143:/rhs/brick1/b1
Brick2: 10.70.47.145:/rhs/brick1/b2
Brick3: 10.70.47.2:/rhs/brick1/b3
Brick4: 10.70.46.99:/rhs/brick1/b4
Options Reconfigured:
features.uss: enable
features.barrier: disable
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
server.allow-insecure: on
features.show-snapshot-directory: on
storage.batch-fsync-delay-usec: 0
performance.stat-prefetch: off
performance.readdir-ahead: on

Comment 2 Anil Shah 2016-02-08 11:04:10 UTC
sosreports uploaded @ 
rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1305474/

Comment 3 rjoseph 2016-02-08 12:08:21 UTC
Please provide more details about the bug. Following details are needed for us to start investigation:

1) Which server/machine is used as cifs mount?
2) What is the quota limit set?
3) How many files (both in terms of count and total size) were copied to volume?
4) Which files are giving permission denied errors?
   - Any special modes or permission present on the file?
   - Are they large files?
   - What is special about these files? Or is it just random?
5) Is the bug scene with Quota disabled?

Comment 4 Anil Shah 2016-02-08 12:30:47 UTC
Set quota limit-limit to 10 GB on / of the volume.
Create 1000 regular data files on 1MB each.
Around 50 files didn't get copied from USS to mount point. Error "Permission Denied".

Will try with quota disable and update.

Comment 5 Anil Shah 2016-02-09 11:30:12 UTC
this bug is reproducible when quota is disabled.

Comment 6 Raghavendra Talur 2016-02-11 09:06:02 UTC
Looking into this. I have managed to reproduce the issue in simple.

Now onto debugging.
Will update soon with more data.

Comment 7 Raghavendra Talur 2016-02-12 06:44:31 UTC
How to reproduce this issue consistently.

1. create a file on normal graph using cifs
2. take snapshot of volume
3. delete the file on normal graph from windows
4. perform a stat on the file on cifs mount.


expected result: stat failed with no such file or directory
actual result:   stat failed with permission denied

Comment 8 Raghavendra Talur 2016-02-12 06:45:22 UTC
backtrace for cbk path for lstat

#0  dht_lookup_cbk (frame=frame@entry=0x7fc971cdb31c,
    cookie=0x7fc971cdb9d4, this=0x7fc96400dd60, op_ret=-1, op_errno=2,
    inode=0x7fc962c7f378, stbuf=stbuf@entry=0x7fc98d74fc30, xattr=0x0,
    postparent=0x7fc98d74fca0) at dht-common.c:2011
#1  0x00007fc96b39e27a in afr_lookup_done (
    frame=frame@entry=0x7fc971cdb9d4, this=this@entry=0x7fc96400bbc0)
    at afr-common.c:1796
#2  0x00007fc96b39eb54 in afr_lookup_metadata_heal_check (
    frame=frame@entry=0x7fc971cdb9d4, this=0x7fc96400bbc0,
    this@entry=0x73a44551f21dc100) at afr-common.c:2043
#3  0x00007fc96b39f29f in afr_lookup_entry_heal (
    frame=frame@entry=0x7fc971cdb9d4, this=0x73a44551f21dc100,
    this@entry=0x7fc96400bbc0) at afr-common.c:2132
#4  0x00007fc96b39f5cd in afr_lookup_cbk (frame=0x7fc971cdb9d4,
    cookie=<optimized out>, this=0x7fc96400bbc0,
    op_ret=<optimized out>, op_errno=<optimized out>,
    inode=<optimized out>, buf=0x7fc969e5a870, xdata=0x0,
    postparent=0x7fc969e5a8e0) at afr-common.c:2180
#5  0x00007fc96b5e7477 in client3_3_lookup_cbk (req=<optimized out>,
    iov=<optimized out>, count=<optimized out>, myframe=0x7fc971cdba80)
    at client-rpc-fops.c:2981
#6  0x00007fc9740e3b20 in rpc_clnt_handle_reply (
    clnt=clnt@entry=0x7fc96410a370, pollin=pollin@entry=0x7fc95c01ffa0)
    at rpc-clnt.c:766
#7  0x00007fc9740e3ddf in rpc_clnt_notify (trans=<optimized out>,
    mydata=0x7fc96410a3a0, event=<optimized out>, data=0x7fc95c01ffa0)
    at rpc-clnt.c:907
#8  0x00007fc9740df913 in rpc_transport_notify (
    this=this@entry=0x7fc964119f00,
    event=event@entry=RPC_TRANSPORT_MSG_RECEIVED,
    data=data@entry=0x7fc95c01ffa0) at rpc-transport.c:545

Comment 9 Avra Sengupta 2016-02-26 06:29:52 UTC
Removing this from 3.1.3. Will be fixing it later.