Bug 1305474 - [USS-CIFS] Unable to copy all the files from USS to mount point , gives error permission denied
[USS-CIFS] Unable to copy all the files from USS to mount point , gives erro...
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: snapshot (Show other bugs)
All Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
: ZStream
Depends On:
  Show dependency treegraph
Reported: 2016-02-08 05:51 EST by Anil Shah
Modified: 2018-02-27 13:17 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Anil Shah 2016-02-08 05:51:34 EST
Description of problem:

While copying files from snapshot to the mount point, can't copy all the files. For few files its gives error " Permission denied".

Version-Release number of selected component (if applicable):

[root@rhs001 glusterfs]# rpm -qa | grep glusterfs

How reproducible:


Steps to Reproduce:
1. Create 2*2 distribute replicated volume
2. Enable quata and set limit usage
3. Enable uss 
4. Do CIFS mount and create some data files
5. Create Snapshot and activate 
6. Delete all the  files from mount point 
7. Copy files from snapshot 

Actual results:

Some files are not copied. Error "Permission denied"

Expected results:

All files should get copied

Additional info:

[root@rhs001 glusterfs]# gluster v info
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 07eed041-3948-44b6-bf36-bcda269fd08f
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Options Reconfigured:
features.uss: enable
features.barrier: disable
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
server.allow-insecure: on
features.show-snapshot-directory: on
storage.batch-fsync-delay-usec: 0
performance.stat-prefetch: off
performance.readdir-ahead: on
Comment 2 Anil Shah 2016-02-08 06:04:10 EST
sosreports uploaded @ 
Comment 3 rjoseph 2016-02-08 07:08:21 EST
Please provide more details about the bug. Following details are needed for us to start investigation:

1) Which server/machine is used as cifs mount?
2) What is the quota limit set?
3) How many files (both in terms of count and total size) were copied to volume?
4) Which files are giving permission denied errors?
   - Any special modes or permission present on the file?
   - Are they large files?
   - What is special about these files? Or is it just random?
5) Is the bug scene with Quota disabled?
Comment 4 Anil Shah 2016-02-08 07:30:47 EST
Set quota limit-limit to 10 GB on / of the volume.
Create 1000 regular data files on 1MB each.
Around 50 files didn't get copied from USS to mount point. Error "Permission Denied".

Will try with quota disable and update.
Comment 5 Anil Shah 2016-02-09 06:30:12 EST
this bug is reproducible when quota is disabled.
Comment 6 Raghavendra Talur 2016-02-11 04:06:02 EST
Looking into this. I have managed to reproduce the issue in simple.

Now onto debugging.
Will update soon with more data.
Comment 7 Raghavendra Talur 2016-02-12 01:44:31 EST
How to reproduce this issue consistently.

1. create a file on normal graph using cifs
2. take snapshot of volume
3. delete the file on normal graph from windows
4. perform a stat on the file on cifs mount.

expected result: stat failed with no such file or directory
actual result:   stat failed with permission denied
Comment 8 Raghavendra Talur 2016-02-12 01:45:22 EST
backtrace for cbk path for lstat

#0  dht_lookup_cbk (frame=frame@entry=0x7fc971cdb31c,
    cookie=0x7fc971cdb9d4, this=0x7fc96400dd60, op_ret=-1, op_errno=2,
    inode=0x7fc962c7f378, stbuf=stbuf@entry=0x7fc98d74fc30, xattr=0x0,
    postparent=0x7fc98d74fca0) at dht-common.c:2011
#1  0x00007fc96b39e27a in afr_lookup_done (
    frame=frame@entry=0x7fc971cdb9d4, this=this@entry=0x7fc96400bbc0)
    at afr-common.c:1796
#2  0x00007fc96b39eb54 in afr_lookup_metadata_heal_check (
    frame=frame@entry=0x7fc971cdb9d4, this=0x7fc96400bbc0,
    this@entry=0x73a44551f21dc100) at afr-common.c:2043
#3  0x00007fc96b39f29f in afr_lookup_entry_heal (
    frame=frame@entry=0x7fc971cdb9d4, this=0x73a44551f21dc100,
    this@entry=0x7fc96400bbc0) at afr-common.c:2132
#4  0x00007fc96b39f5cd in afr_lookup_cbk (frame=0x7fc971cdb9d4,
    cookie=<optimized out>, this=0x7fc96400bbc0,
    op_ret=<optimized out>, op_errno=<optimized out>,
    inode=<optimized out>, buf=0x7fc969e5a870, xdata=0x0,
    postparent=0x7fc969e5a8e0) at afr-common.c:2180
#5  0x00007fc96b5e7477 in client3_3_lookup_cbk (req=<optimized out>,
    iov=<optimized out>, count=<optimized out>, myframe=0x7fc971cdba80)
    at client-rpc-fops.c:2981
#6  0x00007fc9740e3b20 in rpc_clnt_handle_reply (
    clnt=clnt@entry=0x7fc96410a370, pollin=pollin@entry=0x7fc95c01ffa0)
    at rpc-clnt.c:766
#7  0x00007fc9740e3ddf in rpc_clnt_notify (trans=<optimized out>,
    mydata=0x7fc96410a3a0, event=<optimized out>, data=0x7fc95c01ffa0)
    at rpc-clnt.c:907
#8  0x00007fc9740df913 in rpc_transport_notify (
    data=data@entry=0x7fc95c01ffa0) at rpc-transport.c:545
Comment 9 Avra Sengupta 2016-02-26 01:29:52 EST
Removing this from 3.1.3. Will be fixing it later.

Note You need to log in before you can comment on or make changes to this bug.