Description of problem: ================ While browsing the snapshot using USS files are getting displayed as link files Version-Release number of selected component (if applicable): ===================== glusterfs-server-3.7.5-18.e How reproducible: Steps to Reproduce: ============== 1. Create 16x2 volume and then attach 4X2 hot tier and enable USS and quota 2. Mount it on the client using FUSE 3. While IO is going on take snapshot 4. From the FUSE mount go to .snaps/<snap> and list the content but all files are getting displayed as link files (-----T) Actual results: ========== Due to this not able to read the file content Expected results: Additional info: ============== IP of the machine is : 10.70.35.153
On NFS mount, listing files in .snaps directory throws 'stale file handle' errors. Below is the snippet of ec_tier_snap1-3f4f1ddb-8a04-4ef0-b6e1-322b016ce7b7.log [2016-01-30 15:06:41.875879] E [MSGID: 109034] [dht-common.c:1122:dht_lookup_unlink_of_false_linkto_cbk] 0-4469a385507e4344b9e4e4ab8992d27e-tier-dht: Could not unlink the linkto file as either fd is open and/or linkto xattr is set for /client2/renames/dir.1/newfile.349 [Device or resource busy] [2016-01-30 15:06:41.878304] I [MSGID: 109045] [dht-common.c:2124:dht_lookup_cbk] 0-4469a385507e4344b9e4e4ab8992d27e-tier-dht: linkfile not having link subvol for /client2/renames/dir.1/newfile.349 [2016-01-30 15:06:42.110088] I [MSGID: 109069] [dht-common.c:1095:dht_lookup_unlink_of_false_linkto_cbk] 0-4469a385507e4344b9e4e4ab8992d27e-tier-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 117 for /client2/renames/dir.1/newfile.349
RCA: For tiered volume , we send readdirp only to cold tier for performance improvement, since cold tier is the default hash every file or corresponding linkfile should be there in cold tier. For files in hot tier, there will be a linkfile and readdirp from cold tier for such files will not have proper attributes. In normal client stack tier xlators will set inode=null for such entries to force a lookup. In snaps world , since there is no tier layer on client graph, the inode will be passed to fuse kernel as nodeuuid. ie it will print permission as only T bit for files which is in cold tier.
During readdirp call on a tiered volume, listing entries of a snapshot volume through USS will give wrong stat for files in cold tier, which result in printing wrong permissions on mount point. To work-around this issue, we can use either of the options during mounting. Use mount options , For Fuse mount (use any of the option): 1) use-readdirp=no (recommended) 2) attribute-timeout=0 3) entry-timeout=0 For NFS: 1) noac
Rajesh, Can you please verify the doc text provided.
Avra, Can you help verify the doc text?
From the initial analysis, it seems that ideally the fix should be placed in libgfapi, because the same issue can happen on other components like nfs-ganesha and samba.
(In reply to Avra Sengupta from comment #13) > From the initial analysis, it seems that ideally the fix should be placed in > libgfapi, because the same issue can happen on other components like > nfs-ganesha and samba. Could you explain more about the envisioned fix and how it would be done in gfapi? Is it something that needs to be handled in FUSE and Gluster/NFS too?
Rafi conducted the initial RCA. Rafi, Could you please explain the fix.
The ideal fix would be to do a stat/lookup call from gfapi layer for entries having NULL gfid in readdirp_cbk. Then the result of stat call need to be packed into readdirp result and send to application. This will be an intrusive fix and will have a performance impact on a tiered volume (only for tiered volume). In case of fuse, fuse client will do a lookup for entries having a NULL nodeuuid in readdirp result. An for GNFS, we invalidate the such entries before giving the reaadirp output back to nfs client, ie client will force to do a lookup for such entries. RCA is given in comment 3.
In our bug triage it was mentioned this has been fixed and this can be closed.. Rafi can you confirm?
Yes it has been fixed along with bug 1322247 . patch details master : http://review.gluster.org/#/c/14079/ This fix would be available in 3.2 as part of the rebase.
Moving to MODIFIED. Patch available downstream as commit 9423bde.
As tier is not being actively developed, I'm closing this bug. Feel free to open it if necessary.