Bug 1289423

Summary: Regular files are listed as 'T' files on nfs mount
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Bhaskarakiran <byarlaga>
Component: tierAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED ERRATA QA Contact: Bhaskarakiran <byarlaga>
Severity: urgent Docs Contact:
Priority: urgent    
Version: rhgs-3.1CC: jthottan, kkeithle, mzywusko, nchilaka, ndevos, rcyriac, rhs-bugs, rkavunga, sankarshan, skoduri, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.5-12 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1291208 (view as bug list) Environment:
Last Closed: 2016-03-01 06:02:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1291208, 1291212    

Description Bhaskarakiran 2015-12-08 04:19:48 UTC 
Description of problem:
=======================

Create a file with dd command on nfs mount and listed the same with ' ls -l'.  Its shows as 'T' file most of the times. This works correctly on fuse mount.

This is an EC volume as cold tier and 1x2 rep hot tier.

[root@rhs-client17 nfs]# dd if=/dev/urandom of=file1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 98.7191 s, 10.9 MB/s
[root@rhs-client17 nfs]# ls -l file1
-rw-r--r--. 1 root root 1073741824 Dec  8 09:35 file1
[root@rhs-client17 nfs]# ls -l file1
-rw-r--r--. 1 root root 1073741824 Dec  8 09:35 file1
[root@rhs-client17 nfs]# for i in `seq 1 50`; do ls -l ; done
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
[root@rhs-client17 nfs]# ls -l
total 2097152
-rw-r--r--. 1 root root 1073741824 Dec  7 14:46 file
-rw-r--r--. 1 root root 1073741824 Dec  8 09:35 file1
[root@rhs-client17 nfs]# ls -l
total 0
---------T. 1 root root 0 Dec  7 14:04 file
---------T. 1 root root 0 Dec  8 09:27 file1
[root@rhs-client17 nfs]# ls -l
total 2097152
-rw-r--r--. 1 root root 1073741824 Dec  7 14:46 file
-rw-r--r--. 1 root root 1073741824 Dec  8 09:35 file1
[root@rhs-client17 nfs]# ls -l
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
[root@rhs-client17 nfs]# ls -l
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
[root@rhs-client17 nfs]# ls -l
total 0
---------T. 1 root root 0 Dec  7 14:03 file
---------T. 1 root root 0 Dec  8 09:27 file1
[root@rhs-client17 nfs]# 

[root@rhs-client17 nfs]# df . 
Filesystem                    1K-blocks    Used   Available Use% Mounted on
transformers:/disperse_vol1 15971684352 2790400 15968893952   1% /mnt/nfs
[root@rhs-client17 nfs]# 

[root@transformers ~]# gluster v info disperse_vol1
 
Volume Name: disperse_vol1
Type: Tier
Volume ID: a6f27d4d-6838-47bf-ba2d-43fe9c980be3
Status: Started
Number of Bricks: 14
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick1: ninja:/rhs/brick1/vol1-tier2
Brick2: vertigo:/rhs/brick1/vol1-tier1
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick3: transformers:/rhs/brick1/b1
Brick4: interstellar:/rhs/brick1/b2
Brick5: transformers:/rhs/brick2/b3
Brick6: interstellar:/rhs/brick2/b4
Brick7: transformers:/rhs/brick3/b5
Brick8: interstellar:/rhs/brick3/b6
Brick9: transformers:/rhs/brick4/b7
Brick10: interstellar:/rhs/brick4/b8
Brick11: transformers:/rhs/brick5/b9
Brick12: interstellar:/rhs/brick5/b10
Brick13: transformers:/rhs/brick6/b11
Brick14: interstellar:/rhs/brick6/b12
Options Reconfigured:
cluster.watermark-hi: 2
cluster.watermark-low: 1
cluster.tier-mode: cache
features.ctr-enabled: on
server.event-threads: 2
client.event-threads: 2
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
features.uss: on
performance.readdir-ahead: on
[root@transformers ~]# 

Version-Release number of selected component (if applicable):
============================================================
3.7.5-9

How reproducible:
=================
100%

Steps to Reproduce:
===================
As in description

Actual results:
===============
Regular files are listed as 'T' files on nfs mount

Expected results:


Additional info:
Comment 2 Bhaskarakiran 2015-12-08 05:01:52 UTC 
This is seen on RHEL7.2 / NFS.
Comment 5 Jiffin 2015-12-14 10:27:52 UTC 
The http://review.gluster.org/#/c/12960/, fixes the above mentioned problem. But it may need to call an additional getattr for those files from the client side. 

In numbers, for performing readdirp(ls -ltRr ) on dir which contains 10 sub directories , each with 1000 files with 1k size. time taken

before applying the patch:
real	0m32.070s
user	0m0.445s
sys	0m2.213s

after applying the patch:
real	0m34.403s
user	0m0.535s
sys	0m3.046s
Comment 6 Vivek Agarwal 2015-12-17 09:27:08 UTC 
https://code.engineering.redhat.com/gerrit/64016
Comment 7 Bhaskarakiran 2015-12-18 11:35:46 UTC 
verified on glusterfs-3.7.5-12 and have not seen the issue. Marking this as fixed.
Comment 9 errata-xmlrpc 2016-03-01 06:02:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html