Bug 1278384

Summary: 'ls' on client mount lists varying number of files while promotion/demotion
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Vivek Agarwal <vagarwal>
Component: tierAssignee: Dan Lambright <dlambrig>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: byarlaga, dlambrig, nbalacha, rcyriac, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.5-7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1271947 Environment:
Last Closed: 2016-03-01 05:52:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1271947, 1275998    
Bug Blocks: 1260783, 1260923    

Description Vivek Agarwal 2015-11-05 11:34:03 UTC
+++ This bug was initially created as a clone of Bug #1271947 +++

Description of problem:
=======================

This is on NFS mount. Created 6x2 dist-rep volume and attached a rep-2 tier. Created 5000 files and stopped access from client side. While the promotions/demotions are happening, listing the files from client mount and every time it shows different number. 

[root@rhs-client17 5000]# ls file.* | wc -l
4997
[root@rhs-client17 5000]# ls file.* | wc -l
5000
[root@rhs-client17 5000]# ls file.* | wc -l
4995
[root@rhs-client17 5000]# ls file.* | wc -l
4999
[root@rhs-client17 5000]# ls file.* | wc -l
5001
[root@rhs-client17 5000]# ls file.* | wc -l
4997
[root@rhs-client17 5000]# ls file.* | wc -l
4980
[root@rhs-client17 5000]# ls file.* | wc -l
5000
[root@rhs-client17 5000]# ls file.* | wc -l
4992
[root@rhs-client17 5000]# ls file.* | wc -l
4999

Version-Release number of selected component (if applicable):
=============================================================
[root@transformers ~]# gluster --version
glusterfs 3.7.5 built on Oct 12 2015 01:12:54
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

[root@transformers ~]# rpm -qa |grep glusterfs-server
glusterfs-server-3.7.5-0.18.git2def6bb.el7.centos.x86_64

How reproducible:
=================
100%

Steps to Reproduce:
===================
As in description.


Actual results:
===============
Varying number of files are listed from client


Expected results:
=================
Number of files created = listing same number


Additional info:
===============
sosreports will be copied to rhsqe.

Comment 3 Dan Lambright 2015-11-24 13:19:36 UTC
This problem was tracked and solved in bug 1281598.

Comment 4 Nag Pavan Chilakam 2015-11-24 16:35:34 UTC
this bug is same as 1275998.Which means it is a dup
I moved that to verified, hence moving this too

Verified on glusterfs-3.7.5-7:
============================
1)both in test and cache mode 
2)on nfs and fuse
3)while promotes and demotes were happening, the ls -l |wc -l was giving a constant number of files as the count. 

I had about 8000 files and heated some and some got cooled. While they were migrating issued ll|wc -l and was getting constantly the right number of files

[root@zod ~]# gluster v tier testmode status;gluster v rebal testmode status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            3424                 0                    in progress         
10.70.34.43          0                    7234                 in progress         
Tiering Migration Functionality: testmode: success
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             7225        0Bytes         20464             0             0          in progress             626.00
                             10.70.34.43            14134        0Bytes         34876             0             0          in progress             626.00
volume rebalance: testmode: success
[root@zod ~]# gluster v tier testmode status;gluster v rebal testmode status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            3444                 0                    in progress         
10.70.34.43          0                    7234                 in progress         
Tiering Migration Functionality: testmode: success
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             7244        0Bytes         20464             0             0          in progress             627.00
                             10.70.34.43            14134        0Bytes         34876             0             0          in progress             627.00
volume rebalance: testmode: success
[root@zod ~]# 



NFS MOunt:
[root@dhcp35-214 mnt]# cd testmode/
[root@dhcp35-214 testmode]# ll |wc -l
8001
[root@dhcp35-214 testmode]# ll |wc -l
8001
[root@dhcp35-214 testmode]# ll |wc -l
8001
[root@dhcp35-214 testmode]# 

Fuse mount:

[root@localhost testmode]# for i in {1..3000};do echo "hi" >> h$i;done;ll|wc -l
8001

Comment 7 errata-xmlrpc 2016-03-01 05:52:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html