Bug 1278389

Summary: Data Tiering: Tiering deamon is seeing each part of a file in a Disperse cold volume as a different file
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Vivek Agarwal <vagarwal>
Component: tierAssignee: Joseph Elwin Fernandes <josferna>
Status: CLOSED ERRATA QA Contact: Sweta Anandpara <sanandpa>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.1CC: asrivast, dlambrig, josferna, nchilaka, rhs-bugs, sankarshan, storage-qa-internal
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: RHGS 3.1.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.5-10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1262860 Environment:
Last Closed: 2016-03-01 05:52:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1262860    
Bug Blocks: 1260783, 1260923, 1276141    
Attachments:
Description Flags
Server and client logs none

Description Vivek Agarwal 2015-11-05 11:36:59 UTC
+++ This bug was initially created as a clone of Bug #1262860 +++

Description of problem:
========================
On a EC cold volume, when files are promoted or demoted to/from hot tier,
it seems like the tier deamon is seeing each copy or part of the file as a seperate different file. The counter atleast say this.

I had 3 files on a 2 x (4 + 2) = 12 EC cold volume.
When they were promoted or demoted to/from a distrep hot tier,
the stats show each file is counted as 6 times, with 1 time showing the success while the other 5 registering as failure.

Version-Release number of selected component (if applicable):
===========================================================
[root@zod glusterfs]# rpm -qa|grep gluster
glusterfs-client-xlators-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-server-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-cli-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
^[[Aglusterfs-libs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
[root@zod glusterfs]# gluster --version
glusterfs 3.7.4 built on Sep 12 2015 01:35:35
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod glusterfs]# 




Steps to Reproduce:
====================
1.Create a EC cold vol and distrep hot tier
2.Now set promo/demote freq after enabling ctr
3.Now create a file and wait for it to demote and then make it get promoted.
It can be seen that though it is one file, each copy of the file in each brick is considered as different file and the promote/demote counters show them as failed.


Below was the case where I had 3 files(compare promote/demote numbers with failures)
====================================

[root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                9        0Bytes            31            22             0          in progress            1393.00
                                  yarrow                8        0Bytes            36            28             0          in progress            1393.00
volume rebalance: redhat: success: 

Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            9                    0                    in progress         
yarrow               0                    8                    in progress         
volume rebalance: redhat: success: 


[root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                9        0Bytes            33            24             0          in progress            1701.00
                                  yarrow               10        0Bytes            38            28             0          in progress            1701.00
volume rebalance: redhat: success: 
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            9                    0                    in progress         
yarrow               0                    10                   in progress

--- Additional comment from nchilaka on 2015-09-14 09:42:37 EDT ---

volume rebalance: redhat: success: 
[root@zod glusterfs]# gluster v info redhat
 
Volume Name: redhat
Type: Tier
Volume ID: ec61f03a-b9c6-4a43-8aae-a1a3ca65e234
Status: Started
Number of Bricks: 16
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick6/redhat_hot
Brick2: zod:/rhs/brick6/redhat_hot
Brick3: yarrow:/rhs/brick7/redhat_hot
Brick4: zod:/rhs/brick7/redhat_hot
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick5: zod:/rhs/brick1/redhat
Brick6: yarrow:/rhs/brick1/redhat
Brick7: zod:/rhs/brick2/redhat
Brick8: yarrow:/rhs/brick2/redhat
Brick9: zod:/rhs/brick3/redhat
Brick10: yarrow:/rhs/brick3/redhat
Brick11: zod:/rhs/brick4/redhat
Brick12: yarrow:/rhs/brick4/redhat
Brick13: zod:/rhs/brick5/redhat
Brick14: yarrow:/rhs/brick5/redhat
Brick15: yarrow:/rhs/brick6/redhat
Brick16: zod:/rhs/brick6/redhat
Options Reconfigured:
cluster.tier-demote-frequency: 30
cluster.tier-promote-frequency: 50
features.ctr-enabled: on
performance.io-cache: off
performance.quick-read: off
performance.readdir-ahead: on

Comment 3 Nag Pavan Chilakam 2015-11-16 11:57:41 UTC
still this is not fixed:
I promoted 4 files from a ec cold tier, and it can be seen that rebal shows as 12 promoted 

[root@zod ~]# gluster v tier taylor status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            27                   0                    in progress         
yarrow               0                    28                   in progress         
Tiering Migration Functionality: taylor: success
[root@zod ~]# gluster v rebal taylor status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               81        0Bytes           104             3             0          in progress          361303.00
                                  yarrow               28        0Bytes           106             3             0          in progress          361145.00
volume rebalance: taylor: success
[root@zod ~]# 


moving to failed_qa
[root@zod ~]# rpm -qa|grep gluste
rglusterfs-libs-3.7.5-6.el7rhgs.x86_64
glusterfs-fuse-3.7.5-6.el7rhgs.x86_64
glusterfs-3.7.5-6.el7rhgs.x86_64
glusterfs-server-3.7.5-6.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-6.el7rhgs.x86_64
glusterfs-cli-3.7.5-6.el7rhgs.x86_64
glusterfs-api-3.7.5-6.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-6.el7rhgs.x86_64

Comment 4 Joseph Elwin Fernandes 2015-12-05 06:00:48 UTC
https://code.engineering.redhat.com/gerrit/#/c/63091/

Comment 5 Sweta Anandpara 2015-12-10 08:34:27 UTC
Tested and verified this on the build glusterfs-3.7.5-10.el7rhgs.x86_64

Created 100 new files and verified that they were in the hot tier. Checked the output of 'gluster v tier <volname> status' and 'gluster v rebalance <volname> status'.

Promoted 10 files and checked the status. In the next iteration, promoted a single file. After the session time, promoted 2 new files and demoted 1 file. Did a third round of promote/demote. 

Verified the output of the following commands, at every stage:
* gluster v rebalance <volname> status
* gluster v tier <volname> status
* ls -l <hot-tier path>; ls -l <cold-tier path>

No failures were seen in the rebalance output, and the rebalance number was in sync with the promote/demote number shown in tier status, and that in turn was in tune with the files actually heated/cooled.

Moving this bug to verified in 3.1.2. Detailed logs are attached.

Comment 6 Sweta Anandpara 2015-12-10 08:36:52 UTC
Created attachment 1104253 [details]
Server and client logs

Comment 9 errata-xmlrpc 2016-03-01 05:52:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html