Bug 1278389 - Data Tiering: Tiering deamon is seeing each part of a file in a Disperse cold volume as a different file
Data Tiering: Tiering deamon is seeing each part of a file in a Disperse cold...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
high Severity medium
: ---
: RHGS 3.1.2
Assigned To: Joseph Elwin Fernandes
Sweta Anandpara
: Triaged, ZStream
Depends On: 1262860
Blocks: 1260783 1260923 1276141
  Show dependency treegraph
 
Reported: 2015-11-05 06:36 EST by Vivek Agarwal
Modified: 2016-09-17 11:35 EDT (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.7.5-10
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1262860
Environment:
Last Closed: 2016-03-01 00:52:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Server and client logs (30.85 KB, application/vnd.oasis.opendocument.text)
2015-12-10 03:36 EST, Sweta Anandpara
no flags Details

  None (edit)
Description Vivek Agarwal 2015-11-05 06:36:59 EST
+++ This bug was initially created as a clone of Bug #1262860 +++

Description of problem:
========================
On a EC cold volume, when files are promoted or demoted to/from hot tier,
it seems like the tier deamon is seeing each copy or part of the file as a seperate different file. The counter atleast say this.

I had 3 files on a 2 x (4 + 2) = 12 EC cold volume.
When they were promoted or demoted to/from a distrep hot tier,
the stats show each file is counted as 6 times, with 1 time showing the success while the other 5 registering as failure.

Version-Release number of selected component (if applicable):
===========================================================
[root@zod glusterfs]# rpm -qa|grep gluster
glusterfs-client-xlators-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-server-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-cli-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
^[[Aglusterfs-libs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
[root@zod glusterfs]# gluster --version
glusterfs 3.7.4 built on Sep 12 2015 01:35:35
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod glusterfs]# 




Steps to Reproduce:
====================
1.Create a EC cold vol and distrep hot tier
2.Now set promo/demote freq after enabling ctr
3.Now create a file and wait for it to demote and then make it get promoted.
It can be seen that though it is one file, each copy of the file in each brick is considered as different file and the promote/demote counters show them as failed.


Below was the case where I had 3 files(compare promote/demote numbers with failures)
====================================

[root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                9        0Bytes            31            22             0          in progress            1393.00
                                  yarrow                8        0Bytes            36            28             0          in progress            1393.00
volume rebalance: redhat: success: 

Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            9                    0                    in progress         
yarrow               0                    8                    in progress         
volume rebalance: redhat: success: 


[root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                9        0Bytes            33            24             0          in progress            1701.00
                                  yarrow               10        0Bytes            38            28             0          in progress            1701.00
volume rebalance: redhat: success: 
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            9                    0                    in progress         
yarrow               0                    10                   in progress

--- Additional comment from nchilaka on 2015-09-14 09:42:37 EDT ---

volume rebalance: redhat: success: 
[root@zod glusterfs]# gluster v info redhat
 
Volume Name: redhat
Type: Tier
Volume ID: ec61f03a-b9c6-4a43-8aae-a1a3ca65e234
Status: Started
Number of Bricks: 16
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick6/redhat_hot
Brick2: zod:/rhs/brick6/redhat_hot
Brick3: yarrow:/rhs/brick7/redhat_hot
Brick4: zod:/rhs/brick7/redhat_hot
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick5: zod:/rhs/brick1/redhat
Brick6: yarrow:/rhs/brick1/redhat
Brick7: zod:/rhs/brick2/redhat
Brick8: yarrow:/rhs/brick2/redhat
Brick9: zod:/rhs/brick3/redhat
Brick10: yarrow:/rhs/brick3/redhat
Brick11: zod:/rhs/brick4/redhat
Brick12: yarrow:/rhs/brick4/redhat
Brick13: zod:/rhs/brick5/redhat
Brick14: yarrow:/rhs/brick5/redhat
Brick15: yarrow:/rhs/brick6/redhat
Brick16: zod:/rhs/brick6/redhat
Options Reconfigured:
cluster.tier-demote-frequency: 30
cluster.tier-promote-frequency: 50
features.ctr-enabled: on
performance.io-cache: off
performance.quick-read: off
performance.readdir-ahead: on
Comment 3 nchilaka 2015-11-16 06:57:41 EST
still this is not fixed:
I promoted 4 files from a ec cold tier, and it can be seen that rebal shows as 12 promoted 

[root@zod ~]# gluster v tier taylor status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            27                   0                    in progress         
yarrow               0                    28                   in progress         
Tiering Migration Functionality: taylor: success
[root@zod ~]# gluster v rebal taylor status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               81        0Bytes           104             3             0          in progress          361303.00
                                  yarrow               28        0Bytes           106             3             0          in progress          361145.00
volume rebalance: taylor: success
[root@zod ~]# 


moving to failed_qa
[root@zod ~]# rpm -qa|grep gluste
rglusterfs-libs-3.7.5-6.el7rhgs.x86_64
glusterfs-fuse-3.7.5-6.el7rhgs.x86_64
glusterfs-3.7.5-6.el7rhgs.x86_64
glusterfs-server-3.7.5-6.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-6.el7rhgs.x86_64
glusterfs-cli-3.7.5-6.el7rhgs.x86_64
glusterfs-api-3.7.5-6.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-6.el7rhgs.x86_64
Comment 4 Joseph Elwin Fernandes 2015-12-05 01:00:48 EST
https://code.engineering.redhat.com/gerrit/#/c/63091/
Comment 5 Sweta Anandpara 2015-12-10 03:34:27 EST
Tested and verified this on the build glusterfs-3.7.5-10.el7rhgs.x86_64

Created 100 new files and verified that they were in the hot tier. Checked the output of 'gluster v tier <volname> status' and 'gluster v rebalance <volname> status'.

Promoted 10 files and checked the status. In the next iteration, promoted a single file. After the session time, promoted 2 new files and demoted 1 file. Did a third round of promote/demote. 

Verified the output of the following commands, at every stage:
* gluster v rebalance <volname> status
* gluster v tier <volname> status
* ls -l <hot-tier path>; ls -l <cold-tier path>

No failures were seen in the rebalance output, and the rebalance number was in sync with the promote/demote number shown in tier status, and that in turn was in tune with the files actually heated/cooled.

Moving this bug to verified in 3.1.2. Detailed logs are attached.
Comment 6 Sweta Anandpara 2015-12-10 03:36 EST
Created attachment 1104253 [details]
Server and client logs
Comment 9 errata-xmlrpc 2016-03-01 00:52:18 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Note You need to log in before you can comment on or make changes to this bug.