+++ This bug was initially created as a clone of Bug #1262860 +++ Description of problem: ======================== On a EC cold volume, when files are promoted or demoted to/from hot tier, it seems like the tier deamon is seeing each copy or part of the file as a seperate different file. The counter atleast say this. I had 3 files on a 2 x (4 + 2) = 12 EC cold volume. When they were promoted or demoted to/from a distrep hot tier, the stats show each file is counted as 6 times, with 1 time showing the success while the other 5 registering as failure. Version-Release number of selected component (if applicable): =========================================================== [root@zod glusterfs]# rpm -qa|grep gluster glusterfs-client-xlators-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-api-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-fuse-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-server-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 glusterfs-cli-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 ^[[Aglusterfs-libs-3.7.4-0.33.git1d02d4b.el7.centos.x86_64 [root@zod glusterfs]# gluster --version glusterfs 3.7.4 built on Sep 12 2015 01:35:35 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@zod glusterfs]# Steps to Reproduce: ==================== 1.Create a EC cold vol and distrep hot tier 2.Now set promo/demote freq after enabling ctr 3.Now create a file and wait for it to demote and then make it get promoted. It can be seen that though it is one file, each copy of the file in each brick is considered as different file and the promote/demote counters show them as failed. Below was the case where I had 3 files(compare promote/demote numbers with failures) ==================================== [root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 9 0Bytes 31 22 0 in progress 1393.00 yarrow 8 0Bytes 36 28 0 in progress 1393.00 volume rebalance: redhat: success: Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 9 0 in progress yarrow 0 8 in progress volume rebalance: redhat: success: [root@zod glusterfs]# gluster v rebal redhat status; gluster v tier redhat status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 9 0Bytes 33 24 0 in progress 1701.00 yarrow 10 0Bytes 38 28 0 in progress 1701.00 volume rebalance: redhat: success: Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 9 0 in progress yarrow 0 10 in progress --- Additional comment from nchilaka on 2015-09-14 09:42:37 EDT --- volume rebalance: redhat: success: [root@zod glusterfs]# gluster v info redhat Volume Name: redhat Type: Tier Volume ID: ec61f03a-b9c6-4a43-8aae-a1a3ca65e234 Status: Started Number of Bricks: 16 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: yarrow:/rhs/brick6/redhat_hot Brick2: zod:/rhs/brick6/redhat_hot Brick3: yarrow:/rhs/brick7/redhat_hot Brick4: zod:/rhs/brick7/redhat_hot Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick5: zod:/rhs/brick1/redhat Brick6: yarrow:/rhs/brick1/redhat Brick7: zod:/rhs/brick2/redhat Brick8: yarrow:/rhs/brick2/redhat Brick9: zod:/rhs/brick3/redhat Brick10: yarrow:/rhs/brick3/redhat Brick11: zod:/rhs/brick4/redhat Brick12: yarrow:/rhs/brick4/redhat Brick13: zod:/rhs/brick5/redhat Brick14: yarrow:/rhs/brick5/redhat Brick15: yarrow:/rhs/brick6/redhat Brick16: zod:/rhs/brick6/redhat Options Reconfigured: cluster.tier-demote-frequency: 30 cluster.tier-promote-frequency: 50 features.ctr-enabled: on performance.io-cache: off performance.quick-read: off performance.readdir-ahead: on
still this is not fixed: I promoted 4 files from a ec cold tier, and it can be seen that rebal shows as 12 promoted [root@zod ~]# gluster v tier taylor status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 27 0 in progress yarrow 0 28 in progress Tiering Migration Functionality: taylor: success [root@zod ~]# gluster v rebal taylor status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 81 0Bytes 104 3 0 in progress 361303.00 yarrow 28 0Bytes 106 3 0 in progress 361145.00 volume rebalance: taylor: success [root@zod ~]# moving to failed_qa [root@zod ~]# rpm -qa|grep gluste rglusterfs-libs-3.7.5-6.el7rhgs.x86_64 glusterfs-fuse-3.7.5-6.el7rhgs.x86_64 glusterfs-3.7.5-6.el7rhgs.x86_64 glusterfs-server-3.7.5-6.el7rhgs.x86_64 glusterfs-client-xlators-3.7.5-6.el7rhgs.x86_64 glusterfs-cli-3.7.5-6.el7rhgs.x86_64 glusterfs-api-3.7.5-6.el7rhgs.x86_64 glusterfs-debuginfo-3.7.5-6.el7rhgs.x86_64
https://code.engineering.redhat.com/gerrit/#/c/63091/
Tested and verified this on the build glusterfs-3.7.5-10.el7rhgs.x86_64 Created 100 new files and verified that they were in the hot tier. Checked the output of 'gluster v tier <volname> status' and 'gluster v rebalance <volname> status'. Promoted 10 files and checked the status. In the next iteration, promoted a single file. After the session time, promoted 2 new files and demoted 1 file. Did a third round of promote/demote. Verified the output of the following commands, at every stage: * gluster v rebalance <volname> status * gluster v tier <volname> status * ls -l <hot-tier path>; ls -l <cold-tier path> No failures were seen in the rebalance output, and the rebalance number was in sync with the promote/demote number shown in tier status, and that in turn was in tune with the files actually heated/cooled. Moving this bug to verified in 3.1.2. Detailed logs are attached.
Created attachment 1104253 [details] Server and client logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html