Red Hat Bugzilla – Bug 1232625
Data Tiering: Files not getting promoted once demoted
Last modified: 2016-09-17 11:40:59 EDT
Description of problem: ======================= After created a tiered volume, I have added a couple of files which went to the default hot tier as expected. I have set only "features.ctr-enabled" to "ON" and no other tiering parameter. But I noticed that when I don't work on that file for sometime, the file gets demoted to cold tier. But once a file is demoted to cold tier, it is not getting promoted at all. As discussed the default frequency counter is 120. That means if a file in cold tier is modified(i have touched the file) more than 120 times in the given default promote time, it should get promoted automatically. But this is not happening at all Version-Release number of selected component (if applicable): ============================================================== [root@rhsqa14-vm4 glusterfs]# gluster --version glusterfs 3.7.1 built on Jun 14 2015 21:55:41 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@rhsqa14-vm4 glusterfs]# rpm -qa|grep gluster glusterfs-libs-3.7.1-3.el6rhs.x86_64 glusterfs-fuse-3.7.1-3.el6rhs.x86_64 glusterfs-3.7.1-3.el6rhs.x86_64 glusterfs-api-3.7.1-3.el6rhs.x86_64 glusterfs-cli-3.7.1-3.el6rhs.x86_64 glusterfs-geo-replication-3.7.1-3.el6rhs.x86_64 glusterfs-debuginfo-3.7.1-3.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-3.el6rhs.x86_64 glusterfs-server-3.7.1-3.el6rhs.x86_64 glusterfs-rdma-3.7.1-3.el6rhs.x86_64 How reproducible: ================ everytime Steps to Reproduce: =================== 1.create a tiered volume 2.set only "features.ctr-enabled" to "ON" 3.now create a file which will go to hot tier. 4. Wait for some time while keeping the setup and file idle(say 3 min) and you can see file gets demoted It can be seen that file is demoted 5. Now touch the file any number of times back to back and it can be seen that file never gets promoted. Actual behavior: =============== A file once demoted is not getting promoted at all even on modification any number of times back to back and it can be seen that file never gets promoted. Also, if we think we must set all tiering parameters(vol set options), for file to get promoted, then in that case file demotion too should happen only when all params are set. But in this case file demote happens but not promoted back Expected results: =================== file should get promoted if modified the default number of times or more and the default time period. Also, if we think we must set all tiering parameters(vol set options), for file to get promoted, then in that case file demotion too should happen only when all params are set. But in this case file demote happens but not promoted back Additional info: =============== I have used a regular dist-rep volume and mounted on fuse and nfs simultanteously and while pumping IO (copying a huge file of 4gb), i attached a tier. On this kind of volume i reused for this test case
Created attachment 1039790 [details] QA_console logs where the bug was identified
Created attachment 1039805 [details] server#1 logs sosreports
Created attachment 1039806 [details] server#2 logs sosreports
sos reports attached
Created attachment 1039811 [details] QA_console logs where the bug was identified(changed file permissions)
http://review.gluster.org/#/c/11334/
https://code.engineering.redhat.com/gerrit/#/c/51756/
https://code.engineering.redhat.com/gerrit/#/c/49748/
A File in a simple case is now getting promoted and demoted any number of times. Hence moving this bug as verified 1)Created a tier vol 2)set following options Type: Tier Volume ID: 232deda8-c5c1-491e-a20d-4d34d3cffff1 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.42.203:/rhs/brick3/vol2 Brick2: 10.70.42.73:/rhs/brick3/vol2 Cold Tier: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.42.73:/rhs/brick1/vol2 Brick4: 10.70.42.144:/rhs/brick1/vol2 Brick5: 10.70.42.73:/rhs/brick2/vol2 Brick6: 10.70.42.144:/rhs/brick2/vol2 Options Reconfigured: cluster.write-freq-threshold: 0 cluster.read-freq-threshold: 0 features.record-counters: off cluster.tier-demote-frequency: 120 cluster.tier-promote-frequency: 120 features.ctr-enabled: on performance.quick-read: off performance.io-cache: off performance.readdir-ahead: on 3) Now mounted file on nfs 4)created a file 5)waited for some time by keeping file idle 6)the file got demoted after sometime. 7)then touched the file to change ctime 8)now the file got promoted again did this in a loop twice and promotions and demotions were happening Hence moving the bug to verified(fixed) Note: Following were some observations: 1)fusemount 2)when multiple files were played with, some files were not getting migrated. A seperate bug was raised bz#1238944 3) Didn't play with the volume tier settings and options due to the scope of feature 4)Previously a file demoted was not getting promoted at all, but now the problem perse was fixed hence moving to verified Sos reports and logs attached version: ======= [root@nchilaka-tier01 ~]# gluster --version glusterfs 3.7.1 built on Jul 2 2015 21:01:51 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@nchilaka-tier01 ~]# rpm -qa|grep gluster gluster-nagios-common-0.2.0-1.el6rhs.noarch vdsm-gluster-4.16.20-1.2.el6rhs.noarch glusterfs-client-xlators-3.7.1-7.el6rhs.x86_64 glusterfs-server-3.7.1-7.el6rhs.x86_64 nfs-ganesha-gluster-2.2.0-3.el6rhs.x86_64 python-gluster-3.7.1-6.el6rhs.x86_64 glusterfs-3.7.1-7.el6rhs.x86_64 glusterfs-api-3.7.1-7.el6rhs.x86_64 glusterfs-cli-3.7.1-7.el6rhs.x86_64 glusterfs-geo-replication-3.7.1-7.el6rhs.x86_64 glusterfs-rdma-3.7.1-7.el6rhs.x86_64 gluster-nagios-addons-0.2.4-2.el6rhs.x86_64 glusterfs-libs-3.7.1-7.el6rhs.x86_64 glusterfs-fuse-3.7.1-7.el6rhs.x86_64 glusterfs-ganesha-3.7.1-7.el6rhs.x86_64
Created attachment 1049126 [details] QE verified cli log
Created attachment 1049133 [details] rebalance logs for QE verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html