Bug 1265623 - Data Tiering:Promotions and demotions fail after quota hard limits are hit for a tier volume
Data Tiering:Promotions and demotions fail after quota hard limits are hit fo...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: tiering (Show other bugs)
3.7.4
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Vijaikumar Mallikarjuna
bugs@gluster.org
:
Depends On:
Blocks: 1260923 1266841 1267812 glusterfs-3.7.6 1278419
  Show dependency treegraph
 
Reported: 2015-09-23 07:25 EDT by nchilaka
Modified: 2016-05-11 18:48 EDT (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.7.6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1266878 1267812 (view as bug list)
Environment:
Last Closed: 2015-11-17 00:58:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-09-23 07:25:06 EDT
Description of problem:
=====================
Once quota hard limits are hit for a tier volume, promotes and demotes are failing.
Only link files are getting created, but files are not getting migrated.
As tiering migration is internal, it should do promos/demos even on quota filled vols.





Version-Release number of selected component (if applicable):
==============================================================
[root@zod ~]# 
[root@zod ~]# 
[root@zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 





Steps to Reproduce:
==================
1.create a tier volume
2.enable quota and set hardlimit to say 5gb
3.now fill the hardlimit and wait for demotes to happen

It can be seen that only link files are created but files are not really moved.

Errors:
=========

[2015-09-23 10:04:03.249877] E [MSGID: 109039] [dht-helper.c:1176:dht_rebalance_inprogress_task] 0-taiwan-cold-dht: /h.125: failed to get the 'linkto' xattr [No data available]
[2015-09-23 10:04:03.250197] E [MSGID: 109023] [dht-rebalance.c:800:__dht_rebalance_open_src_file] 0-taiwan-tier-dht: failed to set xattr on /h.125 in taiwan-cold-dht (Invalid argument)
[2015-09-23 10:04:03.250228] E [MSGID: 109023] [dht-rebalance.c:1106:dht_migrate_file] 0-taiwan-tier-dht: Migrate file failed: failed to open /h.125 on taiwan-cold-dht
[2015-09-23 10:04:03.250880] E [MSGID: 109037] [tier.c:365:tier_migrate_using_query_file] 0-taiwan-tier-dht: ERROR -28 in current migration h.125 /h.125


[root@zod ~]# gluster v rebal taiwan status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               12        0Bytes           216           204             0          in progress            3203.00
                                  yarrow              103        0Bytes           862           759             0          in progress            3203.00
volume rebalance: taiwan: success:
Comment 1 nchilaka 2015-09-23 07:39:01 EDT
sosreports
[root@rhsqe-repo bug.1265623]# cd ..
[root@rhsqe-repo sosreports]# chmod -R 0777 bug.1265623
[root@rhsqe-repo sosreports]# pwd
/home/repo/sosreports
[root@rhsqe-repo sosreports]# hostname
rhsqe-repo.lab.eng.blr.redhat.com




 username:qe-admin
Comment 2 Vijaikumar Mallikarjuna 2015-10-01 02:28:15 EDT
Patch submitted: http://review.gluster.org/#/c/12266/
Comment 3 Raghavendra Talur 2015-11-17 00:58:48 EST
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.6, please open a new bug report.

glusterfs-3.7.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-November/024359.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.