Bug 1229259 - Data Tiering:command prompt hangs when fetching quota list of a tiered volume
Summary: Data Tiering:command prompt hangs when fetching quota list of a tiered volume
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: tier
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Mohammed Rafi KC
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On: 1214219 1214222 1214666 1219048 1219848 1224075 1224077 1230540 1236128 1238052 1254439
Blocks: qe_tracker_everglades 1202842
TreeView+ depends on / blocked
 
Reported: 2015-06-08 10:37 UTC by Nag Pavan Chilakam
Modified: 2016-09-17 15:44 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1214666
Environment:
Last Closed: 2015-10-30 12:40:16 UTC
Embargoed:


Attachments (Terms of Use)

Description Nag Pavan Chilakam 2015-06-08 10:37:08 UTC
+++ This bug was initially created as a clone of Bug #1214666 +++

Description of problem:
======================
When we enable a quota on a regular volume and then attach a tier, and then try to list quota settings on this tiered volume, the command prompt hangs indefinitely 


Version-Release number of selected component (if applicable):
============================================================
[root@zod ~]# gluster --version
glusterfs 3.7dev built on Apr 17 2015 14:27:16
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# rpm -qa|grep gluster
glusterfs-api-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-cli-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-server-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-fuse-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-libs-3.7dev-0.1009.git8b987be.el7.centos.x86_64

Steps to Reproduce:
==================
1.create a regular volume
2.enable quota on the volume
3.attach a tier to this volume to make it tier volume
4. now issue a "gluster v quota vol3 list" to display quota settings,
it can be seen that the command prompt freezes indefinitely, till the user intervenes with ctrl+c as below
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
^C




Expected results:
================
should be able to get quota settings even for tierd volume

Additional info:
================
[root@zod ~]# gluster v create vol3 replica 2 10.70.35.144:/brick_100G_1/vol3 yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol3 10.70.35.144:/brick_100G_2/vol3 yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol3 force
volume create: vol3: success: please start the volume to access data
[root@zod ~]# gluster v start vol3
volume start: vol3: success
[root@zod ~]# gluster v info vol3
 
Volume Name: vol3
Type: Distributed-Replicate
Volume ID: f324afbd-390c-434c-b0f1-d9222218014a
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.144:/brick_100G_1/vol3
Brick2: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol3
Brick3: 10.70.35.144:/brick_100G_2/vol3
Brick4: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol3

[root@zod ~]# gluster v quota vol3 enable 
glvolume quota : success

[root@zod ~]# gluster v set vol3 features.uss on
volume set: success

[root@zod ~]# gluster v info vol3
 
Volume Name: vol3
Type: Distributed-Replicate
Volume ID: f324afbd-390c-434c-b0f1-d9222218014a
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.144:/brick_100G_1/vol3
Brick2: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol3
Brick3: 10.70.35.144:/brick_100G_2/vol3
Brick4: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol3
Options Reconfigured:
features.uss: on
features.quota: on
[root@zod ~]# gluster v quota vol3 list
quota: No quota configured on volume vol3

[root@zod ~]# gluster v quota vol3 limit-usage / 20GB
volume quota : success
[root@zod ~]# gluster v quota vol3 limit-usage / 20GB 90%
volume quota : success
[root@zod ~]# gluster v quota vol3 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                         20.0GB       90%      0Bytes  20.0GB              No                   No
[root@zod ~]# gluster v quota vol3 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                         20.0GB       90%     200.6MB  19.8GB              No                   No
[root@zod ~]# 
[root@zod ~]# 
[root@zod ~]# gluster v attach-tier vol3 replica 2 moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol3  yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol3 moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol3 yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol3 force
volume add-brick: success
[root@zod ~]# gluster v quota vol3 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
^C
[root@zod ~]# gluster v detach-tier vol3
volume remove-brick unknown: success
[root@zod ~]# gluster v quota vol3 list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/                                         20.0GB       90%       1.2GB  18.8GB              No                   No
[root@zod ~]# gluster v info vol3
 
Volume Name: vol3
Type: Distributed-Replicate
Volume ID: f324afbd-390c-434c-b0f1-d9222218014a
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.144:/brick_100G_1/vol3
Brick2: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol3
Brick3: 10.70.35.144:/brick_100G_2/vol3
Brick4: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol3
Options Reconfigured:
features.uss: on
features.quota: on

--- Additional comment from Joseph Elwin Fernandes on 2015-05-01 02:39:36 EDT ---

This bug depends on these bugs to be fixed

bug 1214219: Data Tiering:Enabling quota command fails with "quota command failed : Commit failed on localhost"
http://review.gluster.org/10474 
This will fix the hang that is caused due to incorrect graph built in vol file of quotad


bug 1214222: Directories are missing on the mount point after attaching tier to distribute replicate volume. 
http://review.gluster.org/10363 
This will fix the disappearing of the directories, and hence disappearing of the qouta list, when a tier is attached.

--- Additional comment from Niels de Vos on 2015-05-15 09:07:42 EDT ---

This change should not be in "ON_QA", the patch posted for this bug is only available in the master branch and not in a release yet. Moving back to MODIFIED until there is an beta release for the next GlusterFS version.

Comment 3 Triveni Rao 2015-06-11 16:48:21 UTC
here quota list doesnot hang but does not show the proper expected output:

[root@rhsqa14-vm1 ~]# gluster v quota mix list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
[root@rhsqa14-vm1 ~]#

Comment 4 Triveni Rao 2015-06-12 11:31:09 UTC
because of the no proper output for the execution of quota list command, this bug is moved to assigned and failed qa 


[root@rhsqa14-vm1 ~]# gluster  v create mix 10.70.47.165:/rhs/brick3/h0 10.70.47.163:/rhs/brick3/h0 
volume create: mix: success: please start the volume to access data
[root@rhsqa14-vm1 ~]# gluster v start mix
volume start: mix: success
[root@rhsqa14-vm1 ~]# gluster v info mix
 
Volume Name: mix
Type: Distribute
Volume ID: 0f1a01aa-9b1a-4cd5-bc9c-64b14630f264
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.47.165:/rhs/brick3/h0
Brick2: 10.70.47.163:/rhs/brick3/h0
Options Reconfigured:
performance.readdir-ahead: on
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]#

[root@rhsqa14-vm1 ~]# gluster v status mix
Status of volume: mix
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.47.165:/rhs/brick3/h0           49176     0          Y       15439
Brick 10.70.47.163:/rhs/brick3/h0           49176     0          Y       14757
NFS Server on localhost                     2049      0          Y       15459
NFS Server on 10.70.47.159                  2049      0          Y       11951
NFS Server on 10.70.46.2                    2049      0          Y       1416 
NFS Server on 10.70.47.163                  2049      0          Y       14781
 
Task Status of Volume mix
------------------------------------------------------------------------------
There are no active volume tasks
 


[root@rhsqa14-vm1 ~]# gluster v attach-tier mix 10.70.47.165:/rhs/brick4/h0 10.70.47.163:/rhs/brick4/h0
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success
volume rebalance: mix: success: Rebalance on mix has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: e5858e08-2ccc-4176-9b2d-bf1419251323

[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# gluster v info mix
 
Volume Name: mix
Type: Tier
Volume ID: 0f1a01aa-9b1a-4cd5-bc9c-64b14630f264
Status: Started
Number of Bricks: 4
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.47.163:/rhs/brick4/h0
Brick2: 10.70.47.165:/rhs/brick4/h0
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 2
Brick3: 10.70.47.165:/rhs/brick3/h0
Brick4: 10.70.47.163:/rhs/brick3/h0
Options Reconfigured:
performance.readdir-ahead: on
[root@rhsqa14-vm1 ~]# ./options.sh mix
volume set: success
volume quota : success
volume set: success
volume quota : success
volume set: success
[root@rhsqa14-vm1 ~]# gluster v quota mix list
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# 


[root@rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.1 built on Jun  9 2015 02:31:54
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.1-1.el6rhs.x86_64
glusterfs-cli-3.7.1-1.el6rhs.x86_64
glusterfs-libs-3.7.1-1.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64
glusterfs-fuse-3.7.1-1.el6rhs.x86_64
glusterfs-server-3.7.1-1.el6rhs.x86_64
glusterfs-api-3.7.1-1.el6rhs.x86_64
[root@rhsqa14-vm1 ~]#


Note You need to log in before you can comment on or make changes to this bug.