Bug 1214219

Summary: Data Tiering:Enabling quota command fails with "quota command failed : Commit failed on localhost"
Product: [Community] GlusterFS Reporter: Nag Pavan Chilakam <nchilaka>
Component: tieringAssignee: Joseph Elwin Fernandes <josferna>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: mainlineCC: bugs, dlambrig, sankarshan
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1219048 (view as bug list) Environment:
Last Closed: 2016-06-16 12:53:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186580, 1214666, 1219048, 1229259, 1260923    

Description Nag Pavan Chilakam 2015-04-22 09:30:44 UTC
Description of problem:
======================
On a fresh 3 node setup, when i tried to enable quota, it failed with following error:
quota command failed : Commit failed on localhost. Please check the log file for more details.

Also, the vol info shows quota as enabled on the local node where the command was executed, but doesnt show the same on other nodes

Note, I was using RHEL 7 with upstream gluster installed.
Logs showed as failed to construct graph

Version-Release number of selected component (if applicable):
=============================================================
[root@zod glusterfs]# gluster --version
glusterfs 3.7dev built on Apr 17 2015 14:27:16
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod glusterfs]# rpm -qa|grep gluster
glusterfs-api-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-cli-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-server-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-fuse-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-libs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
[root@zod glusterfs]# 

Steps to Reproduce:
==================
1.setup a multinode cluster
2.now create a tier volume
3.try to turn on quota using vol quota <vname> enable


Expected results:
=================
quotas should get enabled for all nodes

Additional info:
==============

[root@zod ~]# gluster v create vol1 replica 2 10.70.35.144:/brick_100G_1/vol1 yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1 10.70.35.144:/brick_100G_2/vol1 yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1 force
volume create: vol1: success: please start the volume to access data
[root@zod ~]# gluster v info vol1
 
Volume Name: vol1
Type: Distributed-Replicate
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.144:/brick_100G_1/vol1
Brick2: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick3: 10.70.35.144:/brick_100G_2/vol1
Brick4: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root@zod ~]# gluster v start vol1
volume start: vol1: success
[root@zod ~]# gluster v attach-tier vol1 replica 2 moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1  yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1 moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1 yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
volume add-brick: success
[root@zod ~]# gluster v info vol1
 
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root@zod ~]# gluster v quota vol1 enable
quota command failed : Commit failed on localhost. Please check the log file for more details.
[root@zod ~]# gluster v info vol1
 
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
Options Reconfigured:
features.quota: on
[root@zod ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1                              49155     0          Y       18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1                           49153     0          Y       5007 
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1                              49154     0          Y       18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1                           49152     0          Y       4988 
Brick 10.70.35.144:/brick_100G_1/vol1       49152     0          Y       32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1                                 49152     0          Y       18004
Brick 10.70.35.144:/brick_100G_2/vol1       49153     0          Y       32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1                                 49153     0          Y       18023
NFS Server on localhost                     N/A       N/A        N       N/A  
Quota Daemon on localhost                   N/A       N/A        N       N/A  
NFS Server on moonshine.lab.eng.blr.redhat.
com                                         N/A       N/A        N       N/A  
NFS Server on yarrow.lab.eng.blr.redhat.com N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@zod ~]# 
[root@zod ~]# gluster v status vol1
Locking failed on moonshine.lab.eng.blr.redhat.com. Please check log file for details.
Locking failed on yarrow.lab.eng.blr.redhat.com. Please check log file for details.
[root@zod ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1                              49155     0          Y       18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1                           49153     0          Y       5007 
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1                              49154     0          Y       18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1                           49152     0          Y       4988 
Brick 10.70.35.144:/brick_100G_1/vol1       49152     0          Y       32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1                                 49152     0          Y       18004
Brick 10.70.35.144:/brick_100G_2/vol1       49153     0          Y       32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1                                 49153     0          Y       18023
NFS Server on localhost                     N/A       N/A        N       N/A  
Quota Daemon on localhost                   N/A       N/A        N       N/A  
NFS Server on yarrow.lab.eng.blr.redhat.com N/A       N/A        N       N/A  
NFS Server on moonshine.lab.eng.blr.redhat.
com                                         N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
 

=======================
[root@yarrow ~]# gluster peer status
Number of Peers: 2

Hostname: 10.70.35.144
Uuid: d2e5c2f7-1391-4d0b-80aa-35243c5ad286
State: Peer in Cluster (Connected)

Hostname: moonshine.lab.eng.blr.redhat.com
Uuid: a4f5c421-1636-44ed-a7bf-3772deef9346
State: Peer in Cluster (Connected)
   
[root@yarrow ~]# gluster v info vol1
 
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root@yarrow ~]# 
[root@yarrow ~]# rpm -qa|grep gluster
glusterfs-api-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-cli-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-server-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-fuse-3.7dev-0.1009.git8b987be.el7.centos.x86_64
glusterfs-libs-3.7dev-0.1009.git8b987be.el7.centos.x86_64
[root@yarrow ~]# gluster v info vol1
 
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root@yarrow ~]# gluster v status vol1
Locking failed on 10.70.35.144. Please check log file for details.
Locking failed on moonshine.lab.eng.blr.redhat.com. Please check log file for details.
[root@yarrow ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1                              49155     0          Y       18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1                           49153     0          Y       5007 
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1                              49154     0          Y       18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1                           49152     0          Y       4988 
Brick 10.70.35.144:/brick_100G_1/vol1       49152     0          Y       32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1                                 49152     0          Y       18004
Brick 10.70.35.144:/brick_100G_2/vol1       49153     0          Y       32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1                                 49153     0          Y       18023
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on 10.70.35.144                  N/A       N/A        N       N/A  
Quota Daemon on 10.70.35.144                N/A       N/A        N       N/A  
NFS Server on moonshine.lab.eng.blr.redhat.
com                                         N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@yarrow ~]# gluster v quota vol1 enable
quota command failed : Commit failed on localhost. Please check the log file for more details.
[root@yarrow ~]# cd /var/log/glusterfs/
[root@yarrow glusterfs]# ls
bricks           etc-glusterfs-glusterd.vol.log  quotad.log
cli.log          glustershd.log                  quota-mount-vol1.log
cmd_history.log  nfs.log                         snaps
[root@yarrow glusterfs]# less quota
quotad.log            quota-mount-vol1.log  
[root@yarrow glusterfs]# less quotad.log 
[root@yarrow glusterfs]# 



==============================
[root@moonshine ~]# gluster v info vol1
 
Volume Name: vol1
Type: Tier
Volume ID: 981517ec-60cd-4c28-b854-0443c528e965
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick2: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_2/vol1
Brick3: yarrow.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick4: moonshine.lab.eng.blr.redhat.com:/ssdbricks_75G_1/vol1
Brick5: 10.70.35.144:/brick_100G_1/vol1
Brick6: yarrow.lab.eng.blr.redhat.com:/brick_100G_1/vol1
Brick7: 10.70.35.144:/brick_100G_2/vol1
Brick8: yarrow.lab.eng.blr.redhat.com:/brick_100G_2/vol1
[root@moonshine ~]# gluster v status vol1
Locking failed on 10.70.35.144. Please check log file for details.
Locking failed on yarrow.lab.eng.blr.redhat.com. Please check log file for details.
[root@moonshine ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_2/vol1                              49155     0          Y       18270
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_2/vol1                           49153     0          Y       5007 
Brick yarrow.lab.eng.blr.redhat.com:/ssdbri
cks_75G_1/vol1                              49154     0          Y       18249
Brick moonshine.lab.eng.blr.redhat.com:/ssd
bricks_75G_1/vol1                           49152     0          Y       4988 
Brick 10.70.35.144:/brick_100G_1/vol1       49152     0          Y       32581
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_1/vol1                                 49152     0          Y       18004
Brick 10.70.35.144:/brick_100G_2/vol1       49153     0          Y       32599
Brick yarrow.lab.eng.blr.redhat.com:/brick_
100G_2/vol1                                 49153     0          Y       18023
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on 10.70.35.144                  N/A       N/A        N       N/A  
Quota Daemon on 10.70.35.144                N/A       N/A        N       N/A  
NFS Server on yarrow.lab.eng.blr.redhat.com N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

Comment 1 Nag Pavan Chilakam 2015-04-22 09:40:43 UTC
logs @ rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/1214219/

Comment 2 Anand Avati 2015-04-30 11:39:51 UTC
REVIEW: http://review.gluster.org/10474 (glusterd/quota/tiering: Fixing volgen of quotad) posted (#1) for review on master by Joseph Fernandes (josferna)

Comment 3 Anand Avati 2015-04-30 12:07:36 UTC
REVIEW: http://review.gluster.org/10474 (glusterd/quota/tiering: Fixing volgen of quotad) posted (#2) for review on master by Joseph Fernandes (josferna)

Comment 4 Anand Avati 2015-05-01 14:46:20 UTC
COMMIT: http://review.gluster.org/10474 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit cfb9ea4dc68440a18b7f07422901a715b00776f0
Author: Joseph Fernandes <josferna>
Date:   Thu Apr 30 16:54:16 2015 +0530

    glusterd/quota/tiering: Fixing volgen of quotad
    
    The quotad's graph generation was happening wrongly for
    tiered volume. The check is been inserted.
    
    Change-Id: I5554bc5280b0fbaec750e9008fdd930ad53a774f
    BUG: 1214219
    Signed-off-by: Joseph Fernandes <josferna>
    Reviewed-on: http://review.gluster.org/10474
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Dan Lambright <dlambrig>

Comment 5 Niels de Vos 2015-05-15 13:07:37 UTC
This change should not be in "ON_QA", the patch posted for this bug is only available in the master branch and not in a release yet. Moving back to MODIFIED until there is an beta release for the next GlusterFS version.

Comment 6 Nagaprasad Sathyanarayana 2015-10-25 14:46:03 UTC
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 7 Niels de Vos 2016-06-16 12:53:42 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user