Bug 1412955

Summary:	Quota: After upgrade from 3.1.3 to 3.2 , gluster quota list command shows "No quota configured on volume repvol"
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Anil Shah <ashah>
Component:	quota	Assignee:	Sanoj Unnikrishnan <sunnikri>
Status:	CLOSED ERRATA	QA Contact:	Anil Shah <ashah>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.2	CC:	amukherj, ashah, rcyriac, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.2.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.8.4-13	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1414346 (view as bug list)		Environment:
Last Closed:	2017-03-23 06:03:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1414346
Bug Blocks:	1351528

Description Anil Shah 2017-01-13 09:12:48 UTC

Description of problem:

Quota is enabled,and limit-usage is set on 3.1.3. After upgrade form 3.1.3 to 3.2 quota list command shows "No quota configured on volume repvol "

Version-Release number of selected component (if applicable):

glusterfs-server-3.8.4-11.el7rhgs.x86_64

How reproducible:

1/1

Steps to Reproduce:
1. Create 2*2 dist-repvol vol on 3.1.3
2. Enable quota and set limit-usage and limit-object
3. Do fuse mount and create some date on mount point
4. Now do in-server upgrade of system for 3.2.
5. after upgrade run quota vol list command

Actual results:

gluster v info shows quota is on 
however quota list command shows "quota: No quota configured on volume "

Expected results:

quota list command should work 


Additional info:

Comment 4 Sanoj Unnikrishnan 2017-01-13 09:52:40 UTC

We see that the xattr has been set correctly on all bricks, 
No issues with setting quota limit (so nothing wrong in upgrade path)
[root@rhs-client46 ~]#  getfattr -d -m . -e hex /rhs/brick1/b1/ | grep limit
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.limit-objects.5=0x0000000000001388ffffffffffffffff
trusted.glusterfs.quota.limit-set.5=0x0000001900000000ffffffffffffffff

[root@rhs-client47 ~]# getfattr -d -m . -e hex /rhs/brick1/b2/ | grep limit
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.limit-objects.5=0x0000000000001388ffffffffffffffff
trusted.glusterfs.quota.limit-set.5=0x0000001900000000ffffffffffffffff

[root@rhs-client22 ~]# getfattr -d -m . -e hex /rhs/brick1/b3/ | grep limit
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.limit-objects.5=0x0000000000001388ffffffffffffffff
trusted.glusterfs.quota.limit-set.5=0x0000001900000000ffffffffffffffff


Last login: Fri Jan 13 12:07:24 2017 from dhcp35-176.lab.eng.blr.redhat.com
[root@rhs-hpc-srv3 ~]# getfattr -d -m . -e hex /rhs/brick1/b4/ | grep limit
getfattr: Removing leading '/' from absolute path names
trusted.glusterfs.quota.limit-objects.5=0x0000000000001388ffffffffffffffff
trusted.glusterfs.quota.limit-set.5=0x0000001900000000ffffffffffffffff


We notice that the quota v list command fails to show limit set on root while 
quota v list / does show the limit on root. 

[root@rhs-hpc-srv3 ~]# gluster v quota repvol list 
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/test                                    100.0GB     80%(80.0GB)   0Bytes 100.0GB              No                   No

[root@rhs-hpc-srv3 ~]# 
[root@rhs-hpc-srv3 ~]# gluster v quota repvol list /
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                        100.0GB     80%(80.0GB)    2.4GB  97.6GB              No                   No
[root@rhs-hpc-srv3 ~]# 
[root@rhs-hpc-srv3 ~]# gluster v quota repvol list
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/test                                    100.0GB     80%(80.0GB)   0Bytes 100.0GB              No                   No

This issue is similar to BZ1408359, will confirm if its a duplicate after RCA

Comment 5 Anil Shah 2017-01-16 06:51:26 UTC

Hit this issue again after upgrading from 3.1.3 to 3.2.

Comment 6 Sanoj Unnikrishnan 2017-01-16 07:24:43 UTC

Anil, could you please test with the following changes.

1) Have quota limit configured on multiple paths. So that we know if the listing issue is only with root path (as I suspect).
2) Could you also paste the output of the list command when the path is explicitly given. Just to ensure the issue is seen only when the path is not given. ("gluster v quota repvol list /")
This should help us prioritize.

Comment 7 Sanoj Unnikrishnan 2017-01-16 10:39:35 UTC

Anil and I tested the scenario again, This is indeed an upgrade issue (unlike what was suspected earlier).
It was noticed that with op version bump up quota conf file (v1.2) the conf file is updated and the limits stored in this file get lost.

So, the impact is not limited to root dir (as suspected earlier).

Testing Logs:

The non ASCII characters below correspond to gfid of dirs on which limit is set.

[root@dhcp35-65 mnt]# cat /var/lib/glusterd/vols/v1/quota.conf
GlusterFS Quota conf | version: v1.2
9fEw�bBI�;X������%d�
                        G_��C.Gp��!����Dk��$�
                                              2`�[root@dhcp35-65 mnt]# 

[root@dhcp35-65 mnt]# md5sum /var/lib/glusterd/vols/v1/quota.conf
832388c80269f401dffddc2ec842d15d  /var/lib/glusterd/vols/v1/quota.conf

[root@dhcp35-65 mnt]# gluster v set all cluster.op-version 30901 
volume set: success

[root@dhcp35-65 mnt]# md5sum /var/lib/glusterd/vols/v1/quota.conf 
465e7750229f6ff18a3ddeaec73b3dd8  /var/lib/glusterd/vols/v1/quota.conf

[root@dhcp35-65 mnt]# cat /var/lib/glusterd/vols/v1/quota.conf
GlusterFS Quota conf | version: v1.2

Note that after opversion increase we dont see the gfid (non ASCII values)

Comment 8 Atin Mukherjee 2017-01-19 04:07:39 UTC

upstream mainline patch: http://review.gluster.org/16425

Comment 9 Atin Mukherjee 2017-01-20 06:39:15 UTC

downstream patch : https://code.engineering.redhat.com/gerrit/#/c/95825/

Comment 11 Anil Shah 2017-01-31 11:37:02 UTC

After upgrade to latest build from 3.1.3 GA build , still quota list command is working.

before upgrade 
======================
[root@dhcp46-88 vol0]# gluster  v quota vol0 list
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)  500.0MB   1.5GB              No                   No
/test                                      1.0GB     80%(819.2MB)   0Bytes   1.0GB              No                   No


After upgrade:
=========================

[root@dhcp46-88 fuse]# gluster v quota vol0 list
doesn't show anything .

However quota list with path given works.

[root@dhcp46-88 fuse]# gluster v quota vol0 list /
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)    2.0GB  0Bytes             Yes                  Yes
[root@dhcp46-88 fuse]# 


Hence moving this bug to assigned

Comment 12 Sanoj Unnikrishnan 2017-01-31 18:49:09 UTC

correction in the above logs: the /test dir had been deleted prior to running list (confirmed with Anil). 
So we are expecting list to show limit only on root path

This is not a regression of the previous fix.
The gfid  have been populated as expected. There seems to be a stale gfid, which probably corresponds to /test.

Found that the listing was not happening due to the below list_count check failing.

Breakpoint 8, cli_quotad_getlimit_cbk (req=0x55555587a62c, iov=<optimized out>, count=<optimized out>, myframe=0x5555558003fc) at cli-rpc-ops.c:3918
3918	                if (list_count == max_count) {
3: ret = 0
(gdb) p list_count
$50 = 1
(gdb) p max_count
$51 = 2
(gdb) set var max_count=1
(gdb) n
3919	                        list_for_each_entry_safe (node, tmpnode,
3: ret = 0
(gdb) n
3921	                                dict = node->ptr;
3: ret = 0
(gdb) n
3922	                                print_quota_list_from_quotad (frame, dict);
3: ret = 0
(gdb) n
3921	                                dict = node->ptr;
3: ret = 0
(gdb) n
3922	                                print_quota_list_from_quotad (frame, dict);
3: ret = 0
(gdb) n
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)    2.0GB  0Bytes             Yes                  Yes
3923	                                list_node_del (node);
3: ret = 0
(gdb) n

Need to check whats the rationale for this check in code and how should we go about fixing it.

Comment 13 Sanoj Unnikrishnan 2017-02-01 06:39:57 UTC

I have been able to recreate the issue outside of upgrade path.
The issue could be with stale gfid (after an rmdir).

[root@dhcp-0-139 ~]# gluster v create v1 10.70.1.139:/export/sdb/b1
volume create: v1: success: please start the volume to access data
[root@dhcp-0-139 ~]# 
[root@dhcp-0-139 ~]# gluster v start v1
volume start: v1: success
[root@dhcp-0-139 ~]# mount -t glusterfs 10.70.1.139:v1 /mnt
[root@dhcp-0-139 ~]# gluster v quota v1 enable
volume quota : success
[root@dhcp-0-139 ~]# gluster v quota v1 limit-usage / 1MB
volume quota : success
[root@dhcp-0-139 ~]# mkdir /mnt/test
[root@dhcp-0-139 ~]# gluster v quota v1 limit-usage /test 1MB
volume quota : success
[root@dhcp-0-139 ~]# 
[root@dhcp-0-139 ~]# hexdump /var/lib/glusterd/vols/v1/quota.conf 
0000000 6c47 7375 6574 4672 2053 7551 746f 2061
0000010 6f63 666e 7c20 7620 7265 6973 6e6f 203a
0000020 3176 322e 000a 0000 0000 0000 0000 0000
0000030 0000 0000 0101 b5ae 8664 5994 994d 4d91
0000040 c2ac a4a9 490d 0001                    
0000047
[root@dhcp-0-139 ~]# 
[root@dhcp-0-139 ~]# rm -rf /mnt/test
[root@dhcp-0-139 ~]# gluster v quota v1 list 
[root@dhcp-0-139 ~]# 
[root@dhcp-0-139 ~]# gluster v quota v1 list /
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          1.0MB     80%(819.2KB)   0Bytes   1.0MB              No                   No

Comment 15 Anil Shah 2017-02-01 10:33:42 UTC

After upgrade quota list command works as expected.

Before Upgrade:
======================================

[root@dhcp46-88 yum.repos.d]# gluster v quota vol0 list
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)    1.1GB 948.0MB              No                   No
/test                                      1.0GB     80%(819.2MB)  100.0MB 924.0MB              No                   No
/test/test1                                1.0GB     80%(819.2MB)  100.0MB 924.0MB              No                   No
/testdir                                   1.0GB     80%(819.2MB)   0Bytes   1.0GB              No                   No

[root@dhcp46-88 yum.repos.d]# gluster v quota vol0 list-objects
                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/                                             2500       80%(2000)       1100         6        1394              No                   No
/test                                         2500       80%(2000)        100         2        2398              No                   No

==========================
After upgrade :
==========================

[root@dhcp46-88 yum.repos.d]# gluster v set all op-version 30901
volume set: success

[root@dhcp46-88 yum.repos.d]# gluster v quota vol0 list
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)    1.1GB 948.0MB              No                   No
/test                                      1.0GB     80%(819.2MB)  100.0MB 924.0MB              No                   No
/test/test1                                1.0GB     80%(819.2MB)  100.0MB 924.0MB              No                   No
/testdir                                   1.0GB     80%(819.2MB)   0Bytes   1.0GB              No                   No
[root@dhcp46-88 yum.repos.d]# gluster v quota vol0 list /
                  Path                   Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit exceeded?
-------------------------------------------------------------------------------------------------------------------------------
/                                          2.0GB     80%(1.6GB)    1.1GB 948.0MB              No                   No
[root@dhcp46-88 yum.repos.d]# gluster v quota vol0 list-objects
                  Path                   Hard-limit   Soft-limit      Files       Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/                                             2500       80%(2000)       1100         6        1394              No                   No
/test                                         2500       80%(2000)        100         2        2398              No                   No


[root@dhcp46-88 yum.repos.d]# cat /var/lib/glusterd/vols/vol0/quota.conf
GlusterFS Quota conf | version: v1.2
���w�DM����_1:�u˧�@̋O�#g�Ǌ�z��>Du�u��@a����w�DM����_1:�

[root@dhcp46-88 yum.repos.d]# cat /var/lib/glusterd/vols/vol0/quota.cksum 
cksum=2066491058
version=6

Bug verified on build glusterfs-server-3.8.4-13.el7rhgs.x86_64

Comment 17 errata-xmlrpc 2017-03-23 06:03:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html