Bug 998914

The next time you hit this issue, could you provide the xattrs (and perhaps the logs as well) on each node as it would be difficult to root-cause the problem without the same.

Saurabh,

I am not able to get sos reports. I am getting permission denied errors

You don't have permission to access /sosreports/998914/sosreport-rhsauto032-20130821013955-eca0.tar.xz on this server.

Can you please tell what is the volume configuration?

regards,
Raghavendra.

The bug is not in cli. Hence reassigning the bug to Raghavendra G.

After talking to Saurabh, updating this with steps to re-create the issue:
The directory on which quota limit is set, should be partially filled to observe this issue.

So full steps to re-create the issue is:
1) Create 3x2 dist-rep volume
2) Enable quota
3) Set volume quota of 1GB
4) Fill the directory partially.
5) Check quota list output on all the peers in cluster.

Expected result:
The outputs on all of the nodes should be same.

I tried the steps mentioned in comment-6. It is working fine. 

On all the machines I see the following output:
root@pranith-vm2 - ~ 
04:52:54 :) ⚡ watch gluster volume quota r2 list

Every 2.0s: gluster volume quota r2 list                                         Mon Sep 30 04:57:48 2013

                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0GB       80%     512.1MB 511.9MB

Since the issue is not re-creatable with the steps to re-create. I am moving the bug to ON_QA.

I found a case where the quota sizes will be shown differently not just on two different nodes but also on the same node with multiple invocations of the command "quota list".

Here are the steps:
1) create pure replicate volume r2
2) start the volume and kill one of the bricks immediately
3) enable quota and set limit-usage on / with say 1GB
4) Now create a file of size 1M using "dd of=/mnt/r2/h if=/dev/zero bs=1M count=1"
5) Now if the brick is brought back up the quota size xattrs differ on both the bricks so it also gives different outputs on both the nodes.

xattrs in my run after performing the steps above are:
Please note the quota.size xattrs before, after self-heal.

On bad brick:

root@pranithk-vm1 - /mnt/r2 
16:06:35 :) ⚡ getfattr -d -m. -e hex /brick/r2_0/{,h}
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_0/
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000
trusted.glusterfs.volume-id=0x241701a0e2e3468f946a56149312cfc5

# file: brick/r2_0/h
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0xd3692e0073304d04b6ef62e80f347302
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000100000
trusted.pgfid.00000000-0000-0000-0000-000000000001=0x00000001

On Good brick:
root@pranith-vm2 - ~ 
04:02:28 :) ⚡ getfattr -d -m. -e hex /brick/r2_1/{,h}
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_1/
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000100000
trusted.glusterfs.volume-id=0x241701a0e2e3468f946a56149312cfc5

# file: brick/r2_1/h
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0xd3692e0073304d04b6ef62e80f347302
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000100000
trusted.pgfid.00000000-0000-0000-0000-000000000001=0x00000001



After the fix:
root@pranithk-vm1 - /mnt/r2 
15:46:40 :( ⚡ getfattr -d -m. -e hex /brick/r2_0/
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_0/
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000100000
trusted.glusterfs.volume-id=0x3a1dc83af3cf4a28a3257bf3f9364992


root@pranithk-vm1 - /mnt/r2 
15:46:42 :) ⚡ getfattr -d -m. -e hex /brick/r2_0/h
getfattr: Removing leading '/' from absolute path names
# file: brick/r2_0/h
trusted.afr.r2-client-0=0x000000000000000000000000
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x37b7ebcd4ad3493ab0d5dec4a5ff7ebb
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000100000
trusted.pgfid.00000000-0000-0000-0000-000000000001=0x00000001

Patch is at https://code.engineering.redhat.com/gerrit/13718

patch on u1: https://code.engineering.redhat.com/gerrit/13720

This issue still exist. Re-created the issue on build "glusterfs 3.4.0.35rhs built on Oct 15 2013 14:06:04" 

Output from the quota list command:
================================== 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Source node:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
root@rhs-client13 [Oct-17-2013-12:05:19] >gluster v quota vol_dis_rep list /user2
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/user2                                     1.0GB       80%       2.3GB  0Bytes



root@rhs-client13 [Oct-17-2013-12:05:05] >getfattr -d -e hex -m . /rhs/bricks/brick7/user2
getfattr: Removing leading '/' from absolute path names
# file: rhs/bricks/brick7/user2
trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000
trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000
trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000019800000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000019800000


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Sink node:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

root@rhs-client14 [Oct-17-2013-12:06:14] >gluster v quota vol_dis_rep list /user2
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/user2                                     1.0GB       80%       1.9GB  0Bytes


root@rhs-client14 [Oct-17-2013-12:04:59] >getfattr -d -e hex -m . /rhs/bricks/brick8/user2
getfattr: Removing leading '/' from absolute path names
# file: rhs/bricks/brick8/user2
trusted.afr.vol_dis_rep-client-6=0x000000000000000000000000
trusted.afr.vol_dis_rep-client-7=0x000000000000000000000000
trusted.gfid=0xb4dba3d0d212441e8a3f1f7c57e56cda
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-set=0x0000000040000000ffffffffffffffff
trusted.glusterfs.quota.size=0x0000000000000000

Moving the bug back to assigned state.

hi Shwetha,
       Steps to re-create the bug seem to be different according to the quota sizes of the directory. Could you specify the steps you used to re-create the bug above.
Looking at the quota sizes in the comment above it seems more like the bug 1001556 for which a build is yet to be provided.

Pranith.

Executed the case mentioned in "comment 8" on build "glusterfs 3.4.0.36rhs built on Oct 22 2013 10:56:18" . 

Issue no longer exist. Moving the bug to verified state.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html