Bug 981589 - quota: bricks down with crash
quota: bricks down with crash
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Raghavendra Bhat
Sudhir D
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-05 04:24 EDT by Saurabh
Modified: 2016-01-19 01:12 EST (History)
6 users (show)

See Also:
Fixed In Version: v3.4.0.12rhs.beta4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 18:39:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Saurabh 2013-07-05 04:24:15 EDT
Description of problem:
bricks go down with a crash seen for them, while putting quota limit on a directory which is having I/O over nfs mount,

[root@quota1 ~]# gluster volume info
 
Volume Name: dist-rep
Type: Distributed-Replicate
Volume ID: b1b80b68-b98b-4aab-a563-3c386c39b842
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.37.98:/rhs/bricks/d1r1
Brick2: 10.70.37.174:/rhs/bricks/d1r2
Brick3: 10.70.37.136:/rhs/bricks/d2r1
Brick4: 10.70.37.168:/rhs/bricks/d2r2
Brick5: 10.70.37.98:/rhs/bricks/d3r1
Brick6: 10.70.37.174:/rhs/bricks/d3r2
Brick7: 10.70.37.136:/rhs/bricks/d4r1
Brick8: 10.70.37.168:/rhs/bricks/d4r2
Brick9: 10.70.37.98:/rhs/bricks/d5r1
Brick10: 10.70.37.174:/rhs/bricks/d5r2
Brick11: 10.70.37.136:/rhs/bricks/d6r1
Brick12: 10.70.37.168:/rhs/bricks/d6r2
Options Reconfigured:
features.quota: on
[root@quota1 ~]# 
[root@quota1 ~]# 
[root@quota1 ~]# gluster volume status
Status of volume: dist-rep
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.98:/rhs/bricks/d1r1                      49152   Y       25494
Brick 10.70.37.174:/rhs/bricks/d1r2                     49152   Y       26755
Brick 10.70.37.136:/rhs/bricks/d2r1                     49152   Y       27137
Brick 10.70.37.168:/rhs/bricks/d2r2                     49152   Y       16599
Brick 10.70.37.98:/rhs/bricks/d3r1                      N/A     N       25505
Brick 10.70.37.174:/rhs/bricks/d3r2                     49153   Y       26766
Brick 10.70.37.136:/rhs/bricks/d4r1                     49153   Y       27148
Brick 10.70.37.168:/rhs/bricks/d4r2                     49153   Y       16610
Brick 10.70.37.98:/rhs/bricks/d5r1                      49154   Y       25516
Brick 10.70.37.174:/rhs/bricks/d5r2                     N/A     N       26777
Brick 10.70.37.136:/rhs/bricks/d6r1                     49154   Y       27159
Brick 10.70.37.168:/rhs/bricks/d6r2                     49154   Y       16621
NFS Server on localhost                                 2049    Y       25529
Self-heal Daemon on localhost                           N/A     Y       25537
NFS Server on d6203fcb-6e40-417e-986b-6a74c7f23c42      2049    Y       16641
Self-heal Daemon on d6203fcb-6e40-417e-986b-6a74c7f23c4
2                                                       N/A     Y       16655
NFS Server on cc7bc8ba-fa3a-43d9-a899-114e34d27eb4      2049    Y       26800
Self-heal Daemon on cc7bc8ba-fa3a-43d9-a899-114e34d27eb
4                                                       N/A     Y       26809
NFS Server on 1eb725d7-70d7-4075-a580-b9711fc7affd      2049    Y       27175
Self-heal Daemon on 1eb725d7-70d7-4075-a580-b9711fc7aff
d                                                       N/A     Y       27182
 
There are no active volume tasks
[root@quota1 ~]# 


Version-Release number of selected component (if applicable):
[root@quota1 ~]# rpm -qa | grep glusterfs
glusterfs-fuse-3.4.0.12rhs.beta2-1.el6rhs.x86_64
glusterfs-server-3.4.0.12rhs.beta2-1.el6rhs.x86_64
glusterfs-3.4.0.12rhs.beta2-1.el6rhs.x86_64
[root@quota1 ~]# 

How reproducible:
seen for the first time

Steps to Reproduce:
1. create a volume, start it
2. set quota on the root of the volume
3. mount it over nfs
4. start linux untar and put quota on the directory on which I/O is happening,
in this case,
[root@quota1 ~]# gluster volume quota dist-rep list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                           20GB       90%      10.7MB  20.0GB
/linux-2.6.31.1/Documentation                5GB       90%       1.7MB   5.0GB
[root@quota1 ~]# 


also, infact try to modify the limits for root and directory in consideration to "higher value" than earlier.

Actual results:

2013-07-05 00:48:48.373150] W [common-utils.c:841:gf_string2percent] (-->/usr/lib64/libglusterfs.so.0(+0x5376a) [0x7fc93fcde76a] (-->/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(reconfigure+0x13d) [0x7fc93a3f1abd] (-->/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(quota_parse_limits+0x1c8) [0x7fc93a3f1668]))) 0-dist-rep-quota: argument invalid
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2013-07-05 00:48:48configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.12rhs.beta2
/lib64/libc.so.6[0x325e432920]
/lib64/libc.so.6[0x325e478641]
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x129)[0x7fc93fca4989]
/usr/lib64/libglusterfs.so.0(gf_string2percent+0x111)[0x7fc93fcb1af1]
/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(quota_parse_limits+0x1c8)[0x7fc93a3f1668]
/usr/lib64/glusterfs/3.4.0.12rhs.beta2/xlator/features/quota.so(reconfigure+0x13d)[0x7fc93a3f1abd]
/usr/lib64/libglusterfs.so.0(+0x5376a)[0x7fc93fcde76a]
/usr/lib64/libglusterfs.so.0(+0x536fc)[0x7fc93fcde6fc]
---------
(END) 

Expected results:

quota limit being getting modified or getting added while I/O is happening is a very general scenario, so bricks going down with crash is not acceptable.

Additional info:
Comment 5 Sachidananda Urs 2013-07-18 07:17:55 EDT
If this is fixed in 18th July 2013 build, please move the status to appropriate state.
Comment 7 Scott Haines 2013-09-23 18:39:54 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html
Comment 8 Scott Haines 2013-09-23 18:43:49 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.