Bug 1139473 - quota: bricks coredump while creating data inside a subdir and lookup going on in parallel
Summary: quota: bricks coredump while creating data inside a subdir and lookup going o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: quota
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: RHGS 3.0.1
Assignee: Nithya Balachandran
QA Contact: Saurabh
URL:
Whiteboard:
Depends On:
Blocks: 1139774 1140084 1142411
TreeView+ depends on / blocked
 
Reported: 2014-09-09 03:45 UTC by Saurabh
Modified: 2016-09-17 12:35 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.6.0.29-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1140084 (view as bug list)
Environment:
Last Closed: 2014-09-29 07:46:34 UTC
Embargoed:


Attachments (Terms of Use)
coredump (15.88 MB, application/octet-stream)
2014-09-09 03:50 UTC, Saurabh
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1315 0 normal SHIPPED_LIVE Red Hat Storage 3.0 enhancement and bug fix update #1 2014-09-29 11:46:00 UTC

Description Saurabh 2014-09-09 03:45:42 UTC
Description of problem:
For a volume I have set quota on a subdir to 100GB and I was just trying to create data inside that directory.
Meanwhile I started lookup `find . | xargs stat` in a loop from another mount point.

Although this data creation was I have done rename of files in some other directory and triggered rebalance as well. Infact while data creation as mentioned above was going rename in another directory and rebalance had finished. So, altogether after sometime only data-creation and lookup was going on in parallel and that's also these two operations from different mount-points.

Version-Release number of selected component (if applicable):
glusterfs-3.6.0.28-1.el6rhs.x86_64

How reproducible:
happen to be seen with present exercise

Steps to Reproduce:
1. create a volume of type 6x2, start it
2. enable quota 
3. mount volume over nfs on clients c1, c2
4. create a dir say "qa1dir/dir1" and provide non-root user ownership to qa1dir and dir1 both.
5. set limit of 100GB on qa1dir/dir1
6. create someother directory in mountpoint and create some files in this directory. 
7. start creating data inside qa1dir/dir1 till quota limit is reached.
8. start renaming renaming of files in the directory created in step6.
9. start rebalance.
10. start "find . | xargs stat" in a for loop from client c2.


Actual results:
there are several bricks that hace coredump. 

bricks log,
pending frames:
frame : type(0) op(8)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2014-09-08 01:32:12
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.6.0.28
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x7fd76c2ddf06]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x33f)[0x7fd76c2f859f]
/lib64/libc.so.6[0x3c6be326b0]
/usr/lib64/libglusterfs.so.0(uuid_unpack+0x0)[0x7fd76c319190]
/usr/lib64/libglusterfs.so.0(+0x5ae66)[0x7fd76c318e66]
/usr/lib64/libglusterfs.so.0(uuid_utoa+0x37)[0x7fd76c2f5cb7]
/usr/lib64/glusterfs/3.6.0.28/xlator/features/quota.so(quota_rename_cbk+0x28e)[0x7fd760bb36be]
/usr/lib64/libglusterfs.so.0(default_rename_cbk+0xfd)[0x7fd76c2ec1fd]
/usr/lib64/libglusterfs.so.0(+0x4007a)[0x7fd76c2fe07a]
/usr/lib64/libglusterfs.so.0(call_resume+0xa0)[0x7fd76c2ff830]
/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_rename_done+0x7a)[0x7fd760dd1e4a]
/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_rename_release_newp_lock+0x2b4)[0x7fd760dd2414]
/usr/lib64/libglusterfs.so.0(default_inodelk_cbk+0xb9)[0x7fd76c2ea769]
/usr/lib64/glusterfs/3.6.0.28/xlator/features/locks.so(pl_common_inodelk+0x29f)[0x7fd761613caf]
/usr/lib64/glusterfs/3.6.0.28/xlator/features/locks.so(pl_inodelk+0x27)[0x7fd761614537]
/usr/lib64/libglusterfs.so.0(default_inodelk_resume+0x14d)[0x7fd76c2e640d]
/usr/lib64/libglusterfs.so.0(call_resume+0x2aa)[0x7fd76c2ffa3a]
/usr/lib64/glusterfs/3.6.0.28/xlator/performance/io-threads.so(iot_worker+0x158)[0x7fd7613fe348]
/lib64/libpthread.so.0[0x3c6c6079d1]
/lib64/libc.so.6(clone+0x6d)[0x3c6bee886d]
---------



bt of the coredump.

#0  uuid_unpack (in=0x8 <Address 0x8 out of bounds>, uu=0x7fd6fb6f54c0) at ../../contrib/uuid/unpack.c:44
#1  0x00007fd76c318e66 in uuid_unparse_x (uu=<value optimized out>, out=0x7fd7181529e0 "1a71b01e-22bc-4b60-a0fc-3beac78cb403", fmt=0x7fd76c340d20 "%08x-%04x-%04x-%02x%02x-%02x%02x%02x%02x%02x%02x")
    at ../../contrib/uuid/unparse.c:55
#2  0x00007fd76c2f5cb7 in uuid_utoa (uuid=0x8 <Address 0x8 out of bounds>) at common-utils.c:2138
#3  0x00007fd760bb36be in quota_rename_cbk (frame=0x7fd76b14cd64, cookie=<value optimized out>, this=0x26563c0, op_ret=0, op_errno=61, buf=0x7fd76abb8518, preoldparent=0x7fd76abb8668, 
    postoldparent=0x7fd76abb86d8, prenewparent=0x7fd76abb8748, postnewparent=0x7fd76abb87b8, xdata=0x0) at quota.c:1931
#4  0x00007fd76c2ec1fd in default_rename_cbk (frame=0x7fd76b12de38, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=61, buf=<value optimized out>, preoldparent=0x7fd76abb8668, 
    postoldparent=0x7fd76abb86d8, prenewparent=0x7fd76abb8748, postnewparent=0x7fd76abb87b8, xdata=0x0) at defaults.c:961
#5  0x00007fd76c2fe07a in call_resume_unwind (stub=0x7fd76abb7f18) at call-stub.c:2604
#6  0x00007fd76c2ff830 in call_resume (stub=0x7fd76abb7f18) at call-stub.c:2843
#7  0x00007fd760dd1e4a in marker_rename_done (frame=0x7fd76b12de38, cookie=<value optimized out>, this=0x2654e50, op_ret=<value optimized out>, op_errno=<value optimized out>, xdata=<value optimized out>)
    at marker.c:1035
#8  0x00007fd760dd2414 in marker_rename_release_newp_lock (frame=0x7fd76b12de38, cookie=<value optimized out>, this=0x2654e50, op_ret=<value optimized out>, op_errno=<value optimized out>, 
    xdata=<value optimized out>) at marker.c:1101
#9  0x00007fd76c2ea769 in default_inodelk_cbk (frame=0x7fd76b16215c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, xdata=<value optimized out>) at defaults.c:1175
#10 0x00007fd761613caf in pl_common_inodelk (frame=0x7fd76b12abd4, this=<value optimized out>, volume=<value optimized out>, inode=<value optimized out>, cmd=7, flock=<value optimized out>, 
    loc=0x7fd76abd0b7c, fd=0x0, xdata=0x0) at inodelk.c:792
#11 0x00007fd761614537 in pl_inodelk (frame=<value optimized out>, this=<value optimized out>, volume=<value optimized out>, loc=<value optimized out>, cmd=<value optimized out>, 
    flock=<value optimized out>, xdata=0x0) at inodelk.c:804
#12 0x00007fd76c2e640d in default_inodelk_resume (frame=0x7fd76b16215c, this=0x2651390, volume=0x7fd71801c530 "dist-rep-marker", loc=0x7fd76abd0b7c, cmd=7, lock=0x7fd76abd0c7c, xdata=0x0) at defaults.c:1577
#13 0x00007fd76c2ffa3a in call_resume_wind (stub=0x7fd76abd0b3c) at call-stub.c:2460
#14 call_resume (stub=0x7fd76abd0b3c) at call-stub.c:2841
#15 0x00007fd7613fe348 in iot_worker (data=0x26830c0) at io-threads.c:214
#16 0x0000003c6c6079d1 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003c6bee886d in clone () from /lib64/libc.so.6
(gdb) quit


Expected results:
No core dump exepected. 

Additional info:

Comment 1 Saurabh 2014-09-09 03:50:03 UTC
Created attachment 935536 [details]
coredump

Comment 7 Susant Kumar Palai 2014-09-20 06:33:24 UTC
Tried the steps in reproducer and could not hit the bug. Ran twice.

1. create a volume of type 6x2, start it
2. enable quota 
3. mount volume over nfs on clients c1, c2
4. create a dir say "qa1dir/dir1" and provide non-root user ownership to qa1dir and dir1 both.
5. set limit of 30GB on qa1dir/dir1
6. create someother directory in mountpoint and create some files in this directory. 
7. start creating data inside qa1dir/dir1 till quota limit is reached.
8. start renaming of files in the directory created in step6.
9. start rebalance.
10. start "find . | xargs stat" in a for loop from client c2.


Regards,
Susant/Venkatesh

Comment 8 Saurabh 2014-09-23 09:11:08 UTC
As per the https://bugzilla.redhat.com/show_bug.cgi?id=1139473#c7, the verification post-fix is done by Susant/Venkatesh.

Comment 10 errata-xmlrpc 2014-09-29 07:46:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1315.html


Note You need to log in before you can comment on or make changes to this bug.