Bug 764533 (GLUSTER-2801)

Summary: glusterfsd process crash while running rebalance on simple volume
Product: [Community] GlusterFS Reporter: Amar Tumballi <amarts>
Component: quotaAssignee: Amar Tumballi <amarts>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: mainlineCC: gluster-bugs, jdarcy, rabhat, saurabh, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTNR Mount Type: ---
Documentation: DNR CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Amar Tumballi 2011-04-18 15:00:28 UTC
With patch http://patches.gluster.com/patch/6947/ the crash gets fixed. But this is the log in the server volume file. Any hints would be appreciated. (Anyways I am debugging this rightnow, meantime you are aware of this, can save my time).

Happened during a rebalance.

-----------------
[2011-04-18 23:18:55.296821] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/disable/.usr.bin.firefox.gfs3412: Numerical result out of range
[2011-04-18 23:18:55.296931] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15541: RENAME /etc/apparmor.d/disable/.usr.bin.firefox.gfs3412 (298442) ==> -1 (Cannot allocate memory)
[2011-04-18 23:18:55.322468] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/cache/.usr.sbin.tcpdump.gfs20993: Numerical result out of range
[2011-04-18 23:18:55.322578] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15570: RENAME /etc/apparmor.d/cache/.usr.sbin.tcpdump.gfs20993 (298444) ==> -1 (Cannot allocate memory)
[2011-04-18 23:18:55.346919] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/cache/.usr.bin.evince.gfs635907: Numerical result out of range
[2011-04-18 23:18:55.347029] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15595: RENAME /etc/apparmor.d/cache/.usr.bin.evince.gfs635907 (298446) ==> -1 (Cannot allocate memory)
[2011-04-18 23:18:55.430505] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/abstractions/.winbind.gfs654: Numerical result out of range
[2011-04-18 23:18:55.430616] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15647: RENAME /etc/apparmor.d/abstractions/.winbind.gfs654 (298454) ==> -1 (Cannot allocate memory)

Comment 1 Amar Tumballi 2011-04-18 17:42:04 UTC
This doesn't happen when quota is enabled, but happened when quota is not present in a volume.


Core was generated by `/usr/local/sbin/glusterfsd --xlator-option test1-server.listen-port=24016 -s lo'.
Program terminated with signal 11, Segmentation fault.
#0  marker_quota_removexattr_cbk (frame=0x7fb36dea6234, cookie=0x0, this=0x7fb36dea61b0, op_ret=1860907680, op_errno=19694160)
    at ../../../../../xlators/features/marker/src/marker.c:954
954	        local->oplocal = NULL;
(gdb) bt
#0  marker_quota_removexattr_cbk (frame=0x7fb36dea6234, cookie=0x0, this=0x7fb36dea61b0, op_ret=1860907680, op_errno=19694160)
    at ../../../../../xlators/features/marker/src/marker.c:954
#1  0x00007fb36bc83b62 in iot_removexattr_cbk (frame=0x7fb36dea62b8, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, 
    op_errno=<value optimized out>) at ../../../../../xlators/performance/io-threads/src/io-threads.c:1703
#2  0x00007fb36f50d192 in default_removexattr_cbk (frame=0x7fb36dea633c, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, 
    op_errno=<value optimized out>) at ../../../libglusterfs/src/defaults.c:326
#3  0x00007fb36f50d192 in default_removexattr_cbk (frame=0x7fb36dea63c0, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, 
    op_errno=<value optimized out>) at ../../../libglusterfs/src/defaults.c:326
#4  0x00007fb36c2b7d67 in posix_removexattr (frame=0x7fb36dea6444, this=<value optimized out>, loc=<value optimized out>, name=<value optimized out>)
    at ../../../../../xlators/storage/posix/src/posix.c:3473
#5  0x00007fb36f506abd in default_removexattr (frame=<value optimized out>, this=0x128d7b0, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "")
    at ../../../libglusterfs/src/defaults.c:1040
#6  0x00007fb36f506abd in default_removexattr (frame=<value optimized out>, this=0x128e890, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "")
    at ../../../libglusterfs/src/defaults.c:1040
#7  0x00007fb36bc8628d in iot_removexattr_wrapper (frame=<value optimized out>, this=0x128f970, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "")
    at ../../../../../xlators/performance/io-threads/src/io-threads.c:1712
#8  0x00007fb36f5163eb in call_resume_wind (stub=0x7fb36dbdb024) at ../../../libglusterfs/src/call-stub.c:2297
#9  call_resume (stub=0x7fb36dbdb024) at ../../../libglusterfs/src/call-stub.c:3861
#10 0x00007fb36bc8c6f1 in iot_worker (data=0x12964c0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:129
#11 0x00007fb36eca2971 in start_thread () from /lib/libpthread.so.0
#12 0x00007fb36e9fe92d in clone () from /lib/libc.so.6
#13 0x0000000000000000 in ?? ()

Comment 2 Raghavendra Bhat 2011-04-19 03:48:06 UTC
Any rename operation will make glusterfsd process crash (if quota is not enabled). Tests such as posix compliance tests can trigger this crash. This does not happen on qa14. It happens on git head.(e89f2a1f5daf1513c533a47b90aec8f90702d09f)

Comment 3 Anand Avati 2011-04-19 06:41:42 UTC
PATCH: http://patches.gluster.com/patch/6969 in master (marker: fixes to bypass marker fops in case features are enabled)

Comment 4 Amar Tumballi 2011-04-20 03:15:24 UTC
There is no extra regression test needed. This can happen when a simple rename is done. (which is already present as a part of posix compliance test suite).

internal bug, no documentation required.

Comment 5 Jeff Darcy 2011-04-20 17:38:50 UTC
*** Bug 2823 has been marked as a duplicate of this bug. ***

Comment 6 Saurabh 2011-07-22 09:09:32 UTC
executed posix compliance test, didn't find any crash for glusterd