Hide Forgot
With patch http://patches.gluster.com/patch/6947/ the crash gets fixed. But this is the log in the server volume file. Any hints would be appreciated. (Anyways I am debugging this rightnow, meantime you are aware of this, can save my time). Happened during a rebalance. ----------------- [2011-04-18 23:18:55.296821] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/disable/.usr.bin.firefox.gfs3412: Numerical result out of range [2011-04-18 23:18:55.296931] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15541: RENAME /etc/apparmor.d/disable/.usr.bin.firefox.gfs3412 (298442) ==> -1 (Cannot allocate memory) [2011-04-18 23:18:55.322468] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/cache/.usr.sbin.tcpdump.gfs20993: Numerical result out of range [2011-04-18 23:18:55.322578] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15570: RENAME /etc/apparmor.d/cache/.usr.sbin.tcpdump.gfs20993 (298444) ==> -1 (Cannot allocate memory) [2011-04-18 23:18:55.346919] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/cache/.usr.bin.evince.gfs635907: Numerical result out of range [2011-04-18 23:18:55.347029] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15595: RENAME /etc/apparmor.d/cache/.usr.bin.evince.gfs635907 (298446) ==> -1 (Cannot allocate memory) [2011-04-18 23:18:55.430505] E [posix.c:3464:posix_removexattr] 0-test1-posix: removexattr on /etc/apparmor.d/abstractions/.winbind.gfs654: Numerical result out of range [2011-04-18 23:18:55.430616] I [server3_1-fops.c:918:server_rename_cbk] 0-test1-server: 15647: RENAME /etc/apparmor.d/abstractions/.winbind.gfs654 (298454) ==> -1 (Cannot allocate memory)
This doesn't happen when quota is enabled, but happened when quota is not present in a volume. Core was generated by `/usr/local/sbin/glusterfsd --xlator-option test1-server.listen-port=24016 -s lo'. Program terminated with signal 11, Segmentation fault. #0 marker_quota_removexattr_cbk (frame=0x7fb36dea6234, cookie=0x0, this=0x7fb36dea61b0, op_ret=1860907680, op_errno=19694160) at ../../../../../xlators/features/marker/src/marker.c:954 954 local->oplocal = NULL; (gdb) bt #0 marker_quota_removexattr_cbk (frame=0x7fb36dea6234, cookie=0x0, this=0x7fb36dea61b0, op_ret=1860907680, op_errno=19694160) at ../../../../../xlators/features/marker/src/marker.c:954 #1 0x00007fb36bc83b62 in iot_removexattr_cbk (frame=0x7fb36dea62b8, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=<value optimized out>) at ../../../../../xlators/performance/io-threads/src/io-threads.c:1703 #2 0x00007fb36f50d192 in default_removexattr_cbk (frame=0x7fb36dea633c, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=<value optimized out>) at ../../../libglusterfs/src/defaults.c:326 #3 0x00007fb36f50d192 in default_removexattr_cbk (frame=0x7fb36dea63c0, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=<value optimized out>) at ../../../libglusterfs/src/defaults.c:326 #4 0x00007fb36c2b7d67 in posix_removexattr (frame=0x7fb36dea6444, this=<value optimized out>, loc=<value optimized out>, name=<value optimized out>) at ../../../../../xlators/storage/posix/src/posix.c:3473 #5 0x00007fb36f506abd in default_removexattr (frame=<value optimized out>, this=0x128d7b0, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "") at ../../../libglusterfs/src/defaults.c:1040 #6 0x00007fb36f506abd in default_removexattr (frame=<value optimized out>, this=0x128e890, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "") at ../../../libglusterfs/src/defaults.c:1040 #7 0x00007fb36bc8628d in iot_removexattr_wrapper (frame=<value optimized out>, this=0x128f970, loc=0x7fb36dbdb05c, name=0x7fb364000d90 "") at ../../../../../xlators/performance/io-threads/src/io-threads.c:1712 #8 0x00007fb36f5163eb in call_resume_wind (stub=0x7fb36dbdb024) at ../../../libglusterfs/src/call-stub.c:2297 #9 call_resume (stub=0x7fb36dbdb024) at ../../../libglusterfs/src/call-stub.c:3861 #10 0x00007fb36bc8c6f1 in iot_worker (data=0x12964c0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:129 #11 0x00007fb36eca2971 in start_thread () from /lib/libpthread.so.0 #12 0x00007fb36e9fe92d in clone () from /lib/libc.so.6 #13 0x0000000000000000 in ?? ()
Any rename operation will make glusterfsd process crash (if quota is not enabled). Tests such as posix compliance tests can trigger this crash. This does not happen on qa14. It happens on git head.(e89f2a1f5daf1513c533a47b90aec8f90702d09f)
PATCH: http://patches.gluster.com/patch/6969 in master (marker: fixes to bypass marker fops in case features are enabled)
There is no extra regression test needed. This can happen when a simple rename is done. (which is already present as a part of posix compliance test suite). internal bug, no documentation required.
*** Bug 2823 has been marked as a duplicate of this bug. ***
executed posix compliance test, didn't find any crash for glusterd