| Summary: | glusterfs server crashed due to corrupted call_frame | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra Bhat <rabhat> |
| Component: | core | Assignee: | Amar Tumballi <amarts> |
| Status: | CLOSED WORKSFORME | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.2.5 | CC: | gluster-bugs, vraman |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-06-07 11:15:06 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
anything you notice in logs which is suspicious? 1020 (LOOKUP) [2012-01-10 00:12:39.137479] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.137502] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.137514] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.137526] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.138923] I [server-resolve.c:571:server_resolve] 0-mirror-server: pure path resolution for /167/run11547/test/file 81021 (LOOKUP) [2012-01-10 00:12:39.138955] I [server-resolve.c:571:server_resolve] 0-mirror-server: pure path resolution for /167/run11547/test/file 81022 (LOOKUP) [2012-01-10 00:12:39.138980] I [server-resolve.c:571:server_resolve] 0-mirror-server: pure path resolution for /167/run11547/test/file 81023 (LOOKUP) [2012-01-10 00:12:39.142936] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.142960] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.143259] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.143283] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.143296] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) [2012-01-10 00:12:39.144205] W [marker-quota.c:1805:mq_markdirty] 0-mirror-marker: lock setting failed on / (Resource temporarily unav ailable) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2012-01-10 00:12:39 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 want to have a test run with 3.3.0qa24 (comes out today and see the behavior) Johny, Can you confirm this is the behavior on 3.3.x branch (ie, master branch) too ? If it is not happening on master branch, I want to remove this off '3.3.0beta' milestone. Not seen in 3.3.0 in any testing, but till date I could not run longetivity tests on 3.3.0 to confirmatively say bug exists/does not exist. as per comment #4 not seen with glusterfs-3.3.0 |
Description of problem: Distribted replicate 2x2 setup. Quota enabled volume. Was running longetivity tests. glusterfs server crashed by trying to access the corrputed call_frame. This is the backtrace of the core generated. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/usr/local/sbin/glusterfsd --xlator-option mirror-server.listen-port=24009 -s l'. Program terminated with signal 11, Segmentation fault. #0 0x00007f946d313d3c in default_inodelk_cbk (frame=0x7f94588b37c0, cookie=0x7f94588b5000, this=0x7f94587599d8, op_ret=0, op_errno=0) at ../../../libglusterfs/src/defaults.c:342 342 STACK_UNWIND_STRICT (inodelk, frame, op_ret, op_errno); Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc-4.4.6-3.el6.x86_64 (gdb) bt #0 0x00007f946d313d3c in default_inodelk_cbk (frame=0x7f94588b37c0, cookie=0x7f94588b5000, this=0x7f94587599d8, op_ret=0, op_errno=0) at ../../../libglusterfs/src/defaults.c:342 #1 0x00007f9469eecd87 in grant_blocked_inode_locks (this=0xc0c7b0, pl_inode=0x7f945c0011c0, dom=0x7f945c001290) at ../../../../../xlators/features/locks/src/inodelk.c:361 #2 0x00007f9469eed61e in pl_inode_setlk (this=0xc0c7b0, pl_inode=0x7f945c0011c0, lock=0x7f945484c530, can_block=1, dom=0x7f945c001290) at ../../../../../xlators/features/locks/src/inodelk.c:507 #3 0x00007f9469eedb75 in pl_common_inodelk (frame=0x7f946c4100f0, this=0xc0c7b0, volume=0xc0d8b0 "mirror-marker", inode=0x7f9468880044, cmd=7, flock=0x7f944aae9a30, loc=0x7f9454821b10, fd=0x0) at ../../../../../xlators/features/locks/src/inodelk.c:619 #4 0x00007f9469eedea3 in pl_inodelk (frame=0x7f946c4100f0, this=0xc0c7b0, volume=0xc0d8b0 "mirror-marker", loc=0x7f9454821b10, cmd=7, flock=0x7f944aae9a30) at ../../../../../xlators/features/locks/src/inodelk.c:662 #5 0x00007f946d31e012 in default_inodelk (frame=0x7f946c5b043c, this=0xc0d940, volume=0xc0d8b0 "mirror-marker", loc=0x7f9454821b10, cmd=7, lock=0x7f944aae9a30) at ../../../libglusterfs/src/defaults.c:1060 #6 0x00007f9469aba52c in mq_release_parent_lock (frame=0x7f9454850078, cookie=0x7f946c3f3d4c, this=0xc0eb60, op_ret=0, op_errno=0) at ../../../../../xlators/features/marker/src/marker-quota.c:1386 #7 0x00007f9469cd3820 in iot_setxattr_cbk (frame=0x7f946c3f3d4c, cookie=0x7f946c56bad8, this=0xc0d940, op_ret=0, op_errno=0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:1511 #8 0x00007f946d31335f in default_setxattr_cbk (frame=0x7f946c56bad8, cookie=0x7f946c570c34, this=0xc0c7b0, op_ret=0, op_errno=0) at ../../../libglusterfs/src/defaults.c:273 #9 0x00007f946a0fffa9 in posix_acl_setxattr_cbk (frame=0x7f946c570c34, cookie=0x7f946c4c3f44, this=0xc0b550, op_ret=0, op_errno=0) at ../../../../../xlators/system/posix-acl/src/posix-acl.c:1665 #10 0x00007f946a31811c in posix_setxattr (frame=0x7f946c4c3f44, this=0xc0a120, loc=0x7f946c0cb9a0, dict=0x7f945480e2a0, flags=0) at ../../../../../xlators/storage/posix/src/posix.c:3273 #11 0x00007f946a100227 in posix_acl_setxattr (frame=0x7f946c570c34, this=0xc0b550, loc=0x7f946c0cb9a0, xattr=0x7f945480e2a0, flags=0) at ../../../../../xlators/system/posix-acl/src/posix-acl.c:1684 #12 0x00007f946d31a773 in default_setxattr (frame=0x7f946c56bad8, this=0xc0c7b0, loc=0x7f946c0cb9a0, dict=0x7f945480e2a0, flags=0) at ../../../libglusterfs/src/defaults.c:828 #13 0x00007f9469cd3a54 in iot_setxattr_wrapper (frame=0x7f946c3f3d4c, this=0xc0d940, loc=0x7f946c0cb9a0, dict=0x7f945480e2a0, flags=0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:1520 #14 0x00007f946d331e1e in call_resume_wind (stub=0x7f946c0cb968) at ../../../libglusterfs/src/call-stub.c:2259 #15 0x00007f946d3389da in call_resume (stub=0x7f946c0cb968) at ../../../libglusterfs/src/call-stub.c:3859 #16 0x00007f9469cca213 in iot_worker (data=0xc18010) at ../../../../../xlators/performance/io-threads/src/io-threads.c:129 #17 0x00000033ace077e1 in start_thread () from /lib64/libpthread.so.0 #18 0x00000033acae68ed in clone () from /lib64/libc.so.6 (gdb) f 0 #0 0x00007f946d313d3c in default_inodelk_cbk (frame=0x7f94588b37c0, cookie=0x7f94588b5000, this=0x7f94587599d8, op_ret=0, op_errno=0) at ../../../libglusterfs/src/defaults.c:342 342 STACK_UNWIND_STRICT (inodelk, frame, op_ret, op_errno); (gdb) p frame $1 = (call_frame_t *) 0x7f94588b37c0 (gdb) p frame->root $2 = (call_stack_t *) 0x0 (gdb) p frame->parent $3 = (call_frame_t *) 0xa5 (gdb) info thr 13 Thread 0x7f944aded700 (LWP 22994) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 12 Thread 0x7f944aeee700 (LWP 32712) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 11 Thread 0x7f944a9e9700 (LWP 17167) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 10 Thread 0x7f944afef700 (LWP 32463) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 9 Thread 0x7f94692a8700 (LWP 3295) 0x00000033ace0b7cb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 8 Thread 0x7f946af24700 (LWP 3290) 0x00000033acaaba6d in nanosleep () from /lib64/libc.so.6 7 Thread 0x7f946c0af700 (LWP 3288) 0x00000033ace0f2c5 in sigwait () from /lib64/libpthread.so.0 6 Thread 0x7f944abeb700 (LWP 18063) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 5 Thread 0x7f946ceaf700 (LWP 3287) 0x00000033acae6ee3 in epoll_wait () from /lib64/libc.so.6 4 Thread 0x7f944acec700 (LWP 18057) 0x00000033ace0e074 in __lll_lock_wait () from /lib64/libpthread.so.0 3 Thread 0x7f944b0f0700 (LWP 32320) 0x00000033ace0e205 in __lll_unlock_wake () from /lib64/libpthread.so.0 2 Thread 0x7f944b7f7700 (LWP 14347) list_add_tail (new=0x7f9454758a30, head=0x7f945c00ee28) at ../../../../../libglusterfs/src/list.h:52 * 1 Thread 0x7f944aaea700 (LWP 18076) 0x00007f946d313d3c in default_inodelk_cbk (frame=0x7f94588b37c0, cookie=0x7f94588b5000, this=0x7f94587599d8, op_ret=0, op_errno=0) at ../../../libglusterfs/src/defaults.c:342 (gdb) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: