| Summary: | glusterfsd core found with 3.2.2 in marker | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Lakshmipathi G <lakshmipathi> |
| Component: | quota | Assignee: | Raghavendra Bhat <rabhat> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.2.2 | CC: | gluster-bugs, rabhat |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | 3.2.3 | Category: | --- |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
created a 2dht volume with 3.2.2(ssa) and enabled quota and gsyncd,testing for mem.leak. Volume Name: test2 Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 192.168.1.87:/root/t3 Brick2: 192.168.1.87:/root/t4 Options Reconfigured: features.quota: on geo-replication.indexing: on did kernel untar in a loop.after sometime both glusterfsd are crashed. Core can be found at root.1.87:/core glusterfsd log -------- [2011-08-10 09:48:48.668456] E [posix.c:3127:handle_pair] 0-test2-posix: setxattr on /root/t3/linux-2.6.27.8/arch/powerpc/kernel/ftrace.c failed: No such file or direct ory [2011-08-10 09:48:48.669454] I [marker-quota.c:939:quota_inodelk_cbk] 0-test2-marker: lock setting failed (Success) [2011-08-10 09:48:48.669624] E [posix.c:3127:handle_pair] 0-test2-posix: setxattr on /root/t3/linux-2.6.27.8/arch/powerpc/kernel/vdso32/datapage.S failed: No such file or directory [2011-08-10 09:48:48.696852] E [posix.c:3127:handle_pair] 0-test2-posix: setxattr on /root/t3/linux-2.6.27.8/arch/powerpc/kernel/vdso32/.gitignore failed: No such file or directory pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2011-08-10 09:48:48 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.2.2 /lib64/libc.so.6[0x2b8e2b5d92d0] /opt/glusterfs/3.2.2/lib64/glusterfs/3.2.2/xlator/features/marker.so(quota_forget+0x20)[0x2aaaac024dd0] /opt/glusterfs/3.2.2/lib64/glusterfs/3.2.2/xlator/features/marker.so(marker_forget+0x5b)[0x2aaaac01d14b] /opt/glusterfs/3.2.2/lib64/libglusterfs.so.0[0x2b8e2ab2ac41] /opt/glusterfs/3.2.2/lib64/libglusterfs.so.0(inode_unref+0x3c)[0x2b8e2ab2af4c] /opt/glusterfs/3.2.2/lib64/libglusterfs.so.0(loc_wipe+0x12)[0x2b8e2ab162e2] /opt/glusterfs/3.2.2/lib64/libglusterfs.so.0(call_stub_destroy+0x5ce)[0x2b8e2ab2ccbe] /opt/glusterfs/3.2.2/lib64/glusterfs/3.2.2/xlator/performance/io-threads.so(iot_worker+0x119)[0x2aaaabe14d09] /lib64/libpthread.so.0[0x2b8e2b39373d] /lib64/libc.so.6(clone+0x6d)[0x2b8e2b67d4bd] --------- bt --------- warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffe7dfc000 Core was generated by `/opt/glusterfs/3.2.2/sbin/glusterfsd --xlator-option test2-server.listen-port=2'. Program terminated with signal 11, Segmentation fault. #0 0x00002aaaac024dd0 in quota_forget (this=<value optimized out>, ctx=0x2aaab80399f0) at marker-quota.c:1962 1962 list_for_each_entry_safe (contri, next, &ctx->contribution_head, (gdb) bt #0 0x00002aaaac024dd0 in quota_forget (this=<value optimized out>, ctx=0x2aaab80399f0) at marker-quota.c:1962 #1 0x00002aaaac01d14b in marker_forget (this=0x1e5a9420, inode=<value optimized out>) at marker.c:2034 #2 0x00002b8e2ab2ac41 in __inode_destroy (table=<value optimized out>) at inode.c:330 #3 inode_table_prune (table=<value optimized out>) at inode.c:1127 #4 0x00002b8e2ab2af4c in inode_unref (inode=0x1e60ca80) at inode.c:457 #5 0x00002b8e2ab162e2 in loc_wipe (loc=0x2aaaaafe2d00) at xlator.c:1641 #6 0x00002b8e2ab2ccbe in call_stub_destroy_wind (stub=0x2aaaaafe2cc8) at call-stub.c:3569 #7 call_stub_destroy (stub=0x2aaaaafe2cc8) at call-stub.c:3834 #8 0x00002aaaabe14d09 in iot_worker (data=0x1e5af030) at io-threads.c:129 #9 0x00002b8e2b39373d in start_thread () from /lib64/libpthread.so.0 #10 0x00002b8e2b67d4bd in clone () from /lib64/libc.so.6 CHANGE: http://review.gluster.com/241 (Change-Id: I71bc8a477155cff62c3218fee285a22a44aa54ba) merged in release-3.2 by Anand Avati (avati) Now untarring linux kernel does not crash glusterfs since the corruption which used to happen because of improper refing and unrefing of quota local structure is fixed now. |
The backtrace of the other glusterfsd process is below. Core was generated by `/opt/glusterfs/3.2.2/sbin/glusterfsd --xlator-option test2-server.listen-port=2'. Program terminated with signal 11, Segmentation fault. #0 0x00002ac03aec1cd1 in _int_malloc () from /lib64/libc.so.6 (gdb) bt #0 0x00002ac03aec1cd1 in _int_malloc () from /lib64/libc.so.6 #1 0x00002ac03aec3aad in calloc () from /lib64/libc.so.6 #2 0x00002ac03a3e2bf1 in __gf_default_calloc (nmemb=1, size=1136, type=1256855072) at mem-pool.h:83 #3 __gf_calloc (nmemb=1, size=1136, type=1256855072) at mem-pool.c:135 #4 0x00002aaaac4535b6 in server_alloc_frame (req=0x2aaaac87c494) at server-helpers.c:879 #5 get_frame_from_request (req=0x2aaaac87c494) at server-helpers.c:909 #6 0x00002aaaac468dae in server_stat (req=0x2aaaac87c494) at server3_1-fops.c:2693 #7 0x00002ac03a60c1b1 in rpcsvc_handle_rpc_call (svc=0x1c7882b0, trans=<value optimized out>, msg=0x2aaab0055ab0) at rpcsvc.c:480 #8 0x00002ac03a60c3bc in rpcsvc_notify (trans=0x1c77b260, mydata=0x2aaab0000020, event=<value optimized out>, data=0x2aaab0055ab0) at rpcsvc.c:576 #9 0x00002ac03a60d2e7 in rpc_transport_notify (this=0x2aaab0000020, event=1256855072, data=0x0) at rpc-transport.c:918 #10 0x00002aaaab0fb5ef in socket_event_poll_in (this=0x1c77b260) at socket.c:1647 #11 0x00002aaaab0fb798 in socket_event_handler (fd=<value optimized out>, idx=4, data=0x1c77b260, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762 #12 0x00002ac03a3e1601 in event_dispatch_epoll_handler (event_pool=0x1c7769b0) at event.c:794 #13 event_dispatch_epoll (event_pool=0x1c7769b0) at event.c:856 #14 0x0000000000405617 in main (argc=17, argv=0x7fff4aea1ff8) at glusterfsd.c:1503 (gdb) t 4 [Switching to thread 4 (Thread 14299)]#0 0x00002ac03ac3e732 in ?? () from /lib64/libpthread.so.0 (gdb) bt #0 0x00002ac03ac3e732 in ?? () from /lib64/libpthread.so.0 #1 0x00002ac03a3e2247 in mem_put (pool=0x1c776c50, ptr=0x2aaaaafac158) at mem-pool.c:472 #2 0x00002aaaabe14d09 in iot_worker (data=0x1c78a030) at io-threads.c:129 #3 0x00002ac03ac3973d in start_thread () from /lib64/libpthread.so.0 #4 0x00002ac03af234bd in clone () from /lib64/libc.so.6 (gdb) t 5 [Switching to thread 5 (Thread 12306)]#0 0x00002ac03ac3e735 in ?? () from /lib64/libpthread.so.0 (gdb) bt #0 0x00002ac03ac3e735 in ?? () from /lib64/libpthread.so.0 #1 0x00002ac03a3e23c7 in mem_get (mem_pool=0x1c776c50) at mem-pool.c:388 #2 0x00002ac03a3e24fb in mem_get0 (mem_pool=0x1c776c50) at mem-pool.c:368 #3 0x00002ac03a3d5b6d in stub_new (frame=0x2aaaaaafecf8, wind=1 '\001', fop=GF_FOP_SETXATTR) at call-stub.c:41 #4 0x00002ac03a3db506 in fop_setxattr_stub (frame=0x1c776c68, fn=0x2aaaabe0e980 <iot_setxattr_wrapper>, loc=0x2aaab8040238, dict=0x2aaab805f650, flags=0) at call-stub.c:1242 #5 0x00002aaaabe118f4 in iot_setxattr (frame=0x1c776c68, this=0x1c783360, loc=0x11, dict=0x2aaab805f650, flags=0) at io-threads.c:1533 #6 0x00002aaaac029708 in quota_markdirty (frame=<value optimized out>, cookie=<value optimized out>, this=0x1c784420, op_ret=<value optimized out>, op_errno=<value optimized out>) at marker-quota.c:1370 #7 0x00002ac03a3c9e21 in default_inodelk_cbk (frame=0x2aaaaab4d164, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0) at defaults.c:342 #8 0x00002aaaabc00cd8 in grant_blocked_inode_locks (this=0x1c782320, pl_inode=<value optimized out>, dom=<value optimized out>) at inodelk.c:361 #9 0x00002aaaabc01853 in pl_inode_setlk (frame=0x2aaaaaabd81c, this=0x1c782320, volume=0x1c783d50 "test2-marker", inode=0x2aaaac92315c, cmd=7, flock=0x43603e20, loc=0x2aaab805efb8, fd=0x0) at inodelk.c:502 #10 pl_common_inodelk (frame=0x2aaaaaabd81c, this=0x1c782320, volume=0x1c783d50 "test2-marker", inode=0x2aaaac92315c, cmd=7, flock=0x43603e20, loc=0x2aaab805efb8, fd=0x0) at inodelk.c:614 #11 0x00002aaaabc01c4d in pl_inodelk (frame=0x1c776c68, this=0x1, volume=0x11 <Address 0x11 out of bounds>, loc=<value optimized out>, cmd=0, flock=0x101010101010101) at inodelk.c:657 #12 0x00002ac03a3c2cc5 in default_inodelk (frame=<value optimized out>, this=0x1c783360, volume=0x1c783d50 "test2-marker", loc=0x2aaab805efb8, cmd=7, lock=0x43603e20) at defaults.c:1060 #13 0x00002aaaac02523b in quota_release_parent_lock (frame=<value optimized out>, cookie=<value optimized out>, this=0x1c784420, op_ret=<value optimized out>, op_errno=<value optimized out>) at marker-quota.c:994 #14 0x00002aaaabe0bec1 in iot_setxattr_cbk (frame=0x2aaaaab02a78, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0) at io-threads.c:1511 #15 0x00002ac03a3ca671 in default_setxattr_cbk (frame=0x2aaaaab070f0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0) at defaults.c:273 #16 0x00002aaaab9eac71 in posix_acl_setxattr_cbk () from /opt/glusterfs/3.2.2/lib64/glusterfs/3.2.2/xlator/features/access-control.so #17 0x00002aaaab7d5d55 in posix_setxattr (frame=0x2aaaaaab2a80, this=<value optimized out>, loc=<value optimized out>, dict=<value optimized out>, flags=<value optimized out>) at posix.c:3191 #18 0x00002aaaab9ed3f4 in posix_acl_setxattr () from /opt/glusterfs/3.2.2/lib64/glusterfs/3.2.2/xlator/features/access-control.so #19 0x00002ac03a3c53ed in default_setxattr (frame=<value optimized out>, this=0x1c782320, loc=0x2aaaaafa6c08, dict=0x2aaab8072e00, flags=0) at defaults.c:828 #20 0x00002aaaabe0ea8d in iot_setxattr_wrapper (frame=<value optimized out>, this=0x1c783360, loc=0x2aaaaafa6c08, dict=0x2aaab8072e00, flags=0) at io-threads.c:1520 #21 0x00002ac03a3d35bc in call_resume_wind (stub=0x2aaaaafa6bd0) at call-stub.c:2259 #22 call_resume (stub=0x2aaaaafa6bd0) at call-stub.c:3861 #23 0x00002aaaabe14d09 in iot_worker (data=0x1c78a030) at io-threads.c:129 #24 0x00002ac03ac3973d in start_thread () from /lib64/libpthread.so.0 #25 0x00002ac03af234bd in clone () from /lib64/libc.so.6 (gdb) t 6 [Switching to thread 6 (Thread 12184)]#0 0x00002ac03ac3e732 in ?? () from /lib64/libpthread.so.0 (gdb) bt #0 0x00002ac03ac3e732 in ?? () from /lib64/libpthread.so.0 #1 0x00002ac03a3e23c7 in mem_get (mem_pool=0x1c776c50) at mem-pool.c:388 #2 0x00002ac03a3e24fb in mem_get0 (mem_pool=0x1c776c50) at mem-pool.c:368 #3 0x00002ac03a3d5b6d in stub_new (frame=0x2aaaaaabd964, wind=1 '\001', fop=GF_FOP_SETXATTR) at call-stub.c:41 #4 0x00002ac03a3db506 in fop_setxattr_stub (frame=0x1c776c68, fn=0x2aaaabe0e980 <iot_setxattr_wrapper>, loc=0x1c7bbbd8, dict=0x1c7d3250, flags=0) at call-stub.c:1242 #5 0x00002aaaabe118f4 in iot_setxattr (frame=0x1c776c68, this=0x1c783360, loc=0x11, dict=0x1c7d3250, flags=0) at io-threads.c:1533 #6 0x00002aaaac026b6a in quota_mark_undirty (frame=0x2aaaaad3e128, cookie=<value optimized out>, this=0x1c784420, op_ret=<value optimized out>, op_errno=<value optimized out>, dict=<value optimized out>) at marker-quota.c:1062 #7 0x00002aaaabe0b77b in iot_xattrop_cbk (frame=0x2aaaaaaf752c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=22, xattr=0x2aaab8049700) at io-threads.c:1846 #8 0x00002ac03a3ca24b in default_xattrop_cbk (frame=0x2aaaaab029d4, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=22, dict=0x2aaab8049700) at defaults.c:309 #9 0x00002ac03a3ca24b in default_xattrop_cbk (frame=0x2aaaaab79b0c, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=22, dict=0x2aaab8049700) at defaults.c:309 #10 0x00002aaaab7ddb4d in do_xattrop (frame=0x2aaaaab15ac4, this=0x1c77bef0, loc=0x2aaaaafb1c20, fd=<value optimized out>, optype=GF_XATTROP_ADD_ARRAY64, xattr=0x2aaab8049700) at posix.c:3949 #11 0x00002aaaab7de461 in posix_xattrop (frame=0x1c776c68, this=0x1, loc=0x11, optype=GF_XATTROP_ADD_ARRAY, xattr=0x101010101010101) at posix.c:3958 #12 0x00002ac03a3c32ed in default_xattrop (frame=<value optimized out>, this=0x1c7813a0, loc=0x2aaaaafb1c20, flags=GF_XATTROP_ADD_ARRAY64, dict=0x2aaab8049700) at defaults.c:1022 #13 0x00002ac03a3c32ed in default_xattrop (frame=<value optimized out>, this=0x1c782320, loc=0x2aaaaafb1c20, flags=GF_XATTROP_ADD_ARRAY64, dict=0x2aaab8049700) at defaults.c:1022 #14 0x00002aaaabe0dfcd in iot_xattrop_wrapper (frame=<value optimized out>, this=0x1c783360, loc=0x2aaaaafb1c20, optype=GF_XATTROP_ADD_ARRAY64, xattr=0x2aaab8049700) at io-threads.c:1855 #15 0x00002ac03a3d37a9 in call_resume_wind (stub=0x2aaaaafb1be8) at call-stub.c:2447 #16 call_resume (stub=0x2aaaaafb1be8) at call-stub.c:3861 #17 0x00002aaaabe14d09 in iot_worker (data=0x1c78a030) at io-threads.c:129 #18 0x00002ac03ac3973d in start_thread () from /lib64/libpthread.so.0 #19 0x00002ac03af234bd in clone () from /lib64/libc.so.6 (gdb)