Bug 1654161
Summary: | glusterd crashed with seg fault possibly during node reboot while volume creates and deletes were happening | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> | |
Component: | glusterd | Assignee: | Atin Mukherjee <amukherj> | |
Status: | CLOSED ERRATA | QA Contact: | Nag Pavan Chilakam <nchilaka> | |
Severity: | high | Docs Contact: | ||
Priority: | medium | |||
Version: | rhgs-3.4 | CC: | kiyer, mchangir, nchilaka, rhs-bugs, sankarshan, sheggodu, srakonde, storage-qa-internal, vbellur | |
Target Milestone: | --- | Keywords: | ZStream | |
Target Release: | RHGS 3.4.z Batch Update 3 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.12.2-35 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1654270 (view as bug list) | Environment: | ||
Last Closed: | 2019-02-04 07:41:44 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1654270 |
Description
Nag Pavan Chilakam
2018-11-28 07:32:44 UTC
From "t a a bt" Thread 4 (Thread 0x7ff226972700 (LWP 24785)): #0 0x00007ff223af8ff0 in __do_global_dtors_aux () from /lib64/liblvm2app.so.2.2 #1 0x00007ff22fe59fca in _dl_fini () from /lib64/ld-linux-x86-64.so.2 #2 0x00007ff22e1d7b69 in __run_exit_handlers () from /lib64/libc.so.6 #3 0x00007ff22e1d7bb7 in exit () from /lib64/libc.so.6 #4 0x000055c63d6084af in cleanup_and_exit (signum=15) at glusterfsd.c:1423 #5 0x000055c63d6085a5 in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2145 #6 0x00007ff22e9d3dd5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff22e29bead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff22516f700 (LWP 24788)): #0 0x00007ff22406a0ad in rcu_read_lock_bp () from /lib64/liburcu-bp.so.1 #1 0x00007ff2246efdf0 in gd_peerinfo_find_from_hostname (hoststr=hoststr@entry=0x7ff208970510 "10.70.35.184") at glusterd-peer-utils.c:667 #2 0x00007ff2246f02cd in glusterd_peerinfo_find_by_hostname (hoststr=hoststr@entry=0x7ff208970510 "10.70.35.184") at glusterd-peer-utils.c:110 #3 0x00007ff2246f04b7 in glusterd_hostname_to_uuid (hostname=hostname@entry=0x7ff208970510 "10.70.35.184", uuid=uuid@entry=0x7ff215389300 "") at glusterd-peer-utils.c:154 #4 0x00007ff22462222c in glusterd_volume_brickinfo_get (uuid=uuid@entry=0x0, hostname=0x7ff208970510 "10.70.35.184", path=0x7ff2092558f0 "/var/lib/heketi/mounts/vg_7700284475051d7a420513fffda25002/brick_60470452fbbb1e67e9d1ffe0f7021029/brick", volinfo=volinfo@entry=0x7ff20afa8e30, brickinfo=brickinfo@entry=0x7ff215389360) at glusterd-utils.c:1604 #5 0x00007ff224622409 in glusterd_is_brick_decommissioned (volinfo=volinfo@entry=0x7ff20afa8e30, hostname=<optimized out>, path=<optimized out>) at glusterd-utils.c:1667 #6 0x00007ff224670b1c in _xl_is_client_decommissioned (xl=0x7ff2096adf60, volinfo=volinfo@entry=0x7ff20afa8e30) at glusterd-volgen.c:3464 #7 0x00007ff224670d21 in _xl_has_decommissioned_clients (xl=<optimized out>, volinfo=volinfo@entry=0x7ff20afa8e30) at glusterd-volgen.c:3481 #8 0x00007ff224670cf4 in _xl_has_decommissioned_clients (xl=xl@entry=0x7ff2088bf840, volinfo=volinfo@entry=0x7ff20afa8e30) at glusterd-volgen.c:3490 #9 0x00007ff224670d89 in _graph_get_decommissioned_children (dht=dht@entry=0x7ff20837d1c0, volinfo=volinfo@entry=0x7ff20afa8e30, children=children@entry=0x7ff215389478) at glusterd-volgen.c:3513 #10 0x00007ff224670fa4 in volgen_graph_build_dht_cluster (is_quotad=_gf_false, child_count=1, volinfo=0x7ff20afa8e30, graph=0x7ff20837d1c0) at glusterd-volgen.c:3606 #11 volume_volgen_graph_build_clusters (graph=graph@entry=0x7ff215389740, volinfo=volinfo@entry=0x7ff20afa8e30, is_quotad=is_quotad@entry=_gf_false) at glusterd-volgen.c:3898 #12 0x00007ff224671801 in client_graph_builder (graph=0x7ff215389740, volinfo=0x7ff20afa8e30, set_dict=0x7ff208f475b0, param=<optimized out>) at glusterd-volgen.c:4265 ---Type <return> to continue, or q <return> to quit--- #13 0x00007ff224668a32 in build_graph_generic (graph=graph@entry=0x7ff215389740, volinfo=volinfo@entry=0x7ff20afa8e30, mod_dict=mod_dict@entry=0x7ff20a4203c0, param=param@entry=0x0, builder=builder@entry=0x7ff224671760 <client_graph_builder>) at glusterd-volgen.c:1066 #14 0x00007ff224669084 in build_client_graph (mod_dict=0x7ff20a4203c0, volinfo=0x7ff20afa8e30, graph=0x7ff215389740) at glusterd-volgen.c:4483 #15 generate_single_transport_client_volfile (volinfo=volinfo@entry=0x7ff20afa8e30, filepath=filepath@entry=0x7ff215389890 "/var/lib/glusterd/vols/D-4-5/trusted-D-4-5.tcp-fuse.vol", dict=dict@entry=0x7ff20a4203c0) at glusterd-volgen.c:5711 #16 0x00007ff2246743df in generate_client_volfiles (volinfo=volinfo@entry=0x7ff20afa8e30, client_type=client_type@entry=GF_CLIENT_TRUSTED) at glusterd-volgen.c:5923 #17 0x00007ff2246758bf in glusterd_create_volfiles (volinfo=volinfo@entry=0x7ff20afa8e30) at glusterd-volgen.c:6440 #18 0x00007ff2246759e9 in glusterd_create_volfiles_and_notify_services (volinfo=0x7ff20afa8e30) at glusterd-volgen.c:6468 #19 0x00007ff2246a8706 in glusterd_op_create_volume (dict=dict@entry=0x7ff209b23520, op_errstr=op_errstr@entry=0x7ff21538c098) at glusterd-volume-ops.c:2534 #20 0x00007ff224613153 in glusterd_op_commit_perform (op=GD_OP_CREATE_VOLUME, dict=dict@entry=0x7ff209b23520, op_errstr=op_errstr@entry=0x7ff21538c098, rsp_dict=rsp_dict@entry=0x7ff2090efd10) at glusterd-op-sm.c:6282 #21 0x00007ff22461cdc4 in glusterd_op_ac_commit_op (event=0x7ff2085fd510, ctx=0x7ff2087b6730) at glusterd-op-sm.c:6020 #22 0x00007ff224619d2f in glusterd_op_sm () at glusterd-op-sm.c:8393 #23 0x00007ff2245f3e52 in __glusterd_handle_commit_op (req=req@entry=0x7ff2145d5790) at glusterd-handler.c:1176 #24 0x00007ff2245fb9ce in glusterd_big_locked_handler (req=0x7ff2145d5790, actor_fn=0x7ff2245f3d30 <__glusterd_handle_commit_op>) at glusterd-handler.c:82 #25 0x00007ff22fbadb80 in synctask_wrap () at syncop.c:375 #26 0x00007ff22e1e6010 in ?? () from /lib64/libc.so.6 #27 0x0000000000000000 in ?? () (gdb) q thread 4 is going through cleanup path. at the same time thread 1 is trying to acquire lock on resources which are already freed up by thread 4 as part of cleanup path, that resulted in segmentation fault. Thanks, Sanju upstream patch: https://review.gluster.org/#/c/glusterfs/+/21743 logs,sosreport,health-check-report and core @http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1654161 upstream patch : https://review.gluster.org/#/c/glusterfs/+/21743 ran test mentioned in description on 3.12.2-36 for about 3 days and didnt see any crash. hence moving to verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263 |