Bug 1619857 - glusterd signal 8 - dict_lookup_common
Summary: glusterd signal 8 - dict_lookup_common
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.3
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ---
: RHGS 3.4.z Batch Update 3
Assignee: Sanju
QA Contact: Bala Konda Reddy M
URL:
Whiteboard:
Depends On:
Blocks: 1651547
TreeView+ depends on / blocked
 
Reported: 2018-08-21 23:16 UTC by John Strunk
Modified: 2019-02-04 07:41 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.12.2-33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-04 07:41:25 UTC
Target Upstream Version:


Attachments (Terms of Use)
Full backtrace (41.13 KB, text/plain)
2018-08-21 23:19 UTC, John Strunk
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0263 0 None None None 2019-02-04 07:41:38 UTC

Description John Strunk 2018-08-21 23:16:23 UTC
Description of problem:
Glusterd crashed w/ sig 8 - arithmetic exception

Version-Release number of selected component (if applicable):

RHGS 3.3.1

$ rpm -qa | grep gluster
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-cli-3.8.4-54.10.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-54.10.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-54.10.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-54.10.el7rhgs.x86_64
glusterfs-api-3.8.4-54.10.el7rhgs.x86_64
python-gluster-3.8.4-54.10.el7rhgs.noarch
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
pcp-pmda-gluster-4.1.0-0.201805281909.git68ab4b18.el7.x86_64
glusterfs-libs-3.8.4-54.10.el7rhgs.x86_64
glusterfs-fuse-3.8.4-54.10.el7rhgs.x86_64
vdsm-gluster-4.17.33-1.2.el7rhgs.noarch
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64
glusterfs-3.8.4-54.10.el7rhgs.x86_64
glusterfs-server-3.8.4-54.10.el7rhgs.x86_64
glusterfs-rdma-3.8.4-54.10.el7rhgs.x86_64



How reproducible:
Not reproducible. Happened while cluster was serving normally.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

From glusterd.log:
[2018-08-21 22:18:59.431161] I [MSGID: 106494] [glusterd-handler.c:3087:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume gluster_shared_storage
[2018-08-21 22:18:59.455344] E [MSGID: 106524] [glusterd-op-sm.c:1987:glusterd_op_stage_stats_volume] 0-glusterd: Profile on Volume gluster_shared_storage is not started
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 8
time of crash: 
2018-08-21 22:18:59
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.4
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f2bb1710822]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f2bb171a354]
/lib64/libc.so.6(+0x36280)[0x7f2bafd71280]
/lib64/libglusterfs.so.0(+0x219d2)[0x7f2bb17079d2]
/lib64/libglusterfs.so.0(+0x22f06)[0x7f2bb1708f06]
/lib64/libglusterfs.so.0(dict_get_int32+0x3a)[0x7f2bb170a5ba]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x5d681)[0x7f2ba61e8681]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x70c2e)[0x7f2ba61fbc2e]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x7282a)[0x7f2ba61fd82a]
/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f2bb14d9770]
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1e7)[0x7f2bb14d9a57]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f2bb14d5913]
/usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so(+0x7366)[0x7f2ba33f8366]
/usr/lib64/glusterfs/3.8.4/rpc-transport/socket.so(+0xa042)[0x7f2ba33fb042]
/lib64/libpthread.so.0(+0x7dd5)[0x7f2bb0570dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f2bafe39b3d]
---------



From gdb:
Program terminated with signal 8, Arithmetic exception.
#0  0x00007f2bb17079d2 in dict_lookup_common (this=0x7f2b85ae62a0, 
    key=0x7f2ba62b9a2c "count") at dict.c:283
283                     hashval = SuperFastHash (key, strlen (key)) % this->hash_size;
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 device-
mapper-event-libs-1.02.146-4.el7.x86_64 device-mapper-libs-1.02.146-4.el7.x86_64 elfutils-
libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_6
4 krb5-libs-1.15.1-19.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 lib
com_err-1.42.9-12.el7_5.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-12.el7.x86_64
 libsepol-2.5-8.1.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 lvm2-libs-2.02.177-4.el7.x86_6
4 pcre-8.32-17.el7.x86_64 systemd-libs-219-57.el7.x86_64 userspace-rcu-0.7.9-2.el7rhgs.x86
_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) p this
$1 = (dict_t *) 0x7f2b85ae62a0
(gdb) p this->hash_size
$2 = 0
(gdb) p *this
$3 = {is_static = 0 '\000', hash_size = 0, count = 0, refcount = 0, members = 0x0, 
  members_list = 0x0, 
  extra_free = 0xf00000007000000 <Address 0xf00000007000000 out of bounds>, 
  extra_stdfree = 0x6769726f10000000 <Address 0x6769726f10000000 out of bounds>, lock = {
    spinlock = 1952542313, mutex = {__data = {__lock = 1952542313, __count = 1969189487, 
        __owner = 6580597, __nusers = 3121462155, __kind = 558060240, __spins = 24756, 
        __elision = 1027, __list = {__prev = 0xe000000709dd865, 
          __next = 0x6e61727410000000}}, 
      __size = "inator_uuid\000\213\273\r\272\320RC!\264`\003\004e\330\235p\000\000\000\016\000\000\000\020tran", __align = 8457604448244559465}}, 
  members_internal = 0x5f6e6f6974636173, free_pair = {hash_next = 0x8b63e5f31c006469, 
    prev = 0xb4d335009f20422e, next = 0x7000000f38c3b, value = 0x616e6c6f76170000, 
    key = 0x7473756c6700656d <Address 0x7473756c6700656d out of bounds>}, 
  free_pair_in_use = (_gf_true | unknown: 1935635044)}
(gdb)

Comment 2 John Strunk 2018-08-21 23:19:38 UTC
Created attachment 1477712 [details]
Full backtrace

Comment 5 Sanju 2018-12-04 11:12:33 UTC
upstream patch: https://review.gluster.org/#/c/glusterfs/+/21736/

Comment 19 errata-xmlrpc 2019-02-04 07:41:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0263


Note You need to log in before you can comment on or make changes to this bug.