Bug 1465638 - RHGS 3.1.3 to 3.2 Upgrade - SegFault
Summary: RHGS 3.1.3 to 3.2 Upgrade - SegFault
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: glusterfs
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: RHGS 3.3.0
Assignee: Gaurav Yadav
QA Contact: Vinayak Papnoi
URL:
Whiteboard: 3.3.0-devel-freeze-exception
: 1466621 (view as bug list)
Depends On: 1447523
Blocks: 1417151 1466357
TreeView+ depends on / blocked
 
Reported: 2017-06-27 20:02 UTC by Matthew Robson
Modified: 2020-12-14 08:57 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.8.4-32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1466357 1466621 (view as bug list)
Environment:
Last Closed: 2017-09-21 05:02:13 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2774 0 normal SHIPPED_LIVE glusterfs bug fix and enhancement update 2017-09-21 08:16:29 UTC

Description Matthew Robson 2017-06-27 20:02:47 UTC
Description of problem:
Upgrading RHGS 3.1.3 (glusterfs-3.7.9-12 ) to 3.2 (glusterfs-server-3.8.4-18.4.el7rhgs.x86_64), we hit a seg fault during the yum install and a segfault (11) when trying to start glusterd.

glusterd.log:

[2017-06-27 16:50:37.602265] I [MSGID: 100030] [glusterfsd.c:2412:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.4 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-06-27 16:50:37.608200] I [MSGID: 106478] [glusterd.c:1382:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-06-27 16:50:37.608255] I [MSGID: 106479] [glusterd.c:1431:init] 0-management: Using /var/lib/glusterd as working directory
[2017-06-27 16:50:37.614109] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.8.4/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-06-27 16:50:37.614137] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-06-27 16:50:37.614151] W [rpcsvc.c:1646:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-06-27 16:50:37.614166] E [MSGID: 106243] [glusterd.c:1655:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-06-27 16:50:37.615604] I [MSGID: 106228] [glusterd.c:432:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-06-27 16:50:37.624671] I [MSGID: 106513] [glusterd-store.c:2101:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30712
[2017-06-27 16:50:43.197971] I [MSGID: 106498] [glusterd-handler.c:3662:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-06-27 16:50:43.198104] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
pending frames:
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2017-06-27 16:50:43
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.4


Core Back Trace (has gcc issue BZ-1293594):

Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done.
done.

warning: core file may not match specified executable file.
[New LWP 7262]
[New LWP 7264]
[New LWP 7263]
[New LWP 7265]
[New LWP 7266]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO'.
Program terminated with signal 11, Segmentation fault.
#0  x86_64_fallback_frame_state (context=0x7ffea8d28bc0, context=0x7ffea8d28bc0, fs=0x7ffea8d28cb0) at ./md-unwind-support.h:58
58	  if (*(unsigned char *)(pc+0) == 0x48
(gdb) bt
#0  x86_64_fallback_frame_state (context=0x7ffea8d28bc0, context=0x7ffea8d28bc0, fs=0x7ffea8d28cb0) at ./md-unwind-support.h:58
#1  uw_frame_state_for (context=context@entry=0x7ffea8d28bc0, fs=fs@entry=0x7ffea8d28cb0) at ../../../libgcc/unwind-dw2.c:1253
#2  0x00007f27907f4019 in _Unwind_Backtrace (trace=0x7f279c45d4f0 <backtrace_helper+48>, trace_argument=0x7ffea8d28e70) at ../../../libgcc/unwind.inc:290
#3  0x00007f279c45d666 in __GI___backtrace (array=0x7f279e18d930 <_rtld_local+2352>, array@entry=0x7ffea8d28eb0, size=size@entry=200) at ../sysdeps/x86_64/backtrace.c:103
#4  0x00007f279dca3ce2 in _gf_msg_backtrace_nomem (level=level@entry=GF_LOG_ALERT, stacksize=stacksize@entry=200) at logging.c:1094
#5  0x00007f279dcad884 in gf_print_trace (signum=11, ctx=0x7f279e924010) at common-utils.c:755
#6  <signal handler called>
#7  0x000000018f88af78 in ?? ()
#8  0x0000000000000000 in ?? ()

Yum Update SegFault:

/var/tmp/rpm-tmp.ocZIQX: line 68: 63751 Segmentation fault      (core dumped) glusterd --xlator-option *.upgrade=on -N
  Cleanup    : glusterfs-server-3.7.9-12.el7rhgs.x86_64                    8/14
  Cleanup    : glusterfs-api-3.7.9-12.el7rhgs.x86_64                       9/14
  Cleanup    : glusterfs-fuse-3.7.9-12.el7rhgs.x86_64                     10/14
  Cleanup    : glusterfs-3.7.9-12.el7rhgs.x86_64                          11/14
  Cleanup    : glusterfs-client-xlators-3.7.9-12.el7rhgs.x86_64           12/14
  Cleanup    : glusterfs-cli-3.7.9-12.el7rhgs.x86_64                      13/14
  Cleanup    : glusterfs-libs-3.7.9-12.el7rhgs.x86_64                     14/14
  Verifying  : glusterfs-3.8.4-18.4.el7rhgs.x86_64                         1/14
  Verifying  : glusterfs-fuse-3.8.4-18.4.el7rhgs.x86_64                    2/14
  Verifying  : glusterfs-cli-3.8.4-18.4.el7rhgs.x86_64                     3/14
  Verifying  : glusterfs-api-3.8.4-18.4.el7rhgs.x86_64                     4/14
  Verifying  : glusterfs-libs-3.8.4-18.4.el7rhgs.x86_64                    5/14
  Verifying  : glusterfs-server-3.8.4-18.4.el7rhgs.x86_64                  6/14
  Verifying  : glusterfs-client-xlators-3.8.4-18.4.el7rhgs.x86_64          7/14
  Verifying  : glusterfs-3.7.9-12.el7rhgs.x86_64                           8/14
  Verifying  : glusterfs-server-3.7.9-12.el7rhgs.x86_64                    9/14
  Verifying  : glusterfs-client-xlators-3.7.9-12.el7rhgs.x86_64           10/14
  Verifying  : glusterfs-fuse-3.7.9-12.el7rhgs.x86_64                     11/14
  Verifying  : glusterfs-libs-3.7.9-12.el7rhgs.x86_64                     12/14
  Verifying  : glusterfs-cli-3.7.9-12.el7rhgs.x86_64                      13/14
  Verifying  : glusterfs-api-3.7.9-12.el7rhgs.x86_64                      14/14

Updated:
  glusterfs.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-api.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-cli.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-client-xlators.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-fuse.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-libs.x86_64 0:3.8.4-18.4.el7rhgs
  glusterfs-server.x86_64 0:3.8.4-18.4.el7rhgs

Complete!


Version-Release number of selected component (if applicable):
glusterfs-server-3.8.4-18.4.el7rhgs.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Have: net.ipv4.ip_local_reserved_ports=1002,49152-49652
2. Yum Update
3. Start Gluster  

Actual results:
Fails to start


Expected results:
Starts

Additional info:

Comment 5 Atin Mukherjee 2017-06-28 04:35:26 UTC
upstream patch : https://review.gluster.org/#/c/17359/

Comment 9 Atin Mukherjee 2017-06-30 05:51:54 UTC
*** Bug 1466621 has been marked as a duplicate of this bug. ***

Comment 12 Vinayak Papnoi 2017-07-24 09:20:58 UTC
Build : 3.8.4-35

Followed all the steps in the description. Tried starting glusterd after "yum update" (from glusterfs-3.7.9-12 to glusterfs-3.8.4-35) and glusterd started with no issues.

Hence, moving this to VERIFIED.

Comment 14 errata-xmlrpc 2017-09-21 05:02:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774


Note You need to log in before you can comment on or make changes to this bug.