Bug 1313511

Summary: Erlang Port Mapper crashes at startup
Product: Red Hat OpenStack Reporter: John Eckersberg <jeckersb>
Component: erlangAssignee: John Eckersberg <jeckersb>
Status: CLOSED ERRATA QA Contact: Udi Kalifon <ukalifon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: apevec, erlang, extras-qa, jeckersb, lemenkov, lhh, nlevinki, raul.andresrodriguez, rjones, srevivo, s
Target Milestone: ---Keywords: ZStream
Target Release: 7.0 (Kilo)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: erlang-R16B-03.10min.8.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1312452 Environment:
Last Closed: 2016-06-01 12:32:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1310808, 1312452    
Bug Blocks:    

Description John Eckersberg 2016-03-01 18:11:09 UTC 
+++ This bug was initially created as a clone of Bug #1312452 +++

+++ This bug was initially created as a clone of Bug #1310808 +++

Description of problem:

After upgrading from erlang-R16B-03.11 to erlang-R16B-03.13 epmd has started crashing at start up in some of our machines.

Version-Release number of selected component (if applicable):

erlang-R16B-03.13

How reproducible:

Start epmd daemon in a machine without ipv6 address

Steps to Reproduce:
[root@myhost ~]# epmd -debug
epmd: Wed Feb 17 08:27:23 2016: epmd running - daemon = 0
*** buffer overflow detected ***: epmd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f01fe269ac7]
/lib64/libc.so.6(+0x10bc80)[0x7f01fe267c80]
/lib64/libc.so.6(+0x10da37)[0x7f01fe269a37]
epmd[0x403a96]
epmd[0x401915]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f01fe17db15]
epmd[0x401d0d]
======= Memory map: ========
00400000-0040c000 r-xp 00000000 fd:00 34108967                           /usr/lib64/erlang/erts-5.10.4/bin/epmd
0060b000-0060c000 r--p 0000b000 fd:00 34108967                           /usr/lib64/erlang/erts-5.10.4/bin/epmd
0060c000-0060d000 rw-p 0000c000 fd:00 34108967                           /usr/lib64/erlang/erts-5.10.4/bin/epmd
0149f000-014c0000 rw-p 00000000 00:00 0                                  [heap]
7f01fcc70000-7f01fcc85000 r-xp 00000000 fd:00 50663                      /usr/lib64/libz.so.1.2.7
7f01fcc85000-7f01fce84000 ---p 00015000 fd:00 50663                      /usr/lib64/libz.so.1.2.7
7f01fce84000-7f01fce85000 r--p 00014000 fd:00 50663                      /usr/lib64/libz.so.1.2.7
7f01fce85000-7f01fce86000 rw-p 00015000 fd:00 50663                      /usr/lib64/libz.so.1.2.7
7f01fce86000-7f01fce95000 r-xp 00000000 fd:00 66728                      /usr/lib64/libbz2.so.1.0.6
7f01fce95000-7f01fd094000 ---p 0000f000 fd:00 66728                      /usr/lib64/libbz2.so.1.0.6
7f01fd094000-7f01fd095000 r--p 0000e000 fd:00 66728                      /usr/lib64/libbz2.so.1.0.6
7f01fd095000-7f01fd096000 rw-p 0000f000 fd:00 66728                      /usr/lib64/libbz2.so.1.0.6
7f01fd096000-7f01fd0ba000 r-xp 00000000 fd:00 16885                      /usr/lib64/liblzma.so.5.0.99
7f01fd0ba000-7f01fd2b9000 ---p 00024000 fd:00 16885                      /usr/lib64/liblzma.so.5.0.99
7f01fd2b9000-7f01fd2ba000 r--p 00023000 fd:00 16885                      /usr/lib64/liblzma.so.5.0.99
7f01fd2ba000-7f01fd2bb000 rw-p 00024000 fd:00 16885                      /usr/lib64/liblzma.so.5.0.99
7f01fd2bb000-7f01fd2d0000 r-xp 00000000 fd:00 66743                      /usr/lib64/libelf-0.163.so
7f01fd2d0000-7f01fd4cf000 ---p 00015000 fd:00 66743                      /usr/lib64/libelf-0.163.so
7f01fd4cf000-7f01fd4d0000 r--p 00014000 fd:00 66743                      /usr/lib64/libelf-0.163.so
7f01fd4d0000-7f01fd4d1000 rw-p 00015000 fd:00 66743                      /usr/lib64/libelf-0.163.so
7f01fd4d1000-7f01fd4d5000 r-xp 00000000 fd:00 66753                      /usr/lib64/libattr.so.1.1.0
7f01fd4d5000-7f01fd6d4000 ---p 00004000 fd:00 66753                      /usr/lib64/libattr.so.1.1.0
7f01fd6d4000-7f01fd6d5000 r--p 00003000 fd:00 66753                      /usr/lib64/libattr.so.1.1.0
7f01fd6d5000-7f01fd6d6000 rw-p 00004000 fd:00 66753                      /usr/lib64/libattr.so.1.1.0
7f01fd6d6000-7f01fd6ec000 r-xp 00000000 fd:00 16866                      /usr/lib64/libpthread-2.17.so
7f01fd6ec000-7f01fd8ec000 ---p 00016000 fd:00 16866                      /usr/lib64/libpthread-2.17.so
7f01fd8ec000-7f01fd8ed000 r--p 00016000 fd:00 16866                      /usr/lib64/libpthread-2.17.so
7f01fd8ed000-7f01fd8ee000 rw-p 00017000 fd:00 16866                      /usr/lib64/libpthread-2.17.so
7f01fd8ee000-7f01fd8f2000 rw-p 00000000 00:00 0 
7f01fd8f2000-7f01fd907000 r-xp 00000000 fd:00 142                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fd907000-7f01fdb06000 ---p 00015000 fd:00 142                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb06000-7f01fdb07000 r--p 00014000 fd:00 142                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb07000-7f01fdb08000 rw-p 00015000 fd:00 142                        /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f01fdb08000-7f01fdb4d000 r-xp 00000000 fd:00 82543                      /usr/lib64/libdw-0.163.so
7f01fdb4d000-7f01fdd4c000 ---p 00045000 fd:00 82543                      /usr/lib64/libdw-0.163.so
7f01fdd4c000-7f01fdd4e000 r--p 00044000 fd:00 82543                      /usr/lib64/libdw-0.163.so
7f01fdd4e000-7f01fdd4f000 rw-p 00046000 fd:00 82543                      /usr/lib64/libdw-0.163.so
7f01fdd4f000-7f01fdd53000 r-xp 00000000 fd:00 66757                      /usr/lib64/libcap.so.2.22
7f01fdd53000-7f01fdf52000 ---p 00004000 fd:00 66757                      /usr/lib64/libcap.so.2.22
7f01fdf52000-7f01fdf53000 r--p 00003000 fd:00 66757                      /usr/lib64/libcap.so.2.22
7f01fdf53000-7f01fdf54000 rw-p 00004000 fd:00 66757                      /usr/lib64/libcap.so.2.22
7f01fdf54000-7f01fdf5b000 r-xp 00000000 fd:00 16870                      /usr/lib64/librt-2.17.so
7f01fdf5b000-7f01fe15a000 ---p 00007000 fd:00 16870                      /usr/lib64/librt-2.17.so
7f01fe15a000-7f01fe15b000 r--p 00006000 fd:00 16870                      /usr/lib64/librt-2.17.so
7f01fe15b000-7f01fe15c000 rw-p 00007000 fd:00 16870                      /usr/lib64/librt-2.17.so
7f01fe15c000-7f01fe312000 r-xp 00000000 fd:00 16840                      /usr/lib64/libc-2.17.so
7f01fe312000-7f01fe512000 ---p 001b6000 fd:00 16840                      /usr/lib64/libc-2.17.so
7f01fe512000-7f01fe516000 r--p 001b6000 fd:00 16840                      /usr/lib64/libc-2.17.so
7f01fe516000-7f01fe518000 rw-p 001ba000 fd:00 16840                      /usr/lib64/libc-2.17.so
7f01fe518000-7f01fe51d000 rw-p 00000000 00:00 0 
7f01fe51d000-7f01fe61e000 r-xp 00000000 fd:00 16848                      /usr/lib64/libm-2.17.so
7f01fe61e000-7f01fe81d000 ---p 00101000 fd:00 16848                      /usr/lib64/libm-2.17.so
7f01fe81d000-7f01fe81e000 r--p 00100000 fd:00 16848                      /usr/lib64/libm-2.17.so
7f01fe81e000-7f01fe81f000 rw-p 00101000 fd:00 16848                      /usr/lib64/libm-2.17.so
7f01fe81f000-7f01fe822000 r-xp 00000000 fd:00 16846                      /usr/lib64/libdl-2.17.so
7f01fe822000-7f01fea21000 ---p 00003000 fd:00 16846                      /usr/lib64/libdl-2.17.so
7f01fea21000-7f01fea22000 r--p 00002000 fd:00 16846                      /usr/lib64/libdl-2.17.so
7f01fea22000-7f01fea23000 rw-p 00003000 fd:00 16846                      /usr/lib64/libdl-2.17.so
7f01fea23000-7f01fea25000 r-xp 00000000 fd:00 16874                      /usr/lib64/libutil-2.17.so
7f01fea25000-7f01fec24000 ---p 00002000 fd:00 16874                      /usr/lib64/libutil-2.17.so
7f01fec24000-7f01fec25000 r--p 00001000 fd:00 16874                      /usr/lib64/libutil-2.17.soAborted (core dumped)


Actual results:

epmd crashes

Expected results:

epmd daemon starts

Additional info:

strace shows that before the crash:
bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3, 128)                          = 0
socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
select(4, [3], NULL, NULL, {5, 0})      = 0 (Timeout)
open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 4
writev(4, [{"*** ", 4}, {"buffer overflow detected", 24}, {" ***: ", 6}, {"epmd", 4}, {" terminated\n", 12}], 5*** buffer overflow detected ***: epmd terminated
) = 50

systemctl start epmd.0.0 

works fine.

--- Additional comment from John Eckersberg on 2016-02-22 14:32:37 EST ---

Can you attach the core dump?  Merely starting epmd on a host with no IPv6 addresses doesn't seem to be enough to trigger the bug, I'm unable to reproduce:

[root@centos7 ~]# rpm -q erlang-erts
erlang-erts-R16B-03.13.el7.x86_64
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:87:2d:f5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.147/24 brd 192.168.122.255 scope global dynamic eth0
       valid_lft 2850sec preferred_lft 2850sec
[root@centos7 ~]# epmd -debug
epmd: Mon Feb 22 14:27:20 2016: epmd running - daemon = 0


(no crash for me)

--- Additional comment from Raúl Andrés on 2016-02-23 01:44 EST ---



--- Additional comment from Raúl Andrés on 2016-02-23 02:59:59 EST ---

I have IPv6 support disabled:

[root@myhost tmp]#  cat /proc/net/if_inet6                                
cat: /proc/net/if_inet6: No such file or directory
[root@myhost tmp]#  modprobe -c| grep ipv6
options ipv6 disable=1

--- Additional comment from John Eckersberg on 2016-02-26 12:44:37 EST ---

This also reproduces on OSP:

[root@rhel7 ~]# rpm -q erlang-erts
erlang-erts-R16B-03.10min.5.el7ost.x86_64
[root@rhel7 ~]# epmd -d 2>&1 | head
epmd: Fri Feb 26 12:38:16 2016: epmd running - daemon = 0
*** buffer overflow detected ***: epmd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fc11d6f4b37]
/lib64/libc.so.6(+0x10bcf0)[0x7fc11d6f2cf0]
/lib64/libc.so.6(+0x10daa7)[0x7fc11d6f4aa7]
epmd[0x403a96]
epmd[0x401915]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc11d608b15]
epmd[0x401d0d]
...

Disable IPv6 first as per https://access.redhat.com/solutions/8709?tour=6#rhel7disable in order to reproduce.
Comment 3 Udi Kalifon 2016-05-22 07:24:53 UTC 
Verified in erlang-erts-R16B-03.10min.9.el7ost.x86_64
Comment 5 errata-xmlrpc 2016-06-01 12:32:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1191