Bug 1455142

Summary: rpcbind crash on start [rhel-6]
Product: Red Hat Enterprise Linux 6 Reporter: Achilles Gaikwad <agaikwad>
Component: rpcbindAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: Yongcheng Yang <yoyang>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.9CC: a.gormanly, ajb, brubisch, calum.mackay, cmc, cperry, cww, damien.reyt, darren.miller, dwysocha, james-p, joe.kang, knweiss, m.camen, michiel.dewilde, mjtrangoni, mkolaja, olchansk, pasik, pbunyan, pdudley, pgozart, renaud.haxaire, riehecky, rmj, rstonehouse, steved, swhiteho, toracat, vs, xzhou, yoyang, zagar, zpytela
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rpcbind-0.2.0-15.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1454876
: 1458240 (view as bug list) Environment:
Last Closed: 2018-06-19 05:20:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On: 1454876    
Bug Blocks: 1374441, 1458240    

Comment 4 joe.kang 2017-05-26 15:42:06 UTC
When I hit this problem on a similar system, I happened to run an "rpcbind -d" command. I have the output from it if you would like to see it. It shows "*** glibc detected *** rpcbind: free(): invalid pointer: 0x00007fff67645700 ***"

Comment 5 C.M. Connelly 2017-05-26 20:43:05 UTC
Same here---pmap_rmtcall callit req failed, and then I got the same sort of free() invalid pointer error (at a different address).

Comment 6 Renaud Haxaire 2017-05-30 09:39:44 UTC
Hello,

I cannot reproduce with fastbug glibc installed :
[root@aar057 ~]# rpm -q glibc.x86_64
glibc-2.12-1.209.el6_9.1.x86_64

I let a comment on https://access.redhat.com/solutions/3053461

Regards, Renaud.

Comment 8 Konstantin Olchanski 2017-05-30 14:34:58 UTC
I do confirm that we see the rpcbind crash on el6 machines that have your glibc.

[root@ladd00 ~]# rpm -q glibc
glibc-2.12-1.209.el6_9.1.x86_64
glibc-2.12-1.209.el6_9.1.i686
[root@ladd00 ~]# 

K.O.

BTW, please be aware that your reference to access.redhat.com is password protected, I cannot access it, I cannot read your comments there. if you want maximum feedback from the public, please use publicly accessible resources such as this bugzilla.

Comment 9 Renaud Haxaire 2017-05-30 15:41:04 UTC
More detailed informations for those who cannot access access.redhat.com:

Hello,

I had to patch glibc with fastbug patch because I had a problem running Ansys/hfss with the standard glibc provided with RHEL 6.9 (I had not the problem with RHEL 6.8). I only update glibc on my cluster compute nodes. I noticed that on machines with fastbug glibc, I'm not able to reproduce the issue.

My compute node:

[root@aar057 ~]# rpm -q glibc.x86_64 rpcbind.x86_64
glibc-2.12-1.209.el6_9.1.x86_64
rpcbind-0.2.0-13.el6_9.x86_64
[root@aar057 ~]# rpcbind -d -w
...
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >

My workstation:

[root@kalahari ~]# rpm -q glibc.x86_64 rpcbind.x86_64
glibc-2.12-1.209.el6.x86_64
rpcbind-0.2.0-13.el6_9.x86_64
[root@kalahari ~]# rpcbind -d -w
...
7f665608f000-7f6656090000 rw-p 0000c000 fd:00 151559 /sbin/rpcbind
7f6656090000-7f6656091000 rw-p 00000000 00:00 0
7f66576be000-7f66576df000 rw-p 00000000 00:00 0 [heap]
7ffe0b35d000-7ffe0b386000 rw-p 00000000 00:00 0 [stack]
7ffe0b3c3000-7ffe0b3c4000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted

Another difference btw the two machines is the nscd is activated on compute node but not on workstation:

[root@aar057 ~]# service nscd status
nscd (pid 10037) is running...

[root@kalahari ~]# service nscd status
nscd: unrecognized service

Hope this contibute solving this issue.

Best regards, Renaud.

Comment 10 Andrew Gormanly 2017-05-31 10:53:09 UTC
Also see this, but on 1 server (the other one with RPC services seems fine).

The one that crashes runs nfs, rpcbind, rpcidmapd, ypserv, ypxfrd, yppasswdd; the other runs all these except nfs and does not crash.

Downgrading rpcbind to 0.2.0-13.el6 is a workaround.


[root@excelsior ~]# rpcbind -d
local: 0 lookup routines :
rpcbind : my address is (null)
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for local
udp: 0 lookup routines :
rpcbind : my address is 0.0.0.0.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 2 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for udp
rmtcall fd for udp is 7
tcp: 0 lookup routines :
rpcbind : my address is 0.0.0.0.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 2 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for tcp
udp6: 0 lookup routines :
rpcbind : my address is ::.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for udp6
rmtcall fd for udp6 is 10
tcp6: 0 lookup routines :
rpcbind : my address is ::.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for tcp6
rpcbind debugging enabled.
polling for read on fd < 5 6 7 8 9 10 11 >
poll returned read fds < 6 >
pmap_rmtcall callit req for (100004, 2, 2, udp) from 192.168.99.73.243.83 : not found
*** glibc detected *** rpcbind: free(): invalid pointer: 0x00007ffe7d23ab10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x335ea75dee)[0x7fe4df64fdee]
/lib64/libc.so.6(+0x335ea78c3d)[0x7fe4df652c3d]
/lib64/libtirpc.so.1(xdr_bytes+0xc2)[0x7fe4dfba7202]
rpcbind(+0x6415)[0x7fe4e01e9415]
rpcbind(pmap_service+0x175)[0x7fe4e01ebbe5]
/lib64/libtirpc.so.1(svc_getreq_common+0x231)[0x7fe4dfba0c41]
/lib64/libtirpc.so.1(svc_getreq_poll+0x91)[0x7fe4dfba0cf1]
rpcbind(+0x5c1e)[0x7fe4e01e8c1e]
rpcbind(main+0x478)[0x7fe4e01e79b8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fe4df5f8d1d]
rpcbind(+0x3229)[0x7fe4e01e6229]
======= Memory map: ========
7fe4d8000000-7fe4d8021000 rw-p 00000000 00:00 0 
7fe4d8021000-7fe4dc000000 ---p 00000000 00:00 0 
7fe4deb90000-7fe4deba6000 r-xp 00000000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deba6000-7fe4deda5000 ---p 00016000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deda5000-7fe4deda6000 rw-p 00015000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deda6000-7fe4dedb3000 r-xp 00000000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4dedb3000-7fe4defb2000 ---p 0000d000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb2000-7fe4defb3000 r--p 0000c000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb3000-7fe4defb4000 rw-p 0000d000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb4000-7fe4defb6000 r-xp 00000000 08:05 393295                     /lib64/libdl-2.12.so
7fe4defb6000-7fe4df1b6000 ---p 00002000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b6000-7fe4df1b7000 r--p 00002000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b7000-7fe4df1b8000 rw-p 00003000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b8000-7fe4df1c1000 r-xp 00000000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df1c1000-7fe4df3c0000 ---p 00009000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df3c0000-7fe4df3c1000 rw-p 00008000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df3c1000-7fe4df3d7000 r-xp 00000000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df3d7000-7fe4df5d6000 ---p 00016000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d6000-7fe4df5d7000 r--p 00015000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d7000-7fe4df5d8000 rw-p 00016000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d8000-7fe4df5da000 rw-p 00000000 00:00 0 
7fe4df5da000-7fe4df764000 r-xp 00000000 08:05 393224                     /lib64/libc-2.12.so
7fe4df764000-7fe4df964000 ---p 0018a000 08:05 393224                     /lib64/libc-2.12.so
7fe4df964000-7fe4df968000 r--p 0018a000 08:05 393224                     /lib64/libc-2.12.so
7fe4df968000-7fe4df96a000 rw-p 0018e000 08:05 393224                     /lib64/libc-2.12.so
7fe4df96a000-7fe4df96e000 rw-p 00000000 00:00 0 
7fe4df96e000-7fe4df985000 r-xp 00000000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4df985000-7fe4dfb85000 ---p 00017000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb85000-7fe4dfb86000 r--p 00017000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb86000-7fe4dfb87000 rw-p 00018000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb87000-7fe4dfb8b000 rw-p 00000000 00:00 0 
7fe4dfb8b000-7fe4dfbb3000 r-xp 00000000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfbb3000-7fe4dfdb3000 ---p 00028000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfdb3000-7fe4dfdb5000 rw-p 00028000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfdb5000-7fe4dfdbd000 r-xp 00000000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dfdbd000-7fe4dffbd000 ---p 00008000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbd000-7fe4dffbe000 r--p 00008000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbe000-7fe4dffbf000 rw-p 00009000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbf000-7fe4dffc0000 rw-p 00000000 00:00 0 
7fe4dffc0000-7fe4dffe0000 r-xp 00000000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01c0000-7fe4e01c5000 rw-p 00000000 00:00 0 
7fe4e01dc000-7fe4e01df000 rw-p 00000000 00:00 0 
7fe4e01df000-7fe4e01e1000 r--p 0001f000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01e1000-7fe4e01e2000 rw-p 00021000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01e2000-7fe4e01e3000 rw-p 00000000 00:00 0 
7fe4e01e3000-7fe4e01f0000 r-xp 00000000 08:05 1179747                    /sbin/rpcbind
7fe4e03ef000-7fe4e03f0000 rw-p 0000c000 08:05 1179747                    /sbin/rpcbind
7fe4e03f0000-7fe4e03f1000 rw-p 00000000 00:00 0 
7fe4e1a22000-7fe4e1a43000 rw-p 00000000 00:00 0                          [heap]
7ffe7d227000-7ffe7d24f000 rw-p 00000000 00:00 0                          [stack]
7ffe7d2a2000-7ffe7d2a3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

Comment 13 Jan Stancek 2017-08-02 20:06:26 UTC
*** Bug 1461904 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2018-06-19 05:20:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1901