Bug 1455142

When I hit this problem on a similar system, I happened to run an "rpcbind -d" command. I have the output from it if you would like to see it. It shows "*** glibc detected *** rpcbind: free(): invalid pointer: 0x00007fff67645700 ***"

Same here---pmap_rmtcall callit req failed, and then I got the same sort of free() invalid pointer error (at a different address).

Hello,

I cannot reproduce with fastbug glibc installed :
[root@aar057 ~]# rpm -q glibc.x86_64
glibc-2.12-1.209.el6_9.1.x86_64

I let a comment on https://access.redhat.com/solutions/3053461

Regards, Renaud.

I do confirm that we see the rpcbind crash on el6 machines that have your glibc.

[root@ladd00 ~]# rpm -q glibc
glibc-2.12-1.209.el6_9.1.x86_64
glibc-2.12-1.209.el6_9.1.i686
[root@ladd00 ~]# 

K.O.

BTW, please be aware that your reference to access.redhat.com is password protected, I cannot access it, I cannot read your comments there. if you want maximum feedback from the public, please use publicly accessible resources such as this bugzilla.

More detailed informations for those who cannot access access.redhat.com:

Hello,

I had to patch glibc with fastbug patch because I had a problem running Ansys/hfss with the standard glibc provided with RHEL 6.9 (I had not the problem with RHEL 6.8). I only update glibc on my cluster compute nodes. I noticed that on machines with fastbug glibc, I'm not able to reproduce the issue.

My compute node:

[root@aar057 ~]# rpm -q glibc.x86_64 rpcbind.x86_64
glibc-2.12-1.209.el6_9.1.x86_64
rpcbind-0.2.0-13.el6_9.x86_64
[root@aar057 ~]# rpcbind -d -w
...
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >
polling for read on fd < 5 6 7 8 9 10 11 >

My workstation:

[root@kalahari ~]# rpm -q glibc.x86_64 rpcbind.x86_64
glibc-2.12-1.209.el6.x86_64
rpcbind-0.2.0-13.el6_9.x86_64
[root@kalahari ~]# rpcbind -d -w
...
7f665608f000-7f6656090000 rw-p 0000c000 fd:00 151559 /sbin/rpcbind
7f6656090000-7f6656091000 rw-p 00000000 00:00 0
7f66576be000-7f66576df000 rw-p 00000000 00:00 0 [heap]
7ffe0b35d000-7ffe0b386000 rw-p 00000000 00:00 0 [stack]
7ffe0b3c3000-7ffe0b3c4000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted

Another difference btw the two machines is the nscd is activated on compute node but not on workstation:

[root@aar057 ~]# service nscd status
nscd (pid 10037) is running...

[root@kalahari ~]# service nscd status
nscd: unrecognized service

Hope this contibute solving this issue.

Best regards, Renaud.

Also see this, but on 1 server (the other one with RPC services seems fine).

The one that crashes runs nfs, rpcbind, rpcidmapd, ypserv, ypxfrd, yppasswdd; the other runs all these except nfs and does not crash.

Downgrading rpcbind to 0.2.0-13.el6 is a workaround.


[root@excelsior ~]# rpcbind -d
local: 0 lookup routines :
rpcbind : my address is (null)
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for local
udp: 0 lookup routines :
rpcbind : my address is 0.0.0.0.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 2 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for udp
rmtcall fd for udp is 7
tcp: 0 lookup routines :
rpcbind : my address is 0.0.0.0.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 2 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for tcp
udp6: 0 lookup routines :
rpcbind : my address is ::.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for udp6
rmtcall fd for udp6 is 10
tcp6: 0 lookup routines :
rpcbind : my address is ::.0.111
FUNCTION rbllist_addAdd the prog 100000 vers 3 to the rpcbind list
FUNCTION rbllist_addAdd the prog 100000 vers 4 to the rpcbind list
check binding for tcp6
rpcbind debugging enabled.
polling for read on fd < 5 6 7 8 9 10 11 >
poll returned read fds < 6 >
pmap_rmtcall callit req for (100004, 2, 2, udp) from 192.168.99.73.243.83 : not found
*** glibc detected *** rpcbind: free(): invalid pointer: 0x00007ffe7d23ab10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x335ea75dee)[0x7fe4df64fdee]
/lib64/libc.so.6(+0x335ea78c3d)[0x7fe4df652c3d]
/lib64/libtirpc.so.1(xdr_bytes+0xc2)[0x7fe4dfba7202]
rpcbind(+0x6415)[0x7fe4e01e9415]
rpcbind(pmap_service+0x175)[0x7fe4e01ebbe5]
/lib64/libtirpc.so.1(svc_getreq_common+0x231)[0x7fe4dfba0c41]
/lib64/libtirpc.so.1(svc_getreq_poll+0x91)[0x7fe4dfba0cf1]
rpcbind(+0x5c1e)[0x7fe4e01e8c1e]
rpcbind(main+0x478)[0x7fe4e01e79b8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fe4df5f8d1d]
rpcbind(+0x3229)[0x7fe4e01e6229]
======= Memory map: ========
7fe4d8000000-7fe4d8021000 rw-p 00000000 00:00 0 
7fe4d8021000-7fe4dc000000 ---p 00000000 00:00 0 
7fe4deb90000-7fe4deba6000 r-xp 00000000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deba6000-7fe4deda5000 ---p 00016000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deda5000-7fe4deda6000 rw-p 00015000 08:05 393358                     /lib64/libgcc_s-4.4.7-20120601.so.1
7fe4deda6000-7fe4dedb3000 r-xp 00000000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4dedb3000-7fe4defb2000 ---p 0000d000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb2000-7fe4defb3000 r--p 0000c000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb3000-7fe4defb4000 rw-p 0000d000 08:05 393275                     /lib64/libnss_files-2.12.so
7fe4defb4000-7fe4defb6000 r-xp 00000000 08:05 393295                     /lib64/libdl-2.12.so
7fe4defb6000-7fe4df1b6000 ---p 00002000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b6000-7fe4df1b7000 r--p 00002000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b7000-7fe4df1b8000 rw-p 00003000 08:05 393295                     /lib64/libdl-2.12.so
7fe4df1b8000-7fe4df1c1000 r-xp 00000000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df1c1000-7fe4df3c0000 ---p 00009000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df3c0000-7fe4df3c1000 rw-p 00008000 08:05 393447                     /lib64/libgssglue.so.1.0.0
7fe4df3c1000-7fe4df3d7000 r-xp 00000000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df3d7000-7fe4df5d6000 ---p 00016000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d6000-7fe4df5d7000 r--p 00015000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d7000-7fe4df5d8000 rw-p 00016000 08:05 393693                     /lib64/libnsl-2.12.so
7fe4df5d8000-7fe4df5da000 rw-p 00000000 00:00 0 
7fe4df5da000-7fe4df764000 r-xp 00000000 08:05 393224                     /lib64/libc-2.12.so
7fe4df764000-7fe4df964000 ---p 0018a000 08:05 393224                     /lib64/libc-2.12.so
7fe4df964000-7fe4df968000 r--p 0018a000 08:05 393224                     /lib64/libc-2.12.so
7fe4df968000-7fe4df96a000 rw-p 0018e000 08:05 393224                     /lib64/libc-2.12.so
7fe4df96a000-7fe4df96e000 rw-p 00000000 00:00 0 
7fe4df96e000-7fe4df985000 r-xp 00000000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4df985000-7fe4dfb85000 ---p 00017000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb85000-7fe4dfb86000 r--p 00017000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb86000-7fe4dfb87000 rw-p 00018000 08:05 393225                     /lib64/libpthread-2.12.so
7fe4dfb87000-7fe4dfb8b000 rw-p 00000000 00:00 0 
7fe4dfb8b000-7fe4dfbb3000 r-xp 00000000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfbb3000-7fe4dfdb3000 ---p 00028000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfdb3000-7fe4dfdb5000 rw-p 00028000 08:05 393327                     /lib64/libtirpc.so.1.0.10
7fe4dfdb5000-7fe4dfdbd000 r-xp 00000000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dfdbd000-7fe4dffbd000 ---p 00008000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbd000-7fe4dffbe000 r--p 00008000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbe000-7fe4dffbf000 rw-p 00009000 08:05 393707                     /lib64/libwrap.so.0.7.6
7fe4dffbf000-7fe4dffc0000 rw-p 00000000 00:00 0 
7fe4dffc0000-7fe4dffe0000 r-xp 00000000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01c0000-7fe4e01c5000 rw-p 00000000 00:00 0 
7fe4e01dc000-7fe4e01df000 rw-p 00000000 00:00 0 
7fe4e01df000-7fe4e01e1000 r--p 0001f000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01e1000-7fe4e01e2000 rw-p 00021000 08:05 393220                     /lib64/ld-2.12.so
7fe4e01e2000-7fe4e01e3000 rw-p 00000000 00:00 0 
7fe4e01e3000-7fe4e01f0000 r-xp 00000000 08:05 1179747                    /sbin/rpcbind
7fe4e03ef000-7fe4e03f0000 rw-p 0000c000 08:05 1179747                    /sbin/rpcbind
7fe4e03f0000-7fe4e03f1000 rw-p 00000000 00:00 0 
7fe4e1a22000-7fe4e1a43000 rw-p 00000000 00:00 0                          [heap]
7ffe7d227000-7ffe7d24f000 rw-p 00000000 00:00 0                          [stack]
7ffe7d2a2000-7ffe7d2a3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

*** Bug 1461904 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1901