Bug 1454876

Summary: rpcbind crash on start
Product: Red Hat Enterprise Linux 7 Reporter: Konstantin Olchanski <olchansk>
Component: rpcbindAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: Yongcheng Yang <yoyang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.3CC: ajb, andrew2.hart, calum.mackay, carsten.grohmann, cperry, cww, darren.miller, dkholia, dmoppert, dwysocha, eguan, fsorenso, gedetil, jeremiah, jiyin, jstancek, knweiss, kyle.capatosto, m.camen, michiel.dewilde, mjtrangoni, mulx, nrm, olchansk, pasik, philippe.camps, renaud.haxaire, riehecky, rmj, swhiteho, tbecker, toracat, tru, voetelink, vs, vuiis-sysadmin, xzhou, yoyang, zpytela
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rpcbind-0.2.0-42.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1455142 1457172 (view as bug list) Environment:
Last Closed: 2017-08-01 18:36:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1456511, 1455142, 1457172    

Description Konstantin Olchanski 2017-05-23 16:16:49 UTC
The latest rpcbind update is defective. rpcbind does not run at all, crashes soon after starting.

This is the update:
https://blog.linuxadmins.org/rhsa-20171262-1-important-rpcbind-security-update/
[root@iris01 ~]# rpm -q rpcbind
rpcbind-0.2.0-38.el7_3.x86_64

This is maybe the same bug in fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1450765

This is the secret bug tracking rpcbind updates:
https://bugzilla.redhat.com/show_bug.cgi?id=1449462

This is what I see when I start rpcbind: (the crash is right after connection from an el6 machine)

[root@iris01 ~]# rpcbind -d -w -f
libtirpc: debug level 1
rpcbind: local: 0 lookup routines :

rpcbind: rpcbind : my address is (null)

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for local

rpcbind: udp: 0 lookup routines :

rpcbind: rpcbind : my address is 0.0.0.0.0.111

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 2 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for udp

rpcbind: rmtcall fd for udp is 7

rpcbind: tcp: 0 lookup routines :

rpcbind: udp6: 0 lookup routines :

rpcbind: rpcbind : my address is ::.0.111

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for udp6

rpcbind: rmtcall fd for udp6 is 11

rpcbind: tcp6: 0 lookup routines :

rpcbind: debugging enabled.
rpcbind: using '/run/rpcbind/rpcbind.xdr' startup file
rpcbind: will start from scratch
rpcbind: using '/run/rpcbind/portmap.xdr' startup file
rpcbind: will start from scratch
rpcbind: pmap_rmtcall callit req for (100004, 2, 2, udp) from 142.90.103.115.236.60 : 
rpcbind: not found

*** Error in `rpcbind': free(): invalid pointer: 0x00007fff48459d90 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f78f142a503]
/lib64/libtirpc.so.1(xdr_bytes+0x8b)[0x7f78f1bb20ab]
rpcbind(+0x672b)[0x7f78f1fe972b]
rpcbind(+0x422d)[0x7f78f1fe722d]
/lib64/libtirpc.so.1(svc_getreq_common+0x251)[0x7f78f1bac511]
/lib64/libtirpc.so.1(svc_getreq_poll+0x8b)[0x7f78f1bac6ab]
rpcbind(+0x7832)[0x7f78f1fea832]
rpcbind(+0x3600)[0x7f78f1fe6600]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f78f13cfb35]
rpcbind(+0x3800)[0x7f78f1fe6800]
======= Memory map: ========
7f78e8000000-7f78e8021000 rw-p 00000000 00:00 0 

K.O.

Comment 2 Konstantin Olchanski 2017-05-23 16:20:03 UTC
I confirm that "yum downgrade" to rpcbind-0.2.0-38.el7.x86_64 resolves the problem, rpcbind runs without crash. K.O.

Comment 3 Konstantin Olchanski 2017-05-23 16:43:55 UTC
Oops, posted a bogus link to the rpcbind update notice, correct link is this:
http://rhn.redhat.com/errata/RHSA-2017-1262.html
K.O.

Comment 4 Steve Dickson 2017-05-23 17:25:25 UTC
I'm not able to reproduce this... 

What command are you using to create that call to rpcbind?

Comment 5 Konstantin Olchanski 2017-05-23 18:04:35 UTC
on my side, rpcbind crashes right out from systemd startup scripts, I tried "-d -w -f" to look at the crash.

on the remote side, I do not know now to read the 6-number "from" reported by rpcbind. the first 4 digits look like a valid IP address of a local machine running el6. the el6 machine afait, does not issue any special rpcbind calls other than the usual NFS and NIS stuff. At each invocation, the IP address is different, but in my 3 tries it was always another el6 machine.

my guess? it is an NIS broadcast, the 100004 printed by rpcbind before the crash is ypserv.

the good news? afaik, impossible to firewall rpcbind to selectively block nis broadcasts...

the bad news? I think I need rpcbind for nfsv3 mounts to work. (yes, confirmed, on machines with dead rpcbind, nfsv3 mounts hang, do not work).

K.O.

Comment 6 Michiel De Wilde 2017-05-24 08:24:33 UTC
I'm having the rpcbind crash too here, both on RHEL6 and RHEL7 after that security update to rpcbind and libtirpc. The rpcbind service crashes when ypbind tries to bind to the NIS domain.

Comment 7 Marcus Camen 2017-05-24 08:54:29 UTC
Just to confirm the issue:
At our customers site nearly every NFS server is affected.
The backtrace is the same and a rollback to rpcbind-0.2.0-38.el7.x86_64 does resolve the problem.

Comment 8 Volker Schäfer 2017-05-24 10:26:51 UTC
Same here, rpcbind dies in combination with yp after the security update to rpcbind-0.2.0-38.el7_3

Comment 9 Konstantin Olchanski 2017-05-24 14:44:34 UTC
Finally found the tracker for the original problem that the update was supposed to fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1448124
K.O.

Comment 13 Clifford Perry 2017-05-24 19:38:38 UTC
Hi all, 
This is a reminder that bugzilla is not a support tool. IF you have not yet opened a support case to allow Red Hat to correctly track this for you, please do open a ticket by logging into access.redhat.com and opening a new case with a pointer to this bugzilla and CVE-2017-8779. 

Regards,

Comment 17 Murphy Zhou 2017-05-25 07:56:31 UTC
This crash was first seen on May 16th
https://bugzilla.redhat.com/show_bug.cgi?id=1450765#c11

Comment 18 Murphy Zhou 2017-05-25 09:07:00 UTC
Note to reproduce:

Terminal 0: # rpcbind -d -w -f

Terminal 1: # rpcinfo -b 100004 2

Comment 23 Steve Dickson 2017-05-25 11:24:52 UTC
Yes, I did realize what as happening... But I was playing
around with letting the xdr routines do the allocation
but there seems to a problem there too... 

I'll test the patch out today... thanks!

Comment 24 VUIIS SysAdmin 2017-05-25 18:38:56 UTC
I am having the same issue with ypbind crashing on both clients and NIS servers. I see this in both RHEL6 and RHEL7. Downgrading to previous versions of rpcbind and libtirpc
do not solve the problem.

Comment 25 andrew2.hart 2017-05-26 10:52:52 UTC
@ VUIIS SysAdmin 
yum downgrade rpcbind 
+ restarting rpcbind and ypbind should work.
I didn't need to downgrade libtirpc.
Maybe you could run:
rpcbind -df
to see if it is the same problem you have.

Comment 26 VUIIS SysAdmin 2017-05-26 14:50:43 UTC
@ (In reply to andrew2.hart from comment #25)
> @ VUIIS SysAdmin 
> yum downgrade rpcbind 
> + restarting rpcbind and ypbind should work.
> I didn't need to downgrade libtirpc.
> Maybe you could run:
> rpcbind -df
> to see if it is the same problem you have.

Not sure what I am looking for but this is what I got:

# rpcbind -df
libtirpc: debug level 1
rpcbind: local: 0 lookup routines :

rpcbind: rpcbind : my address is (null)

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for local

rpcbind: udp: 0 lookup routines :

rpcbind: rpcbind : my address is 0.0.0.0.0.111

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 2 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for udp

rpcbind: rmtcall fd for udp is 7

rpcbind: tcp: 0 lookup routines :

rpcbind: udp6: 0 lookup routines :

rpcbind: rpcbind : my address is ::.0.111

rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 3 to the rpcbind list
rpcbind: FUNCTION rbllist_add
rpcbind: Add the prog 100000 vers 4 to the rpcbind list
rpcbind: check binding for udp6

rpcbind: rmtcall fd for udp6 is 11

rpcbind: tcp6: 0 lookup routines :

rpcbind: debugging enabled.
rpcbind: PMAP_GETPORT req for (100004, 2, udp) from 10.140.19.237.2.255 :
rpcbind: port = 0

rpcbind: PMAP_GETPORT req for (100004, 2, udp) from 10.140.19.211.3.57 :
rpcbind: port = 0

Comment 27 Frank Sorenson 2017-05-26 18:58:03 UTC
@VUIIS SysAdmin 

Just a reminder that bugzilla is not a support tool.  If you continue having problems, please open a support ticket at access.redhat.com

Frank

Comment 32 Yongcheng Yang 2017-05-31 13:22:43 UTC
Moving to VERIFIED according to the test logs of Comment #31.

Comment 33 Aymeric 2017-06-07 15:15:18 UTC
This issue also affect RHEL6.9 (at least)
If needed we have captured an UDP packet that case rcpbind to crash (just ask me).
Regards,
Aymeric

Comment 34 Frank Sorenson 2017-06-07 20:29:31 UTC
(In reply to Aymeric from comment #33)
> This issue also affect RHEL6.9 (at least)
> If needed we have captured an UDP packet that case rcpbind to crash (just
> ask me).

The reproducer is well understood.  The bugzilla tracking this issue for 6.9 is bz1458240

Comment 35 J. Bruce Fields 2017-06-09 14:14:12 UTC
*** Bug 1451651 has been marked as a duplicate of this bug. ***

Comment 36 Steve Dickson 2017-06-15 15:58:07 UTC
*** Bug 1457963 has been marked as a duplicate of this bug. ***

Comment 37 jeremiah 2017-06-20 01:15:15 UTC
Looks like only x86 was fixed but not also ARM platform. Do I need to make a new bug report for that?

Comment 38 Yongcheng Yang 2017-06-21 04:24:12 UTC
(In reply to jeremiah from comment #37)
> Looks like only x86 was fixed but not also ARM platform. Do I need to make a
> new bug report for that?

I have checked this issue fixed in rpcbind-0.2.0-42.el7.aarch64.

Would you please have a test within your ARM platform?

------------------------------------------------------
[21:18:45 root@ ~~]# service_rpcbind restart
Redirecting to /bin/systemctl restart rpcbind.service
[21:18:45 root@ ~~]# ps aux | grep [r]pcbind
rpc      10956  4.0  0.0  11712  7232 ?        Ss   21:18   0:00 /sbin/rpcbind -w
[21:18:45 root@ ~~]# rpcinfo -b 100004 2
10.12.0.163.2.150	pluto.lab.eng.rdu.redhat.com
10.12.0.161.2.250	eagle.lab.eng.rdu.redhat.com
10.12.0.158.2.124	bsod-bdc.lab.eng.rdu.redhat.com
10.12.0.159.2.139	bsod2.lab.eng.rdu.redhat.com
10.12.0.162.3.33	longhaul.lab.eng.rdu.redhat.com
10.12.0.157.3.60	bsod.lab.eng.rdu.redhat.com
10.12.0.159.2.139	bsod2.lab.eng.rdu.redhat.com
10.12.0.163.2.150	pluto.lab.eng.rdu.redhat.com
10.12.0.161.2.250	eagle.lab.eng.rdu.redhat.com
10.12.0.158.2.124	bsod-bdc.lab.eng.rdu.redhat.com
10.12.0.162.3.33	longhaul.lab.eng.rdu.redhat.com
10.12.0.157.3.60	bsod.lab.eng.rdu.redhat.com
[21:18:58 root@ ~~]# ps aux | grep [r]pcbind
rpc      10956  0.2  0.0  11712  7232 ?        Ss   21:18   0:00 /sbin/rpcbind -w

Comment 40 jeremiah 2017-06-27 22:18:56 UTC
> I have checked this issue fixed in rpcbind-0.2.0-42.el7.aarch64.
>
> Would you please have a test within your ARM platform?

Ah. Apparently this was never processed downstream to any other distribution (namely CentOS). That is, of course, not RedHat's responsibility so pardon my noise & thanks for the reply!

Comment 41 errata-xmlrpc 2017-08-01 18:36:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1992