Bug 1448124 - (CVE-2017-8779) CVE-2017-8779 rpcbind, libtirpc, libntirpc: Memory leak when failing to parse XDR strings or bytearrays
CVE-2017-8779 rpcbind, libtirpc, libntirpc: Memory leak when failing to parse...
Status: CLOSED ERRATA
Product: Security Response
Classification: Other
Component: vulnerability (Show other bugs)
unspecified
All Linux
high Severity high
: ---
: ---
Assigned To: Red Hat Product Security
impact=important,public=20170503,repo...
: Security
Depends On: 1448125 1448126 1449456 1449460 1449463 1449464 1448127 1448128 1448862 1449310 1449458 1449459 1449461 1449462
Blocks: 1448130
  Show dependency treegraph
 
Reported: 2017-05-04 11:03 EDT by Andrej Nemec
Modified: 2017-07-04 13:21 EDT (History)
36 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
It was found that due to the way rpcbind uses libtirpc (libntirpc), a memory leak can occur when parsing specially crafted XDR messages. An attacker sending thousands of messages to rpcbind could cause its memory usage to grow without bound, eventually causing it to be terminated by the OOM killer.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-16 10:53:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Minimal patch for libtirpc (1.21 KB, patch)
2017-05-08 04:58 EDT, Dag-Erling Smørgrav
no flags Details | Diff
svc_freeargs patch for rpcbind (2.85 KB, patch)
2017-05-09 03:16 EDT, Doran Moppert
no flags Details | Diff
svc_freeargs patch for rpcbind (4.61 KB, application/mbox)
2017-05-11 02:14 EDT, Doran Moppert
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Debian BTS 861836 None None None 2017-05-06 10:36 EDT

  None (edit)
Description Andrej Nemec 2017-05-04 11:03:14 EDT
Rpcbind does not consider the maximum RPC data size during memory allocation for XDR strings, which allows remote attackers to cause a denial of service (memory consumption with no subsequent free) via a crafted UDP packet to port 111, aka rpcbomb.

References:

http://seclists.org/oss-sec/2017/q2/209
https://guidovranken.wordpress.com/2017/05/03/rpcbomb-remote-rpcbind-denial-of-service-patches/
Comment 1 Andrej Nemec 2017-05-04 11:03:37 EDT
Created libntirpc tracking bugs for this issue:

Affects: epel-all [bug 1448125]
Affects: fedora-all [bug 1448126]


Created libtirpc tracking bugs for this issue:

Affects: fedora-all [bug 1448127]


Created rpcbind tracking bugs for this issue:

Affects: fedora-all [bug 1448128]
Comment 3 Doran Moppert 2017-05-04 22:37:01 EDT
The affected component is actually libtirpc / libntirpc, which fails to free the allocated buffer if XDR parsing fails.  rpcbind incorrectly specifies a maximum message length of UINT_MAX rather than RPC_MAXDATASIZE, but with a fixed libtirpc this only results in a temporary oversized allocation.
Comment 6 Doran Moppert 2017-05-04 22:59:11 EDT
Statement:

In the default system configuration, with the sysctl variable vm.overcommit_memory set to either 0 (the default) or 1, an attack would take a not-insignificant amount of time to exhaust the system's memory. If vm.overcommit_memory is set to a value of 2, the time required to exhaust system memory is sufficiently reduced. It was further noticed that, a 32-bit system would have its memory exhausted faster than a 64-bit system.
Comment 8 Dag-Erling Smørgrav 2017-05-05 10:56:05 EDT
(In reply to Doran Moppert from comment #6)
> In the default system configuration with sysctl vm.overcommit_memory=0 (or
> 1), this vulnerabilty has minimal impact on a Linux system. While the
> process's reported VmSize can increase without bound, the allocated pages
> are never actually touched and available system memory is not affected.

1) The exploit can easily tweaked to consume the *entire* address space of rpcbind, rendering it incapable of handling legitimate requests.

2) A similar but less efficient attack can be mounted against NFS, which is a far more serious threat since it lives in the kernel.  I suspect, but have not yet been able to verify, that anyone who can run "showmount -e" against an NFS server will be able to attack it, even if they are not authorized to mount any of the filesystems it exports.
Comment 14 Raphael Groner 2017-05-06 10:32:58 EDT
Thanks to the debian guys, they managed to fix it (partly) in sid. Patches are available: https://github.com/guidovranken/rpcbomb/
Comment 16 Wade Mealing 2017-05-07 21:53:19 EDT
> 1) The exploit can easily tweaked to consume the *entire* address space of 
> rpcbind, rendering it incapable of handling legitimate requests.

Interested third party here, in this case, does it actually write to each page or just try a malloc() ?  The latter will succeed as overcommit allows processes to malloc more memory than they need.  

Otherwise this exploit is not going to be impactful on modern systems (in the default shipping vm configuration).
Comment 17 Doran Moppert 2017-05-07 23:56:32 EDT
(In reply to Dag-Erling Smørgrav from comment #8)
> 1) The exploit can easily tweaked to consume the *entire* address space of
> rpcbind, rendering it incapable of handling legitimate requests.

On 32-bit systems this may be possible.  Our original testing was on 64-bit architecture, where the number of messages to cause virtual address space exhaustion would exceed one billion, making such an attack infeasible.  Tests with 32-bit architectures are in progress.

> 2) A similar but less efficient attack can be mounted against NFS, which is
> a far more serious threat since it lives in the kernel.  I suspect, but have
> not yet been able to verify, that anyone who can run "showmount -e" against
> an NFS server will be able to attack it, even if they are not authorized to
> mount any of the filesystems it exports.

Indeed there is XDR parsing code present in the kernel nfs module.  Our kernel security team are looking into this to determine if a similar vulnerabilty may exist.

If you have further information about the exploitability of either of these vectors or anything not yet discussed in public, please contact secalert@redhat.com.
Comment 22 Dag-Erling Smørgrav 2017-05-08 04:54:37 EDT
(In reply to Wade Mealing from comment #16)
> Interested third party here, in this case, does it actually write to each
> page or just try a malloc() ?  The latter will succeed as overcommit allows
> processes to malloc more memory than they need.  
> 
> Otherwise this exploit is not going to be impactful on modern systems (in
> the default shipping vm configuration).

It will still shut down rpcbind, and as mentioned above, I believe a similar (but slower) attack can be mounted against NFS itself.  I am working on proof-of-concept code for both (DoS against rpcbind + kernel address space exhaustion).
Comment 23 Dag-Erling Smørgrav 2017-05-08 04:58 EDT
Created attachment 1277059 [details]
Minimal patch for libtirpc

This is a minimal patch for the memory leak only, with a slightly different approach than Vranken's.  The rest of Vranken's patch is mitigation and cleanup which I consider orthogonal to the bug.
Comment 25 Doran Moppert 2017-05-08 20:22:38 EDT
Mitigation:

rpcbind should be protected by iptables so that only trusted hosts that require access can reach it (eg, nfs clients).  Applying per-IP rate limits in iptables will also significantly limit the impact of this attack.  The default iptables rules in the system-config-firewall or firewalld package deny all remote access to rpcbind.

If you elect to run your system with overcommit turned off, daemons should have memory limits enforced by the init system to ensure stability.  With systemd, use directives such as LimitAS in unit files.  With upstart, place ulimit commands in /etc/sysconfig/$daemon.
Comment 26 Doran Moppert 2017-05-09 03:16 EDT
Created attachment 1277229 [details]
svc_freeargs patch for rpcbind

I think this may be a better patch for rpcbind.  Per comments on the glibc mailing list (and having looked at several other users of libtirpc that don't seem to suffer such leaks), calls to xdr_* parsers should be matched with xdr_free, even in the error case.  There are some convenience layers for sunrpc in particular to help with this:  clnt_call/clnt_destroy and svc_getargs/svc_freeargs.  rpcbind in these spots is using the latter, but can bypass svc_freeargs in the error case.

I think the patch to libtirpc is still a good idea as it hardens the library against such errors.  The one in comment 23 looks good to me.

In more complex cases, client code calling the free/destroy/freeargs methods appropriately is even more important:  I think a structure containing two xdr_strings could parse the first one correctly, then reject the second one, leaving allocated memory that only the caller can release.  For a UDP protocol, the chunks will be small, but they will be a persistent leak.
Comment 27 Dag-Erling Smørgrav 2017-05-09 05:46:05 EDT
(In reply to Doran Moppert from comment #26)
> I think the patch to libtirpc is still a good idea as it hardens the library
> against such errors.  The one in comment 23 looks good to me.

I think you got it a little backward.  The memory leak is in libtirpc and *must* be fixed, everything else in Vranken's patch is just mitigation and *should* be fixed.  If I understand correctly, your rpcbind patch fixes a separate issue where rpcbind uses the library incorrectly.

There are a couple of style issues in the patch:  "rc = (TRUE)" and "rc = (FALSE)" should be "rc = TRUE" and "rc = FALSE" respectively, and "return rc" should be "return (rc)" to conform to the BSD coding standards.

As for mitigation: you can protect the machine, but not the service.  A well executed attack will cause rpcbind to either stop responding or die.  If it is restarted, it will be useless since all the services that depend on it must also be restarted in order to re-register, because the state file is only created upon clean shutdown.
Comment 32 Doran Moppert 2017-05-10 03:20:06 EDT
(In reply to Dag-Erling Smørgrav from comment #27)
> I think you got it a little backward.  The memory leak is in libtirpc and
> *must* be fixed, everything else in Vranken's patch is just mitigation and
> *should* be fixed.  If I understand correctly, your rpcbind patch fixes a
> separate issue where rpcbind uses the library incorrectly.

It is my understanding that the sunrpc interfaces in libtirpc expect cleanup routines to be called even if they report XDR parsing failure.  I suspect an interface that gets two strings as input, where only the second is truncated, would still leak with the libtirpc patch alone.

That's what my patch tries to address in rpcbind - the svc_getargs() calls are not consistently matched with svc_freeargs() in the error case.  With this patch I'm no longer able to reproduce the original memory leak.

To be sure, I think patching both components is a good idea.  I'll defer to package maintainers more intimate with the interfaces to determine what combination of mitigation / logic fix should be applied, and we'll test carefully with variants of the attack.
Comment 33 Doran Moppert 2017-05-11 02:14 EDT
Created attachment 1277755 [details]
svc_freeargs patch for rpcbind
Comment 35 Doran Moppert 2017-05-11 21:00:14 EDT
External References:

https://access.redhat.com/solutions/3025811/
Comment 36 Raphael Groner 2017-05-15 13:37:36 EDT
@Wade, what information do you need from my side (comment #16)? I just wanted to reference the externally available information. Please ask there for details.
Comment 38 errata-xmlrpc 2017-05-21 22:52:48 EDT
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7

Via RHSA-2017:1262 https://access.redhat.com/errata/RHSA-2017:1262
Comment 39 errata-xmlrpc 2017-05-21 23:43:05 EDT
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7

Via RHSA-2017:1263 https://access.redhat.com/errata/RHSA-2017:1263
Comment 40 jacco 2017-05-22 18:49:26 EDT
It looks like both updates don't have their version number updated. Systems do see this as an update because the 'dist' part of the version changed (from el7 to el7_3).
Comment 41 errata-xmlrpc 2017-05-23 03:38:44 EDT
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6

Via RHSA-2017:1267 https://access.redhat.com/errata/RHSA-2017:1267
Comment 42 errata-xmlrpc 2017-05-23 04:39:13 EDT
This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6

Via RHSA-2017:1268 https://access.redhat.com/errata/RHSA-2017:1268
Comment 43 Konstantin Olchanski 2017-05-24 10:41:49 EDT
The cure is worse than the disease, the updated rpcbind crashes soon after start, both el6 and el7 are affected (this breaks nis and nfs3 rpc.mountd). The solution is to revert to the old (vulnerable) version of rpcbind.

https://bugzilla.redhat.com/show_bug.cgi?id=1454876
https://bugzilla.redhat.com/show_bug.cgi?id=1455142

K.O.
Comment 44 VUIIS SysAdmin 2017-05-24 13:20:58 EDT
Unlike for Konstantin Olchanski 

reverting to
 rpcbind.x86_64 0.2.0-38.el7
and 
 libtirpc.x86_64 0.2.4-0.8.el7

Does not fix the problem for me.
Otherwise I am seeing the same issue.
Comment 45 andrew2.hart 2017-05-25 07:21:41 EDT
rpcbind: pmap_rmtcall callit req for (100004, 2, 2, udp) from ########: 
rpcbind: not found

*** Error in `rpcbind': free(): invalid pointer: 0x00007fff3cd8fd80 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7feba03a6503]
/lib64/libtirpc.so.1(xdr_bytes+0x8b)[0x7feba0b2e0ab]
rpcbind(+0x672b)[0x7feba0f6572b]
rpcbind(+0x422d)[0x7feba0f6322d]
/lib64/libtirpc.so.1(svc_getreq_common+0x251)[0x7feba0b28511]
/lib64/libtirpc.so.1(svc_getreq_poll+0x8b)[0x7feba0b286ab]
rpcbind(+0x7832)[0x7feba0f66832]
rpcbind(+0x3600)[0x7feba0f62600]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7feba034bb35]
rpcbind(+0x3800)[0x7feba0f62800]
Comment 46 Dimitri Maziuk 2017-05-26 18:21:11 EDT
Another "yum downgrade" here because of rpcbind crashes. I've reverted only rpcbind, not libtirpc, and that seems to work for me.
I'm on centos 6.9.
Comment 47 errata-xmlrpc 2017-06-06 03:58:08 EDT
This issue has been addressed in the following products:

  Red Hat Gluster Storage 3.2 for RHEL 6
  Red Hat Gluster Storage 3.2 for RHEL 7

Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395
Comment 48 Damien REYT 2017-06-06 10:46:16 EDT
same behaviour on RHEL Server 6.9 using rpcbind-0.2.0-13.el6_9.x86_64 :

*** glibc detected *** rpcbind: free(): invalid pointer: 0x00007ffd6929baa0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x3aaa675dee)[0x7ff74d00adee]
/lib64/libc.so.6(+0x3aaa678c3d)[0x7ff74d00dc3d]
/lib64/libtirpc.so.1(xdr_bytes+0xc2)[0x7ff74d562202]
rpcbind(+0x6415)[0x7ff74dba4415]
rpcbind(pmap_service+0x175)[0x7ff74dba6be5]
/lib64/libtirpc.so.1(svc_getreq_common+0x231)[0x7ff74d55bc41]
/lib64/libtirpc.so.1(svc_getreq_poll+0x91)[0x7ff74d55bcf1]
rpcbind(+0x5c1e)[0x7ff74dba3c1e]
rpcbind(main+0x478)[0x7ff74dba29b8]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7ff74cfb3d1d]
rpcbind(+0x3229)[0x7ff74dba1229]

downgrading for now to rpcbind.x86_64 0:0.2.0-13.el6 seems to fix the issue.

Thanks Dimitri.
Comment 49 Dimitri Maziuk 2017-06-06 10:48:12 EDT
(In reply to errata-xmlrpc from comment #47)
> This issue has been addressed in the following products:
> 
>   Red Hat Gluster Storage 3.2 for RHEL 6
>   Red Hat Gluster Storage 3.2 for RHEL 7
> 
> Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395

I'm sure this means something. I guess it means I'm writing a salt state to update yum.conf to exclude rpcbind and libtirpc so we can start updating our 6 & 7 computers again.
Comment 51 Clifford Perry 2017-06-06 11:55:56 EDT
As mentioned in comment #43 there was an error within the fix for this CVE which caused a breakage for some customers. Customer cases are being attached and tracked for this breakage within the following bugs below. The bugs are in a Verified state and passed testing. Those fixes are being tracked as RHBA - bugfix advisories. 

Current bugs being tracked are: 

RHEL 6.9.z rpcbind: 
 - Bug 1458240 - rpcbind crash on start [rhel-6.9.z]
 - https://bugzilla.redhat.com/show_bug.cgi?id=1458240

RHEL 7.3.z rpcbind: 
 - Bug 1457172 - rpcbind crash on start [rhel-7.3.z] 
 - https://bugzilla.redhat.com/show_bug.cgi?id=1457172

Tracking for future releases we have the 2 bugs mentioned in comment 43, which were cloned to the two above bugs: 

RHEL 6 - https://bugzilla.redhat.com/show_bug.cgi?id=1455142
RHEL 7 - https://bugzilla.redhat.com/show_bug.cgi?id=1454876

We will be updating the text of the previously released rpcbind RHSA's to note them as well. 

(In reply to Dimitri Maziuk from comment #49)
> (In reply to errata-xmlrpc from comment #47)
> > This issue has been addressed in the following products:
> > 
> >   Red Hat Gluster Storage 3.2 for RHEL 6
> >   Red Hat Gluster Storage 3.2 for RHEL 7
> > 
> > Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395
> 
> I'm sure this means something. I guess it means I'm writing a salt state to
> update yum.conf to exclude rpcbind and libtirpc so we can start updating our
> 6 & 7 computers again.

Those fixed packages have been released with the Red Hat Gluster Storage products and not within the main RHEL channels, which as noted above, are being tracked and in a verified state. 

Regards,
Cliff
Comment 52 Dimitri Maziuk 2017-06-06 12:52:05 EDT
Thank you Cliff, now this makes sense.
Comment 53 Clifford Perry 2017-06-16 09:17:13 EDT
Hi to follow up from my previous comment. I'm pleased to say that this week Red Hat released RHBA's to fix the regression introduced with this CVE fix. 
 - You should be able to apply the following RHBA's, which include this security fix: 

Red Hat Enterprise Linux 6.9.z
 https://access.redhat.com/errata/RHBA-2017:1435 
 ( Released for bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1458240 )

Red Hat Enterprise Linux 7.3.z
 https://access.redhat.com/errata/RHBA-2017:1436
 ( Released for bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1457172 )

Regards,
Cliff
Comment 54 Siddharth Sharma 2017-07-04 13:21:47 EDT
This issue has been addressed in the following products:

  Red Hat Ceph Storage 2

Via RHBA-2017:1497 https://access.redhat.com/errata/RHBA-2017:1497

Note You need to log in before you can comment on or make changes to this bug.