Rpcbind does not consider the maximum RPC data size during memory allocation for XDR strings, which allows remote attackers to cause a denial of service (memory consumption with no subsequent free) via a crafted UDP packet to port 111, aka rpcbomb. References: http://seclists.org/oss-sec/2017/q2/209 https://guidovranken.wordpress.com/2017/05/03/rpcbomb-remote-rpcbind-denial-of-service-patches/
Created libntirpc tracking bugs for this issue: Affects: epel-all [bug 1448125] Affects: fedora-all [bug 1448126] Created libtirpc tracking bugs for this issue: Affects: fedora-all [bug 1448127] Created rpcbind tracking bugs for this issue: Affects: fedora-all [bug 1448128]
The affected component is actually libtirpc / libntirpc, which fails to free the allocated buffer if XDR parsing fails. rpcbind incorrectly specifies a maximum message length of UINT_MAX rather than RPC_MAXDATASIZE, but with a fixed libtirpc this only results in a temporary oversized allocation.
Statement: In the default system configuration, with the sysctl variable vm.overcommit_memory set to either 0 (the default) or 1, an attack would take a not-insignificant amount of time to exhaust the system's memory. If vm.overcommit_memory is set to a value of 2, the time required to exhaust system memory is sufficiently reduced. It was further noticed that, a 32-bit system would have its memory exhausted faster than a 64-bit system.
(In reply to Doran Moppert from comment #6) > In the default system configuration with sysctl vm.overcommit_memory=0 (or > 1), this vulnerabilty has minimal impact on a Linux system. While the > process's reported VmSize can increase without bound, the allocated pages > are never actually touched and available system memory is not affected. 1) The exploit can easily tweaked to consume the *entire* address space of rpcbind, rendering it incapable of handling legitimate requests. 2) A similar but less efficient attack can be mounted against NFS, which is a far more serious threat since it lives in the kernel. I suspect, but have not yet been able to verify, that anyone who can run "showmount -e" against an NFS server will be able to attack it, even if they are not authorized to mount any of the filesystems it exports.
Thanks to the debian guys, they managed to fix it (partly) in sid. Patches are available: https://github.com/guidovranken/rpcbomb/
https://guidovranken.wordpress.com/2017/05/03/rpcbomb-remote-rpcbind-denial-of-service-patches/
> 1) The exploit can easily tweaked to consume the *entire* address space of > rpcbind, rendering it incapable of handling legitimate requests. Interested third party here, in this case, does it actually write to each page or just try a malloc() ? The latter will succeed as overcommit allows processes to malloc more memory than they need. Otherwise this exploit is not going to be impactful on modern systems (in the default shipping vm configuration).
(In reply to Dag-Erling Smørgrav from comment #8) > 1) The exploit can easily tweaked to consume the *entire* address space of > rpcbind, rendering it incapable of handling legitimate requests. On 32-bit systems this may be possible. Our original testing was on 64-bit architecture, where the number of messages to cause virtual address space exhaustion would exceed one billion, making such an attack infeasible. Tests with 32-bit architectures are in progress. > 2) A similar but less efficient attack can be mounted against NFS, which is > a far more serious threat since it lives in the kernel. I suspect, but have > not yet been able to verify, that anyone who can run "showmount -e" against > an NFS server will be able to attack it, even if they are not authorized to > mount any of the filesystems it exports. Indeed there is XDR parsing code present in the kernel nfs module. Our kernel security team are looking into this to determine if a similar vulnerabilty may exist. If you have further information about the exploitability of either of these vectors or anything not yet discussed in public, please contact secalert.
(In reply to Wade Mealing from comment #16) > Interested third party here, in this case, does it actually write to each > page or just try a malloc() ? The latter will succeed as overcommit allows > processes to malloc more memory than they need. > > Otherwise this exploit is not going to be impactful on modern systems (in > the default shipping vm configuration). It will still shut down rpcbind, and as mentioned above, I believe a similar (but slower) attack can be mounted against NFS itself. I am working on proof-of-concept code for both (DoS against rpcbind + kernel address space exhaustion).
Created attachment 1277059 [details] Minimal patch for libtirpc This is a minimal patch for the memory leak only, with a slightly different approach than Vranken's. The rest of Vranken's patch is mitigation and cleanup which I consider orthogonal to the bug.
Mitigation: rpcbind should be protected by iptables so that only trusted hosts that require access can reach it (eg, nfs clients). Applying per-IP rate limits in iptables will also significantly limit the impact of this attack. The default iptables rules in the system-config-firewall or firewalld package deny all remote access to rpcbind. If you elect to run your system with overcommit turned off, daemons should have memory limits enforced by the init system to ensure stability. With systemd, use directives such as LimitAS in unit files. With upstart, place ulimit commands in /etc/sysconfig/$daemon.
Created attachment 1277229 [details] svc_freeargs patch for rpcbind I think this may be a better patch for rpcbind. Per comments on the glibc mailing list (and having looked at several other users of libtirpc that don't seem to suffer such leaks), calls to xdr_* parsers should be matched with xdr_free, even in the error case. There are some convenience layers for sunrpc in particular to help with this: clnt_call/clnt_destroy and svc_getargs/svc_freeargs. rpcbind in these spots is using the latter, but can bypass svc_freeargs in the error case. I think the patch to libtirpc is still a good idea as it hardens the library against such errors. The one in comment 23 looks good to me. In more complex cases, client code calling the free/destroy/freeargs methods appropriately is even more important: I think a structure containing two xdr_strings could parse the first one correctly, then reject the second one, leaving allocated memory that only the caller can release. For a UDP protocol, the chunks will be small, but they will be a persistent leak.
(In reply to Doran Moppert from comment #26) > I think the patch to libtirpc is still a good idea as it hardens the library > against such errors. The one in comment 23 looks good to me. I think you got it a little backward. The memory leak is in libtirpc and *must* be fixed, everything else in Vranken's patch is just mitigation and *should* be fixed. If I understand correctly, your rpcbind patch fixes a separate issue where rpcbind uses the library incorrectly. There are a couple of style issues in the patch: "rc = (TRUE)" and "rc = (FALSE)" should be "rc = TRUE" and "rc = FALSE" respectively, and "return rc" should be "return (rc)" to conform to the BSD coding standards. As for mitigation: you can protect the machine, but not the service. A well executed attack will cause rpcbind to either stop responding or die. If it is restarted, it will be useless since all the services that depend on it must also be restarted in order to re-register, because the state file is only created upon clean shutdown.
(In reply to Dag-Erling Smørgrav from comment #27) > I think you got it a little backward. The memory leak is in libtirpc and > *must* be fixed, everything else in Vranken's patch is just mitigation and > *should* be fixed. If I understand correctly, your rpcbind patch fixes a > separate issue where rpcbind uses the library incorrectly. It is my understanding that the sunrpc interfaces in libtirpc expect cleanup routines to be called even if they report XDR parsing failure. I suspect an interface that gets two strings as input, where only the second is truncated, would still leak with the libtirpc patch alone. That's what my patch tries to address in rpcbind - the svc_getargs() calls are not consistently matched with svc_freeargs() in the error case. With this patch I'm no longer able to reproduce the original memory leak. To be sure, I think patching both components is a good idea. I'll defer to package maintainers more intimate with the interfaces to determine what combination of mitigation / logic fix should be applied, and we'll test carefully with variants of the attack.
Created attachment 1277755 [details] svc_freeargs patch for rpcbind
External References: https://access.redhat.com/solutions/3025811/
@Wade, what information do you need from my side (comment #16)? I just wanted to reference the externally available information. Please ask there for details.
This issue has been addressed in the following products: Red Hat Enterprise Linux 7 Via RHSA-2017:1262 https://access.redhat.com/errata/RHSA-2017:1262
This issue has been addressed in the following products: Red Hat Enterprise Linux 7 Via RHSA-2017:1263 https://access.redhat.com/errata/RHSA-2017:1263
It looks like both updates don't have their version number updated. Systems do see this as an update because the 'dist' part of the version changed (from el7 to el7_3).
This issue has been addressed in the following products: Red Hat Enterprise Linux 6 Via RHSA-2017:1267 https://access.redhat.com/errata/RHSA-2017:1267
This issue has been addressed in the following products: Red Hat Enterprise Linux 6 Via RHSA-2017:1268 https://access.redhat.com/errata/RHSA-2017:1268
The cure is worse than the disease, the updated rpcbind crashes soon after start, both el6 and el7 are affected (this breaks nis and nfs3 rpc.mountd). The solution is to revert to the old (vulnerable) version of rpcbind. https://bugzilla.redhat.com/show_bug.cgi?id=1454876 https://bugzilla.redhat.com/show_bug.cgi?id=1455142 K.O.
Unlike for Konstantin Olchanski reverting to rpcbind.x86_64 0.2.0-38.el7 and libtirpc.x86_64 0.2.4-0.8.el7 Does not fix the problem for me. Otherwise I am seeing the same issue.
rpcbind: pmap_rmtcall callit req for (100004, 2, 2, udp) from ########: rpcbind: not found *** Error in `rpcbind': free(): invalid pointer: 0x00007fff3cd8fd80 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7c503)[0x7feba03a6503] /lib64/libtirpc.so.1(xdr_bytes+0x8b)[0x7feba0b2e0ab] rpcbind(+0x672b)[0x7feba0f6572b] rpcbind(+0x422d)[0x7feba0f6322d] /lib64/libtirpc.so.1(svc_getreq_common+0x251)[0x7feba0b28511] /lib64/libtirpc.so.1(svc_getreq_poll+0x8b)[0x7feba0b286ab] rpcbind(+0x7832)[0x7feba0f66832] rpcbind(+0x3600)[0x7feba0f62600] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7feba034bb35] rpcbind(+0x3800)[0x7feba0f62800]
Another "yum downgrade" here because of rpcbind crashes. I've reverted only rpcbind, not libtirpc, and that seems to work for me. I'm on centos 6.9.
This issue has been addressed in the following products: Red Hat Gluster Storage 3.2 for RHEL 6 Red Hat Gluster Storage 3.2 for RHEL 7 Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395
same behaviour on RHEL Server 6.9 using rpcbind-0.2.0-13.el6_9.x86_64 : *** glibc detected *** rpcbind: free(): invalid pointer: 0x00007ffd6929baa0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x3aaa675dee)[0x7ff74d00adee] /lib64/libc.so.6(+0x3aaa678c3d)[0x7ff74d00dc3d] /lib64/libtirpc.so.1(xdr_bytes+0xc2)[0x7ff74d562202] rpcbind(+0x6415)[0x7ff74dba4415] rpcbind(pmap_service+0x175)[0x7ff74dba6be5] /lib64/libtirpc.so.1(svc_getreq_common+0x231)[0x7ff74d55bc41] /lib64/libtirpc.so.1(svc_getreq_poll+0x91)[0x7ff74d55bcf1] rpcbind(+0x5c1e)[0x7ff74dba3c1e] rpcbind(main+0x478)[0x7ff74dba29b8] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7ff74cfb3d1d] rpcbind(+0x3229)[0x7ff74dba1229] downgrading for now to rpcbind.x86_64 0:0.2.0-13.el6 seems to fix the issue. Thanks Dimitri.
(In reply to errata-xmlrpc from comment #47) > This issue has been addressed in the following products: > > Red Hat Gluster Storage 3.2 for RHEL 6 > Red Hat Gluster Storage 3.2 for RHEL 7 > > Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395 I'm sure this means something. I guess it means I'm writing a salt state to update yum.conf to exclude rpcbind and libtirpc so we can start updating our 6 & 7 computers again.
As mentioned in comment #43 there was an error within the fix for this CVE which caused a breakage for some customers. Customer cases are being attached and tracked for this breakage within the following bugs below. The bugs are in a Verified state and passed testing. Those fixes are being tracked as RHBA - bugfix advisories. Current bugs being tracked are: RHEL 6.9.z rpcbind: - Bug 1458240 - rpcbind crash on start [rhel-6.9.z] - https://bugzilla.redhat.com/show_bug.cgi?id=1458240 RHEL 7.3.z rpcbind: - Bug 1457172 - rpcbind crash on start [rhel-7.3.z] - https://bugzilla.redhat.com/show_bug.cgi?id=1457172 Tracking for future releases we have the 2 bugs mentioned in comment 43, which were cloned to the two above bugs: RHEL 6 - https://bugzilla.redhat.com/show_bug.cgi?id=1455142 RHEL 7 - https://bugzilla.redhat.com/show_bug.cgi?id=1454876 We will be updating the text of the previously released rpcbind RHSA's to note them as well. (In reply to Dimitri Maziuk from comment #49) > (In reply to errata-xmlrpc from comment #47) > > This issue has been addressed in the following products: > > > > Red Hat Gluster Storage 3.2 for RHEL 6 > > Red Hat Gluster Storage 3.2 for RHEL 7 > > > > Via RHSA-2017:1395 https://access.redhat.com/errata/RHSA-2017:1395 > > I'm sure this means something. I guess it means I'm writing a salt state to > update yum.conf to exclude rpcbind and libtirpc so we can start updating our > 6 & 7 computers again. Those fixed packages have been released with the Red Hat Gluster Storage products and not within the main RHEL channels, which as noted above, are being tracked and in a verified state. Regards, Cliff
Thank you Cliff, now this makes sense.
Hi to follow up from my previous comment. I'm pleased to say that this week Red Hat released RHBA's to fix the regression introduced with this CVE fix. - You should be able to apply the following RHBA's, which include this security fix: Red Hat Enterprise Linux 6.9.z https://access.redhat.com/errata/RHBA-2017:1435 ( Released for bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1458240 ) Red Hat Enterprise Linux 7.3.z https://access.redhat.com/errata/RHBA-2017:1436 ( Released for bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1457172 ) Regards, Cliff
This issue has been addressed in the following products: Red Hat Ceph Storage 2 Via RHBA-2017:1497 https://access.redhat.com/errata/RHBA-2017:1497