Acknowledgments:

Name: Qualys Research Labs

Statement:

This issue does not affect 32-bit systems as they do not have a large enough address space to exploit this flaw.

Systems with less than 32GB of memory are very unlikely to be affected by this issue due to memory demands during exploitation.

This issue does not affect the versions of Linux kernel as shipped with Red Hat Enterprise Linux 5.

This issue affects the version of the kernel packages as shipped with Red Hat Enterprise Linux 6, 7 and Red Hat Enterprise MRG 2. Future kernel updates for Red Hat Enterprise Linux 6, 7 and Red Hat Enterprise MRG 2 will address this issue.

Mitigation:

To mitigate the issue:

Enable and install kernel-debuginfo packages as per https://access.redhat.com/solutions/666123

1) On the host, save the following in a file with the ".stp" extension:

// CVE-2018-14634
//
// Theory of operations: adjust the thread's # rlimit-in-effect around
// calls to the vulnerable get_arg_page() function so as to encompass
// the newly required _STK_LIM / 4 * 3 maximum.

// Complication: the rlimit is stored in a current-> structure that
// is shared across the threads of the process.  They may concurrently
// invoke this operation.

function clamp_stack_rlim_cur:long ()
%{
  struct rlimit *rlim = current->signal->rlim;
  unsigned long rlim_cur = READ_ONCE(rlim[RLIMIT_STACK].rlim_cur);

  unsigned long limit = _STK_LIM / 4 * 3;
  limit *= 4; // multiply it back up, to the scale used by rlim_cur

  if (rlim_cur > limit) {
    WRITE_ONCE(rlim[RLIMIT_STACK].rlim_cur, limit);
    STAP_RETURN(limit);
  } else
    STAP_RETURN(0);
%}

probe kernel.function("copy_strings").call
{
  l = clamp_stack_rlim_cur()
   if (l)
     printf("lowered process %s(%d) STACK rlim_cur to %p\n",
            execname(), pid(), l)
}

probe begin {
	printf("CVE-2018-14634 mitigation loaded\n")

}

probe end {
	printf("CVE-2018-14634 mitigation unloaded\n")
}

2) Install the "systemtap" package and any required dependencies. Refer
to the "2. Using SystemTap" chapter in the Red Hat Enterprise Linux
"SystemTap Beginners Guide" document, available from docs.redhat.com,
for information on installing the required -debuginfo and matching kernel-devel packages

3) Run the "stap -g [filename-from-step-1].stp" command as root.

If the host is rebooted, the changes will be lost and the script must be
run again.


Alternatively, build the systemtap script on a development system with
"stap -g -p 4 [filename-from-step-1].stp", distribute the resulting
kernel module to all affected systems, and run "staprun -L <module>" on those.
When using this approach only systemtap-runtime package is required on
the affected systems. Please notice that the kernel version must be the same
across all systems.


This may not be a suitable workaround if your application uses massive amounts of stack space. Please consider this if there are any adverse affects when running this mitigation.

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7

Via RHSA-2018:2748 https://access.redhat.com/errata/RHSA-2018:2748

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7

Via RHSA-2018:2763 https://access.redhat.com/errata/RHSA-2018:2763

Petr, am I correct that the systemtap mitigation is racy, and can be bypassed by a more elaborate exploit that would reset rlim[RLIMIT_STACK].rlim_cur back to a high enough value from another thread of the invoking process (perhaps it'd be doing that in a busy loop)? This came up in discussions of a similar kernel patch, resulting in patches to the patch, last year. I wonder if we can come up with a simple yet non-racy systemtap mitigation. Maybe also trap setrlimit() and impose a higher limit there that's enough to prevent the attack yet unlikely to break desirable uses.

Also, I'm curious - why exactly is RHEL5 unaffected? Thanks!

Trying the SystemTap mitigation script in our EL6 x86_64 environment, it seems to set the stack limit at shown by "ulimit -s" to 30720 regardless of the initial setting in /etc/security/limits.conf where we have "unlimited".

Setting a big number instead of unlimited in /etc/security/limits.conf for stack does not seem to matter; still set to 30720 by the SystemTap script.

Before applying:

# ssh n1 ulimit -s
unlimited

# ssh n1 staprun -L CVE-2018-14634-mitigation/2.6.32-754.3.5.el6.x86_64/stap_4d64dccb8f09ef10871f7b00522c1ff7_3076.ko

# ssh n1 ulimit -s
30720

Is it supposed to be like this?

I can't speak about the intent, but the behavior Kent describes is consistent with my reading of the SystemTap mitigation script. The specific number 30720 is a result of what I think is a bug-turned-feature, in "limit *= 4; // multiply it back up, to the scale used by rlim_cur". This doesn't make sense to me in its current context (I guess this line is a leftover from prior edits of the script), but the multiplication results in the limit being not so low (would have been 3/4 of the default 10 MiB stack limit otherwise, which is more likely to break some applications and is unnecessarily strict to prevent exploitation of this specific issue). I think a forced limit of 30 MiB is sane, unless you have specific applications that require a higher limit (in which case you can edit the SystemTap script accordingly - a limit of a few hundred MiB would still mitigate the original issue).

This will not modifiy the live limit shown by ulimit command.

Gday Alexander,

I take responsibility for the systemtap script, Petr just made it public. 


> am I correct that the systemtap mitigation is racy, and can be bypassed by a more elaborate exploit that would reset rlim[RLIMIT_STACK].rlim_cur back to a high enough value from another thread of the invoking process (perhaps it'd be doing that in a busy loop)? 

You seem to be correct, I was not part of this discussion.  I had assumed that copy_strings would be the correct time, but much like most situations it is never as simple as it seems.


> This came up in discussions of a similar kernel patch, resulting in patches to
> the patch, last year. I wonder if we can come up with a simple yet non-racy 
> systemtap mitigation.

I had a few different angles, both of which appear to suffer the same race condition flaw that you've pointed out here.

I was pondering if it would be better to apply it directly to get_arg_page, but I think that will require additional investigation.    Feedback welcome on your behalf.  I think that working on this may have diminishing returns as the errata has already started shipping.


I had a reproducer that worked on el6 and el7, it did not reproduce on el5, but now you have me concerned I'm going to test it again.  Will post back with results.

Thanks, Wade.

I don't understand what you mean by "This will not modifiy the live limit shown by ulimit command." If you're referring to the effect of the SystemTap mitigation, then it's actually expected to modify that, visibly I think due to bash calling getrlimit(), and it's been reported that way in a comment above.

In last year's discussions (and in the grsecurity patch much earlier), the only reason for the rlimit to be lowered on exec was to apply that selectively to SUID/SGID exec, rather than to all execs. Since you're doing it for all execs anyway, a simple workaround and end-user can use is to lower the hard limit in /etc/security/limits.conf with a line like this:

* hard stack 30720

This will apply to user logins (e.g., via SSH), but unfortunately not to various privsep child processes of services, etc., unless and until those services are manually restarted by someone logged in as root. It might or might not apply to various non-login ways that a user can invoke a command - e.g., via .forward files - depending on whether those services start PAM sessions for those command invocations or not. Thus, a more complete fix would (also) include adding "ulimit -Hs 30720" to an early system bootup script from where all other service startup scripts are invoked, but do we still have anything like that with systemd? I'm not familiar with systemd.

And this brings us back to a possible SystemTap, where I think you could lower both the hard and the soft limit. There's no point in allowing a higher hard limit if you'd be resetting the soft limit on each exec anyway. Not to be racy against a concurrent setrlimit(), which might be past its hard limit check already, you'd need to also intercept setrlimit() calls (is there a compat version of that syscall? if so, that as well) and impose the limit on its inputs. OTOH, if someone doesn't mind rebooting with the SystemTap loaded early on at bootup, then the race wouldn't matter anymore once we're setting both the hard and the soft limit on each exec - the hard limit would get inherited by all child processes, thereby preventing a concurrent setrlimit() attack on the already booted up system.

Yes, "working on this may have diminishing returns as the errata has already started shipping." Maybe it's more important to prepare better instructions for setup of SystemTap-based mitigations for next time one is needed. How can one have a SystemTap-based mitigation re-applied on system bootup early on? A hack I once used on a RHEL6'ish system is this:

# tail -4 /etc/sysconfig/sshd
MOD=/lib/modules/`uname -r`/extra/dirtycow.ko
if [ -e $MOD ]; then
	staprun -L $MOD
fi

Perhaps Red Hat should start recommending something like this, or come up with an official way for SystemTap-based mitigations to be loaded on system bootup (e.g., have a "service" that would do that, and would be guaranteed to start up before things such as sshd, crond, MTAs, etc. - those that may invoke user-supplied programs as part of their normal functionality).

Regarding RHEL5, I know why our OpenVZ/RHEL5-based kernels on Owl are unaffected since we migrated to those back in 2010 - I lowered MAX_ARG_STRINGS and introduced an extra check into get_arg_pages() back then (no, this isn't in OpenVZ upstream, it's Owl-specific unfortunately) - but I was surprised to hear yours are unaffected either. Maybe there's something else preventing the attack on those kernels (good) or on the specific system you tested on (bad; e.g., 32 GB could somehow not be allocated in the kernel when total RAM isn't a lot higher than that).

Thanks again for your replies.

External References:

https://access.redhat.com/security/vulnerabilities/mutagen-astronomy
https://www.openwall.com/lists/oss-security/2018/09/25/4

Created attachment 1490730 [details]
extra mitigation - setrlimit race condition

Alexander, thanks, good point regarding the race condition between a concurrent setrlimit(2).  This extra snippet of stap script aims to intercept those syscalls at a lower level worker function (do_prlimit), and to impose the same limiting value on RLIMIT_STACK calls.

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6

Via RHSA-2018:2846 https://access.redhat.com/errata/RHSA-2018:2846

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6.6 Advanced Update Support
  Red Hat Enterprise Linux 6.6 Telco Extended Update Support

Via RHSA-2018:2924 https://access.redhat.com/errata/RHSA-2018:2924

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6.5 Advanced Update Support

Via RHSA-2018:2933 https://access.redhat.com/errata/RHSA-2018:2933

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6.7 Extended Update Support

Via RHSA-2018:2925 https://access.redhat.com/errata/RHSA-2018:2925

This issue has been addressed in the following products:

  Red Hat Enterprise MRG 2

Via RHSA-2018:3586 https://access.redhat.com/errata/RHSA-2018:3586

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7.4 Extended Update Support

Via RHSA-2018:3540 https://access.redhat.com/errata/RHSA-2018:3540

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7.2 Advanced Update Support
  Red Hat Enterprise Linux 7.2 Update Services for SAP Solutions
  Red Hat Enterprise Linux 7.2 Telco Extended Update Support

Via RHSA-2018:3590 https://access.redhat.com/errata/RHSA-2018:3590

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7.3 Extended Update Support

Via RHSA-2018:3591 https://access.redhat.com/errata/RHSA-2018:3591

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 6.4 Advanced Update Support

Via RHSA-2018:3643 https://access.redhat.com/errata/RHSA-2018:3643