Bug 585433 - String functions use stack allocations when they should use heap causing stack overflow with certain string sizes.
String functions use stack allocations when they should use heap causing stac...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: glibc (Show other bugs)
5.2
All Linux
low Severity medium
: rc
: ---
Assigned To: Andreas Schwab
qe-baseos-tools
:
Depends On:
Blocks: 758797
  Show dependency treegraph
 
Reported: 2010-04-23 21:03 EDT by Scott Cameron
Modified: 2016-11-24 11:14 EST (History)
6 users (show)

See Also:
Fixed In Version: glibc-2.5-69
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-21 01:32:47 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0260 normal SHIPPED_LIVE glibc bug fix update 2012-02-20 10:06:42 EST

  None (edit)
Description Scott Cameron 2010-04-23 21:03:30 EDT
Description of problem:

This problem is present in two string functions of glibc:

* strcoll_l
* strxfrm

In these functions, the code needs to choose whether to allocate its internal working buffers using the stack or the heap.  It makes this decision by calling __libc_use_alloca -- if it returns true it uses ::alloca, otherwise it uses ::malloc.

The problem is that the value it passes into __libc_use_alloca is not the number of bytes it will need to allocate, but rather a count of the number of *characters*.

The result is that, for example, if I strcoll two strings of 5 characters each, the function will do __libc_use_alloca(10) and then subsequently proceed to allocate 5 times this number on the stack.

If you have a thread stack size of 256KB and you compare 2 strings that are 30,000 characters each, the function will do __libc_use_alloca(60000), which returns true because 60,000 is under the threshold of 64KB.  Then it will alloca a total of 300,000 bytes (num_chars * (sizeof(int32_t)+1)), which will overflow the stack.

The functions should be changed to pass in the true number of bytes that will be allocated when calling __libc_use_alloca.
Comment 1 Frank Danapfel 2011-07-18 05:11:07 EDT
Hello Scott,

I'm the Onsite Engineer from Red Hat at the SAP LinuxLab. While reviewing the Bugzilla request opened by SAP in the past I found this bug reported by you.

Since this bug is already over a year old could you let us know if this is still an issue or if this does not occur any more on later versions of RHEL5?

Thanks and regards,
Frank Danapfel
Red Hat Onsite Engineer @ SAP LinuxLab
Comment 2 Scott Cameron 2011-07-18 14:02:24 EDT
Hi Frank,

I'm not in a position at this point to reproduce the problem without putting in some effort.  It's not really necessary, though, because the bug is very obvious from simply looking at the code.

If you crack open one of the functions I reference above (e.g. strcoll_l) and look for the value it passes into __libc_use_alloca.  If it is number of characters, the bug is still present.  If it is number of bytes, the bug was fixed.

Judging by the response time on this bug report that I submitted over a year ago, I'm going to take a wild guess and wager that the bug has not been fixed (and likely the surrounding code has not even been touched).

Thanks,
scott
Comment 6 errata-xmlrpc 2012-02-21 01:32:47 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0260.html

Note You need to log in before you can comment on or make changes to this bug.