Bug 585433 - String functions use stack allocations when they should use heap causing stack overflow with certain string sizes.
Summary: String functions use stack allocations when they should use heap causing stac...
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: glibc
Version: 5.2
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Andreas Schwab
QA Contact: qe-baseos-tools-bugs
Depends On:
Blocks: 758797
TreeView+ depends on / blocked
Reported: 2010-04-24 01:03 UTC by Scott Cameron
Modified: 2016-11-24 16:14 UTC (History)
6 users (show)

Fixed In Version: glibc-2.5-69
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2012-02-21 06:32:47 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0260 0 normal SHIPPED_LIVE glibc bug fix update 2012-02-20 15:06:42 UTC

Description Scott Cameron 2010-04-24 01:03:30 UTC
Description of problem:

This problem is present in two string functions of glibc:

* strcoll_l
* strxfrm

In these functions, the code needs to choose whether to allocate its internal working buffers using the stack or the heap.  It makes this decision by calling __libc_use_alloca -- if it returns true it uses ::alloca, otherwise it uses ::malloc.

The problem is that the value it passes into __libc_use_alloca is not the number of bytes it will need to allocate, but rather a count of the number of *characters*.

The result is that, for example, if I strcoll two strings of 5 characters each, the function will do __libc_use_alloca(10) and then subsequently proceed to allocate 5 times this number on the stack.

If you have a thread stack size of 256KB and you compare 2 strings that are 30,000 characters each, the function will do __libc_use_alloca(60000), which returns true because 60,000 is under the threshold of 64KB.  Then it will alloca a total of 300,000 bytes (num_chars * (sizeof(int32_t)+1)), which will overflow the stack.

The functions should be changed to pass in the true number of bytes that will be allocated when calling __libc_use_alloca.

Comment 1 Frank Danapfel 2011-07-18 09:11:07 UTC
Hello Scott,

I'm the Onsite Engineer from Red Hat at the SAP LinuxLab. While reviewing the Bugzilla request opened by SAP in the past I found this bug reported by you.

Since this bug is already over a year old could you let us know if this is still an issue or if this does not occur any more on later versions of RHEL5?

Thanks and regards,
Frank Danapfel
Red Hat Onsite Engineer @ SAP LinuxLab

Comment 2 Scott Cameron 2011-07-18 18:02:24 UTC
Hi Frank,

I'm not in a position at this point to reproduce the problem without putting in some effort.  It's not really necessary, though, because the bug is very obvious from simply looking at the code.

If you crack open one of the functions I reference above (e.g. strcoll_l) and look for the value it passes into __libc_use_alloca.  If it is number of characters, the bug is still present.  If it is number of bytes, the bug was fixed.

Judging by the response time on this bug report that I submitted over a year ago, I'm going to take a wild guess and wager that the bug has not been fixed (and likely the surrounding code has not even been touched).


Comment 6 errata-xmlrpc 2012-02-21 06:32:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.