Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 585433

Summary:	String functions use stack allocations when they should use heap causing stack overflow with certain string sizes.
Product:	Red Hat Enterprise Linux 5	Reporter:	Scott Cameron <scott.cameron>
Component:	glibc	Assignee:	Andreas Schwab <schwab>
Status:	CLOSED ERRATA	QA Contact:	qe-baseos-tools-bugs
Severity:	medium	Docs Contact:
Priority:	low
Version:	5.2	CC:	fche, fdanapfe, fweimer, mfranc, patrickm, pmuller
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	glibc-2.5-69	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-02-21 06:32:47 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	758797

Description Scott Cameron 2010-04-24 01:03:30 UTC

Description of problem:

This problem is present in two string functions of glibc:

* strcoll_l
* strxfrm

In these functions, the code needs to choose whether to allocate its internal working buffers using the stack or the heap.  It makes this decision by calling __libc_use_alloca -- if it returns true it uses ::alloca, otherwise it uses ::malloc.

The problem is that the value it passes into __libc_use_alloca is not the number of bytes it will need to allocate, but rather a count of the number of *characters*.

The result is that, for example, if I strcoll two strings of 5 characters each, the function will do __libc_use_alloca(10) and then subsequently proceed to allocate 5 times this number on the stack.

If you have a thread stack size of 256KB and you compare 2 strings that are 30,000 characters each, the function will do __libc_use_alloca(60000), which returns true because 60,000 is under the threshold of 64KB.  Then it will alloca a total of 300,000 bytes (num_chars * (sizeof(int32_t)+1)), which will overflow the stack.

The functions should be changed to pass in the true number of bytes that will be allocated when calling __libc_use_alloca.

Comment 1 Frank Danapfel 2011-07-18 09:11:07 UTC

Hello Scott,

I'm the Onsite Engineer from Red Hat at the SAP LinuxLab. While reviewing the Bugzilla request opened by SAP in the past I found this bug reported by you.

Since this bug is already over a year old could you let us know if this is still an issue or if this does not occur any more on later versions of RHEL5?

Thanks and regards,
Frank Danapfel
Red Hat Onsite Engineer @ SAP LinuxLab

Comment 2 Scott Cameron 2011-07-18 18:02:24 UTC

Hi Frank,

I'm not in a position at this point to reproduce the problem without putting in some effort.  It's not really necessary, though, because the bug is very obvious from simply looking at the code.

If you crack open one of the functions I reference above (e.g. strcoll_l) and look for the value it passes into __libc_use_alloca.  If it is number of characters, the bug is still present.  If it is number of bytes, the bug was fixed.

Judging by the response time on this bug report that I submitted over a year ago, I'm going to take a wild guess and wager that the bug has not been fixed (and likely the surrounding code has not even been touched).

Thanks,
scott

Comment 6 errata-xmlrpc 2012-02-21 06:32:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0260.html