Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 5 product line. The current stable release is 5.10. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 448011

Summary: Upgrade to RHEL5.2 - SELinux Stopping Apache Worker from Loading
Product: Red Hat Enterprise Linux 5 Reporter: Chris Stankaitis <cstankaitis>
Component: glibcAssignee: Andreas Schwab <schwab>
Status: CLOSED ERRATA QA Contact: qe-baseos-tools-bugs
Severity: urgent Docs Contact:
Priority: high    
Version: 5.2CC: cww, drepper, dwalsh, ebachalo, fweimer, jakub, jlau, ksnider, mbelangia, mfranc, mpoole, patdung100+redhat, pmuller, tao
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: glibc-2.5-64 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-21 08:25:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
strace of httpd startup
none
2nd strace with -f flag none

Description Chris Stankaitis 2008-05-22 20:57:36 UTC 
Description of problem:

Since Patching server to 5.2 I can not get apache to load because SELinux is
stopping the worker module.

Version-Release number of selected component (if applicable):

selinux-policy-2.4.6-137.el5.noarch
selinux-policy-targeted-2.4.6-137.el5.noarch
httpd-2.2.3-11.el5_1.3.x86_64

Actual results:


May 22 16:50:22 static1 kernel: audit(1211489422.302:90): avc:  denied  {
execmem } for  pid=5243 comm="httpd.worker" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:httpd_t:s0 tclass
=process
May 22 16:50:22 static1 kernel: audit(1211489422.304:91): avc:  denied  {
execmem } for  pid=5245 comm="httpd.worker" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:httpd_t:s0 tclass
=process
May 22 16:50:22 static1 kernel: audit(1211489422.306:92): avc:  denied  {
execmem } for  pid=5247 comm="httpd.worker" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:httpd_t:s0 tclass
=process
May 22 16:50:22 static1 kernel: audit(1211489422.313:93): avc:  denied  {
execmem } for  pid=5249 comm="httpd.worker" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:httpd_t:s0 tclass
=process


Additional info:

System has been rebooted.... I have done a full /.autorelabel to try and fix the
issue.
Comment 1 Chris Stankaitis 2008-05-22 21:04:35 UTC 
BTW also:

# getsebool allow_execmem
allow_execmem --> on
Comment 2 Chris Stankaitis 2008-05-22 21:09:32 UTC 
# ll -lZ /usr/sbin/httpd.worker
-rwxr-xr-x  root root system_u:object_r:httpd_exec_t   /usr/sbin/httpd.worker
Comment 3 Daniel Walsh 2008-05-23 16:09:06 UTC 
allow_execmem only applies to unconfined domains.

httpd.worker should not require execmem  this is a bug in httpd.worker or some
library that it is using.

http://people.redhat.com/~drepper/selinux-mem.html

Do you have some special modules loaded?
Comment 4 Chris Stankaitis 2008-05-23 22:30:43 UTC 
The following modules are loaded up by that server's apache.  My First though
would have been mem_cache... but I commented it out and tried to fire apache
back up and it would not.  The list of modules below has not changed from our
5.1 boxes we have 7 other servers @ 5.1 that are in the same class as this one,
and have not had to do any special SELinux under 5.1


LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule headers_module modules/mod_headers.so
LoadModule mime_module modules/mod_mime.so
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule cache_module modules/mod_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
Comment 5 Ulrich Drepper 2008-05-23 23:27:08 UTC 
The execmem error is generated by mmap calls.  Try to determine the command line
of httpd.worker and then replicate it under strace.  Use

  strace -i -o LOG /usr/sbin/httpd.worker ...the args...

In the LOG file you then look at the mmap calls which fail.  Once you found the
call you hopefully are able through the context able to determine which module
is responsible.

I don't know whether it makes sense ahead of time to look with
  nm -D --undef
at all the DSOs and determine where there are callers to mmap.  This is not
entrire conclusive since the use might be indirect.
Comment 6 Daniel Walsh 2008-05-24 09:36:36 UTC 
I am moving this to httpd since this is not an SELinux bug and potentially it is
a httpd bug.
Comment 7 Chris Stankaitis 2008-05-26 13:25:36 UTC 
I did an strace of /etc/init.d/httpd start and have attached it.. if you want
more specific command line args... please provide me with an exact command line
to execute.
Comment 8 Chris Stankaitis 2008-05-26 13:26:18 UTC 
Created attachment 306674 [details]
strace of httpd startup
Comment 9 Ulrich Drepper 2008-05-26 16:10:33 UTC 
(In reply to comment #7)
> I did an strace of /etc/init.d/httpd start and have attached it.. if you want
> more specific command line args... please provide me with an exact command line
> to execute.

What you traced is only the startup script itself, not the httpd.worker binary.
 You'll have to add the -f option to strace.  This could lead to a very large
file.  Consider compressing it using bzip2 or even 7za before attaching it.
Comment 10 Chris Stankaitis 2008-05-26 18:37:00 UTC 
Created attachment 306695 [details]
2nd strace with -f flag

I ran the strace with the -f flag as recommended in the above comment.	here is
the attachment.   The process actually never finished.. I had to kill -9 it
from another window after I had let the strace run for > 30 mins.
Comment 11 Joe Orton 2008-05-27 10:28:32 UTC 
Nothing in the standard set of httpd modules should be failing like this; I
can't reproduce the problem here on a 5.2 installation.

Your strace output does not show an mmap failure here either.  Does it work if
you use prefork rather than worker?
Comment 12 Chris Stankaitis 2008-05-27 14:43:14 UTC 
yes if I turn off the worker option in /etc/syconfig/httpd and use prefork I am
able to start up the web server.  These servers process a lot of hits and we
really do need to use the worker mode, so I am still not able to put this server
back into production until we get to the bottom of the worker issue.
Comment 13 Chris Stankaitis 2008-05-27 14:44:28 UTC 
I am not sure if this could have cause strace to not show you info that would
have been important to your troubleshooting but when I was running the 2nd
strace above the following AVC's were thrown many many times.

May 26 14:32:38 static1 kernel: audit(1211826758.215:213897): avc:  denied  {
signal } for  pid=8853 comm="strace" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:unconfined_t:s0 
tclass=process
May 26 14:32:43 static1 kernel: printk: 672 messages suppressed.
May 26 14:32:43 static1 kernel: audit(1211826763.215:214122): avc:  denied  {
signal } for  pid=8853 comm="strace" scontext=user_u:system_r:httpd_t:s0
tcontext=user_u:system_r:unconfined_t:s0 
tclass=process
Comment 14 Joe Orton 2008-05-27 14:55:29 UTC 
I suggest you open a support ticket for this so that support can help diagnose
this further, especially if it is urgent.
Comment 15 Daniel Walsh 2008-05-27 18:31:03 UTC 
BTW you can temporarily allow http to do execmem by using 

# grep execmem /var/log/audit/audit.log | audit2allow -M myhttp
# semodule -i myhttp.pp

This should only be temporary until we get to the bottom of the problem.
Comment 16 Daniel Walsh 2008-05-27 19:14:22 UTC 
Also run the strace in permissive mode.
Comment 17 Chris Stankaitis 2008-05-27 19:29:38 UTC 
I have found the smoking gun... our Worker MPM config section...

<IfModule worker.c>
ThreadLimit        100
ServerLimit         60
StartServers         5
MaxClients        6000
MinSpareThreads    199
MaxSpareThreads    499
ThreadsPerChild    100
MaxRequestsPerChild  50000
</IfModule>


The directive that is causing all this is the "ThreadsPerChild"  if I comment
that out... and ONLY that single line Apache will start in worker mode fine.

I have replicated this same issue on 3 servers (one production, and two dev) and
the ThreadsPerChild config directive is the issue.

I now put the ball in your court.... Why, and how can this be fixed.
Comment 18 Ulrich Drepper 2008-05-27 19:51:58 UTC 
(In reply to comment #17)
> The directive that is causing all this is the "ThreadsPerChild"  if I comment
> that out... and ONLY that single line Apache will start in worker mode fine.

If the default is to have one thread only then the problem is with the stack
allocation for all the other 99 threads.  Executable stacks must be avoided at
all costs but in this case at least one DSO or the program itself requests an
executable stack.

It's easy enough to verify: look at all the DSOs which are loaded and run

  eu-readelf -l BINARY | fgrep GNU_STACK

on them.  The second to  last field should always be RW.  If it is RWX somewhere
you found the culprit.
Comment 19 Chris Stankaitis 2008-05-27 20:03:23 UTC 

Starting httpd: WARNING: MaxClients of 6000 would require 240 servers,
 and would exceed the ServerLimit value of 60.
 Automatically lowering MaxClients to 1500.  To increase,
 please see the ServerLimit directive.
Comment 20 Chris Stankaitis 2008-05-28 16:46:56 UTC 
still waiting on an answer as to why "ThreadsPerChild" is causing Httpd.worker
to throw and SELinux AVC and stopping it from loading.
Comment 21 Ulrich Drepper 2008-05-28 17:02:45 UTC 
(In reply to comment #20)
> still waiting on an answer as to why "ThreadsPerChild" is causing Httpd.worker
> to throw and SELinux AVC and stopping it from loading.

What are you talking about?  Read comment #18.  You have to provide information
before anything else can happen.
Comment 22 Chris Stankaitis 2008-05-28 19:40:21 UTC 
might I ask, how do I tell what DSOs are loaded?
Comment 23 Ulrich Drepper 2008-05-28 21:11:23 UTC 
(In reply to comment #22)
> might I ask, how do I tell what DSOs are loaded?

Just look at /proc/<PID>/maps with <PID> being replaced by the PID of the process.
Comment 24 Chris Stankaitis 2008-05-29 13:34:11 UTC 
here is the output of my cat /proc and the pid for the main httpd.worker parent


/usr/sbin/httpd.worker
/dev/zero
/usr/sbin/httpd.worker
/lib64/ld-2.5.so
/lib64/ld-2.5.so
/lib64/ld-2.5.so
/lib64/libm-2.5.so
/lib64/libm-2.5.so
/lib64/libm-2.5.so
/lib64/libm-2.5.so
/lib64/libpcre.so.0.0.1
/lib64/libpcre.so.0.0.1
/lib64/libpcre.so.0.0.1
/lib64/libselinux.so.1
/lib64/libselinux.so.1
/lib64/libselinux.so.1
/usr/lib64/libaprutil-1.so.0.2.7
/usr/lib64/libaprutil-1.so.0.2.7
/usr/lib64/libaprutil-1.so.0.2.7
/lib64/libcrypt-2.5.so
/lib64/libcrypt-2.5.so
/lib64/libcrypt-2.5.so
/lib64/libcrypt-2.5.so
/usr/lib64/libldap-2.3.so.0.2.15
/usr/lib64/libldap-2.3.so.0.2.15
/usr/lib64/libldap-2.3.so.0.2.15
/usr/lib64/liblber-2.3.so.0.2.15
/usr/lib64/liblber-2.3.so.0.2.15
/usr/lib64/liblber-2.3.so.0.2.15
/lib64/libdb-4.3.so
/lib64/libdb-4.3.so
/lib64/libdb-4.3.so
/lib64/libexpat.so.0.5.0
/lib64/libexpat.so.0.5.0
/lib64/libexpat.so.0.5.0
/usr/lib64/libapr-1.so.0.2.7
/usr/lib64/libapr-1.so.0.2.7
/usr/lib64/libapr-1.so.0.2.7
/lib64/libpthread-2.5.so
/lib64/libpthread-2.5.so
/lib64/libpthread-2.5.so
/lib64/libpthread-2.5.so
/lib64/libdl-2.5.so
/lib64/libdl-2.5.so
/lib64/libdl-2.5.so
/lib64/libdl-2.5.so
/lib64/libc-2.5.so
/lib64/libc-2.5.so
/lib64/libc-2.5.so
/lib64/libc-2.5.so
/lib64/libsepol.so.1
/lib64/libsepol.so.1
/lib64/libsepol.so.1
/lib64/libuuid.so.1.2
/lib64/libuuid.so.1.2
/lib64/libuuid.so.1.2
/usr/lib64/libpq.so.4.1
/usr/lib64/libpq.so.4.1
/usr/lib64/libpq.so.4.1
/usr/lib64/libsqlite3.so.0.8.6
/usr/lib64/libsqlite3.so.0.8.6
/usr/lib64/libsqlite3.so.0.8.6
/lib64/libresolv-2.5.so
/lib64/libresolv-2.5.so
/lib64/libresolv-2.5.so
/lib64/libresolv-2.5.so
/usr/lib64/libsasl2.so.2.0.22
/usr/lib64/libsasl2.so.2.0.22
/usr/lib64/libsasl2.so.2.0.22
/lib64/libssl.so.0.9.8b
/lib64/libssl.so.0.9.8b
/lib64/libssl.so.0.9.8b
/lib64/libcrypto.so.0.9.8b
/lib64/libcrypto.so.0.9.8b
/lib64/libcrypto.so.0.9.8b
/usr/lib64/libkrb5.so.3.3
/usr/lib64/libkrb5.so.3.3
/usr/lib64/libkrb5.so.3.3
/lib64/libnsl-2.5.so
/lib64/libnsl-2.5.so
/lib64/libnsl-2.5.so
/lib64/libnsl-2.5.so
/usr/lib64/libgssapi_krb5.so.2.2
/usr/lib64/libgssapi_krb5.so.2.2
/usr/lib64/libgssapi_krb5.so.2.2
/lib64/libcom_err.so.2.1
/lib64/libcom_err.so.2.1
/lib64/libcom_err.so.2.1
/usr/lib64/libk5crypto.so.3.1
/usr/lib64/libk5crypto.so.3.1
/usr/lib64/libk5crypto.so.3.1
/usr/lib64/libz.so.1.2.3
/usr/lib64/libz.so.1.2.3
/usr/lib64/libz.so.1.2.3
/usr/lib64/libkrb5support.so.0.1
/usr/lib64/libkrb5support.so.0.1
/usr/lib64/libkrb5support.so.0.1
/lib64/libkeyutils-1.2.so
/lib64/libkeyutils-1.2.so
/lib64/libkeyutils-1.2.so
/usr/lib64/httpd/modules/mod_authn_default.so
/usr/lib64/httpd/modules/mod_authn_default.so
/usr/lib64/httpd/modules/mod_authn_default.so
/usr/lib64/httpd/modules/mod_authz_host.so
/usr/lib64/httpd/modules/mod_authz_host.so
/usr/lib64/httpd/modules/mod_authz_host.so
/usr/lib64/httpd/modules/mod_authz_default.so
/usr/lib64/httpd/modules/mod_authz_default.so
/usr/lib64/httpd/modules/mod_authz_default.so
/usr/lib64/httpd/modules/mod_log_config.so
/usr/lib64/httpd/modules/mod_log_config.so
/usr/lib64/httpd/modules/mod_log_config.so
/usr/lib64/httpd/modules/mod_headers.so
/usr/lib64/httpd/modules/mod_headers.so
/usr/lib64/httpd/modules/mod_headers.so
/usr/lib64/httpd/modules/mod_mime.so
/usr/lib64/httpd/modules/mod_mime.so
/usr/lib64/httpd/modules/mod_mime.so
/usr/lib64/httpd/modules/mod_dir.so
/usr/lib64/httpd/modules/mod_dir.so
/usr/lib64/httpd/modules/mod_dir.so
/usr/lib64/httpd/modules/mod_alias.so
/usr/lib64/httpd/modules/mod_alias.so
/usr/lib64/httpd/modules/mod_alias.so
/usr/lib64/httpd/modules/mod_rewrite.so
/usr/lib64/httpd/modules/mod_rewrite.so
/usr/lib64/httpd/modules/mod_rewrite.so
/usr/lib64/httpd/modules/mod_proxy.so
/usr/lib64/httpd/modules/mod_proxy.so
/usr/lib64/httpd/modules/mod_proxy.so
/usr/lib64/httpd/modules/mod_proxy_http.so
/usr/lib64/httpd/modules/mod_proxy_http.so
/usr/lib64/httpd/modules/mod_proxy_http.so
/usr/lib64/httpd/modules/mod_cache.so
/usr/lib64/httpd/modules/mod_cache.so
/usr/lib64/httpd/modules/mod_cache.so
/usr/lib64/httpd/modules/mod_mem_cache.so
/usr/lib64/httpd/modules/mod_mem_cache.so
/usr/lib64/httpd/modules/mod_mem_cache.so
/lib64/libnss_files-2.5.so
/lib64/libnss_files-2.5.so
/lib64/libnss_files-2.5.so
/lib64/libnss_files-2.5.so


and here are those same dso's run through the  eu-readelf -l BINARY | fgrep
GNU_STACK command..

for i in `cat dsolist.txt` ; do echo "$i - "; eu-readelf -l $i | fgrep GNU_STACK
; done

/usr/sbin/httpd.worker - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/dev/zero - 
eu-readelf: input file is empty
/usr/sbin/httpd.worker - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/ld-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/ld-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/ld-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libm-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libm-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libm-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libm-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpcre.so.0.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpcre.so.0.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpcre.so.0.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libselinux.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libselinux.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libselinux.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libaprutil-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libaprutil-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libaprutil-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypt-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypt-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypt-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypt-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libldap-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libldap-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libldap-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/liblber-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/liblber-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/liblber-2.3.so.0.2.15 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdb-4.3.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdb-4.3.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdb-4.3.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libexpat.so.0.5.0 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libexpat.so.0.5.0 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libexpat.so.0.5.0 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libapr-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libapr-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libapr-1.so.0.2.7 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpthread-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpthread-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpthread-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libpthread-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libdl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libc-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libc-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libc-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libc-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libsepol.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libsepol.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libsepol.so.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libuuid.so.1.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libuuid.so.1.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libuuid.so.1.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libpq.so.4.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libpq.so.4.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libpq.so.4.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsqlite3.so.0.8.6 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsqlite3.so.0.8.6 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsqlite3.so.0.8.6 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libresolv-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libresolv-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libresolv-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libresolv-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsasl2.so.2.0.22 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsasl2.so.2.0.22 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libsasl2.so.2.0.22 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libssl.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libssl.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libssl.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypto.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypto.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcrypto.so.0.9.8b - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5.so.3.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5.so.3.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5.so.3.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnsl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnsl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnsl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnsl-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libgssapi_krb5.so.2.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libgssapi_krb5.so.2.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libgssapi_krb5.so.2.2 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcom_err.so.2.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcom_err.so.2.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libcom_err.so.2.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libk5crypto.so.3.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libk5crypto.so.3.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libk5crypto.so.3.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libz.so.1.2.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libz.so.1.2.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libz.so.1.2.3 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5support.so.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5support.so.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/libkrb5support.so.0.1 - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libkeyutils-1.2.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libkeyutils-1.2.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libkeyutils-1.2.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authn_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authn_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authn_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_host.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_host.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_host.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_authz_default.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_log_config.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_log_config.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_log_config.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_headers.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_headers.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_headers.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mime.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mime.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mime.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_dir.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_dir.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_dir.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_alias.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_alias.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_alias.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_rewrite.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_rewrite.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_rewrite.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy_http.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy_http.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_proxy_http.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mem_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mem_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/usr/lib64/httpd/modules/mod_mem_cache.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnss_files-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnss_files-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnss_files-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
/lib64/libnss_files-2.5.so - 
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8
Comment 25 Chris Stankaitis 2008-05-29 13:35:25 UTC 
As you can see from the above, if I followed your instructions properly.. there
are no RWX DSO's
Comment 26 Ken Snider 2008-05-30 17:54:01 UTC 
This is now attached to RH service ticket 1831132.
Comment 27 Joe Orton 2008-06-05 11:50:20 UTC 
Thanks.  The error_log information implies pthread_create() is failing:

[Thu May 29 14:44:19 2008] [alert] (13)Permission denied: apr_thread_create:
unable to create worker thread

So this is an x86_64 xen guest.  This has regressed between 5.1 and 5.2 in
exactly the same OS install, kernel, httpd configuration?

Can you reproducing this when booted into the 2.6.18-53.1.21.el5xen for purposes
of elimination?
Comment 28 Chris Stankaitis 2008-06-05 13:01:39 UTC 
Xen kernel vs. normal kernel has nothing to do with it... I can reproduce this
on kernel-2.6.18-92.el5.

Our previous kernel - kernel-2.6.18-53.1.4.el5 did not have this problem.
Comment 29 Ken Snider 2008-06-05 15:15:06 UTC 
To Clarify:

The Xen Guest isn't the only server we've seen this behaviour with - we've seen
it with our normal (non-xen) servers as well, all running, at minimum, 5.1, a
few servers with newer kernels as well.
Comment 31 Chris Stankaitis 2008-06-13 12:40:11 UTC 
It's been over a week, could we please get an update on this issue?
Comment 32 Ulrich Drepper 2008-06-13 14:21:36 UTC 
The problem has been referred to our support engineers to reproduce it etc who
haven't picked it up yet.

Anyway, Try  one thing: take the libpthread.so file from F9 (or just replace the
entire glibc ).  Only do this on a sacrificial machine, of course.

There has been one bug I've never seen in the wild which I fixed in glibc.  The
patch might not have been propagated to RHEL5 yet.  It caused libpthread to
incorrectly use the PROC_EXEC permissions for the stack to be used in certain
situations.  Esp with many threads and potentially large amounts of data this
could very well be it.
Comment 35 Joe Orton 2008-07-02 15:29:27 UTC 
Have you been able to try Ulrich's suggestion to use an F9 libpthread.so/glibc?
Comment 36 Chris Stankaitis 2008-07-31 20:57:18 UTC 
we do not have the extra servers or time to destroy a box in the way suggested.
 These are core production servers, we've built a custom SELinux module to work
around this issue for now, however we really do need RH to fix this problem in
an errata going forward.
Comment 37 Ulrich Drepper 2008-07-31 21:46:17 UTC 
(In reply to comment #36)
> we do not have the extra servers or time to destroy a box in the way suggested.
>  These are core production servers, we've built a custom SELinux module to work
> around this issue for now, however we really do need RH to fix this problem in
> an errata going forward.

If you cannot test any code the best that can happen is that our guess is right
and the bug is fixed thusly.  If you want to have assurance have your RH account
manager open a bug.
Comment 42 RHEL Program Management 2009-10-23 11:15:46 UTC 
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.
Comment 51 Joe Orton 2010-04-06 12:48:47 UTC 
I would presume the trigger is the mmap(, PROT_EXEC|PROT_WRITE system call failing with EACCESS.  That failure is contemporary with an mmap failure due to ENOMEM, which could be a significant coincidence, but may be irrelevant.

5026  mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0 <unfinished ...>
5027  <... clone resumed> child_stack=0x7ecd51d0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, paren
t_tidptr=0x7ecd59d0, tls=0x7ecd5940, child_tidptr=0x7ecd59d0) = 5223
5026  <... mmap resumed> )              = -1 ENOMEM (Cannot allocate memory)
5027  mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0 <unfinished ...>
5026  mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
5027  <... mmap resumed> )              = 0x7ecd6000
5026  <... mmap resumed> )              = -1 EACCES (Permission denied)
5027  mprotect(0x7ecd6000, 4096, PROT_NONE <unfinished ...>
5026  write(10, "[Mon Mar 29 23:41:20 2010] [aler"..., 108 <unfinished ...>
5027  <... mprotect resumed> )          = 0

There is a lot of mod_python stuff going on in that trace.  There are known issues where the Python ctypes extension uses libffi and can use mmap PROT_WRITE|PROT_EXEC in a similar way to the above, though I don't believe this affects the version of Python in RHEL5.  Does this go away if mod_python is removed?

Using systemtap or gdb to get a stack trace from the failing mmap call would be useful.
Comment 63 RHEL Program Management 2010-08-09 18:25:35 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 65 RHEL Program Management 2011-01-11 20:36:42 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 66 RHEL Program Management 2011-01-11 22:37:19 UTC 
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 67 Joe Orton 2011-02-04 15:55:34 UTC 
To summarize the current analysis of this issue, I hope this is all correct:

a) You can configure httpd such that thread stack allocation hits a memory limit. 

b) In this failure case, an SELinux failure is being triggered due to glibc's use of mmap with PROT_EXEC.

c) On upgrade from RHEL 5.1 to 5.2 some previously-working httpd configurations now hit the memory limit and fail as in (a) and (b).

Point (a) is expected behaviour.  ThreadStackSize can be used as a specific work around to reduce stack sizes, or simply using fewer threads.

Point (b) I am unclear about whether the glibc folks are saying this is expected behaviour or not, perhaps they can clarify.

Point (c) is certainly unfortunate but it is not clear whether there is much we can  about it.  If a configuration is right at the "tipping point" of memory usage, some unrelated change (anywhere; kernel, libc, whatever) could cause the transition to failure; only if we could guarantee never to increase memory usage by one byte could would guarantee to avoid that transition for a working configuration.
Comment 68 John Lau 2011-02-07 03:37:12 UTC 
I agree that (c) is unfortunate, and since it can be workaround by kernel parameter I think it is acceptable. 

I hope that someone (probably glibc folks) can further investigate (b) because  it is quite confusing because a memory overflow issue would disguise as a security violation.
Comment 76 errata-xmlrpc 2011-07-21 08:25:44 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1034.html