Red Hat Bugzilla – Bug 82142
Apache leaking file descriptors to cgi-bin programs.
Last modified: 2007-04-18 12:50:07 EDT
Description of problem: Apache appears to be leaking some pretty crucial file descriptors to programs invoked via mod_cgi. Version-Release number of selected component (if applicable): httpd-2.0.40-11 How reproducible: Always. Steps to Reproduce: 1. Compile and install the following program in cgi-bin: #include <stdio.h> #include <unistd.h> int main() { char buffer[256]; printf("Content-Type: text/plain\n\n"); fflush(stdout); sprintf(buffer, "ls -al /proc/%d/fd", getpid()); system(buffer); return (0); } 2. Use the browser to run this cgi-bin application. Actual results: total 0 dr-x------ 2 apache apache 0 Jan 17 21:42 . dr-xr-xr-x 3 apache apache 0 Jan 17 21:42 .. lr-x------ 1 apache apache 64 Jan 17 21:42 0 -> pipe:[236258] l-wx------ 1 apache apache 64 Jan 17 21:42 1 -> pipe:[236259] l-wx------ 1 apache apache 64 Jan 17 21:42 10 -> /var/log/httpd/ssl_access_log l-wx------ 1 apache apache 64 Jan 17 21:42 11 -> /var/log/httpd/ssl_request_log l-wx------ 1 apache apache 64 Jan 17 21:42 2 -> pipe:[236260] lr-x------ 1 apache apache 64 Jan 17 21:42 5 -> pipe:[236171] l-wx------ 1 apache apache 64 Jan 17 21:42 6 -> pipe:[236171] lrwx------ 1 apache apache 64 Jan 17 21:42 7 -> /var/log/httpd/error_log lrwx------ 1 apache apache 64 Jan 17 21:42 8 -> /var/log/httpd/ssl_error_log l-wx------ 1 apache apache 64 Jan 17 21:42 9 -> /var/log/httpd/access_log Expected results: Should not be seeing these file descriptors. Additional info: I added the following code to main(): FILE *fp=fdopen(8, "w+"); if (fp) { fprintf(fp, "rosebud\n"); fflush(fp); printf("Wrote it\n"); } else { printf("Couldn't open it\n"); } This worked, and the cgi-bin program, invoked as uid nobody, scribbled all over a log file that only root is supposed to be able to write (the file descriptor was opened for writing before httpd dropped root).
This bug is very bad. I have published much information on the web about it since October. It has been the subject of a vuln-dev thread located here: http://marc.theaimsgroup.com/?l=vuln-dev&m=104585997219471&w=2 With access to these descriptors, you can find out who else shares the machine with you, get an idea what the directory structure is, read your neighbor's log files, delete your neighbor's log files, or write bad entries in your neighbor's log files. This presents a severe problem since programs should never be able to write to the access logs. This renders log analysis useless. Also, there have been log analysis programs that have been vulnerable to various buffer overflows, too. The combination of being able to write to log files & vulnerable log analysis programs is a real problem. Next we have the pipe descriptors, these have been traced by Bjoern A. Zeeb to be from server/mpm_common.c:ap_mpm_pod_open:401. This is known as the pipe of death. With mpm=worker or mpm=threadpool one may send a graceful or restart via them to some of the processes/threads (tested on linux). This allows anyone with a cgi account on the machine to start playing games with the apache threads. As far as I can see, there is nothing a common admin can do to protect their system from this if they give any kind of scripting access to their clients. Chroot Jails or Sandboxes will not solve the problem since the file descriptors are inheritted open. Now let's take alook from another angle, you have probably heard of code injection vulnerabilities. For example, look at the one for cutenews on Bugtraq, http://marc.theaimsgroup.com/?l=bugtraq&m=104619051400775&w=2. The cutenews problem does not stand alone. There are many PHP programs that could have this problem and programs in other scripting languages could have this problem. By using a code injection exploit, any server running apache 2x could have its logs viewed/deleted/modified or pipes written to. Writing directly to logs also circumvents the CAN-2003-0020 fixups, too. Christian Kratzer has opened the following bug report with apache: <http://nagoya.apache.org/bugzilla/show_bug.cgi?id=17206> In it he identifies the problem with the leaked descriptors as being a typo: *_inherit_set() and *_inherit_unset() The following is a patch that we (myself, Christian, & Bjoern) are testing for this bug: ---------------------------------------------------------- diff -ur httpd-2.0.40/modules/loggers/mod_log_config.c httpd- 2.0.40a/modules/loggers/mod_log_config.c --- httpd-2.0.40/modules/loggers/mod_log_config.c 2002-07-17 12:08:53.000000000 -0400 +++ httpd-2.0.40a/modules/loggers/mod_log_config.c 2003-03-02 09:23:41.000000000 -0500 @@ -1112,7 +1112,7 @@ "could not open transfer log file %s.", fname); exit(1); } - apr_file_inherit_set(cls->log_fd); + apr_file_inherit_unset(cls->log_fd); } #ifdef BUFFERED_LOGS cls->outcnt = 0; diff -ur httpd-2.0.40/server/log.c httpd-2.0.40a/server/log.c --- httpd-2.0.40/server/log.c 2002-07-10 02:01:11.000000000 -0400 +++ httpd-2.0.40a/server/log.c 2003-03-02 09:24:09.000000000 -0500 @@ -312,7 +312,7 @@ exit(1); } - apr_file_inherit_set(s->error_log); + apr_file_inherit_unset(s->error_log); } } diff -ur httpd-2.0.40/srclib/apr/file_io/unix/pipe.c httpd- 2.0.40a/srclib/apr/file_io/unix/pipe.c --- httpd-2.0.40/srclib/apr/file_io/unix/pipe.c 2002-07-11 18:15:26.000000000 - 0400 +++ httpd-2.0.40a/srclib/apr/file_io/unix/pipe.c 2003-03-02 09:27:18.000000000 -0500 @@ -202,9 +202,9 @@ #endif apr_pool_cleanup_register((*in)->pool, (void *)(*in), apr_unix_file_cleanup, - apr_pool_cleanup_null); + apr_unix_file_cleanup); apr_pool_cleanup_register((*out)->pool, (void *)(*out), apr_unix_file_cleanup, - apr_pool_cleanup_null); + apr_unix_file_cleanup); return APR_SUCCESS; } It would be good to get some feedback by the maintainers on this patch. -Steve Grubb
Many thanks for sending in the analysis and patch - I'm currently looking into this further; we'll keep you updated with progress.
I'm a bit worried about changing the pipe cleanups inside APR, I'd rather just do this in Apache. Removing the inherit_set lines is equivalent to calling _unset, but faster. I'll attach the patch I'm testing.
Created attachment 90474 [details] fd leak fixes
My patch omitted the log.c change by mistake, and doesn't cure the pod leak.
Please DON'T do a vendor fix for this. work it out together with the apache/apr security team which already has been notified.
We always try to work with upstream maintainers on security fixes. Current version of my patch below.
Created attachment 90478 [details] fd leak fixes #2
My initial testing shows that mod_cgi is fixed by the above patch. However, mod_php still leaks. Try this: <html><head> <title>PHP Test</title></head> <body><?php $cmd = "/bin/ls -l /proc/$$/fd"; exec($cmd, $dir_listing, $status); foreach($dir_listing as $item) { echo $item; echo "<br>"; } ?> </body></html> Remember, *everything* is leaking. Is the apr the right place to fix it?
Yes, this is just fixing the first class of bug, where pool cleanups should and can be handling leaks correctly. The fact that fd's are leaked to processes fork+exec'd from embedded scripting languages is a separate class of bug, really, where, as you, say, everything leaks. I'm continuing my analysis of this second class of bug. A simpler version of your above is just <?php system("/bin/ls /proc/$$/fd"); ?>. PHP's "safe mode", so far as it goes, mitigates this type of issue by allowing script authors less trust.
Created attachment 90492 [details] fd leaks (against 2.0 HEAD)
(That's what I submitted upstream)
>I'm continuing my analysis of this second class of bug. So, is the leaking fd's into mod_php & other scripting languages going to be fixed under this bug report or is there another one that it will be fixed under? Do you see the fix for this one yet? Just wondering...
It occurred to me that it's actually rather pointless fixing any of these bugs: if you allow CGI script authors to run arbitrary code from CGI scripts, they can arrange for the httpd server children to run arbitrary code via ptrace().
... but should those child processes also be able to scribble over root-owned log files?
Yes, the Apache httpd children can write over any root-owned logs file - they have open fd's for all error logs, that's by design. (Ideally, since the logs were opened with O_APPEND by root, then httpd could only *append* to the files, but O_APPEND can be cleared by fcntl, unfortunately. )
Wrong attitude. You are doing good so far. It is important to fix these for several reasons, each open descriptor consumes some memory in the machine, files cannot actually be closed unless all programs close their reference to the file, and this circumvents chroot jails. I cannot believe that this is by design. It is a mistake and has been corrected in the 1.3 branch...it is only the 2.0 branch that currently has this problem. (I can show you vuln-dev mailing from a couple years ago reporting that 1.3.14 had similar problems.) How is it intuitive that descriptor 14 should be used to write to my error log? stderr should be the way or using my own logfile. I can lock down cgi programs with chroot jails to where they don't have access to system resources, but it is terribly inefficient to call child processes through an intermediate program that sanitizes the environment. Its better to continue fixing the problem. Let me also offer this scenario...PHP code injection + writing to descriptors + Inverse Logfile Lookup exploits (http://marc.theaimsgroup.com/? l=bugtraq&m=104680367329779&w=2) or Terminal escape sequences or the file exploit + system admin taking a look at the log file or the report from his account. This does not involve a level of trust or ptrace, it can start from PHP code injection and get much worse from there.
Further more this might be a problem with resource limits when too many file descriptors are open. Me using a nice patch for suexec that limits these on FreeBSD. What I am reading in #14,16 from you btw could be a quotation from a private mail from wrowe...
To clarify, when I wrote "httpd server children" I am talking about the httpd processes themselves, not scripts fork+execed from them. Yes, by design, these httpd processes must have all log files open. I agree completely that by design, CGI scripts should just have three pipes on fds 0, 1, 2, and the current 2.0.40 behaviour is a bug. My point is that merely fixing these fd leaks does not improve security if you don't trusted your CGI script authors, since a malicious CGI script can use ptrace() to run arbitrary code in an httpd process, thereby gaining access to all the log file fds and more. If you allow PHP scripts to exec/system/etc arbitrary binaries on your system, then the same applies. PHP safe mode mitigates this somewhat, as I've said above.
A forthcoming erratum is going to fix the fd leaks addressed in the attached patches. I've opened an RFE at bug 88182 to track fd leaks which are caused by not setting CLOEXEC, since that is really a separate issue. Thanks for persisting with this issue.
An errata has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2003-139.html