Bug 82142 - Apache leaking file descriptors to cgi-bin programs.
Summary: Apache leaking file descriptors to cgi-bin programs.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: httpd
Version: 8.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Joe Orton
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-01-18 02:50 UTC by Sam Varshavchik
Modified: 2007-04-18 16:50 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2003-04-09 16:31:49 UTC
Embargoed:


Attachments (Terms of Use)
fd leak fixes (1.19 KB, patch)
2003-03-05 11:04 UTC, Joe Orton
no flags Details | Diff
fd leak fixes #2 (2.29 KB, patch)
2003-03-05 16:44 UTC, Joe Orton
no flags Details | Diff
fd leaks (against 2.0 HEAD) (2.68 KB, patch)
2003-03-06 15:47 UTC, Joe Orton
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2003:139 0 high SHIPPED_LIVE : Updated httpd packages fix security vulnerabilities. 2003-04-09 04:00:00 UTC

Description Sam Varshavchik 2003-01-18 02:50:21 UTC
Description of problem:

Apache appears to be leaking some pretty crucial file descriptors to programs
invoked via mod_cgi.

Version-Release number of selected component (if applicable):

httpd-2.0.40-11

How reproducible:

Always.

Steps to Reproduce:

1. Compile and install the following program in cgi-bin:

#include <stdio.h>
#include <unistd.h>

int main()
{
        char buffer[256];

        printf("Content-Type: text/plain\n\n");
        fflush(stdout);

        sprintf(buffer, "ls -al /proc/%d/fd", getpid());
        system(buffer);
        return (0);
}

2. Use the browser to run this cgi-bin application.

Actual results:

total 0
dr-x------    2 apache   apache          0 Jan 17 21:42 .
dr-xr-xr-x    3 apache   apache          0 Jan 17 21:42 ..
lr-x------    1 apache   apache         64 Jan 17 21:42 0 -> pipe:[236258]
l-wx------    1 apache   apache         64 Jan 17 21:42 1 -> pipe:[236259]
l-wx------    1 apache   apache         64 Jan 17 21:42 10 ->
/var/log/httpd/ssl_access_log
l-wx------    1 apache   apache         64 Jan 17 21:42 11 ->
/var/log/httpd/ssl_request_log
l-wx------    1 apache   apache         64 Jan 17 21:42 2 -> pipe:[236260]
lr-x------    1 apache   apache         64 Jan 17 21:42 5 -> pipe:[236171]
l-wx------    1 apache   apache         64 Jan 17 21:42 6 -> pipe:[236171]
lrwx------    1 apache   apache         64 Jan 17 21:42 7 ->
/var/log/httpd/error_log
lrwx------    1 apache   apache         64 Jan 17 21:42 8 ->
/var/log/httpd/ssl_error_log
l-wx------    1 apache   apache         64 Jan 17 21:42 9 ->
/var/log/httpd/access_log


Expected results:

Should not be seeing these file descriptors.

Additional info:

I added the following code to main():

     FILE *fp=fdopen(8, "w+");

     if (fp)
     {
           fprintf(fp, "rosebud\n");
           fflush(fp);
           printf("Wrote it\n");
     }
     else
     {
           printf("Couldn't open it\n");
     }

This worked, and the cgi-bin program, invoked as uid nobody, scribbled all over
a log file that only root is supposed to be able to write (the file descriptor
was opened for writing before httpd dropped root).

Comment 1 Steve Grubb 2003-03-03 17:19:29 UTC
This bug is very bad. I have published much information on the web about it 
since October. It has been the subject of a vuln-dev thread located here: 
http://marc.theaimsgroup.com/?l=vuln-dev&m=104585997219471&w=2

With access to these descriptors, you can find out who else shares the machine 
with you, get an idea what the directory structure is, read your neighbor's log 
files, delete your neighbor's log files, or write bad entries in your
neighbor's log files. This presents a severe problem since programs should 
never be able to write to the access logs. This renders log analysis useless. 
Also, there have been log analysis programs that have been vulnerable to various
buffer overflows, too. The combination of being able to write to log files & 
vulnerable log analysis programs is a real problem.

Next we have the pipe descriptors, these have been traced by Bjoern A. Zeeb to 
be from server/mpm_common.c:ap_mpm_pod_open:401. This is known as the pipe of 
death. With mpm=worker or mpm=threadpool one may send a graceful or restart via 
them to some of the processes/threads (tested on linux). This allows anyone
with a cgi account on the machine to start playing games with the apache 
threads.

As far as I can see, there is nothing a common admin can do to protect their 
system from this if they give any kind of scripting access to their clients. 
Chroot Jails or Sandboxes will not solve the problem since the file
descriptors are inheritted open.

Now let's take alook from another angle, you have probably heard of code 
injection vulnerabilities. For example, look at the one for cutenews on Bugtraq,
http://marc.theaimsgroup.com/?l=bugtraq&m=104619051400775&w=2. The cutenews 
problem does not stand alone. There are many PHP programs that could have this 
problem and programs in other scripting languages could have this problem. By 
using a code injection exploit, any server running apache 2x could have its 
logs viewed/deleted/modified or pipes written to.

Writing directly to logs also circumvents the CAN-2003-0020 fixups, too. 

Christian Kratzer has opened the following bug report with apache:
<http://nagoya.apache.org/bugzilla/show_bug.cgi?id=17206>  In it he identifies 
the problem with the leaked descriptors as being a typo: *_inherit_set() and 
*_inherit_unset()

The following is a patch that we (myself, Christian, & Bjoern) are testing for 
this bug:
----------------------------------------------------------
diff -ur httpd-2.0.40/modules/loggers/mod_log_config.c httpd-
2.0.40a/modules/loggers/mod_log_config.c
--- httpd-2.0.40/modules/loggers/mod_log_config.c	2002-07-17 
12:08:53.000000000 -0400
+++ httpd-2.0.40a/modules/loggers/mod_log_config.c	2003-03-02 
09:23:41.000000000 -0500
@@ -1112,7 +1112,7 @@
                          "could not open transfer log file %s.", fname);
             exit(1);
         }
-        apr_file_inherit_set(cls->log_fd);
+        apr_file_inherit_unset(cls->log_fd);
     }
 #ifdef BUFFERED_LOGS
     cls->outcnt = 0;
diff -ur httpd-2.0.40/server/log.c httpd-2.0.40a/server/log.c
--- httpd-2.0.40/server/log.c	2002-07-10 02:01:11.000000000 -0400
+++ httpd-2.0.40a/server/log.c	2003-03-02 09:24:09.000000000 -0500
@@ -312,7 +312,7 @@
             exit(1);
         }
 
-        apr_file_inherit_set(s->error_log);
+        apr_file_inherit_unset(s->error_log);
     }
 }
 
diff -ur httpd-2.0.40/srclib/apr/file_io/unix/pipe.c httpd-
2.0.40a/srclib/apr/file_io/unix/pipe.c
--- httpd-2.0.40/srclib/apr/file_io/unix/pipe.c	2002-07-11 18:15:26.000000000 -
0400
+++ httpd-2.0.40a/srclib/apr/file_io/unix/pipe.c	2003-03-02 
09:27:18.000000000 -0500
@@ -202,9 +202,9 @@
 #endif
 
     apr_pool_cleanup_register((*in)->pool, (void *)(*in), 
apr_unix_file_cleanup,
-                         apr_pool_cleanup_null);
+                         apr_unix_file_cleanup);
     apr_pool_cleanup_register((*out)->pool, (void *)(*out), 
apr_unix_file_cleanup,
-                         apr_pool_cleanup_null);
+                         apr_unix_file_cleanup);
     return APR_SUCCESS;
 }
 

It would be good to get some feedback by the maintainers on this patch.

-Steve Grubb

Comment 2 Joe Orton 2003-03-03 17:24:27 UTC
Many thanks for sending in the analysis and patch - I'm currently looking into
this further; we'll keep you updated with progress.

Comment 3 Joe Orton 2003-03-05 11:04:03 UTC
I'm a bit worried about changing the pipe cleanups inside APR, I'd rather just
do this in Apache. Removing the inherit_set lines is equivalent to calling
_unset, but faster.  I'll attach the patch I'm testing.

Comment 4 Joe Orton 2003-03-05 11:04:55 UTC
Created attachment 90474 [details]
fd leak fixes

Comment 5 Joe Orton 2003-03-05 11:23:53 UTC
My patch omitted the log.c change by mistake, and doesn't cure the pod leak.



Comment 6 Bjoern A. Zeeb 2003-03-05 14:43:07 UTC
Please DON'T do a vendor fix for this. work it out together with the apache/apr
security team which already has been notified.

Comment 7 Joe Orton 2003-03-05 16:43:39 UTC
We always try to work with upstream maintainers on security fixes. Current
version of my patch below.

Comment 8 Joe Orton 2003-03-05 16:44:57 UTC
Created attachment 90478 [details]
fd leak fixes #2

Comment 9 Steve Grubb 2003-03-05 18:40:54 UTC
My initial testing shows that mod_cgi is fixed by the above patch. However, 
mod_php still leaks. Try this:

<html><head> <title>PHP Test</title></head>
<body><?php
   $cmd = "/bin/ls -l /proc/$$/fd";
   exec($cmd, $dir_listing, $status);
   foreach($dir_listing as $item) {
      echo $item;
      echo "<br>";
   }
?> </body></html>

Remember, *everything* is leaking. Is the apr the right place to fix it?

Comment 10 Joe Orton 2003-03-05 19:48:11 UTC
Yes, this is just fixing the first class of bug, where pool cleanups should and
can be handling leaks correctly.

The fact that fd's are leaked to processes fork+exec'd from embedded scripting
languages is a separate class of bug, really, where, as you, say, everything
leaks. I'm continuing my analysis of this second class of bug.

A simpler version of your above is just <?php system("/bin/ls /proc/$$/fd"); ?>.
 PHP's "safe mode", so far as it goes, mitigates this type of issue by allowing
script authors less trust.


Comment 11 Joe Orton 2003-03-06 15:47:03 UTC
Created attachment 90492 [details]
fd leaks (against 2.0 HEAD)

Comment 12 Joe Orton 2003-03-06 15:55:02 UTC
(That's what I submitted upstream)

Comment 13 Steve Grubb 2003-03-11 21:36:01 UTC
>I'm continuing my analysis of this second class of bug.

So, is the leaking fd's into mod_php & other scripting languages going to be 
fixed under this bug report or is there another one that it will be fixed 
under? Do you see the fix for this one yet? Just wondering...


Comment 14 Joe Orton 2003-03-13 12:37:28 UTC
It occurred to me that it's actually rather pointless fixing any of these bugs:
if you allow CGI script authors to run arbitrary code from CGI scripts, they can
arrange for the httpd server children to run arbitrary code via ptrace().

Comment 15 Sam Varshavchik 2003-03-13 12:54:10 UTC
... but should those child processes also be able to scribble over root-owned
log files?


Comment 16 Joe Orton 2003-03-13 13:31:57 UTC
Yes, the Apache httpd children can write over any root-owned logs file - they
have open fd's for all error logs, that's by design.

(Ideally, since the logs were opened with O_APPEND by root, then httpd could
only *append* to the files, but O_APPEND can be cleared by fcntl, unfortunately. )

Comment 17 Steve Grubb 2003-03-13 14:56:59 UTC
Wrong attitude. You are doing good so far. It is important to fix these for 
several reasons, each open descriptor consumes some memory in the machine, 
files cannot actually be closed unless all programs close their reference to 
the file, and this circumvents chroot jails.

I cannot believe that this is by design. It is a mistake and has been corrected 
in the 1.3 branch...it is only the 2.0 branch that currently has this problem. 
(I can show you vuln-dev mailing from a couple years ago reporting that 1.3.14 
had similar problems.) How is it intuitive that descriptor 14 should be used to 
write to my error log? stderr should be the way or using my own logfile.

I can lock down cgi programs with chroot jails to where they don't have access 
to system resources, but it is terribly inefficient to call child processes 
through an intermediate program that sanitizes the environment. Its better to 
continue fixing the problem.

Let me also offer this scenario...PHP code injection + writing to descriptors + 
Inverse Logfile Lookup exploits (http://marc.theaimsgroup.com/?
l=bugtraq&m=104680367329779&w=2) or Terminal escape sequences or the file 
exploit + system admin taking a look at the log file or the report from his 
account.

This does not involve a level of trust or ptrace, it can start from PHP code 
injection and get much worse from there.

Comment 18 Bjoern A. Zeeb 2003-03-13 16:06:14 UTC
Further more this might be a problem with resource limits when
too many file descriptors are open.
Me using a nice patch for suexec that limits these on FreeBSD.

What I am reading in #14,16 from you btw could be a quotation from
a private mail from wrowe...

Comment 19 Joe Orton 2003-03-13 16:21:12 UTC
To clarify, when I wrote "httpd server children" I am talking about the httpd
processes themselves, not scripts fork+execed from them.   Yes, by design, these
httpd processes must have all log files open.

I agree completely that by design, CGI scripts should just have three pipes on
fds 0, 1, 2, and the current 2.0.40 behaviour is a bug.

My point is that merely fixing these fd leaks does not improve security if you 
don't trusted your CGI script authors, since a malicious CGI script can use
ptrace() to run arbitrary code in an httpd process, thereby gaining access to
all the log file fds and more.

If you allow PHP scripts to exec/system/etc arbitrary binaries on your system,
then the same applies.  PHP safe mode mitigates this somewhat, as I've said above.


Comment 20 Joe Orton 2003-04-07 13:08:11 UTC
A forthcoming erratum is going to fix the fd leaks addressed in the attached
patches.   I've opened an RFE at bug 88182 to track fd leaks which are caused by
not setting CLOEXEC, since that is really a separate issue.

Thanks for persisting with this issue.

Comment 21 Mark J. Cox 2003-04-09 16:31:49 UTC
An errata has been issued which should help the problem described in this bug report. 
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen 
this bug report if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2003-139.html



Note You need to log in before you can comment on or make changes to this bug.