An apache child can't read it's own /proc/self/smaps if it's not root, because the permissions on smaps are: -r-------- 1 root root 0 2007-10-08 01:46 smaps For all apache children, even those that are running as "apache:apache". This causes a mod_perl problem: Apache::SizeLimit ideally should use Linux::Smaps, but Linux::Smaps can't read /proc/self/smaps under Apache.
Hi Max, A couple of questions... Is this a RHEL5-specific issue, or does the upstream kernel also exhibit the same behavior? Also, I'm trying to reproduce this with this test program -- which first as a parent, and then as its child, reads both /proc/<pid>/maps and /proc/self/smaps: /* mytest.c */ #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> void do_func(char *, char *); main() { int status; do_func("parent", "pid"); do_func("parent", "self"); if (fork() == 0) { do_func("child", "pid"); do_func("child", "self"); } else wait(&status); } void do_func(char *who, char *which) { int fd, i, bytes; char c; char filename[100]; char command[100]; printf("%s pid: %d [uid: %d euid: %d gid: %d egid: %d]\n", who, getpid(), getuid(), geteuid(), getgid(), getegid()); if (strcmp(which, "self") == 0) sprintf(filename, "/proc/self/smaps"); else if (strcmp(which, "pid") == 0) sprintf(filename, "/proc/%d/smaps", getpid()); sprintf(command, "ls -l %s", filename); system(command); if ((fd = open(filename, O_RDONLY)) < 0) { perror("open"); exit(1); } bytes = 0; while (read(fd, &c, 1) == 1) { // printf("%c", c); bytes++; } printf("%d bytes read\n\n", bytes); } I created an "apache" user with a uid of 500. When the test program above runs without doing a root setuid operation on the binary, it works as expected: $ make mytest cc mytest.c -o mytest $ ls -l mytest -rwxrwxr-x 1 apache apache 9063 Nov 8 15:53 mytest $ ./mytest parent pid: 17722 [uid: 500 euid: 500 gid: 500 egid: 500] -r-------- 1 apache apache 0 Nov 8 15:53 /proc/17722/smaps 3398 bytes read parent pid: 17722 [uid: 500 euid: 500 gid: 500 egid: 500] -r-------- 1 apache apache 0 Nov 8 15:53 /proc/self/smaps 3398 bytes read child pid: 17725 [uid: 500 euid: 500 gid: 500 egid: 500] -r-------- 1 apache apache 0 Nov 8 15:53 /proc/17725/smaps 3398 bytes read child pid: 17725 [uid: 500 euid: 500 gid: 500 egid: 500] -r-------- 1 apache apache 0 Nov 8 15:53 /proc/self/smaps 3398 bytes read $ Then I made the binary setuid root, but it still runs OK. Although what's kind of interesting is that /proc/self/maps doesn't show the root ownership the same way as /proc/<pid>/maps does: $ ls -l mytest -rwsrwsr-x 1 root root 9063 Nov 8 15:53 mytest $ ./mytest parent pid: 17734 [uid: 500 euid: 0 gid: 500 egid: 0] -r-------- 1 root root 0 Nov 8 15:54 /proc/17734/smaps 3398 bytes read parent pid: 17734 [uid: 500 euid: 0 gid: 500 egid: 0] -r-------- 1 apache apache 0 Nov 8 15:54 /proc/self/smaps 3398 bytes read child pid: 17737 [uid: 500 euid: 0 gid: 500 egid: 0] -r-------- 1 root root 0 Nov 8 15:54 /proc/17737/smaps 3398 bytes read child pid: 17737 [uid: 500 euid: 0 gid: 500 egid: 0] -r-------- 1 apache apache 0 Nov 8 15:54 /proc/self/smaps 3398 bytes read $ What do you think?
...and also, what are the file permissions on the executable that is failing for you?
Hi Dave. I don't have an upstream kernel to test with, at the moment, and my only RHEL5 server is a production machine. I suspect it's a RHEL-only issue, though, because I think the problem was introduced in bug 176687. I think your tests at the bottom of comment 1 are reversed from what I'm experiencing. That is, you have a binary that does setuid root, and I have Apache, which starts as root and does a setuid to the "apache" user. Also, I'm pretty sure that Apache is doing the setuid inside of its C code, as opposed to having the binary set setuid, which might make a difference?
This is trivially reproducible, use a CGI script as below. SELinux will prevent access to /proc/$PPID by default, but even with "setenforce 0", most of /proc/$PPID shows up as owned by root. In e.g. 2.6.23.1-21.fc7 most of /proc/$PPID shows up owned by apache. #!/bin/sh echo Content-Type: text/plain echo ls -l /proc/$PPID
> I think your tests at the bottom of comment 1 are reversed from what I'm > experiencing. That is, you have a binary that does setuid root, and I have > Apache, which starts as root and does a setuid to the "apache" user. Then, at least with respect to the smaps file, it appears to have something to do with my other test observation, where the /proc/self/maps permissions do not "follow suit" with /proc/<pid>/maps: > $ ls -l mytest > -rwsrwsr-x 1 root root 9063 Nov 8 15:53 mytest > $ ./mytest > parent pid: 17734 [uid: 500 euid: 0 gid: 500 egid: 0] > -r-------- 1 root root 0 Nov 8 15:54 /proc/17734/smaps > 3398 bytes read > > parent pid: 17734 [uid: 500 euid: 0 gid: 500 egid: 0] > -r-------- 1 apache apache 0 Nov 8 15:54 /proc/self/smaps > 3398 bytes read > > child pid: 17737 [uid: 500 euid: 0 gid: 500 egid: 0] > -r-------- 1 root root 0 Nov 8 15:54 /proc/17737/smaps > 3398 bytes read > > child pid: 17737 [uid: 500 euid: 0 gid: 500 egid: 0] > -r-------- 1 apache apache 0 Nov 8 15:54 /proc/self/smaps > 3398 bytes read So in your case, there would be a permissions violation.
> Then, at least with respect to the smaps file, it appears to have > something to do with my other test observation, where the /proc/self/maps > permissions do not "follow suit" with /proc/<pid>/maps: I take that back -- both files show root ownership. I changed the mytest.c file above to return instead of exiting if the open() failed: # diff mytest.c.orig mytest.c 47c47 < exit(1); --- > return; # And changed it so that the test was setuid "apache". When run as root, this happens: # ls -l mytest -rwsrwsr-x 1 apache apache 8933 Nov 9 10:09 mytest # ./mytest parent pid: 20609 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:10 /proc/20609/smaps open: Permission denied parent pid: 20609 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:10 /proc/self/smaps open: Permission denied child pid: 20612 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:10 /proc/20612/smaps open: Permission denied child pid: 20612 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:10 /proc/self/smaps open: Permission denied #
> This is trivially reproducible, use a CGI script as below. SELinux will > prevent access to /proc/$PPID by default, but even with "setenforce 0", > most of /proc/$PPID shows up as owned by root. Yeah -- when the kernel is booted with selinux turned off: # dmesg | grep -i selinux Command line: ro root=/dev/VolGroup00/LogVol00 console=ttyS0,115200 selinux=0 Kernel command line: ro root=/dev/VolGroup00/LogVol00 console=ttyS0,115200 selinux=0 SELinux: Disabled at boot. # ... the results are the same: # ls -l mytest -rwsrwsr-x 1 apache apache 8933 Nov 9 10:09 mytest # ./mytest parent pid: 3089 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:33 /proc/3089/smaps open: Permission denied parent pid: 3089 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:33 /proc/self/smaps open: Permission denied child pid: 3092 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:33 /proc/3092/smaps open: Permission denied child pid: 3092 [uid: 0 euid: 500 gid: 0 egid: 500] -r-------- 1 root root 0 Nov 9 10:33 /proc/self/smaps open: Permission denied #
My initial wild-ass guess was that it had something to do with the linux-2.6-execshield.patch, which does change the permissions to the "smaps" file: # grep smaps linux-2.6-execshield.patch - E(PROC_TGID_SMAPS, "smaps", S_IFREG|S_IRUGO), + E(PROC_TGID_SMAPS, "smaps", S_IFREG|S_IRUSR), - E(PROC_TID_SMAPS, "smaps", S_IFREG|S_IRUGO), + E(PROC_TID_SMAPS, "smaps", S_IFREG|S_IRUSR), # But I can't find anything in there that addresses the ownership of the file.
As it turns out, my original test program was flawed w/respect to my claim that there was a difference in behaviour between /proc/self/maps and /proc/<pid>/maps, probably due to the fact that I was doing a system("ls") of the files, which would fork a shell, which would exec the "ls". Sorry for the confusion... So just to get on the same page here, a more relevant test program is this, which stat()'s the files to get their ownership and permissions bits: /* mystat.c */ #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> void file_check(char *); main(int argc, char **argv) { char filename[100]; struct stat sbuf; sprintf(filename, "ls -l %s", argv[0]); system(filename); printf("pid: %d -> uid: %d gid: %d euid: %d egid: %d\n", getpid(), getuid(), getgid(), geteuid(), getegid()); sprintf(filename, "/proc/self/smaps"); file_check(filename); sprintf(filename, "/proc/%d/smaps", getpid()); file_check(filename); } void file_check(char *filename) { int fd; struct stat sbuf; if (stat(filename, &sbuf) != 0) { perror(filename); return; } fd = open(filename, O_RDONLY); printf("%s: st_mode: %o st_uid: %d st_gid: %d open: %s\n", filename, sbuf.st_mode, sbuf.st_uid, sbuf.st_gid, fd < 0 ? "FAILED" : "OK"); if (fd >= 0) close(fd); } And it simply shows that when the binary is setuid-root, and run as a normal user "apache", the proc file's ownership reflects the upgrade, and they can be opened OK: apache> ./mystat -rwsr-sr-x 1 root root 8643 Nov 9 12:10 ./mystat pid: 3918 -> uid: 500 gid: 500 euid: 0 egid: 0 /proc/self/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: OK /proc/3918/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: OK apache> But when the binary is setuid-apache, and run as root, the proc file's ownership does not get downgraded, and they cannot be opened: root> ./mystat -rwsr-sr-x 1 apache apache 8643 Nov 9 12:10 ./mystat pid: 3927 -> uid: 0 gid: 0 euid: 500 egid: 500 /proc/self/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: FAILED /proc/3927/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: FAILED root>
Any progress on this? I'm setting up a new bugzilla.gnome.org and it would be really nice to have this fixed for mod_perl performance reasons. (The Apache2::SizeLimit module ideally needs to use Linux::Smaps to see if the httpd is too big after the request, and shut it down. Right now it can't use smaps so it thinks that each individual httpd child is using the entire virt size and always shuts them down after every request, requiring a new httpd to be spawned after every request.)
No.
We are also wanting to move this installation of Bugzilla over to mod_perl but now I think we should wait til this problem is fixed. Our production webapps are currently running on RHEL5 so we also would be affected by this.
(In reply to comment #12) > We are also wanting to move this installation of Bugzilla over to mod_perl but > now I think we should wait til this problem is fixed. Our production webapps > are currently running on RHEL5 so we also would be affected by this. In this location: http://people.redhat.com/anderson/BZ_322881 are the following files: kernel-2.6.18-177.el5.smaps1.i686.rpm kernel-2.6.18-177.el5.smaps1.src.rpm kernel-2.6.18-177.el5.smaps1.x86_64.rpm kernel-PAE-2.6.18-177.el5.smaps1.i686.rpm linux-kernel-test.patch If a different kernel/architecture is required for testing, please let me know, or build it from the kernel-2.6.18-177.el5.smaps1.src.rpm file above. Please advise here that your application runs as expected. When the "mystat.c" test file in comment #9 is setuid-apache and run as root, its /proc/self/smaps and /proc/<pid>/smaps cannot be opened: # uname -r 2.6.18-164.el5 # ./mystat -rwsrwsr-x 1 apache apache 8790 Dec 7 11:01 ./mystat pid: 3367 -> uid: 0 gid: 0 euid: 500 egid: 500 /proc/self/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: FAILED /proc/3367/smaps: st_mode: 100400 st_uid: 0 st_gid: 0 open: FAILED # uname -a Running the test kernel: # uname -r 2.6.18-177.el5.smaps1 # ./mystat -rwsrwsr-x 1 apache apache 8790 Dec 7 11:01 ./mystat pid: 2636 -> uid: 0 gid: 0 euid: 500 egid: 500 /proc/self/smaps: st_mode: 100444 st_uid: 0 st_gid: 0 open: OK /proc/2636/smaps: st_mode: 100444 st_uid: 0 st_gid: 0 open: OK # Thanks, Dave Anderson
I tried this on a test RHEL5 server with updated packages. Unfortuntely I do not see a difference from what I can tell. Every page hit even index.cgi causes the httpd child to exit. kernel-2.6.18-164.6.1.el5.x86_64 (stock): # tail /var/log/httpd/error_log [Thu Dec 10 00:24:41 2009] (3108) Apache2::SizeLimit httpd process too big, exiting at SIZE=325636/0 KB SHARE=3776/0 KB UNSHARED=321860/140000 KB REQUESTS=2 LIFETIME=0 seconds kernel-2.6.18-177.el5.smaps1.x86_64: # tail /var/log/httpd/error_log [Thu Dec 10 00:41:01 2009] (2917) Apache2::SizeLimit httpd process too big, exiting at SIZE=337264/0 KB SHARE=3984/0 KB UNSHARED=333280/240000 KB REQUESTS=2 LIFETIME=0 seconds Only by increasing the $Apache2::SizeLimit::MAX_UNSHARED_SIZE value to some suitably large size such as 340000 will the error go away for each page hit. Dave
So whatever problem that apache is running into, it has nothing to do with the issue reported by this BZ (i.e. the permissions of /proc/<pid>/smaps).
(In reply to comment #15) > So whatever problem that apache is running into, it has nothing to do with > the issue reported by this BZ (i.e. the permissions of /proc/<pid>/smaps). I very well could be looking at something different than what Max was originally reporting about. Max, are you able to apply the test kernel to your RHEL5 environment to see if it solves the problem for you? Thanks Dave
dkl: You have to also install the Linux::Smaps package. I will also test.
I am getting the same results with Linux::Smaps installed (latest version from CPAN). Hopefully you have better results than I. Let me know if there is anything else I can try or provide to help diagnose this. Dave
Okay, I ran the test kernel today, and I can confirm that this bug is in fact fixed. :-) Apache can now correctly tell the size of its shared memory, though Linux::Smaps seems to be miscalculating the size of the unshared memory, somehow, and I have to look into that. For reference, here's what my processes used to say when they died: (4151) Apache2::SizeLimit httpd process too big, exiting at SIZE=409720/0 KB SHARE=4832/0 KB UNSHARED=404888/70000 KB REQUESTS=2 LIFETIME=0 seconds And now they say something like: [Tue Feb 2 15:12:27 2010] (2907) Apache2::SizeLimit httpd process too big, exiting at SIZE=430968/0 KB SHARE=53604/0 KB UNSHARED=377364/70000 KB REQUESTS=2 LIFETIME=0 seconds As you can see, the SHARE size is much more realistic.
I found the issue in Apache2::SizeLimit, and described it here, for anybody who's interested: http://marc.info/?l=apache-modperl&m=126514792920784&w=2 So in fact both the fix described in this bug *and* the fix described in my above email are required to fix the issue for Bugzilla.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
In order to complete the CCFR format of the errata associated with this bug, and to facilitate docs approval, please provide more information on the following: Cause What actions or circumstances cause this bug to present. Consequence What happens when the bug presents. Fix What was done to fix the bug. Result What now happens when the actions or circumstances above occur. Note: this is not the same as the bug doesn’t present anymore. Please note, this information is for use by non-developers. Thanks
Cause: The /proc/<pid>/smaps file gets created with S_IRUSR permissions (-r--------). Consequence: If a non-root setuid binary is run as root, its /proc/<pid>/smaps file cannot be read because the file's permissions only allow allowing access from a task with the original root UID value. Fix: Create /proc/<pid>/smaps file with S_IRUGO permissions (-r--r--r--) Result: The /proc/<pid>/smaps file can be read even when running a setuid binary.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: The /proc/<pid>/smaps file gets created with S_IRUSR permissions (-r--------). Consequence: If a non-root setuid binary is run as root, its /proc/<pid>/smaps file cannot be read because the file's permissions only allow allowing access from a task with the original root UID value. Fix: Create /proc/<pid>/smaps file with S_IRUGO permissions (-r--r--r--) Result: The /proc/<pid>/smaps file can be read even when running a setuid binary.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0178.html