Red Hat Bugzilla – Bug 88047
/proc/<pid>/cmdline is empty in RHL8
Last modified: 2005-10-31 17:00:50 EST
Description of problem:
Inexplicably, with the new 2.4.20 kernel, some processes (usually daemons) have
an empty (zero-byte) /proc/<pid>/cmdline file. Consequently, "ps ax" will not
show the processes' command lines.
I read that this can happen if processes are swapped out of memory, so I tried
disabling swap (swapoff -a) and restarting the processes, but the problem was
Version-Release number of selected component (if applicable):
Steps to Reproduce:
# ps ax | grep sendmail
1634 ? S 0:00 [sendmail]
1643 ? S 0:00 [sendmail]
# cat /proc/1634/cmdline
# cat /proc/1643/cmdline
Expected Results: "ps" and "cat" should show the full command lines, as was
the case in Red Hat Linux 8 and earlier.
The problem is also present in Red Hat 8.0's 2.4.18-27.8.0 errata kernel.
The stock 2.4.18-14 kernel included with Red Hat 8.0 does not have this problem.
JR, could you please raise the priority of this bug to "High"?
Not only does this bug make it impossible to see status information from
affected processes (e.g., sendmail), but init.d scripts fail because long
program names are truncated:
1083 ? S 0:00 [mimedefang-mult]
1112 ? S 0:15 \_ /usr/bin/perl -w /usr/bin/mimedefang.pl -server
1989 ? S 0:01 \_ /usr/bin/perl -w /usr/bin/mimedefang.pl -server
1101 ? S 0:00 [mimedefang]
1109 ? S 0:00 \_ [mimedefang]
1110 ? S 0:00 \_ [mimedefang]
1127 ? S 0:00 [sendmail]
1136 ? S 0:00 [sendmail]
(The "[mimedefang-mult]" process is mimedefang-multiplexor. Because the
/etc/init.d/mimedefang script (understandable) calls "killproc
mimedefang-multiplexor", running "service mimedefang stop" always fails.)
I'm not sure what the common element here is. (It had looked like only
processes that attempted to modify argv were broken, but a test program I
wrote worked just fine, so that's not the case.)
Ouch, you're right! Changing the priority...
BTW, I haven't verified it but I *suspect* this bug has something to do with
the recent fix for the ptrace security hole. See for example:
Yeah, that's the problem:
From your link, as of 2003-04-04T19:00-0500, it looks like people are still
arguing on lkml about how to properly fix the problem.
If you see a patch float by that Alan, Arjan, et. al. accept, please attach it
here. (I'll be watching as well.)
Actually, from skimming through the past few weeks of the lkml archive, I think
it's pretty clear that there's going to be another version of the ptrace patch,
as the current one is broken:
Hopefully the /proc/*/cmdline issues can be addressed in the second patch...
Yes, this was introduced in the ptrace patch. Both the /proc/pid/cmdline and
/proc/pid/environ pseudo-files (at least) are affected, and return an empty
string, and also indirectly affects the output of ps(1). This happens whenever
the process is considered non-dumpable, and occurs for daemons (like sendmail)
whenever the process euid != uid. This is unrelated to a process changing its
argv or environ arrays.
The problem was introduced in the access_process_vm() function in
linux/kernel/ptrace.c, which now returns -EPERM when the process is not
dumpable. And that function is used by both proc_pid_cmdline() and
proc_pid_environ() functions in the fs/proc/base.c file for the proc filesystem.
I believe this is not platform specific, should this change in this bug?
A purported patch was just made by Tachino Nobuhiro on the linux.kernel mailing
list. Looks reasonable to me, here's the thread to follow...
I'm not at all convinced this is safe.
The thing is, euid!=uid programs are "special" security wise and ANY access to
them needs to be very very careful. Including seeing the arguments as user,
since that sort of is privileged already.
Yes, there is certainly the argument that even read-only access is not valid,
but that would be a change in the way Linux has traditionally behaved (except
perhaps "trusted" linux variants). But that argument applies to not only setuid
processes, but also to say processes where euid=uid=0; why is a setuid process
more deserving of command-line hiding than a normal root process? The
kmod-ptrace hole was certainly a very critical bug, but a read-only check seems
to still keep that particular hole closed while making a mimimal change in other
Some other Unixes solve this by keeping the globally visible command line in
kernel space, providing a pstat() system call to access/modify it. But in Linux
you have to actually access the user-space of a process to read it's command
line. It also seems to me that the /proc fs is not really a ptrace() type
function and probably should have different access rules. At a minimum though,
if you're root (or have the CAP_SYS_PTRACE capability) you should always be able
to see the command line, and with the current ptrace patch you can not. This is
a major policy and bahavior change! Of course it's best left to the kernel
gurus to determine a better fix, but the current patch is breaking software and
I agree with Deron in comment #9.
Historically speaking, (regular processes, root processes) have been able to see
the cmdline for (regular, root, euid!=uid) processes. Over the years, I think
this has been more of an implicit assumption than a conscious decision.
Perhaps, now that this issue has been explicitly examined, a different consensus
might be reached.
Regardless, much software and many users depend on the access to the cmdline
that the ptrace patch took away. Removing the access is a vast sweeping change,
and it should *not* occur in anything short of the 2.5 kernel, or perhaps even
the 2.7 kernel. (And even then, I'd argue that the same functionality--a
process being able to make some very small synopsis of its command-line
arguments and/or current state publicly available--should be provided via some
other mechanism. Linux setproctitle(), anyone?)
Discussion about the ptrace patch seems to have mostly died off on lkml. The
only patch which came out of the discussion which wasn't obviously wrong was
Tachino Nobuhiro's patch:
It might not be the best solution, but it's *a* solution, and it doesn't seem to
re-open the ptrace vulnerability.
can be marked as duplicate of this, as this bug has some work for fixing.
An errata has been issued which should help the problem described in this bug report.
This report is therefore being closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files, please follow the link below. You may reopen
this bug report if the solution does not work for you.
I tested this out, and everything seems to WORKFORME.
As an added bonus, the root user can now strace (effective id != real id!)
processes. (That fix wasn't mentioned in the errata, but I know I've seen other
Bugzilla bugs about it.)
Much thanks for fixing this.
This problem is also being addressed in the 87638 thread.
I just installed kernel-smp-2.4.20-13.7 that came out yesterday, and I can
assure everyone that the problem _is not fixed_.
This is fixed in Red Hat 9, but it is still broken in 8.0 and earlier. Thus the
errata page RHSA-2003:172-23 describing the new kernel is somewhat misleading.
In particular the following kernels,
Red Hat 9 - 2.4.20-13.9 - FIXED
Red Hat 8.0 - 2.4.20-13.8 - BROKEN
Red Hat 7.3 - 2.4.20-13.7 - BROKEN
I didn't dive into the source code yet to see what was done, or not done.
Arjan, please reopen this bug for 8.0 (or create a new bug, since technically
this bug was reported against release 9 which is fixed). Thanks.
For me too the problem are solved in 9 and appeared in 7.3
Before upgrade 7.3 was with all cmdlines, and after kernel upgrade many
processes now appear as swapped and they are not swapped.
just updated to kernel 2.4.20-13.7 [RHSA-2003:172-00] on redhat 7.2 and my
machine now has this problem. cat /proc/*/cmdline is empty for child processes. eg:
root 1016 1 0 02:51 ? 00:00:03 /usr/sbin/httpd -DHAVE_ACCESS
nobody 2723 1016 0 04:02 ? 00:00:02 \_ [httpd]
nobody 2724 1016 0 04:02 ? 00:00:02 \_ [httpd]
nobody 2725 1016 0 04:02 ? 00:00:02 \_ [httpd]
this is on a pentium (x86).
just installed kernel-2.4.20-18.7 [RHSA-2003:187-01] and this appears to be
fixed on my redhat 7.2 box.