Bug 158277

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.7.6) Gecko/20050321 Firefox/1.0.2

Description of problem:
This problem was first seen on EL 2.1.  The ps command, when a single pid is specified, can sometimes report nothing, wrongly suggesting that the pid has died.  This can happen on systems where there are a large number of short-lived processes. This is a major problem for software that monitors the health of certain appications on the system because it can result in the monitoring software thinking that the application has died.  The chances of this problem happening on EL 3 and 4 are significantly less.

The ps command, regardless of whether it is asked to look at a single process or 
all processes on the system, will look at the details of all processes on the 
system.  What it does is look at all processes and select only the one(s) which 
qualify for the command.

ps uses the /proc/ filesystem to look at the process information using  
interfaces in libproc.  This involves an opendir of /proc and then readdir 
calls to get all of the entries in there.  For each process entry that readdir 
returns, additional information is collected from the "files" in the /proc/<pid>/ directory.  In the case of a ps command for a single pid, most of this information is collected and thrown away.

readdir buffers up information from the /proc directory.  It calls getdents64() 
which returns entries from the directory in blocks.  These blocks are 20 
entries in size (once it's reading pid entries from the filesystem).  The /proc 
root directory consists of the /proc stuff for information other than pids, 
following by the pid information.  When directory reads are done, the pid 
directories are generated on the fly from the current task list in the kernel. 
proc_pid_readdir does this.  This routine will return up to 20 "directory" 
entries in the /proc file system.  Since it is not possible to return all pid 
directories in one call, it is necessary for the proc filesystem layer to keep 
track of the current "file" pointer.  It is this which is causing the problem.

On RHEL 2.1 the first call to getdents64() will return the first 20 pids beginning at the start of the system's task list (let's call these entry numbers 1..20).  The second call will return entry numbers 21..40 and so on.  The problem is that if a process dies, then the relative position in the list of all processes after it moves and so it's possible for the reading of the list of directories to skip an entry.

For example, let's say that pids 1..20 are the first 20 in the task list, pids 
21..40 are the following 20 pids and 41..50 are the last 10.  In the first read 
of the pids getdents64 will return pids 1..20.  If pid 18 dies after this, but 
before the second call to getdents64, then the first 20 pids becomes 
1..17,19,20,21 so the second call to getdents64 will return pids 22..41.  i.e. 
to the caller of getdents64() pid 21 has been skipped.  This scenario can take 
place even when a "ps 21" command is done.

The routine get_pid_list() in fs/proc/base.c is where this is done.

I notice that in RHEL3 and above, the concept of a cursor was added to 
get_pid_list.  In these releases, get_pid_list() stores away a pointer to the 
last pid entry in the private_data field in the file struct.  This allows the reading of the /proc directory to continue on from where it left off without falling into the above problem....  With the exception of the case where the last pid died.  In my above scenario, if pid 20 had died, then the same problem would happen, even with this enhanced code in get_pid_list().

My suggestion would be to enhance ps so that if a single pid is passed as an argument then it just gets information about that pid and does not search the whole /proc/ directory.

As a workaround, I have recommended that the ps command not be used to monitor applications and that the contents of /proc/<pid>/stat be checked instead.

Version-Release number of selected component (if applicable):
procps-2.0.17-13

How reproducible:
Sometimes

Steps to Reproduce:
On EL 2.1, run 10 copies of this script at the same time.  One of the ps commands will fail to find mysqld after a few hours:
  #!/bin/bash
  PROC="mysqld"
  PID_FILE="mysql.pid"
  RETRY_INTERVAL=500000
  counter=0
  error_count=0
  p_pid=`cat $PID_FILE`
  if [ -z "$p_pid" ]; then
    		  echo "$(date '+%b %e %T')" >> loop.sh.log
		  echo "PID file empty!!!" >> loop.sh.log
  fi
  while :; do
	  pid=`ps $p_pid | grep $PROC | awk '{print $1}'`

	  if [ -z "$pid" ]; then
		  echo "$(date '+%b %e %T')" >> loop.sh.log
		  echo "mysqld not found!!!" >> loop.sh.log
              echo "p_pid output" >> loop.sh.log
              ps $p_pid >> loop.sh.log
              echo "full ps output" >> loop.sh.log
              ps -ef >> loop.sh.log
	  fi
	  usleep $RETRY_INTERVAL        
  done

On EL3 and EL4 I was not able to duplicate the problem with the above script. However, I believe that it would just take more time before the problem was seen (and the "right" process died during one of the ps commands).

Additional info: