Created attachment 1144898 [details]
Close descriptor when return value is POLLERR.

Description of problem:

Under some circumstances, rpc.gssd spikes to 100% CPU when a Kerberos context disappears from disk, due to a corner case in how things are handled internally when the return struct of poll() is analyzed.

The internet is full of reports from people having this issue, admittedly several years old, as it seems that it is no longer a problem after the big refactor that took out the inotify and poll logics.  Still, it's a problem for us because we use RHEL and CentOS 6.

After spending some quality time with gdb, I found that rpc.gssd hits a corner case when the clntXXX/gssd named pipe is deleted while rpc.gssd is still attached to it (but the directory is still there). In that situation the poll() function in the main loop returns a value of POLLERR|POLLHUP, but there's no handling for POLLERR in scan_poll_results() other than re-read all the contents of /var/lib/nfs/rpc_pipefs/ again for changes. Sadly, the containing directory is still there but empty, so the deleting logic is not triggered and the problem remains, so the server goes to 100% CPU reading those directories over, and over and over again (poll()'ing in the meantime).


Version-Release number of selected component (if applicable):

All of them up to latest release (1.2.3-64).


How reproducible:
So far this has happened to us only with mosh/screen/tmux sessions, so I suspect that there's something in these that triggers the behaviour (probably the user is there but the ticket is expired).  It happens when the directory clntXXX is there and rpc.gssd has a fd pointing to the corresponding gssd named pipe (see fd 24):

[...]
lrwx------. 1 root root 64 Mar 25 10:26 2 -> /dev/null
lr-x------. 1 root root 64 Mar 25 10:26 20 -> /var/lib/nfs/rpc_pipefs/nfs/clnt14e1
lr-x------. 1 root root 64 Mar 25 10:26 21 -> /var/lib/nfs/rpc_pipefs/nfs/clnt173b
lr-x------. 1 root root 64 Mar 25 10:26 22 -> /var/lib/nfs/rpc_pipefs/nfs/clnt1aea
lr-x------. 1 root root 64 Mar 25 10:26 23 -> /var/lib/nfs/rpc_pipefs/nfs/clnt18c4
lrwx------. 1 root root 64 Mar 25 10:26 24 -> /var/lib/nfs/rpc_pipefs/nfs/clnt14e1
lr-x------. 1 root root 64 Mar 25 10:26 25 -> /var/lib/nfs/rpc_pipefs/gssd/clntXX
lrwx------. 1 root root 64 Mar 25 10:26 26 -> /var/lib/nfs/rpc_pipefs/gssd/clntXX/gssd
lrwx------. 1 root root 64 Mar 25 10:26 27 -> /var/lib/nfs/rpc_pipefs/nfs/clnt14e1/gssd (deleted)
lr-x------. 1 root root 64 Mar 28 11:42 28 -> /var/lib/nfs/rpc_pipefs/nfs/clnt1aec
[...]

To make sure, I ran a "memset(&pollarray[i], 0, sizeof(struct pollfd))" under gdb and watched how rpc.gssd returned to normal operations (strace showed everything fine, ls -l /proc/PID/fd didn't show anything odd).

Patch attached applies to nfs-utils-1.2.3-64.el6.  I haven't seen a single appearance of this bug ever since I patched our internal binary.