Bug 618995

Run pstree -p to get a list of all the parent processes back up to init, then run lsof on all those processes (so lsof -p 1;... lsof -p 6297)  and see where that fd is coming from.

(or get the same info by browsing /proc)

All commands executed in runlevel 1 right after boot.

# pvs
File descriptor 9 (pipe:[10526]) leaked on pvs invocation. Parent PID 996: /bin/bash
  PV         VG       Fmt  Attr PSize PFree
  /dev/vda2  vg_one   lvm2 a-   9.76g    0 
  /dev/vda5  VolGroup lvm2 a-   9.25g    0 

[root@localhost /]# pstree -p
init(1)─┬─bash(996)───pstree(1025)
        └─udevd(494)─┬─udevd(893)
                     └─udevd(894)

[root@localhost /]# lsof -p 996
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    996 root  cwd    DIR  253,0     4096      2 /
bash    996 root  rtd    DIR  253,0     4096      2 /
bash    996 root  txt    REG  253,0   943248   5149 /bin/bash
bash    996 root  mem    REG  253,0   150672  15098 /lib64/ld-2.12.so
bash    996 root  mem    REG  253,0    22536  15100 /lib64/libdl-2.12.so
bash    996 root  mem    REG  253,0  1838296  15099 /lib64/libc-2.12.so
bash    996 root  mem    REG  253,0   138280   4952 /lib64/libtinfo.so.5.7
bash    996 root  mem    REG  253,0    26050 131015 /usr/lib64/gconv/gconv-modules.cache
bash    996 root  mem    REG  253,0 99158752   2463 /usr/lib/locale/locale-archive
bash    996 root  mem    REG  253,0    61624   2490 /lib64/libnss_files-2.12.so
bash    996 root    0u   CHR    5,1      0t0   4821 /dev/console
bash    996 root    1u   CHR    5,1      0t0   4821 /dev/console
bash    996 root    2u   CHR    5,1      0t0   4821 /dev/console
bash    996 root    9r  FIFO    0,8      0t0  10526 pipe
bash    996 root  255u   CHR    5,1      0t0   4821 /dev/console

[root@localhost /]# lsof -p 1
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
init      1 root  cwd    DIR              253,0     4096      2 /
init      1 root  rtd    DIR              253,0     4096      2 /
init      1 root  txt    REG              253,0   137808 259553 /sbin/init
init      1 root  mem    REG              253,0   150672  15098 /lib64/ld-2.12.so
init      1 root  mem    REG              253,0   106360   3274 /lib64/libnih.so.1.0.0
init      1 root  mem    REG              253,0  1838296  15099 /lib64/libc-2.12.so
init      1 root  mem    REG              253,0   145672  15116 /lib64/libpthread-2.12.so
init      1 root  mem    REG              253,0    40080   8143 /lib64/libnih-dbus.so.1.0.0
init      1 root  mem    REG              253,0    47064  15123 /lib64/librt-2.12.so
init      1 root  mem    REG              253,0   268200  15138 /lib64/libdbus-1.so.3.4.0
init      1 root  mem    REG              253,0    93320  15142 /lib64/libgcc_s-4.4.4-20100713.so.1
init      1 root  mem    REG              253,0    61624   2490 /lib64/libnss_files-2.12.so
init      1 root    0u   CHR                1,3      0t0   3434 /dev/null
init      1 root    1u   CHR                1,3      0t0   3434 /dev/null
init      1 root    2u   CHR                1,3      0t0   3434 /dev/null
init      1 root    3r  FIFO                0,8      0t0   8154 pipe
init      1 root    4w  FIFO                0,8      0t0   8154 pipe
init      1 root    5r   DIR               0,10        0      1 inotify
init      1 root    6r   DIR               0,10        0      1 inotify
init      1 root    7u  unix 0xffff8800379f4680      0t0   8155 socket

This is not lvm2 bug.

See man page:
On invocation, lvm requires that only the standard file descriptors stdin, stdout and stderr are available.  If others are found, they get closed and messages are issued warning about the leak.

So there is bug in some runlevel script which invoke bash after switching to runlevel 1 but forgot to close all unused descriptors.

If you run "pvdisplay 9>&-" to close it (where 9 is the reported leaked descriptor), no warning is printed.

Reassigning to upstart, but it is possible that it is some intermediate script which leaks it...

(and this should not be blocker IMHO)

Upstart sends job scripts to shell via pipe represented by /dev/fd/<fd> (/proc/self/fd/) file in proc filesystem:

/bin/sh -e /dev/fd/9

This fd stays open and pvdisplay complaints about that. If it is closed on exec then /dev/fd/9 won't exist and shell won't have place to read from. 

Possible solution would be send first closing pipe fd command to shell:

+		close_fd = NIH_MUST (nih_sprintf (NULL, "<&%d- ;", fds[0]));
+		NIH_ZERO (nih_io_write (io, close_fd, strlen (close_fd)));
+
 		NIH_ZERO (nih_io_write (io, script, strlen (script)));
 		nih_io_shutdown (io);

Created attachment 452680 [details]
send close command for pipe fd to shell

This is upstart workaround patch which first sends close fd command to shell before script defined in job.

I can reproduce this problem without upstart using named fifo as script source instead of file:

# bash <( echo pvdisplay)
File descriptor 63 (pipe:[20578493]) leaked on pvdisplay invocation. Parent PID 27181: bash
...

# bash <( echo lsof -p \$\$)
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    27310 root  cwd    DIR  253,0     4096    90113 /root
bash    27310 root  rtd    DIR  253,0     4096        2 /
bash    27310 root  txt    REG  253,0   943248     6347 /bin/bash
bash    27310 root  mem    REG  253,0   150672     5069 /lib64/ld-2.12.so
bash    27310 root  mem    REG  253,0  1838296     7190 /lib64/libc-2.12.so
bash    27310 root  mem    REG  253,0    22536    10714 /lib64/libdl-2.12.so
bash    27310 root  mem    REG  253,0   138280    11019 /lib64/libtinfo.so.5.7
bash    27310 root  mem    REG  253,0 99158752   141962 /usr/lib/locale/locale-archive
bash    27310 root  mem    REG  253,0    26050   141827 /usr/lib64/gconv/gconv-modules.cache
bash    27310 root    0u   CHR  136,6      0t0        9 /dev/pts/6
bash    27310 root    1u   CHR  136,6      0t0        9 /dev/pts/6
bash    27310 root    2u   CHR  136,6      0t0        9 /dev/pts/6
bash    27310 root   63r  FIFO    0,8      0t0 20697063 pipe
^^^^^^^ shouldn't be here
bash    27310 root  255r  FIFO    0,8      0t0 20697063 pipe

# cat test_fd.sh
lsof -p $$

# bash test_fd.sh
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    27357 root  cwd    DIR  253,2     4096 122882 /home/plautrba/tmp
bash    27357 root  rtd    DIR  253,0     4096      2 /
bash    27357 root  txt    REG  253,0   943248   6347 /bin/bash
bash    27357 root  mem    REG  253,0   150672   5069 /lib64/ld-2.12.so
bash    27357 root  mem    REG  253,0  1838296   7190 /lib64/libc-2.12.so
bash    27357 root  mem    REG  253,0    22536  10714 /lib64/libdl-2.12.so
bash    27357 root  mem    REG  253,0   138280  11019 /lib64/libtinfo.so.5.7
bash    27357 root  mem    REG  253,0 99158752 141962 /usr/lib/locale/locale-archive
bash    27357 root  mem    REG  253,0    26050 141827 /usr/lib64/gconv/gconv-modules.cache
bash    27357 root    0u   CHR  136,6      0t0      9 /dev/pts/6
bash    27357 root    1u   CHR  136,6      0t0      9 /dev/pts/6
bash    27357 root    2u   CHR  136,6      0t0      9 /dev/pts/6
bash    27357 root  255r   REG  253,2       21   2188 /home/plautrba/tmp/test_fd.sh

# rpm -q bash
bash-4.1.2-3.el6.x86_64

It is standard behaviour of bash. Bash keeps input file open; in this case pipe, which is fd 63.
What about to use named pipe (FIFO)?

In case of pipe, it's opened twice
bash    27310 root   63r  FIFO    0,8      0t0 20697063 pipe
bash    27310 root  255r  FIFO    0,8      0t0 20697063 pipe

but regular file is opened only once
bash    27357 root  255r   REG  253,2       21   2188 /home/plautrba/tmp/test_fd.sh


This is reproducer using /dev/fd/<fd>:

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>

int main() {
	char buf[] = "pvdisplay; lsof -p $$";
	char fd_name[256];
	int fd[2], s;

	pipe(fd);

	if (fork()) {
		close(fd[0]);
		write(fd[1], buf, sizeof(buf));
		close(fd[1]);
		wait(&s);
		return 0;
	}

	close(fd[1]);
	sprintf(fd_name, "/dev/fd/%d", fd[0]);
	execl("/bin/bash", "bash", fd_name, NULL);
}

# mkfifo test_fd; echo 'pvdisplay; lsof -p $$' > test_fd & bash test_fd

works ok

> bash    27310 root   63r  FIFO    0,8      0t0 20697063 pipe
> bash    27310 root  255r  FIFO    0,8      0t0 20697063 pipe

here comes opened fd 63 from parent process. bash (or child generally) probably shouldn't close opened fd from parent process as it can be intended.

Comments #15, #16 => VERIFIED

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Upstart sends jobs to the shell via file descriptors (/dev/fd files) in the /proc file system. One of these file descriptors was not closed, which resulted in a message about a file descriptor leak when operating at run level 1. This file descriptor is now closed correctly.

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Upstart sends jobs to the shell via file descriptors (/dev/fd files) in the /proc file system. One of these file descriptors was not closed, which resulted in a message about a file descriptor leak when operating at run level 1. This file descriptor is now closed correctly.+Upstart sends jobs to the shell via pipe represented by /proc/self/fd/<fd> file in /proc filesystem. This file descriptor was not, which resulted in a message about a file descriptor leak when operating at run level 1. This file descriptor is now closed correctly.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0531.html