Created attachment 353837 [details] reproducer Description of problem: Parallels Virtuozzo Containers/OpenVZ linux kernel team found that sometimes NFS temporary (silly-rename: .nfs0000*) files can be left on a filesystem when no process has them open. Kernel version affected: checked 2.6.18-8.el5, 2.6.18-128.2.1.el5 x86_64, both are affected. Reproducer source (test_nfs_exit_thr4.c) is in attach. Brief description: create a thread which will create and unlink files in a cycle, then exit main thread without waiting for children. How to reproduce: [root@tom nfs]# uname -a Linux HOSTNAME 2.6.18-128.2.1.el5 #1 SMP Wed Jul 8 11:54:47 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@tom ~]# mount NFSSERVERNAME:/vz/export/tom /mnt/nfs/ [root@tom ~]# cd /mnt/nfs/ [root@tom nfs]# mount |grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) NFSSERVERNAME:/vz/export/tom on /mnt/nfs type nfs (rw,addr=NFSSERVERIP) [root@tom nfs]# gcc /tmp/test_nfs_exit_thr4.c -pthread -o /tmp/test [root@tom nfs]# ls -Flaio total 12 13403720 drwxr-xr-x 2 root 4096 Jul 15 18:00 ./ 1498497 drwxr-xr-x 3 root 4096 Jul 15 16:40 ../ [root@tom nfs]# /tmp/test main: exit ok! [root@tom nfs]# /tmp/test main: exit ok! [root@tom nfs]# ls -Flaio total 12 13403720 drwxr-xr-x 2 root 4096 Jul 15 17:55 ./ 1498497 drwxr-xr-x 3 root 4096 Jul 15 16:40 ../ 13403721 ---------- 1 root 0 Jan 5 1970 ashfilecs2QiM 13403722 -rw------- 1 root 0 Jul 15 17:55 .nfs0000000000cc864a000005a8 [root@tom nfs]# ps axf |grep test 3147 pts/0 S+ 0:00 \_ grep test Note1: ashfiles* files left are ok, they were created but not unlinked before the thread killed, but .nfs* files left seems to be wrong. Note2: the reproducer does not trigger .nfs* files appearance in 100% cases but still very often.
(In reply to comment #0) > Created an attachment (id=353837) [details] > reproducer > > Description of problem: > > Parallels Virtuozzo Containers/OpenVZ linux kernel team found that sometimes > NFS temporary (silly-rename: .nfs0000*) files can be left on a filesystem when > no process has them open. > Kernel version affected: checked 2.6.18-8.el5, 2.6.18-128.2.1.el5 x86_64, both > are affected. > > Reproducer source (test_nfs_exit_thr4.c) is in attach. > Brief description: create a thread which will create and unlink files in a > cycle, then exit main thread without waiting for children. > > How to reproduce: > > [root@tom nfs]# uname -a > Linux HOSTNAME 2.6.18-128.2.1.el5 #1 SMP Wed Jul 8 11:54:47 EDT 2009 x86_64 > x86_64 x86_64 GNU/Linux > [root@tom ~]# mount NFSSERVERNAME:/vz/export/tom /mnt/nfs/ > [root@tom ~]# cd /mnt/nfs/ > [root@tom nfs]# mount |grep nfs > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) > NFSSERVERNAME:/vz/export/tom on /mnt/nfs type nfs (rw,addr=NFSSERVERIP) > [root@tom nfs]# gcc /tmp/test_nfs_exit_thr4.c -pthread -o /tmp/test > [root@tom nfs]# ls -Flaio > total 12 > 13403720 drwxr-xr-x 2 root 4096 Jul 15 18:00 ./ > 1498497 drwxr-xr-x 3 root 4096 Jul 15 16:40 ../ > [root@tom nfs]# /tmp/test > main: exit ok! > [root@tom nfs]# /tmp/test > main: exit ok! > [root@tom nfs]# ls -Flaio > total 12 > 13403720 drwxr-xr-x 2 root 4096 Jul 15 17:55 ./ > 1498497 drwxr-xr-x 3 root 4096 Jul 15 16:40 ../ > 13403721 ---------- 1 root 0 Jan 5 1970 ashfilecs2QiM > 13403722 -rw------- 1 root 0 Jul 15 17:55 .nfs0000000000cc864a000005a8 > [root@tom nfs]# ps axf |grep test > 3147 pts/0 S+ 0:00 \_ grep test > > Note1: ashfiles* files left are ok, they were created but not unlinked before > the thread killed, but .nfs* files left seems to be wrong. > Note2: the reproducer does not trigger .nfs* files appearance in 100% cases but > still very often. I didn't manage to create .nfs file with your test. But I noticed with problem on my server.
> I didn't manage to create .nfs file with your test. But I noticed with problem > on my server. Don't consider my previous comment. I've managed to create .nfs* hidden files but it take me a dozen of trials.
In my test, it looks like a signal got in the way: NFS call rename ashfileQ2ibsq -> .nfs00000000000a00030000008e nfs: flush(0:18/655363) NFS reply rename: -512 NFS: dentry_delete(sillyrename/.nfs00000000000a00030000008e, 0) NFS: dentry_delete(sillyrename/ashfileQ2ibsq, 0) ...in this case, the file *was* renamed but it looks like the reply was dropped on the floor due to a signal popping in. We probably just need to make these calls use uninterruptible sleeps.
FWIW, I've been able to reproduce this on 2.6.34-ish kernels too. The race window there seems to be slightly smaller for reasons that aren't exactly clear to me, but it's still present there.
cc'ing Trond in case he has thoughts on this... We can't easily make this function use an uninterruptible sleep. What we may actually need to do is make the rename asynchronous, and have the sillyrename thread wait on its completion. That way, if the thread is killed, everything still should proceed to completion.
I've backported the patchset that I pushed upstream for this and have incorporated the patches into my test kernels here: http://people.redhat.com/jlayton/ ...if you have a non-critical place to test them, please do so and report the results.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
in kernel-2.6.18-247.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed.
reproduced on RHEL5-Server-U6: # uname -a Linux intel-s3e3144-01.rhts.eng.nay.redhat.com 2.6.18-238.el5 #1 SMP Sun Dec 19 14:22:44 EST 2010 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/exports /tmp *(rw) # mount -t nfs intel-s3e3144-01.rhts.eng.nay.redhat.com:/tmp /mnt # cat /proc/mounts intel-s3e3144-01.rhts.eng.nay.redhat.com:/tmp /mnt nfs rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=intel-s3e3144-01.rhts.eng.nay.redhat.com 0 0 # cd /mnt # gcc ~/test_nfs_exit_thr4.c -pthread -o test # ./test (run it several times and then trigger the issue) [root@intel-s3e3144-01 mnt]# ls -Flaio total 132 3080193 drwxrwxrwt 4 root 4096 Jun 3 03:02 ./ 2 drwxr-xr-x 24 root 4096 Jun 3 01:41 ../ 3080207 -rw------- 1 65534 0 Jun 3 03:02 ashfile1mEK7b 3080204 -rw------- 1 65534 0 Jun 3 03:01 ashfileIRmw8f 3080205 -rw------- 1 65534 0 Jun 3 03:01 ashfileKtFdKg 3080199 -rw-r--r-- 1 root 113 Jun 3 01:42 beah-beaker-backend.out 3080200 -rw-r--r-- 1 root 48 Jun 3 01:42 beah-fwd-backend.out 3080198 -rw-r--r-- 1 root 45 Jun 3 01:42 beah-srv.out 3080196 drwxrwxrwt 2 root 4096 Jun 3 01:42 .font-unix/ 3080195 drwxrwxrwt 2 root 4096 Jun 3 01:41 .ICE-unix/ 3080194 -rwx------ 1 root 8153 Jun 3 01:34 ks-script-2YUiCI* 3080206 -rw------- 1 65534 0 Jun 3 03:02 .nfs00000000002f000e00000058 3080210 -rwxr-xr-x 1 65534 8633 Jun 3 03:00 test* 3080201 -r-x------ 1 root 1868 Jun 3 01:42 tmpc7Pcgj.sh* 3080202 -rw------- 1 root 9262 Jun 3 01:42 tmp.rN3325 3080203 -r-x------ 1 root 1868 Jun 3 01:43 tmpW3EtRe.sh* verified on RHEL5.7-Server-20110525.0: # uname -a Linux hp-dl360g4-01.rhts.eng.bos.redhat.com 2.6.18-264.el5 #1 SMP Tue May 24 15:18:55 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux run the reproducer in a loop for several hours and never ended up with a leftover .nfs* file. # while true; do ./test;sleep 2; ls -Flaio |tee -a log; done;
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-1065.html