Red Hat Bugzilla – Bug 246268
NFS failover causes VM deadlock
Last modified: 2007-11-30 17:07:45 EST
Description of problem:
I have 2 x86 boxes, f5 and f6, running RHEL5 with shared storage. The
shared storage contains an ext3 filesystem. I mount the shared storage locally
on f5 and NFS export it. On f6, it's mounted as NFS. I have an IP alias
for the nfs server that fails over too. I start an I/O job on f6 writing to
a file on the NFS filesystem. While the application is writing, I stop NFS
on f5 and unmount the filesystem. I attempt to mount the filesystem and
start the NFS server on f6. Either during the process of bringing up the server
or shortly afterwards, f6 will "hang". If I had windows previously open on f6,
I can extract some information. If I cause any write to the disk, the login
session will block uninterruptible.
I first saw this behavior with EL4 and exchanged some e-mail with Larry
Woodman. I have now reproduced it with EL5. The stack indicates both ext3
and nfsd threads are blocked waiting for pages to be freed. To capture
information, I ran typescript and executed 'cat meminfo' every 10 seconds
while I reproduced it. I'll attach the files. I left it overnight and it's
still hung with the following output from meminfo:
Dirty: 0 kB
Writeback: 614792 kB
AnonPages: 14084 kB
Mapped: 6240 kB
Slab: 48900 kB
PageTables: 1396 kB
NFS_Unstable: 279040 kB
Bounce: 0 kB
CommitLimit: 3078040 kB
Committed_AS: 57688 kB
To correlate the meminfo output, the nfs failover happened around 16:54,
Version-Release number of selected component (if applicable):
How reproducible: very
Steps to Reproduce:
easiest reproduction was using 2 machines and shared storage. I was able
to reproduce it with one machine by stopping NFS, waiting a few minutes and
trying to start it again.
1. nfs exporting a shared storage filesystem from 1 machine, NFS mounting on
2. Start heavy writing app (iozone -i 0 -n 0 -s 4G -r 1M -w -f
/mnt/NFS/iozone.dat) on the nfs client writing over the NFS mount.
3. stop the NFS server and try to start it on the client machine.
client machine locks up (processes hang)
The I/O would complete.
Created attachment 158232 [details]
typescript of loop doing cat meminfo every 10 seconds
I forced a stack trace dump and executed a dmesg at the end of the meminfo
The reproducer here involves a known problematic configuration -- namely,
attempting to relocate an NFS service onto a client of that service. There are
quite a few problems associated with this configuration -- both in kernel and
userspace and so it's not a situation that we support.
Peter Zijlstra has some changes upstream which may help some with this. However,
they are pervasive and not a good candidate for backporting to any of our
Development Management has reviewed and declined this request. You may appeal
this decision by reopening this request.
What specific part of the configuration is unsupported? I can reproduce the
hang using only 1 box. filesystem locally mounted and NFS exported. The
same machine NFS mounts it.
/dev/sda on /mnt/local type ext3 (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
nfsserver:/mnt/local on /mnt/NFS type nfs (rw,intr,addr=192.168.2.100)
[root@f6 2.6.18-8.el5-i686]# cat /etc/exports
Start writing a 4G file to the NFS mount. Stop NFS and try to unmount the
local filesystem. It will block waiting for pages.
umount D 50D9BE9E 1928 10010 9972 (NOTLB)
f3635cec 00000082 00000001 50d9be9e 000227e5 000227d3 00000007 c21fe000
c2138aa0 50d9d052 000227e5 000011b4 00000001 c21fe10c c20144e0 c042cdf2
c213e000 f3635cf4 00000286 c042cf03 00000000 00000286 243cdf63 243cdf63
[<f888ee89>] ext3_file_write+0x19/0x83 [ext3]
The part that is unsupported and does not work reliably is the part
where the client and server are running on the same system, in the
same operating system. This is a well known problem and is very
difficult to fix and will not be fixed in an existing RHEL release.
Thank you. Could you please confirm running the NFS client and server on the
same box IS supported if they are not the same filesystem.
Yes, there's no problem running a host that is a NFS client and server, as long
as it's not serving to itself.