From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; FunWebProducts) Description of problem: 1. Whenver nfs client does good amount of io , mounts remain busy even after unexporting all the filesystems. 2.fuser does not show any user for the busy mount. 3.After around 20-30 min , filesystem can be unmounted 4.Umount can unmount filesystem in only one call if nfsd is killed manually Version-Release number of selected component (if applicable): nfs-utils-1.0.6-46 How reproducible: Always Steps to Reproduce: 1.exported filesystems to client 2.On client side , mount the exported filesystem and do io on it using multiple threads 3.Wait for around 2 min , unexport the filessytems and then try to umount filesystem. Actual Results: Its was not possible to umount the filesystems mounted by client. Expected Results: After unexporting filessytems , umounting should happen without any problem and this should not require killing of nfsd Additional info:
What is the kernel version?
Tested on two kernel versions Machine 1 (RHEL4 GA): ---------- $ uname -a Linux vcslinux6.vxindia.veritas.com 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST $ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant) Machine 2 (RHEL4U1 Beta): ---------- $ uname -a Linux thorpci25.veritas.com 2.6.9-6.28.ELv #1 SMP Wed Mar 23 20:53:13 GMT 2005 ia64 ia64 ia64 GNU/Linux $ cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant)
Just to make it clear. "Mounts are busy on machine running nfs server"
Again, just to be clear.... Your unexporting the the filsystems on the server and then the unmount from the clients are hanging, correct?
i am unexporting filesystems on server and then umount on --server-- are hanging
Ok... Does "server nfs stop" allow the local (to the server) filesystems to be unmouted?
Yes, after runnnig server nfs stop , umount will happen. As i said earlier , after killing nfsd , umount happens properly. But its not possible to umount without killing nfsd
OK... So the expected action is for the umount to fail with EBUSY instead of just hang.... At least that's what I would expect...
Currently , umount fails with EBUSY. Shouldn't umount flush all buffers(used by nfs) on disk and cleanly umount the disk? Why is it failing with EBUSY ?
Please let us know when can we get fix for this ?
Unless I'm misunderstanding something, this is not a problem. The server has to ensure the filesystems its exporting stay mounted. Do other NFS implementation allow you unmount exported filesystems?
Hi Steve , In "Steps to Reproduce" I have mentioned that filesystem is unexported before umount is called. One filessytem is unexported , umount should succeed.But due to some reasons mount remains busy.
Its reproducible everytime the "Steps to Reproduce" are followed. Were you able to reproduce this issue ?
Sorry for the confusion... exportfs -u does seem to be broken.
Thanks Steve. when can we expect the fix
As far as i know , exportfs maintains/manipulates table of exported filesystems. It is not at all concerned with io. Problem occurs when client does io on mounted FS and not otherwise. If problem is with exportfs , it should also occur when client does not do io. Its not clear how exportfs is responsible for this problem.
any update on this ?
Steve , Please let us know if the problem has been identified/fixed. This fix is very important for us.
I am not able to raise priority from normal->high.
Unfortunately I have not been able to reproduce this problem. Here are the step I'm taking: server> service nfs start server> exportfs -v /home <world>(rw,wdelay,nohide,root_squash,fsid=2) /usr <world>(rw,wdelay,nohide,root_squash,fsid=1) / <world>(ro,wdelay,root_squash,fsid=0) client> mount server:/home /mnt/server/home client> cat /tmp/100m > /mnt/server/home/100m server> exportfs -u \*:/home exportfs -v /usr <world>(rw,wdelay,nohide,root_squash,fsid=1) / <world>(ro,wdelay,root_squash,fsid=0) server> umount /home [the filesystem unmounts] What am I not doing to cause this hang?
Let client do i "continuously" for 5 min and then while io is in progress try unexporting and unmounting. This should be enough to cause mount busy Even the "sync" option for exportfs could not solve this problem.
Created attachment 114627 [details] program to do io Steve , You can use the attached program to do io on client side. syntax: >./disfk <no. of threads> mountpt1 mountpt2 mountpt3...... Eg: ./diskf 3 /mnt/mntpt1
did the program help to reproduce the issue ?
PM ACK for U2.
Created attachment 114989 [details] Upstream patch I've been working with this upstream patch that helps but does not completely take care of the the problem. The patch does allow the local filesystem to be umounted after the exportfs -u when the exported filesystem is idle, but when the filesystem is busy, a second exportfs -u is needed to remove an extra reference... I'm still looking into where that extra reference is coming from.
If filesystem is busy , FS does not necessarily get umounted after second exportfs -u. Most(90-95%) of the times even 25-30 exportfs -u do not help in unmounting the Filesystem.
This is when your patch is not applied
I've finally gotten back to this and it appears the patch in Comment #24 is most definitely needed since it eliminates a bit flag what was just getting in the way... Now the "extra reference" talked about in Comment #24 is not really an extra references at all... Its more like a missed reference... Here's what appears to be happening (using a patched kernel). When the exportfs -u is done on a busy export (i.e. the client is witting to a file in the exported directory), the kernel export code notices that the export is still referenced (because one of the nfsd has not finished yet), so exportfs only takes the export out of the exported namespace (meaning clients and not longer see it but there is still a reference on the mount point). Next the nfsd finishes and dereferences the export, which means at this point another exportfs -u is need to dereference the export which in turn will cause the a dereference of the mount point which allows the mount point to be unmounted. I'm currently looking into way of making the exportfs command wait until the exported directory is complete de-exported before returning. Unfortunately, it appears this is not a trivial task...
Hi Steve, What i undertsand from your comment is that exportfs holds a reference to the exported filesystem and if we try to unexport a busy exported filesystem, exportfs removes fs from exports table but still preserves reference to that fs. Queries: 1. Why should exportfs hold reference to exported filesystem ? 2. Should exportfs just maintain exports table without concerning status(busy/not busy) of exported filesytem ? ------------------------------------------------------------------------------ Let me know if there is problem with this approach : 1.Exportfs should maintain exports table and should hold no references to exported filesystem(otherwise we will need second exportfs -u to remove the reference as in your comment) 2.When user wants to unexport the fs, remove it from exports table From here onwards if nfs client tries to access the fs, it will get permission denied. 3.If user wants to unmount the fs, flush all the buffers used for that filesystem and unmount it cleanly.As exportfs has not reference to this fs, there should not be any problem unmounting it. Let me know if i am missing something
> 1. Why should exportfs hold reference to exported filesystem ? Well exportfs is not hold a reference on the filesystem ( I guess I was not very clear with that), its the kernel holding the reference on the filesystem , which happens when the client mounts the filesystem. Note: There are actually two reference counts I was referring to in Comment #24; One on the mounted filesystem (mnt_count) and the other on cached export entry in the kernel (refcnt, in cache_head of the svc_export structure). So when I said "the kernel export code notices that the export is still referenced" I meant the refcnt in the cache_head of the svc_export structure). >2. Should exportfs just maintain exports table without concerning status(busy/not busy) of exported filesystem ? It does. But when it comes time to for exportfs to flush the cached exports and the cached refcnt is >1 (because an nfsd is still using it), the cache fill not be flushed. Note: the reference count on the filesystem (i.e. mnt_count) is only decremented when the cached export entry is not in use (i.e. refcnt == 1). >1.Exportfs should maintain exports table and should hold no references to > exported filesystem(otherwise we will need second exportfs -u to remove the > reference as in your comment) This does happen... again the refcnt is on the export and mount point are held by the kernel. > 2.When user wants to unexport the fs, remove it from exports table > From here onwards if nfs client tries to access the fs, it will get > permission denied. This does happen... >3.If user wants to unmount the fs, flush all the buffers used for that > filesystem and unmount it cleanly.As exportfs has not reference to this > fs, there should not be any problem unmounting it. Again its not exportfs with the reference, its the nfsd with the references on the cached export which in turn has a reference on the filesystem... Once the cached reference goes to 1 and a cach_flush() is done (i.e by exportfs), the filesystem will be unmount-able I hope this help... BTW, did you get a chance to test the patch?
I have not yet tested the patch Will do it next week.
Created attachment 120119 [details] Updated Patch for a 2.6.9-22 kernel Here is updated patch that can be used on RHEL4 kernels.
Tested the patch. As you told, Additional exportfs -u is required to umount the filesystem. /dev/sdb2 is mounted on /share [root@vcslinux176 ~]# umount /share umount: /share: device is busy umount: /share: device is busy [root@vcslinux176 ~]# iostat /dev/sdb2 Linux 2.6.9-prep (vcslinux176.vxindia.veritas.com) 10/19/2005 avg-cpu: %user %nice %sys %iowait %idle 0.60 0.00 1.87 7.60 89.93 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb2 891.26 10.80 3886.24 13565 4880962 [root@vcslinux176 ~]# sleep 100 [root@vcslinux176 ~]# iostat /dev/sdb2 Linux 2.6.9-prep (vcslinux176.vxindia.veritas.com) 10/19/2005 avg-cpu: %user %nice %sys %iowait %idle 0.51 0.00 1.59 6.45 91.45 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb2 756.44 9.17 3298.37 13565 4880962 [root@vcslinux176 ~]# umount /share umount: /share: device is busy umount: /share: device is busy [root@vcslinux176 ~]# exportfs -ua [root@vcslinux176 ~]# umount /share <<< umount successful Though nfs server had written all the client's data on the disk /dev/sdb2 (for 100seconds there was no write to disk), umount was possible only after an additional exportfs -u was executed.
Good... I'm glad your seeing the same thing I am... So the next step is to make exportfs -u wait for the nfsd to un-busy themselves so it can take the last reference off both the cached export entry and mount point. Thanks for taking the time to test this!
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2006-0132.html