Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 154387 - (IT_70798) umount fails on nfs server side when nfs client does heavy io
umount fails on nfs server side when nfs client does heavy io
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: nfs-utils (Show other bugs)
4.0
All Linux
medium Severity high
: ---
: ---
Assigned To: Steve Dickson
Ben Levenson
:
Depends On:
Blocks: 168429
  Show dependency treegraph
 
Reported: 2005-04-11 07:23 EDT by kiran mehta
Modified: 2007-11-30 17:07 EST (History)
5 users (show)

See Also:
Fixed In Version: RHSA-2006-0132
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-03-07 13:52:42 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
program to do io (21.70 KB, application/octet-stream)
2005-05-20 10:26 EDT, kiran mehta
no flags Details
Upstream patch (5.53 KB, patch)
2005-05-31 09:29 EDT, Steve Dickson
no flags Details | Diff
Updated Patch for a 2.6.9-22 kernel (4.65 KB, patch)
2005-10-18 06:36 EDT, Steve Dickson
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2005:808 normal SHIPPED_LIVE Important: kernel security update 2005-10-27 00:00:00 EDT
Red Hat Product Errata RHSA-2006:0132 qe-ready SHIPPED_LIVE Moderate: Updated kernel packages available for Red Hat Enterprise Linux 4 Update 3 2006-03-09 11:31:00 EST

  None (edit)
Description kiran mehta 2005-04-11 07:23:57 EDT
From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; FunWebProducts)

Description of problem:
1. Whenver nfs client does good amount of io , mounts remain busy even after 
   unexporting all the filesystems.

2.fuser does not show any user for the busy mount.

3.After around 20-30 min , filesystem can be unmounted

4.Umount can unmount filesystem in only one call if nfsd is killed manually

Version-Release number of selected component (if applicable):
nfs-utils-1.0.6-46

How reproducible:
Always

Steps to Reproduce:
1.exported filesystems to client
2.On client side  , mount the exported filesystem and do io on it using 
  multiple threads
3.Wait for around 2 min , unexport the filessytems and then try to umount filesystem.
  

Actual Results:  Its was not possible to umount the filesystems mounted by client.

Expected Results:  After unexporting filessytems , umounting should happen without any problem
and this should not require killing of nfsd

Additional info:
Comment 1 Steve Dickson 2005-04-11 10:16:57 EDT
What is the kernel version? 
Comment 2 kiran mehta 2005-04-11 10:29:46 EDT
Tested on two kernel versions

Machine 1 (RHEL4 GA):
----------
$ uname -a
Linux vcslinux6.vxindia.veritas.com 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST
$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant)

Machine 2 (RHEL4U1 Beta):
----------
$ uname -a
Linux thorpci25.veritas.com 2.6.9-6.28.ELv #1 SMP Wed Mar 23 20:53:13 GMT 2005 
ia64 ia64 ia64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant)
Comment 3 kiran mehta 2005-04-12 02:43:53 EDT
Just to make it clear.
  "Mounts are busy on machine running nfs server"
Comment 4 Steve Dickson 2005-04-13 08:26:59 EDT
Again, just to be clear....

Your unexporting the the filsystems on the server
and then the unmount from the clients are hanging, correct?
Comment 5 kiran mehta 2005-04-13 08:54:54 EDT
i am unexporting filesystems on server and then umount on --server-- are hanging
Comment 6 Steve Dickson 2005-04-13 10:18:35 EDT
Ok... Does "server nfs stop" allow the local (to the server) 
filesystems to be unmouted? 
Comment 7 kiran mehta 2005-04-13 10:28:48 EDT
Yes, after runnnig server nfs stop , umount will happen.
As i said earlier , after killing nfsd  , umount happens properly.
But its not possible to umount  without killing nfsd
 
Comment 8 Steve Dickson 2005-04-13 15:25:59 EDT
OK... So the expected action is for the umount to
fail with EBUSY instead of just hang.... At least
that's what I would expect...
Comment 9 kiran mehta 2005-04-13 23:49:36 EDT
Currently , umount fails with EBUSY.
Shouldn't umount flush all buffers(used by nfs) on disk and cleanly umount
the disk?
Why is it failing with EBUSY ?
Comment 10 kiran mehta 2005-04-18 11:34:32 EDT
Please let us know when can we get fix for this ?
Comment 11 Steve Dickson 2005-04-28 05:52:53 EDT
Unless I'm misunderstanding something, this is not a problem.
The server has to ensure the filesystems its exporting stay mounted.
Do other NFS implementation allow you unmount exported filesystems?
Comment 12 kiran mehta 2005-04-28 07:11:32 EDT
Hi Steve , 
  In "Steps to Reproduce"
  I have mentioned that filesystem is unexported  before umount is called.
  One filessytem is unexported , umount should succeed.But due to some
  reasons mount remains busy. 
Comment 13 kiran mehta 2005-04-29 03:25:23 EDT
Its reproducible everytime the "Steps to Reproduce" are followed.
Were you able to reproduce this issue ?
Comment 14 Steve Dickson 2005-04-29 08:29:01 EDT
Sorry for the confusion... exportfs -u does seem to be broken.
Comment 15 kiran mehta 2005-04-29 08:42:22 EDT
Thanks Steve.
 when can we expect the fix
Comment 16 kiran mehta 2005-05-02 08:26:23 EDT
As far as i know , exportfs maintains/manipulates table of exported  
filesystems.  
It is not at all concerned with io.  
Problem occurs  when client does io on mounted FS and not otherwise.  
If problem is with exportfs , it should also occur when client  
does not do io.  
Its not clear how exportfs is responsible for this problem.  
  
Comment 17 kiran mehta 2005-05-12 00:40:55 EDT
any update on this ? 
Comment 18 kiran mehta 2005-05-19 01:14:01 EDT
Steve , 
 Please let us know if the problem has been identified/fixed.
 This fix is very important for us.
Comment 19 kiran mehta 2005-05-19 01:15:56 EDT
I am not able to raise priority from normal->high.
Comment 20 Steve Dickson 2005-05-20 09:25:49 EDT
Unfortunately I have not been able to reproduce this problem. Here
are the step I'm taking:

server> service nfs start
server> exportfs -v
/home           <world>(rw,wdelay,nohide,root_squash,fsid=2)
/usr            <world>(rw,wdelay,nohide,root_squash,fsid=1)
/               <world>(ro,wdelay,root_squash,fsid=0)

client> mount server:/home /mnt/server/home
client> cat /tmp/100m > /mnt/server/home/100m

server> exportfs -u \*:/home
exportfs -v
/usr            <world>(rw,wdelay,nohide,root_squash,fsid=1)
/               <world>(ro,wdelay,root_squash,fsid=0)

server> umount /home
[the filesystem unmounts]

What am I not doing to cause this hang?
Comment 21 kiran mehta 2005-05-20 09:38:54 EDT
Let client do i "continuously" for 5 min and then while io is 
in progress try unexporting and unmounting. 
This should be enough to cause mount  busy 
Even the "sync" option for exportfs could not solve this problem. 
Comment 22 kiran mehta 2005-05-20 10:26:45 EDT
Created attachment 114627 [details]
program to do io

Steve , 
	   You can use the attached program to do io on client side.
      syntax: >./disfk <no. of threads> mountpt1 mountpt2 mountpt3......

	   Eg: ./diskf 3  /mnt/mntpt1
Comment 23 kiran mehta 2005-05-25 09:04:20 EDT
did the program help to reproduce the issue ?  
Comment 24 Marty Wesley 2005-05-26 02:55:16 EDT
PM ACK for U2.
Comment 25 Steve Dickson 2005-05-31 09:29:08 EDT
Created attachment 114989 [details]
Upstream patch

I've been working with this upstream patch that helps
but does not completely take care of the the problem.

The patch does allow the local filesystem to be umounted
after the exportfs -u when the exported filesystem is idle,
but when the filesystem is busy, a second exportfs -u is needed
to remove an extra reference... I'm still looking into where
that extra reference is coming from.
Comment 26 kiran mehta 2005-05-31 10:03:29 EDT
If filesystem is busy ,  FS does not necessarily get umounted after second
exportfs -u.
Most(90-95%) of the times even 25-30 exportfs -u do not help in unmounting the 
Filesystem.
Comment 27 kiran mehta 2005-05-31 10:17:02 EDT
This is when your patch is not applied
Comment 33 Steve Dickson 2005-10-12 14:27:20 EDT
I've finally gotten back to this and it appears the patch
in Comment #24 is most definitely needed since it eliminates
a bit flag what was just getting in the way...

Now the "extra reference" talked about in Comment #24 is
not really an extra references at all... Its more like a missed
reference... Here's what appears to be happening (using a
patched kernel).

When the exportfs -u is done on a busy export  (i.e. the client
is witting to a file in the exported directory), the kernel export
code notices that the export is still referenced (because one
of the nfsd has not finished yet), so exportfs only takes the
export out of the exported namespace (meaning clients and not
longer see it but there is still a reference on the mount point).

Next the nfsd finishes and dereferences the export, which means
at this point another exportfs -u is need to dereference the export
which in turn will cause the a dereference of the  mount point which
allows the mount point to be unmounted.

I'm currently looking into way of making the exportfs command
wait until the exported directory is complete de-exported before
returning. Unfortunately, it appears this is not a trivial task...
Comment 34 kiran mehta 2005-10-13 23:09:21 EDT
Hi Steve,
    What i undertsand from your comment is that exportfs holds a reference
    to the exported filesystem and if we try to unexport a busy exported
    filesystem, exportfs removes fs from exports table but still preserves
    reference to that fs.

Queries:
1. Why should exportfs hold reference to exported filesystem ?
2. Should exportfs just maintain exports table without concerning
   status(busy/not busy) of exported filesytem ?

------------------------------------------------------------------------------

Let me know if there is problem with this approach :

1.Exportfs should maintain exports table and should hold no references to 
  exported filesystem(otherwise we will need second exportfs -u to remove the 
  reference as in your comment)
2.When user wants to unexport the fs, remove it from exports table
  From here onwards if nfs client tries to access the fs, it will get
  permission denied.
3.If user wants to unmount the fs, flush all the buffers used for that
  filesystem and unmount it cleanly.As exportfs has not reference to this
  fs, there should not be any problem unmounting it.

Let me know if i am missing something   

Comment 35 Steve Dickson 2005-10-14 10:06:54 EDT
> 1. Why should exportfs hold reference to exported filesystem ?
Well exportfs is not  hold a reference on the filesystem ( I guess I
was not very clear with that), its the kernel holding the reference on
the filesystem , which happens when the  client mounts the filesystem.

Note: There are actually two reference counts I was referring to in
Comment #24; One on the mounted filesystem (mnt_count) and the other
on cached export entry in the kernel (refcnt, in cache_head of the
svc_export structure). So when I said "the kernel export code notices
that the export is still referenced" I meant the refcnt in the cache_head
of the svc_export structure).

>2. Should exportfs just maintain exports table without concerning
      status(busy/not busy) of exported filesystem ?
It does. But when it comes time to for exportfs to flush the cached
exports and the cached refcnt is >1 (because an nfsd is still using it),
the cache fill not be flushed. Note: the reference count on the filesystem
(i.e. mnt_count) is only decremented when the cached export entry
is not in use (i.e. refcnt == 1).

>1.Exportfs should maintain exports table and should hold no references to 
>  exported filesystem(otherwise we will need second exportfs -u to remove the 
>  reference as in your comment)
This does happen... again the refcnt is on the export and mount point
are held by the kernel.

> 2.When user wants to unexport the fs, remove it from exports table
>  From here onwards if nfs client tries to access the fs, it will get
>  permission denied.
This does happen... 

>3.If user wants to unmount the fs, flush all the buffers used for that
>  filesystem and unmount it cleanly.As exportfs has not reference to this
>  fs, there should not be any problem unmounting it.
Again its not exportfs with the reference, its the nfsd with the
references on the cached export which in turn has a reference on the
filesystem... Once the cached reference goes to 1 and a cach_flush()
is done (i.e by exportfs), the filesystem will be unmount-able

I hope this help... 

BTW, did you get a chance to test the patch?



Comment 36 kiran mehta 2005-10-14 10:26:25 EDT
I have not yet tested the patch
Will  do it next week.

Comment 38 Steve Dickson 2005-10-18 06:36:50 EDT
Created attachment 120119 [details]
Updated Patch for a 2.6.9-22 kernel

Here is updated patch that can be used on RHEL4 kernels.
Comment 40 kiran mehta 2005-10-19 01:28:42 EDT
Tested the patch.
As you told,
Additional exportfs -u  is required to umount the filesystem.
/dev/sdb2 is mounted on /share

[root@vcslinux176 ~]# umount /share
umount: /share: device is busy
umount: /share: device is busy
[root@vcslinux176 ~]# iostat /dev/sdb2
Linux 2.6.9-prep (vcslinux176.vxindia.veritas.com)      10/19/2005

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.60    0.00    1.87    7.60   89.93

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb2            891.26        10.80      3886.24      13565    4880962

[root@vcslinux176 ~]# sleep 100

[root@vcslinux176 ~]# iostat /dev/sdb2
Linux 2.6.9-prep (vcslinux176.vxindia.veritas.com)      10/19/2005

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.51    0.00    1.59    6.45   91.45

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb2            756.44         9.17      3298.37      13565    4880962

[root@vcslinux176 ~]# umount /share
umount: /share: device is busy
umount: /share: device is busy
[root@vcslinux176 ~]# exportfs -ua
[root@vcslinux176 ~]# umount /share    <<< umount successful

Though nfs server had written all the
client's data on the disk /dev/sdb2
(for 100seconds there was no write to disk), umount was possible
only after an additional exportfs -u was executed.
Comment 41 Steve Dickson 2005-10-20 10:51:13 EDT
Good... I'm glad your seeing the same thing I am... So
the next step is to make exportfs -u wait for the nfsd to
un-busy themselves so it can take the last reference off
both the cached export entry and mount point.

Thanks for taking the time to test this!
Comment 48 Red Hat Bugzilla 2006-03-07 13:52:42 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0132.html

Note You need to log in before you can comment on or make changes to this bug.