Bug 61794

Summary: cannot umount after using nfs mount to loop mount
Product: [Retired] Red Hat Linux Reporter: Gene Czarcinski <gczarcinski>
Component: kernelAssignee: Arjan van de Ven <arjanv>
Status: CLOSED RAWHIDE QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3   
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2002-04-09 10:09:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 61901    
Attachments:
Description Flags
output from mount on the fresh-install system
none
output from mount on the 7.2->beta3 install system
none
fresh install install.log
none
fresh install anaconda-ks.cfg
none
fresh install strace umount
none
upgrade install 7.2 install.log
none
upgrade install beta3 upgrade.log
none
upgrade install anaconda-ks.cfg
none
upgrade install strace umount output none

Description Gene Czarcinski 2002-03-24 17:01:25 UTC
I am not sure what component to report this against.

The problem described below occured on a UPGRADE system (7.2 -> beta3).  I could
NOT get it to happen EVERY TIME on a similar FRESH install system (similar
components installed) -- but it has happened.


Description of Problem:
Having mounted a directory tree as a nfs export from a 7.2 system, I mounted an
iso image in the nfs mounted tree as a loopback mount.  I then did a rpm -qip
against a file on the loopback mount.  I then umounted the loopback and tried to
umount the nfs mount but got "umount: /mnt/images: device busy"

Version-Release number of selected component (if applicable):
7.2 upgraded to beta3

How Reproducible:
every time I tried

Steps to Reproduce:
1. upgrade install beta3
2. nfs mount (ro) a directory exported on a 7.2 system with iso images on it
3. loop mount (ro) an iso image
4. do rpm -qip on some package on the loop mounted tree
5. umount the loop mount
6. umount the nfs mount

Actual Results:
umount ... device busy

Expected Results:
nfs mount umounted.

Additional Information:
	
On the fresh install system, nautilus started for the loop mounted iso9660 tree
BUT on the upgraded system it DID NOT start.  I killed magicdev in case that was
the problem.

I am suspicious of nautilus.

Comment 1 Elliot Lee 2002-03-24 19:33:08 UTC
There are three possible reasons I know of why a filesystem is considered to be
 "busy":
1. A process is using it (either current directory or an open file).
 '/sbin/fuser -muv /the/mount/point' run as root will tell you what that process is.

2. Another filesystem is mounted in a subdirectory of the filesystem

3. The filesystem is being exported by knfsd (not the case here).

Please let me know which the problem is.

Comment 2 Gene Czarcinski 2002-03-24 20:57:59 UTC
1. The sequence of commands are the only ones executed before the last umount
was issued.

2. I tried logging off and then logging in as root but still cannot umount.

---

3. running /sbin/fuser -muv /mnt/images
produced no output.

4. df shows only /, /dev/shm, and /mnt/images (nfs mount)

5. /etc/exports is a null file.

Comment 3 Elliot Lee 2002-03-24 21:21:03 UTC
Run 'mount' or 'cat /proc/mounts' to show all mounted filesystems, just in case.

But, I can't reproduce the problem here with a beta3-ish system. I need evidence
that the problem isn't just an oversight on your end.

Comment 4 Gene Czarcinski 2002-03-24 22:06:59 UTC
I am going to put something out on the public beta mailing list to see if anyone
else sees this problem.

Comment 5 Gene Czarcinski 2002-03-24 22:08:09 UTC
Created attachment 50005 [details]
output from mount on the fresh-install system

Comment 6 Gene Czarcinski 2002-03-24 22:08:49 UTC
Created attachment 50006 [details]
output from mount on the 7.2->beta3 install system

Comment 7 Gene Czarcinski 2002-03-25 09:29:05 UTC
Sorry but this is a bug. See testers mailing list.

Comment 8 Gene Czarcinski 2002-03-25 09:37:43 UTC
I was FINALLY able to get rid of the mount by going to single user mode --

telinit 1



Comment 9 Gene Czarcinski 2002-03-25 10:40:56 UTC
I have been playing with this some more.  Sometimes it happens and then
sometimes it does not.

I get get the mount to go away by switching to telinit 1 or telinit 2.  During
the switch, the first message from nfs says that "umount2 failed because devise
busy" and is immediately followed by a message from "mount" that "/mnt/images
not mounted".

This is followed by a message that nfs (retry) OK.  But at the end of the
switch, there are three more nfs (retry) OK messages.

Any suggestions on determining what this is will be helpful.

Comment 10 Elliot Lee 2002-03-27 18:41:30 UTC
If it is a bug, I need information on reproducing it on my system. So far that
hasn't been possible :(

Comment 11 Gene Czarcinski 2002-03-27 19:32:42 UTC
OK, I can get this problem to occur on two separate systems -- one a fresh
install and the other a 7.2->beta3 upgrade.  I will attach the install and
upgrade logs for the two installs, the anaconda-ks.cfg files from the two
systems, and strace output from the failing umount for each system.

I was hoping you could have me do something to help isolate the problem.  I
expect it is something strange since I assume that redhat uses nfs a lot.

Please note that it is only after I access some file on the loop mounted
filesystem (pointing to an iso file on the nfs mounted filesystem) that the
problem occurs.

Comment 12 Gene Czarcinski 2002-03-27 19:33:52 UTC
Created attachment 50964 [details]
fresh install install.log

Comment 13 Gene Czarcinski 2002-03-27 19:34:26 UTC
Created attachment 50965 [details]
fresh install anaconda-ks.cfg

Comment 14 Gene Czarcinski 2002-03-27 19:35:14 UTC
Created attachment 50966 [details]
fresh install strace umount

Comment 15 Gene Czarcinski 2002-03-27 19:36:10 UTC
Created attachment 50967 [details]
upgrade install 7.2 install.log

Comment 16 Gene Czarcinski 2002-03-27 19:36:47 UTC
Created attachment 50968 [details]
upgrade install beta3 upgrade.log

Comment 17 Gene Czarcinski 2002-03-27 19:37:12 UTC
Created attachment 50969 [details]
upgrade install anaconda-ks.cfg

Comment 18 Gene Czarcinski 2002-03-27 19:37:41 UTC
Created attachment 50970 [details]
upgrade install strace umount output

Comment 19 Elliot Lee 2002-03-27 19:45:44 UTC
It's clear that the umount is failing because _something_ is still using the
filesystem, because there's no reason yet to assume that the kernel is lying :)

I think it just requires sufficient poking with fuser, etc. If going to runlevel
1 solves the problem, and you can't get fuser to produce useful information, try
stopping services/processes one by one until the umount succeeds, etc. etc.

Comment 20 Gene Czarcinski 2002-03-27 20:29:40 UTC
OK, switching to run level 1 (telinit 1) does free up the device according to
"df".  However, /proc/mounts still shows the mount.  In fact, it shows a whole
bunch of them (I did a bunch of runlevel switching).

Executing /etc/init.d/netfs stop fails but the runlevel switch says it fails the
first time but the succeeds.  However, it keeps doing retrys since there are
still entries in /proc/mounts.

I have tried stopping everything in sight with no effect.  In all cases, fuser
-vamu /mnt/images shows NOTHING.

Comment 21 Jay Turner 2002-03-27 21:10:15 UTC
OK, the good news is that we are able to replicate this now.  The bad news is
that it is a kernel bug.  More good news, there's a workaround in the mean time.

Just to clarify, here are the steps to reproduce:
1) mount an NFS export
2) on your local machine, navigate to an ISO image in that NFS export
3) loopback mount this ISO (so you have loopback mounted a NFS-exported file)
4) touch a file in the loopback mount with rpm or one of its derivatives
5) unmount the loopback mount
6) attempt to unmount the NFS export

You will get a message that the device or resource is busy, but fuser shows
nothing for that filesystem.

The workaround is to just remount the loopback mount, then immediately unmount
it.  You will then be able to unmount the NFS export.

Comment 22 Elliot Lee 2002-03-27 21:13:31 UTC
jkt reproduced it, the trick seems to be to use rpm to access some files
inside the loopback-mounted ISO.

Clearly a kernel bug, SEP.

Comment 23 Gene Czarcinski 2002-04-08 17:33:54 UTC
Confirmed it is still present in beta4

Comment 24 Gene Czarcinski 2002-04-08 20:16:32 UTC
Something odd happened.  Running under kde, I did the nfs mount, loop mount iso
image thing and everything accessed fine.  I then umounted the loop mount and
remounted it but this time the kde equivalent of nautilus did not come up.  I
could umount the loop mount but the nfs mount was "busy".

I then logged off and logged in as root under gnome.  I remounted the loop mount
and nautilus came up (closed it) and umounted the loop mount. I could then
umount the nfs mount.

Comment 25 Arjan van de Ven 2002-04-09 10:09:37 UTC
question: does
rpm -e autorun

help for the kde case ?

Comment 26 Arjan van de Ven 2002-04-14 12:40:34 UTC
Ok culprit found, identified and exterminated.
Should appear in rawhide as 2.4.18-0.25