Bug 144786

Do you happen to run amd as well on these systems?  What other file system types
are automounted?

Since I'm not sure what you mean by "amd", I think not.

We are automounting the same kind of file systems from various linux
servers.  We also automount two mvfs filesystems for our clearcase
product.

What precise file systems are being mounted?  Ext2?  Ext3?  Samba shares? 
(we've already identified that nfs and mvfs file systems are being mounted)  I'm
asking because some filesystems implement their own version of 'mount,' which
may or may not get the locking on /etc/mtab correct.

You say you mount mvfs?  What command is actually run to mount that fs? 
/sbin/mount?  /sbin/mount.mvfs?  My guess is there is /etc/mtab corruption.

When this occurs, could you please capture the contents of /etc/mtab and
/proc/mounts and attach them to this bugzilla?

Thanks!

All of the filesystems are EXT3 being exported from Linux WS 3 systems.

All of the filesystems are mounted through the autofs system, and when I do a
which mount I find it in /bin/mount not /sbin/mount.

acnlin82 54:file /bin/mount
/bin/mount: setuid ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for
GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped

I will update this again with the /proc/mounts data and /etc/mtab data when the
problem next shows up.

here is the output from a machine which has this problem now.

[root@acnlin89 root]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md2               8111932   2944276   4755592  39% /
/dev/md0                101018     15087     80715  16% /boot
/dev/md5              20746036     32828  19659352   1% /home
none                   2062008         0   2062008   0% /dev/shm
/dev/md3                124323      5242    112662   5% /trip
/dev/md1               2063440    180280   1778344  10% /var
/dev/mvfs              8111932   2944276   4755592  39% /view

Here is the /proc/mounts and /etc/mtab listing.

cfsb.pbn.bnl.gov:/cfsb
                     183658384  47136300 127192740  28% /cfs/b
cfsx.pbn.bnl.gov:/cfsx
                     288467296 163681172 110132840  60% /cfs/x
acnlin80.pbn.bnl.gov:/cfsad
                             1         1         1   0% /cfs/ad
acnlin81.pbn.bnl.gov:/viewstore
                     418273764  33061904 385211860   8% /misc/viewstore
vobs.pbn.bnl.gov:/vobstore
                             1         1         1   0% /misc/vobstore
cfsy.pbn.bnl.gov:/cfsy
                     576934468 173304920 374322976  32% /cfs/y
[root@acnlin89 root]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
none /dev/pts devpts rw 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md3 /trip ext3 rw 0 0
/dev/md1 /var ext3 rw 0 0
automount(pid3213) /cfs autofs rw 0 0
automount(pid3211) /misc autofs rw 0 0
/dev/mvfs /view mvfs rw 0 0
/dev/mvfs /vobs/apps mvfs rw,nosuid 0 0
/dev/mvfs /vobs/admin mvfs rw,nosuid 0 0
/dev/mvfs /vobs/libs mvfs rw,nosuid 0 0
/dev/mvfs /vobs/store mvfs rw,nosuid 0 0
/dev/mvfs /vobs/servers mvfs rw,nosuid 0 0
/dev/mvfs /vobs/ados mvfs rw,nosuid 0 0
/dev/mvfs /vobs/drivers mvfs rw,nosuid 0 0
/dev/mvfs /vobs/firmware mvfs rw,nosuid 0 0
/dev/mvfs /vobs/japps mvfs rw,nosuid 0 0
/dev/mvfs /vobs/jservers mvfs rw,nosuid 0 0
/dev/mvfs /vobs/jpkgs mvfs rw,nosuid 0 0
/dev/mvfs /vobs/agsfec mvfs rw,nosuid 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0
acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin81.pbn.bnl.gov 0 0
vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=vobs.pbn.bnl.gov 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0
[root@acnlin89 root]# cat /etc/mtab
/dev/md2 / ext3 rw 0 0
none /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md0 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md3 /trip ext3 rw 0 0
/dev/md1 /var ext3 rw 0 0
automount(pid3213) /cfs autofs rw,fd=4,pgrp=3213,minproto=2,maxproto=4 0 0
automount(pid3211) /misc autofs rw,fd=4,pgrp=3211,minproto=2,maxproto=4 0 0
/dev/mvfs /view mvfs rw,viewroot 0 0
/vobstore/apps.vbs /vobs/apps mvfs
uuid=e71b7218.e27d11d0.b478.00:01:80:7f:13:0f,nosuid 0 0
/vobstore/admin.vbs /vobs/admin mvfs
uuid=1a03ba3e.077e11d1.bd31.00:01:80:7f:13:0f,nosuid 0 0
/vobstore/libs.vbs /vobs/libs mvfs
uuid=7616b72c.e1ca11d0.b625.00:01:80:7f:13:0f,exportid=1,nosuid 0 0
/vobstore/store.vbs /vobs/store mvfs
uuid=1d5f9121.e31c11d0.a1c2.00:01:80:7f:13:0f,nosuid 0 0
/vobstore/servers.vbs /vobs/servers mvfs
uuid=dbbee030.f14f11d0.ac2a.00:01:80:7f:13:0f,nosuid 0 0
/vobstore/ados.vbs /vobs/ados mvfs
uuid=b0e6d1ff.aed811d1.a0c7.00:01:80:88:d7:50,nosuid 0 0
/vobstore/drivers.vbs /vobs/drivers mvfs
uuid=d3a2974c.171e11d2.a788.00:01:80:75:73:ea,nosuid 0 0
/vobstore/firmware.vbs /vobs/firmware mvfs
uuid=ac117aae.bb7611d2.bc1e.00:01:80:92:ff:71,nosuid 0 0
/vobstore/japps.vbs /vobs/japps mvfs
uuid=2a9846cd.63f811d5.baae.00:01:80:e5:e7:e6,nosuid 0 0
/vobstore/jservers.vbs /vobs/jservers mvfs
uuid=6df84890.63f811d5.babb.00:01:80:e5:e7:e6,nosuid 0 0
/vobstore/jpkgs.vbs /vobs/jpkgs mvfs
uuid=8eb8496c.63f811d5.bac9.00:01:80:e5:e7:e6,nosuid 0 0
/vobstore/agsfec.vbs /vobs/agsfec mvfs
uuid=675fcd28.e58b11d8.9ea9.00:07:e9:06:81:a7,nosuid 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0
acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.63 0 0
vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.97 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0

Actually this occurrance is a variation. The other variation is you can cd into
the directory and it is just empty.

Now a cd into the offending directory gives back a Permission Denied message.

[root@acnlin89 x]# fuser -u /cfs/ad
/cfs/ad: Permission denied
[root@acnlin89 x]# lsof  /cfs/ad
lsof: WARNING: can't stat() nfs file system /cfs/ad
      Output information may be incomplete.
[root@acnlin89 root]# cd /cfs/ad
-bash: cd: /cfs/ad: Permission denied

Created attachment 109974 [details]
Fix for the BUG: fs already mounted

This patch was posted upstream recently, and claims to solve this problem for
the /net filesystem.  Please give it a try and see if it fixes things in your
environment.

Note the following from /proc/mounts:

acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs 
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0

and this from the mtab:

acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0

So the kernel thinks it is mounted, too!  This rules out mtab corruption.

Please try the attached patch.

What attached patch? Patch number?

https://bugzilla.redhat.com/beta/attachment.cgi?id=109974

Could you point me to some documentation on what I should to with this
"autofs-4.1.3-multi-over.patch: ASCII C program text" file?

Never done this before.  Thanks.

Okay, sorry about that.  I've built a package for you.  You can find it here:
  http://people.redhat.com/jmoyer/.bz144786/

Note that this package is *not* checked into CVS.  As such, please don't
reference it when reporting other bugs.  ;-)  If this fixes your problem, I will
make a point to get this fix into the next release.

just to document the empty directory phenomenon.

[root@acnlin86 cfsb]# cd /misc/opfecs
[root@acnlin86 opfecs]# ls
[root@acnlin86 opfecs]# df .
Filesystem           1K-blocks      Used Available Use% Mounted on
cfsb.pbn.bnl.gov:/operations
                             0         0         0   -  /misc/opfecs
[root@acnlin86 opfecs]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
none /dev/pts devpts rw 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3759) /cfs autofs rw 0 0
automount(pid3757) /misc autofs rw 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0
owl.pbn.bnl.gov:/rap /misc/rap nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0
owl.pbn.bnl.gov:/track /misc/track nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0
cfsa.pbn.bnl.gov:/diag /misc/diag nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsa.pbn.bnl.gov 0 0
acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin81.pbn.bnl.gov 0 0
vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=vobs.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0
acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0
acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0
acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0
cfsz.pbn.bnl.gov:/cfsz /cfs/z nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsz.pbn.bnl.gov 0 0
[root@acnlin86 opfecs]# cat /etc/mtab
/dev/md0 / ext3 rw 0 0
none /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3759) /cfs autofs rw,fd=4,pgrp=3759,minproto=2,maxproto=4 0 0
automount(pid3757) /misc autofs rw,fd=4,pgrp=3757,minproto=2,maxproto=4 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0
acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs
ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0
acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
cfsb.pbn.bnl.gov:/operations /misc/opfecs nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0
owl.pbn.bnl.gov:/rap /misc/rap nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0
owl.pbn.bnl.gov:/track /misc/track nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0
cfsa.pbn.bnl.gov:/diag /misc/diag nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.154 0 0
acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.63 0 0
vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.97 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0
acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs
ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0
acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs
ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0
acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
cfsz.pbn.bnl.gov:/cfsz /cfs/z nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.207 0 0

I will be applying the patch to this machine today.

Ok we are still having the problem:
Here is your patch installed:
[root@acnlin86 k]# rpm -qa | grep autofs
autofs-4.1.3-12.1
The bad mount is now on /cfs/k.   In /var/log/messages we have
[root@acnlin86 k]# tail /var/log/messages
Jan 21 13:06:13 acnlin86 automount[11693]: BUG: /cfs/k already mounted
Jan 21 13:06:13 acnlin86 automount[11696]: BUG: /cfs/k already mounted
Jan 21 13:06:13 acnlin86 automount[11698]: BUG: /cfs/k already mounted

Here are the tables
[root@acnlin86 k]# pwd
/cfs/k
[root@acnlin86 k]# ls
[root@acnlin86 k]# ls -la
total 0
dr-xr-xr-x    2 root     root            0 Jan 21 10:54 .
drwxr-xr-x    9 root     root            0 Jan 20 14:40 ..
[root@acnlin86 k]# df .
Filesystem           1K-blocks      Used Available Use% Mounted on
acnlin33.pbn.bnl.gov:/cfsk
                             0         0         0   -  /cfs/k
[root@acnlin86 k]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
none /dev/pts devpts rw 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3730) /misc autofs rw 0 0
automount(pid3732) /cfs autofs rw 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0
acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0
acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0
acnlin34.pbn.bnl.gov:/cfsg /cfs/g nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin34.pbn.bnl.gov 0 0
acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0
acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0
acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0
[root@acnlin86 k]# cat /etc/mntab
cat: /etc/mntab: No such file or directory
[root@acnlin86 k]# cat /etc/mtab
/dev/md0 / ext3 rw 0 0
none /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3730) /misc autofs rw,fd=4,pgrp=3730,minproto=2,maxproto=4 0 0
automount(pid3732) /cfs autofs rw,fd=4,pgrp=3732,minproto=2,maxproto=4 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0
acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs
ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0
acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs
ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0
acnlin34.pbn.bnl.gov:/cfsg /cfs/g nfs
ro,intr,noquota,proto=udp,addr=130.199.111.198 0 0
acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs
ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0
acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs
ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0
[root@acnlin86 k]# grep cfsk /etc/auto.cfs
k    -ro,intr,noquota,proto=udp                acnlin33.pbn.bnl.gov:/cfsk

ok so here's how it gets fixed when everything is works.

[root@acnlin86 k]# pwd
/cfs/k
[root@acnlin86 k]# ls
[root@acnlin86 k]# ls -la
total 0
dr-xr-xr-x    2 root     root            0 Jan 21 10:54 .
drwxr-xr-x    9 root     root            0 Jan 20 14:40 ..
[root@acnlin86 k]# df .
Filesystem           1K-blocks      Used Available Use% Mounted on
acnlin33.pbn.bnl.gov:/cfsk
                             0         0         0   -  /cfs/k
[root@acnlin86 k]#
[root@acnlin86 k]# cd /
[root@acnlin86 /]# umount /cfs/k
umount: /cfs/k: not mounted
[root@acnlin86 /]# cd /cfs/k
[root@acnlin86 k]# ls
Archive  lost+found  restore
[root@acnlin86 k]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
none /dev/pts devpts rw 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3730) /misc autofs rw 0 0
automount(pid3732) /cfs autofs rw 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0
acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs
ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin33.pbn.bnl.gov 0 0
[root@acnlin86 k]# cat /etc/mtab
/dev/md0 / ext3 rw 0 0
none /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
usbdevfs /proc/bus/usb usbdevfs rw 0 0
/dev/md3 /boot ext3 rw 0 0
/dev/md5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/md1 /trip ext3 rw 0 0
/dev/md4 /var ext3 rw 0 0
automount(pid3730) /misc autofs rw,fd=4,pgrp=3730,minproto=2,maxproto=4 0 0
automount(pid3732) /cfs autofs rw,fd=4,pgrp=3732,minproto=2,maxproto=4 0 0
cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0
acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0
cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0
www.cadops.bnl.gov:/http /misc/http nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0
owl.pbn.bnl.gov:/home/owl /misc/owl nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0
cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0
cfsb.pbn.bnl.gov:/usr/public /misc/public nfs
rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0
cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0
cfse.pbn.bnl.gov:/cfse /cfs/e nfs
rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0
acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs
ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0
[root@acnlin86 k]#

Jan 24:

Is there any other steps I can pursue to gather additional information?

Roger

Well, the next step is to enable debugging for the autofs daemon.  You can
accomplish this as follows:

1) Edit /etc/sysconfig/autofs.  You want to add an option to DAEMONOPTIONS like so:
  DAEMONOPTIONS="--debug"

2) Next, you want to make sure debug messages are logged somewhere.  To do this,
modify /etc/syslog.conf.  You can simply add a line that looks like this:
  *.debug           /var/log/debug

3) restart syslogd:
  service syslogd restart

4) restart autofs
  service autofs restart

The next time the problem occurs, please attach the debug log (/var/log/debug)
to this bugzilla.

Thanks.

Created attachment 110433 [details]
syslog debug file

did you get this attached file?  anything else we can do?

It doesn't look like you ever successfully restarted the automounter.  Please
reboot and try again.

Thanks.

Thanks, you were right.  Started to get a good log....no failures...then the
system got up2dated and autofs-4.1.3-47 was installed over your
autofs-4.1.3-12.1.i386.rpm and --debug turned off.

Should I go back to your version with debug on or just turn debug on with this
newer version?

Please test with the newer version of the package.

Thanks!

ok so since turning on debuging and using autofs-4.1.3-47 we have not seen a
reoccurrance of this bug....so I guess we can close this one out....

Have other problems with automount...but should probably open another bugzilla
about them....

Thanks

Have you seen any recurrences of the problem?  If not, then I'll close out this bug.

Thanks.

As I said above in comment #23 you can close this out

OK, thanks.

Can we re-open this bug, or should I open a new one?

I've started seeing this on RHEL 3.0 U4 systems running autofs-4.1.3-47. (A user
just complained about a filesystem, and going through my systems I'm finding
numerous instances of this problem.)

% rpm -q autofs
autofs-4.1.3-47

% cd /var/log

% grep BUG messages*
messages:Jun  6 09:30:06 mm2dev65 automount[27261]: BUG: /home/rsopha1 already
mounted
messages.1:Jun  1 04:02:18 mm2dev65 automount[4823]: BUG: /usr/prod/vobstore116
already mounted
messages.1:Jun  2 04:02:20 mm2dev65 automount[20946]: BUG: /usr/prod/vobstore116
already mounted
messages.1:Jun  3 13:39:11 mm2dev65 automount[24747]: BUG: /home/amohapa1
already mounted
messages.1:Jun  3 13:39:11 mm2dev65 automount[24748]: BUG: /home/amohapa1
already mounted
messages.1:Jun  3 13:39:16 mm2dev65 automount[24781]: BUG: /home/amohapa1
already mounted
messages.1:Jun  3 13:39:16 mm2dev65 automount[24782]: BUG: /home/amohapa1
already mounted
messages.1:Jun  3 13:39:16 mm2dev65 automount[24786]: BUG: /home/amohapa1
already mounted
messages.1:Jun  4 04:02:20 mm2dev65 automount[17559]: BUG: /usr/prod/vobstore116
already mounted
messages.2:May 23 10:00:32 mm2dev65 automount[13573]: BUG: /usr/prod/ldsapplic
already mounted
messages.2:May 23 17:09:58 mm2dev65 automount[32710]: BUG: /usr/test/IN_fa6
already mounted
messages.2:May 23 17:10:06 mm2dev65 automount[32767]: BUG: /home/nandi already
mounted
messages.2:May 24 12:42:51 mm2dev65 automount[2687]: BUG: /usr/prod/rvG4FOA1_dom
already mounted
messages.2:May 25 10:54:31 mm2dev65 automount[13304]: BUG: /usr/prod/vobstore116
already mounted
messages.2:May 26 04:02:03 mm2dev65 automount[8615]: BUG: /usr/prod/vobstore116
already mounted
messages.2:May 27 04:02:19 mm2dev65 automount[8350]: BUG: /usr/prod/vobstore116
already mounted

% ls /usr/prod/vobstore116
./  ../

% df -k /usr/prod/vobstore116
Filesystem           1K-blocks      Used Available Use% Mounted on
clearcase18:/dsk/v0/prod/vobstore116
                             0         0         0   -  /usr/prod/vobstore116

% mount | grep vobstore116
/usr/prod/vobstore116/csas.vbs on /vobs/csas type mvfs
(uuid=7502abec.7b9311d7.a5df.00:01:80:95:f9:07,nosuid)
clearcase18:/dsk/v0/prod/vobstore116 on /usr/prod/vobstore116 type nfs
(rw,grpid,nobrowse,intr,addr=136.182.8.219)

% grep vobstore116 /proc/mounts
[no output]

% umount /usr/prod/vobstore116
umount: /usr/prod/vobstore116: not mounted

% ls /usr/prod/vobstore116
[mount successful and contents properly list]

Some observations:

A) When this happens, the mount appears to be in an indeterminate state -- the
mount command thinks it's mounted but /proc/mounts doesn't.

B) This behaviour is exactly the same as behaviour that I described in bug
132831, which was a result of using an autofs 3.x init script with autofs 4.x,
which resulted in the improper autofs module being loaded. That's *not* the case
here, but the behaviour appears to be the same.

C) I'm wondering if this may have something to do with ClearCase, as Roger (the
bug opener) seems to have originally experienced this problem on a system with
ClearCase loaded (mvfs filesystems are ClearCase multi-version filesystems).
Note, however, that this problem, when experienced, is not limited to
filesystems used by ClearCase.

The following will spit out a list of mounts where /etc/mtab and /proc/mounts
are not in sync, revealing any instances of this problem on a system:

cat /etc/mtab /proc/mounts | awk '{print $2}' | sort | uniq -u

E.g.:

% cat /etc/mtab /proc/mounts | awk '{print $2}' | sort | uniq -u
/home/amohapa1
/home/nandi
/home/qpz000
/home/rsopha1
/project/uisdocs
/usr/prod/ldsapplic
/usr/prod/rvG4FOA1_dom
/usr/test/IN_fa6
/usr/test/userimage

Setting the state to reopened.

Paul, could you please enable debugging and attach the debug logs showing the
problem?  Also, I would like to know if you can reproduce the problem with the
current autofs, version 4.1.3-130, from Update 5.

Thanks.

I'm trying to come back with some meaningful data here, but I'm encountering
issues when I enable debug mode in autofs.

For testing purposes I've created a perl script which automounts 500
directories, waits three minutes to let them expire, then checks for "dropped"
mounts (lather, rinse, repeat until dropped mounts are detected).

This seems to run okay normally, but when I enable debug level logging, I appear
to be getting random hangs in the automount daemons -- it will stop expiring
automounts and will not do any further automounts. I've seen this both with
--debug in /etc/sysconfig/autofs (thus applying to all maps) as well as when
I've killed the daemon for a single map and restarted it manually with --debug
(to cut down on logging). I've seen this in U4 and U5 systems with and without
ClearCase installed in VMWare virtual machines as well as in a real physical
machine with U5+ClearCase. (I started working in VMWare, but when I encountered
problems tried running on a real machine with the same effect.)

Any suggestions? (Other than opening another bug... :)

This is a known issue.  We call syslog from a signal handler, which is illegal
(and causes these hangs).  I've got another patch that addresses that problem.

I can either send you the patch, or an updated package.  Pick your poison.

An updated package is generally more convenient, but either's fine.

Are we talking about a patch to autofs or syslog here? If autofs, I'd like to
get a patch/patched version of both autofs-4.1.3-130 and autofs-4.1.3-47 so that
I can test this on both U4 and U5.

It's a patch to autofs.  I will not backport it to 4.1.3-47, I simply don't see
any justification for doing so.

A new package is posted on my people page:

  http://people.redhat.com/jmoyer/autofs/rhel3/autofs-4.1.3-136/

I'll attach the syslog patch shortly, in case you would like to backport it.

Created attachment 115293 [details]
Patch to defer syslogs from signal handler context.

This patch will fix hangs when autofs is run with debugging enabled.  The hangs
are due to unsafe calls to syslog from a signal handler, which result in
deadlocks.  This patch is NOT a fix for this bug, but rather a means for
collecting more information.

All right. The -136 rpm has definitely resolved the problem with the automount
daemon hanging when running with --debug turned on.

I'm now running in my script on two U5 systems -- one with ClearCase installed
and one without. If/when I see a drop, I'll attach relevant debug and normal log
info.

We here are are experience the same issue with autofs4.1.3-47 on RHEL3 WS (U4) 
systems as described by Paul Waterman. To me it looks like a mtab corruption, 
to verify i linked /etc/mtab to /proc/mounts and even though i get flush errors 
for the mtab file, the system is stable and the mounts are not disappearing as 
they were earlier.

I'd give autofs4.1.3-136 a try now and report it's stability.

Just adding an FYI... To date I have been unable to find a test case that
reliably reproduces the problem -- it appears to happen extremely sporadically
and unpredictably.

I started by writing a perl script which autmounts ~500 filesystems, waits for
unmount, checks for dropped mounts, and then repeats. I racked up ~1M
mount/unmounts on three different test systems (U4+ClearCase, U5, and
U5+ClearCase), and did not observe the problem on any of the filesystems in my
test. Oddly, I did observe the problem on the U4 test system in my automounted
home directory, but not in any of the other filesystems that I was repeatedly
automounting.

I then modified my test to actually read files from each filesystem after
automounting, but again was unable to cause the problem to appear despite ~100k
mount/read/unmount cycles on the U4+ClearCase system.

I have not yet seen this dropped mount problem on any of our test U5 systems, so
I *suspect* that the problem may be resolved in autofs4.1.3-130. However, as we
don't currently have any production U5 systems I'm not willing to swear to it.

Despite racking up ~3M mount/unmounts on test systems the problem did not occur. 

I can now verify that moving to autofs4.1.4 on U4 has rectified this issue on 
almost all systems provided with the fix.

After the upgrade to autofs4.1.4,a few systems worked without issue for 2 
weeks. However again the issue of missing mounts has cropped up. This time the 
log files dont show these as a 'BUG', but just report that filesystem is 
already mounted. There is a clear descrepance in the number of available mounts 
in mtab(106) as compared to /proc/mounts(83)

On a few other systems, instead of upgrading i had just linked /etc/mtab 
to /proc/mounts and those system are looking good uptill now.

A lot our user's are requesting downgrading to RHEl U1 which doesn't exhibhit 
this bug.

Could someone comment on this issue and what should we be doing to fix this?

I personally would recommend against downgrading to U1, since there are a *lot*
of kernel/stability issues that have been resolved since then.

A couple suggestions:

1) You can increase the automount timeout (edit DAEMONOPTIONS in
/etc/sysconfig/autofs) value. This will reduce the number of unmounts of
automount filesystems and thus, in theory reduce the number of dropped mounts.
RHEL defaults to 60 seconds; I'd suggest increasing this to 3600 seconds (one
hour). Setting it to zero will eliminate the problem entirely, at the expense of
never automatically unmounting unused automount filesystems.

2) You can set up a cron job to look for dropped mounts and clean them up. The
following quick and dirty one-liner cleans up dropped mounts nicely:

/bin/umount `/bin/cat /etc/mtab /proc/mounts | /bin/awk '{print $2}' | /bin/sort
| /usr/bin/uniq -u`

Has anyone been able to track down the cause of this bug yet?  Will there be a
patched autofs package from RedHat anytime soon, maybe U6?

A) I haven't seen this problem on RHEL 3.0 U5, which uses autofs-4.1.3-130, so I
suspect it was resolved somewhere between -47 and -130 (although it would be
nice to know what *exactly* fixed it).

B) U6 will include additional autofs updates -- the beta of U6 currently
available includes 4.1.3-150.

I am not sure exactly what triggers the mtab corruption, but we definitely see
the problem in the U5 package (autofs-4.1.3-130).  We updated to an earlier U6
beta package (autofs-4.1.3-136 from comment #33) and now no longer have this
problem.

> We updated to an earlier U6 beta package (autofs-4.1.3-136 from comment #33)
> and now no longer have this problem.

This is surprising news.  Autofs doesn't directly modify the mtab file.  It
leaves this up to the mount and umount commands.

To all of the reporters, can you tell me whether clearcase is involved in every
instance of this problem?

I have not observed this on systems that are not running ClearCase (although I
only have a couple that aren't, and they don't see much automount action).

I *have* observed this on systems running both ClearCase 5.0 and 6.0.

We are not using ClearCase, I am not sure about the other BNL group.

Are folks still seeing this problem?  If so, please provide me with all of the
information requested in the section entitled, "Filing bug reports" on my people
page:  http://people.redhat.com/jmoyer/

I will need all of the information requested in order to diagnose the problem.

Thanks.