From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.2) Gecko/20040301 Description of problem: automount fails to mount fileysystem. States it is already mounted. We see this in /var/log/messages Jan 11 09:34:00 acnlin86 automount[8590]: BUG: /cfs/h already mounted You can cd in /cfs/h. ls produces a zero list. A df . produces this.[root@acnlin86 h]# df . Filesystem 1K-blocks Used Available Use% Mounted on acnlin34.pbn.bnl.gov:/cfsh 0 0 0 - /cfs/h The /etc/auto.cfs entry looks like this: h -ro,intr,noquota,proto=udp acnlin34.pbn.bnl.gov:/cfsh The problem can sometimes be fixed by doing this: root@acnlin86 /]# umount /cfs/h umount: /cfs/h: not mounted [root@acnlin86 /]# cd /cfs/h [root@acnlin86 h]# ls Archive lost+found [root@acnlin86 h]# df . Filesystem 1K-blocks Used Available Use% Mounted on acnlin34.pbn.bnl.gov:/cfsh 1427662076 32836 1427629240 1% /cfs/h Version-Release number of selected component (if applicable): autofs-4.1.3-12 How reproducible: Didn't try Additional info: I can't reproduce this at whim. It occurs on various machines for various automounted filesystems from various clients. All running the same version of RH. We are constantly just checking the machines themselves to see if any particular filesystem has fallen into this state.
Do you happen to run amd as well on these systems? What other file system types are automounted?
Since I'm not sure what you mean by "amd", I think not. We are automounting the same kind of file systems from various linux servers. We also automount two mvfs filesystems for our clearcase product.
What precise file systems are being mounted? Ext2? Ext3? Samba shares? (we've already identified that nfs and mvfs file systems are being mounted) I'm asking because some filesystems implement their own version of 'mount,' which may or may not get the locking on /etc/mtab correct. You say you mount mvfs? What command is actually run to mount that fs? /sbin/mount? /sbin/mount.mvfs? My guess is there is /etc/mtab corruption. When this occurs, could you please capture the contents of /etc/mtab and /proc/mounts and attach them to this bugzilla? Thanks!
All of the filesystems are EXT3 being exported from Linux WS 3 systems. All of the filesystems are mounted through the autofs system, and when I do a which mount I find it in /bin/mount not /sbin/mount. acnlin82 54:file /bin/mount /bin/mount: setuid ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped I will update this again with the /proc/mounts data and /etc/mtab data when the problem next shows up.
here is the output from a machine which has this problem now. [root@acnlin89 root]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/md2 8111932 2944276 4755592 39% / /dev/md0 101018 15087 80715 16% /boot /dev/md5 20746036 32828 19659352 1% /home none 2062008 0 2062008 0% /dev/shm /dev/md3 124323 5242 112662 5% /trip /dev/md1 2063440 180280 1778344 10% /var /dev/mvfs 8111932 2944276 4755592 39% /view Here is the /proc/mounts and /etc/mtab listing. cfsb.pbn.bnl.gov:/cfsb 183658384 47136300 127192740 28% /cfs/b cfsx.pbn.bnl.gov:/cfsx 288467296 163681172 110132840 60% /cfs/x acnlin80.pbn.bnl.gov:/cfsad 1 1 1 0% /cfs/ad acnlin81.pbn.bnl.gov:/viewstore 418273764 33061904 385211860 8% /misc/viewstore vobs.pbn.bnl.gov:/vobstore 1 1 1 0% /misc/vobstore cfsy.pbn.bnl.gov:/cfsy 576934468 173304920 374322976 32% /cfs/y [root@acnlin89 root]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw 0 0 none /dev/pts devpts rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md0 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md3 /trip ext3 rw 0 0 /dev/md1 /var ext3 rw 0 0 automount(pid3213) /cfs autofs rw 0 0 automount(pid3211) /misc autofs rw 0 0 /dev/mvfs /view mvfs rw 0 0 /dev/mvfs /vobs/apps mvfs rw,nosuid 0 0 /dev/mvfs /vobs/admin mvfs rw,nosuid 0 0 /dev/mvfs /vobs/libs mvfs rw,nosuid 0 0 /dev/mvfs /vobs/store mvfs rw,nosuid 0 0 /dev/mvfs /vobs/servers mvfs rw,nosuid 0 0 /dev/mvfs /vobs/ados mvfs rw,nosuid 0 0 /dev/mvfs /vobs/drivers mvfs rw,nosuid 0 0 /dev/mvfs /vobs/firmware mvfs rw,nosuid 0 0 /dev/mvfs /vobs/japps mvfs rw,nosuid 0 0 /dev/mvfs /vobs/jservers mvfs rw,nosuid 0 0 /dev/mvfs /vobs/jpkgs mvfs rw,nosuid 0 0 /dev/mvfs /vobs/agsfec mvfs rw,nosuid 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0 acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin81.pbn.bnl.gov 0 0 vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=vobs.pbn.bnl.gov 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0 [root@acnlin89 root]# cat /etc/mtab /dev/md2 / ext3 rw 0 0 none /proc proc rw 0 0 none /dev/pts devpts rw,gid=5,mode=620 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md0 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md3 /trip ext3 rw 0 0 /dev/md1 /var ext3 rw 0 0 automount(pid3213) /cfs autofs rw,fd=4,pgrp=3213,minproto=2,maxproto=4 0 0 automount(pid3211) /misc autofs rw,fd=4,pgrp=3211,minproto=2,maxproto=4 0 0 /dev/mvfs /view mvfs rw,viewroot 0 0 /vobstore/apps.vbs /vobs/apps mvfs uuid=e71b7218.e27d11d0.b478.00:01:80:7f:13:0f,nosuid 0 0 /vobstore/admin.vbs /vobs/admin mvfs uuid=1a03ba3e.077e11d1.bd31.00:01:80:7f:13:0f,nosuid 0 0 /vobstore/libs.vbs /vobs/libs mvfs uuid=7616b72c.e1ca11d0.b625.00:01:80:7f:13:0f,exportid=1,nosuid 0 0 /vobstore/store.vbs /vobs/store mvfs uuid=1d5f9121.e31c11d0.a1c2.00:01:80:7f:13:0f,nosuid 0 0 /vobstore/servers.vbs /vobs/servers mvfs uuid=dbbee030.f14f11d0.ac2a.00:01:80:7f:13:0f,nosuid 0 0 /vobstore/ados.vbs /vobs/ados mvfs uuid=b0e6d1ff.aed811d1.a0c7.00:01:80:88:d7:50,nosuid 0 0 /vobstore/drivers.vbs /vobs/drivers mvfs uuid=d3a2974c.171e11d2.a788.00:01:80:75:73:ea,nosuid 0 0 /vobstore/firmware.vbs /vobs/firmware mvfs uuid=ac117aae.bb7611d2.bc1e.00:01:80:92:ff:71,nosuid 0 0 /vobstore/japps.vbs /vobs/japps mvfs uuid=2a9846cd.63f811d5.baae.00:01:80:e5:e7:e6,nosuid 0 0 /vobstore/jservers.vbs /vobs/jservers mvfs uuid=6df84890.63f811d5.babb.00:01:80:e5:e7:e6,nosuid 0 0 /vobstore/jpkgs.vbs /vobs/jpkgs mvfs uuid=8eb8496c.63f811d5.bac9.00:01:80:e5:e7:e6,nosuid 0 0 /vobstore/agsfec.vbs /vobs/agsfec mvfs uuid=675fcd28.e58b11d8.9ea9.00:07:e9:06:81:a7,nosuid 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0 acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.63 0 0 vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.97 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0
Actually this occurrance is a variation. The other variation is you can cd into the directory and it is just empty. Now a cd into the offending directory gives back a Permission Denied message. [root@acnlin89 x]# fuser -u /cfs/ad /cfs/ad: Permission denied [root@acnlin89 x]# lsof /cfs/ad lsof: WARNING: can't stat() nfs file system /cfs/ad Output information may be incomplete. [root@acnlin89 root]# cd /cfs/ad -bash: cd: /cfs/ad: Permission denied
Created attachment 109974 [details] Fix for the BUG: fs already mounted This patch was posted upstream recently, and claims to solve this problem for the /net filesystem. Please give it a try and see if it fixes things in your environment.
Note the following from /proc/mounts: acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0 and this from the mtab: acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0 So the kernel thinks it is mounted, too! This rules out mtab corruption. Please try the attached patch.
What attached patch? Patch number?
https://bugzilla.redhat.com/beta/attachment.cgi?id=109974
Could you point me to some documentation on what I should to with this "autofs-4.1.3-multi-over.patch: ASCII C program text" file? Never done this before. Thanks.
Okay, sorry about that. I've built a package for you. You can find it here: http://people.redhat.com/jmoyer/.bz144786/ Note that this package is *not* checked into CVS. As such, please don't reference it when reporting other bugs. ;-) If this fixes your problem, I will make a point to get this fix into the next release.
just to document the empty directory phenomenon. [root@acnlin86 cfsb]# cd /misc/opfecs [root@acnlin86 opfecs]# ls [root@acnlin86 opfecs]# df . Filesystem 1K-blocks Used Available Use% Mounted on cfsb.pbn.bnl.gov:/operations 0 0 0 - /misc/opfecs [root@acnlin86 opfecs]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw 0 0 none /dev/pts devpts rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3759) /cfs autofs rw 0 0 automount(pid3757) /misc autofs rw 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0 owl.pbn.bnl.gov:/rap /misc/rap nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0 owl.pbn.bnl.gov:/track /misc/track nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0 cfsa.pbn.bnl.gov:/diag /misc/diag nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsa.pbn.bnl.gov 0 0 acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin81.pbn.bnl.gov 0 0 vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=vobs.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0 acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0 acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0 acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0 cfsz.pbn.bnl.gov:/cfsz /cfs/z nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsz.pbn.bnl.gov 0 0 [root@acnlin86 opfecs]# cat /etc/mtab /dev/md0 / ext3 rw 0 0 none /proc proc rw 0 0 none /dev/pts devpts rw,gid=5,mode=620 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3759) /cfs autofs rw,fd=4,pgrp=3759,minproto=2,maxproto=4 0 0 automount(pid3757) /misc autofs rw,fd=4,pgrp=3757,minproto=2,maxproto=4 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0 acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0 acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 cfsb.pbn.bnl.gov:/operations /misc/opfecs nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0 owl.pbn.bnl.gov:/rap /misc/rap nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0 owl.pbn.bnl.gov:/track /misc/track nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0 cfsa.pbn.bnl.gov:/diag /misc/diag nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.154 0 0 acnlin81.pbn.bnl.gov:/viewstore /misc/viewstore nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.63 0 0 vobs.pbn.bnl.gov:/vobstore /misc/vobstore nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.104.97 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0 acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0 acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0 acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 cfsz.pbn.bnl.gov:/cfsz /cfs/z nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.207 0 0 I will be applying the patch to this machine today.
Ok we are still having the problem: Here is your patch installed: [root@acnlin86 k]# rpm -qa | grep autofs autofs-4.1.3-12.1 The bad mount is now on /cfs/k. In /var/log/messages we have [root@acnlin86 k]# tail /var/log/messages Jan 21 13:06:13 acnlin86 automount[11693]: BUG: /cfs/k already mounted Jan 21 13:06:13 acnlin86 automount[11696]: BUG: /cfs/k already mounted Jan 21 13:06:13 acnlin86 automount[11698]: BUG: /cfs/k already mounted Here are the tables [root@acnlin86 k]# pwd /cfs/k [root@acnlin86 k]# ls [root@acnlin86 k]# ls -la total 0 dr-xr-xr-x 2 root root 0 Jan 21 10:54 . drwxr-xr-x 9 root root 0 Jan 20 14:40 .. [root@acnlin86 k]# df . Filesystem 1K-blocks Used Available Use% Mounted on acnlin33.pbn.bnl.gov:/cfsk 0 0 0 - /cfs/k [root@acnlin86 k]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw 0 0 none /dev/pts devpts rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3730) /misc autofs rw 0 0 automount(pid3732) /cfs autofs rw 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0 acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0 acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0 acnlin34.pbn.bnl.gov:/cfsg /cfs/g nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin34.pbn.bnl.gov 0 0 acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin31.pbn.bnl.gov 0 0 acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0 acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin35.pbn.bnl.gov 0 0 [root@acnlin86 k]# cat /etc/mntab cat: /etc/mntab: No such file or directory [root@acnlin86 k]# cat /etc/mtab /dev/md0 / ext3 rw 0 0 none /proc proc rw 0 0 none /dev/pts devpts rw,gid=5,mode=620 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3730) /misc autofs rw,fd=4,pgrp=3730,minproto=2,maxproto=4 0 0 automount(pid3732) /cfs autofs rw,fd=4,pgrp=3732,minproto=2,maxproto=4 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0 acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0 acnlin35.pbn.bnl.gov:/cfsn /cfs/n nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 acnlin31.pbn.bnl.gov:/cfsi /cfs/i nfs ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0 acnlin34.pbn.bnl.gov:/cfsg /cfs/g nfs ro,intr,noquota,proto=udp,addr=130.199.111.198 0 0 acnlin31.pbn.bnl.gov:/cfsj /cfs/j nfs ro,intr,noquota,proto=udp,addr=130.199.110.176 0 0 acnlin35.pbn.bnl.gov:/cfsm /cfs/m nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 acnlin35.pbn.bnl.gov:/cfso /cfs/o nfs ro,intr,noquota,proto=udp,addr=130.199.109.107 0 0 [root@acnlin86 k]# grep cfsk /etc/auto.cfs k -ro,intr,noquota,proto=udp acnlin33.pbn.bnl.gov:/cfsk
ok so here's how it gets fixed when everything is works. [root@acnlin86 k]# pwd /cfs/k [root@acnlin86 k]# ls [root@acnlin86 k]# ls -la total 0 dr-xr-xr-x 2 root root 0 Jan 21 10:54 . drwxr-xr-x 9 root root 0 Jan 20 14:40 .. [root@acnlin86 k]# df . Filesystem 1K-blocks Used Available Use% Mounted on acnlin33.pbn.bnl.gov:/cfsk 0 0 0 - /cfs/k [root@acnlin86 k]# [root@acnlin86 k]# cd / [root@acnlin86 /]# umount /cfs/k umount: /cfs/k: not mounted [root@acnlin86 /]# cd /cfs/k [root@acnlin86 k]# ls Archive lost+found restore [root@acnlin86 k]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw 0 0 none /dev/pts devpts rw 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3730) /misc autofs rw 0 0 automount(pid3732) /cfs autofs rw 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsx.pbn.bnl.gov 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin80.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=www.cadops.bnl.gov 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=owl.pbn.bnl.gov 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsy.pbn.bnl.gov 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsb.pbn.bnl.gov 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfsf.pbn.bnl.gov 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=cfse.pbn.bnl.gov 0 0 acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs ro,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=acnlin33.pbn.bnl.gov 0 0 [root@acnlin86 k]# cat /etc/mtab /dev/md0 / ext3 rw 0 0 none /proc proc rw 0 0 none /dev/pts devpts rw,gid=5,mode=620 0 0 usbdevfs /proc/bus/usb usbdevfs rw 0 0 /dev/md3 /boot ext3 rw 0 0 /dev/md5 /home ext3 rw 0 0 none /dev/shm tmpfs rw 0 0 /dev/md1 /trip ext3 rw 0 0 /dev/md4 /var ext3 rw 0 0 automount(pid3730) /misc autofs rw,fd=4,pgrp=3730,minproto=2,maxproto=4 0 0 automount(pid3732) /cfs autofs rw,fd=4,pgrp=3732,minproto=2,maxproto=4 0 0 cfsx.pbn.bnl.gov:/cfsx /cfs/x nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.201 0 0 acnlin80.pbn.bnl.gov:/cfsad /cfs/ad nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.104.97 0 0 cfsb.pbn.bnl.gov:/cfsb /cfs/b nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.45 0 0 www.cadops.bnl.gov:/http /misc/http nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.110.134 0 0 owl.pbn.bnl.gov:/home/owl /misc/owl nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.90.129 0 0 cfsy.pbn.bnl.gov:/cfsy /cfs/y nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.238 0 0 cfsb.pbn.bnl.gov:/usr/public /misc/public nfs rw,hard,intr,rsize=8192,wsize=8192,addr=130.199.108.45 0 0 cfsf.pbn.bnl.gov:/cfsf /cfs/f nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.108.165 0 0 cfse.pbn.bnl.gov:/cfse /cfs/e nfs rw,hard,intr,rsize=8192,wsize=8192,proto=udp,addr=130.199.110.2 0 0 acnlin33.pbn.bnl.gov:/cfsk /cfs/k nfs ro,intr,noquota,proto=udp,addr=130.199.110.178 0 0 [root@acnlin86 k]#
Jan 24: Is there any other steps I can pursue to gather additional information? Roger
Well, the next step is to enable debugging for the autofs daemon. You can accomplish this as follows: 1) Edit /etc/sysconfig/autofs. You want to add an option to DAEMONOPTIONS like so: DAEMONOPTIONS="--debug" 2) Next, you want to make sure debug messages are logged somewhere. To do this, modify /etc/syslog.conf. You can simply add a line that looks like this: *.debug /var/log/debug 3) restart syslogd: service syslogd restart 4) restart autofs service autofs restart The next time the problem occurs, please attach the debug log (/var/log/debug) to this bugzilla. Thanks.
Created attachment 110433 [details] syslog debug file
did you get this attached file? anything else we can do?
It doesn't look like you ever successfully restarted the automounter. Please reboot and try again. Thanks.
Thanks, you were right. Started to get a good log....no failures...then the system got up2dated and autofs-4.1.3-47 was installed over your autofs-4.1.3-12.1.i386.rpm and --debug turned off. Should I go back to your version with debug on or just turn debug on with this newer version?
Please test with the newer version of the package. Thanks!
ok so since turning on debuging and using autofs-4.1.3-47 we have not seen a reoccurrance of this bug....so I guess we can close this one out.... Have other problems with automount...but should probably open another bugzilla about them.... Thanks
Have you seen any recurrences of the problem? If not, then I'll close out this bug. Thanks.
As I said above in comment #23 you can close this out
OK, thanks.
Can we re-open this bug, or should I open a new one? I've started seeing this on RHEL 3.0 U4 systems running autofs-4.1.3-47. (A user just complained about a filesystem, and going through my systems I'm finding numerous instances of this problem.) % rpm -q autofs autofs-4.1.3-47 % cd /var/log % grep BUG messages* messages:Jun 6 09:30:06 mm2dev65 automount[27261]: BUG: /home/rsopha1 already mounted messages.1:Jun 1 04:02:18 mm2dev65 automount[4823]: BUG: /usr/prod/vobstore116 already mounted messages.1:Jun 2 04:02:20 mm2dev65 automount[20946]: BUG: /usr/prod/vobstore116 already mounted messages.1:Jun 3 13:39:11 mm2dev65 automount[24747]: BUG: /home/amohapa1 already mounted messages.1:Jun 3 13:39:11 mm2dev65 automount[24748]: BUG: /home/amohapa1 already mounted messages.1:Jun 3 13:39:16 mm2dev65 automount[24781]: BUG: /home/amohapa1 already mounted messages.1:Jun 3 13:39:16 mm2dev65 automount[24782]: BUG: /home/amohapa1 already mounted messages.1:Jun 3 13:39:16 mm2dev65 automount[24786]: BUG: /home/amohapa1 already mounted messages.1:Jun 4 04:02:20 mm2dev65 automount[17559]: BUG: /usr/prod/vobstore116 already mounted messages.2:May 23 10:00:32 mm2dev65 automount[13573]: BUG: /usr/prod/ldsapplic already mounted messages.2:May 23 17:09:58 mm2dev65 automount[32710]: BUG: /usr/test/IN_fa6 already mounted messages.2:May 23 17:10:06 mm2dev65 automount[32767]: BUG: /home/nandi already mounted messages.2:May 24 12:42:51 mm2dev65 automount[2687]: BUG: /usr/prod/rvG4FOA1_dom already mounted messages.2:May 25 10:54:31 mm2dev65 automount[13304]: BUG: /usr/prod/vobstore116 already mounted messages.2:May 26 04:02:03 mm2dev65 automount[8615]: BUG: /usr/prod/vobstore116 already mounted messages.2:May 27 04:02:19 mm2dev65 automount[8350]: BUG: /usr/prod/vobstore116 already mounted % ls /usr/prod/vobstore116 ./ ../ % df -k /usr/prod/vobstore116 Filesystem 1K-blocks Used Available Use% Mounted on clearcase18:/dsk/v0/prod/vobstore116 0 0 0 - /usr/prod/vobstore116 % mount | grep vobstore116 /usr/prod/vobstore116/csas.vbs on /vobs/csas type mvfs (uuid=7502abec.7b9311d7.a5df.00:01:80:95:f9:07,nosuid) clearcase18:/dsk/v0/prod/vobstore116 on /usr/prod/vobstore116 type nfs (rw,grpid,nobrowse,intr,addr=136.182.8.219) % grep vobstore116 /proc/mounts [no output] % umount /usr/prod/vobstore116 umount: /usr/prod/vobstore116: not mounted % ls /usr/prod/vobstore116 [mount successful and contents properly list] Some observations: A) When this happens, the mount appears to be in an indeterminate state -- the mount command thinks it's mounted but /proc/mounts doesn't. B) This behaviour is exactly the same as behaviour that I described in bug 132831, which was a result of using an autofs 3.x init script with autofs 4.x, which resulted in the improper autofs module being loaded. That's *not* the case here, but the behaviour appears to be the same. C) I'm wondering if this may have something to do with ClearCase, as Roger (the bug opener) seems to have originally experienced this problem on a system with ClearCase loaded (mvfs filesystems are ClearCase multi-version filesystems). Note, however, that this problem, when experienced, is not limited to filesystems used by ClearCase.
The following will spit out a list of mounts where /etc/mtab and /proc/mounts are not in sync, revealing any instances of this problem on a system: cat /etc/mtab /proc/mounts | awk '{print $2}' | sort | uniq -u E.g.: % cat /etc/mtab /proc/mounts | awk '{print $2}' | sort | uniq -u /home/amohapa1 /home/nandi /home/qpz000 /home/rsopha1 /project/uisdocs /usr/prod/ldsapplic /usr/prod/rvG4FOA1_dom /usr/test/IN_fa6 /usr/test/userimage
Setting the state to reopened. Paul, could you please enable debugging and attach the debug logs showing the problem? Also, I would like to know if you can reproduce the problem with the current autofs, version 4.1.3-130, from Update 5. Thanks.
I'm trying to come back with some meaningful data here, but I'm encountering issues when I enable debug mode in autofs. For testing purposes I've created a perl script which automounts 500 directories, waits three minutes to let them expire, then checks for "dropped" mounts (lather, rinse, repeat until dropped mounts are detected). This seems to run okay normally, but when I enable debug level logging, I appear to be getting random hangs in the automount daemons -- it will stop expiring automounts and will not do any further automounts. I've seen this both with --debug in /etc/sysconfig/autofs (thus applying to all maps) as well as when I've killed the daemon for a single map and restarted it manually with --debug (to cut down on logging). I've seen this in U4 and U5 systems with and without ClearCase installed in VMWare virtual machines as well as in a real physical machine with U5+ClearCase. (I started working in VMWare, but when I encountered problems tried running on a real machine with the same effect.) Any suggestions? (Other than opening another bug... :)
This is a known issue. We call syslog from a signal handler, which is illegal (and causes these hangs). I've got another patch that addresses that problem. I can either send you the patch, or an updated package. Pick your poison.
An updated package is generally more convenient, but either's fine. Are we talking about a patch to autofs or syslog here? If autofs, I'd like to get a patch/patched version of both autofs-4.1.3-130 and autofs-4.1.3-47 so that I can test this on both U4 and U5.
It's a patch to autofs. I will not backport it to 4.1.3-47, I simply don't see any justification for doing so. A new package is posted on my people page: http://people.redhat.com/jmoyer/autofs/rhel3/autofs-4.1.3-136/ I'll attach the syslog patch shortly, in case you would like to backport it.
Created attachment 115293 [details] Patch to defer syslogs from signal handler context. This patch will fix hangs when autofs is run with debugging enabled. The hangs are due to unsafe calls to syslog from a signal handler, which result in deadlocks. This patch is NOT a fix for this bug, but rather a means for collecting more information.
All right. The -136 rpm has definitely resolved the problem with the automount daemon hanging when running with --debug turned on. I'm now running in my script on two U5 systems -- one with ClearCase installed and one without. If/when I see a drop, I'll attach relevant debug and normal log info.
We here are are experience the same issue with autofs4.1.3-47 on RHEL3 WS (U4) systems as described by Paul Waterman. To me it looks like a mtab corruption, to verify i linked /etc/mtab to /proc/mounts and even though i get flush errors for the mtab file, the system is stable and the mounts are not disappearing as they were earlier. I'd give autofs4.1.3-136 a try now and report it's stability.
Just adding an FYI... To date I have been unable to find a test case that reliably reproduces the problem -- it appears to happen extremely sporadically and unpredictably. I started by writing a perl script which autmounts ~500 filesystems, waits for unmount, checks for dropped mounts, and then repeats. I racked up ~1M mount/unmounts on three different test systems (U4+ClearCase, U5, and U5+ClearCase), and did not observe the problem on any of the filesystems in my test. Oddly, I did observe the problem on the U4 test system in my automounted home directory, but not in any of the other filesystems that I was repeatedly automounting. I then modified my test to actually read files from each filesystem after automounting, but again was unable to cause the problem to appear despite ~100k mount/read/unmount cycles on the U4+ClearCase system. I have not yet seen this dropped mount problem on any of our test U5 systems, so I *suspect* that the problem may be resolved in autofs4.1.3-130. However, as we don't currently have any production U5 systems I'm not willing to swear to it. Despite racking up ~3M mount/unmounts on test systems the problem did not occur.
I can now verify that moving to autofs4.1.4 on U4 has rectified this issue on almost all systems provided with the fix.
After the upgrade to autofs4.1.4,a few systems worked without issue for 2 weeks. However again the issue of missing mounts has cropped up. This time the log files dont show these as a 'BUG', but just report that filesystem is already mounted. There is a clear descrepance in the number of available mounts in mtab(106) as compared to /proc/mounts(83) On a few other systems, instead of upgrading i had just linked /etc/mtab to /proc/mounts and those system are looking good uptill now. A lot our user's are requesting downgrading to RHEl U1 which doesn't exhibhit this bug. Could someone comment on this issue and what should we be doing to fix this?
I personally would recommend against downgrading to U1, since there are a *lot* of kernel/stability issues that have been resolved since then. A couple suggestions: 1) You can increase the automount timeout (edit DAEMONOPTIONS in /etc/sysconfig/autofs) value. This will reduce the number of unmounts of automount filesystems and thus, in theory reduce the number of dropped mounts. RHEL defaults to 60 seconds; I'd suggest increasing this to 3600 seconds (one hour). Setting it to zero will eliminate the problem entirely, at the expense of never automatically unmounting unused automount filesystems. 2) You can set up a cron job to look for dropped mounts and clean them up. The following quick and dirty one-liner cleans up dropped mounts nicely: /bin/umount `/bin/cat /etc/mtab /proc/mounts | /bin/awk '{print $2}' | /bin/sort | /usr/bin/uniq -u`
Has anyone been able to track down the cause of this bug yet? Will there be a patched autofs package from RedHat anytime soon, maybe U6?
A) I haven't seen this problem on RHEL 3.0 U5, which uses autofs-4.1.3-130, so I suspect it was resolved somewhere between -47 and -130 (although it would be nice to know what *exactly* fixed it). B) U6 will include additional autofs updates -- the beta of U6 currently available includes 4.1.3-150.
I am not sure exactly what triggers the mtab corruption, but we definitely see the problem in the U5 package (autofs-4.1.3-130). We updated to an earlier U6 beta package (autofs-4.1.3-136 from comment #33) and now no longer have this problem.
> We updated to an earlier U6 beta package (autofs-4.1.3-136 from comment #33) > and now no longer have this problem. This is surprising news. Autofs doesn't directly modify the mtab file. It leaves this up to the mount and umount commands. To all of the reporters, can you tell me whether clearcase is involved in every instance of this problem?
I have not observed this on systems that are not running ClearCase (although I only have a couple that aren't, and they don't see much automount action). I *have* observed this on systems running both ClearCase 5.0 and 6.0.
We are not using ClearCase, I am not sure about the other BNL group.
Are folks still seeing this problem? If so, please provide me with all of the information requested in the section entitled, "Filing bug reports" on my people page: http://people.redhat.com/jmoyer/ I will need all of the information requested in order to diagnose the problem. Thanks.