Bug 745775

Summary: Unable to unmount autofs filesystems inside a container
Product: Red Hat Enterprise Linux 6 Reporter: Daniel Berrangé <berrange>
Component: kernelAssignee: Ian Kent <ikent>
Status: CLOSED ERRATA QA Contact: yanfu,wang <yanwang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.3CC: ikent, jmoyer, rwheeler, yanwang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-2.6.32-238.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 920036 1048056 (view as bug list) Environment:
Last Closed: 2012-06-20 07:57:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 920036, 1048056    
Attachments:
Description Flags
Demo to illustrate umount problems inside a container
none
Patch - vfs: automount should ignore LOOKUP_FOLLOW
none
Patch - vfs pathname lookup: Add LOOKUP_AUTOMOUNT flag
none
Patch - VFS: Fix the remaining automounter semantics regressions
none
Patch - vfs: remove LOOKUP_NO_AUTOMOUNT flag
none
Corrected problem demo program none

Description Daniel Berrangé 2011-10-13 11:29:21 UTC
Created attachment 527958 [details]
Demo to illustrate umount problems inside a container

Description of problem:
One of the first things that is required when setting up a container with a private root filesystem, is to unmount all filesystems inherited from the host OS.

For most filesystems this works without trouble, but for autofs this is not the case.

For demonstration, consider an autofs map with one direct mount and one indirect mount:

# cat /etc/auto.master 
/net	-hosts
/-        /etc/auto.marrow

# cat /etc/auto.marrow 
/mnt/demo marrow.example.com:/var/lib/libvirt/images

When autofs initially starts these mount points are visible

# grep -E ' (/net|/mnt)' /proc/mounts 
-hosts /net autofs rw,relatime,fd=6,pgrp=2938,timeout=300,minproto=5,maxproto=5,indirect 0 0
/etc/auto.marrow /mnt/demo autofs rw,relatime,fd=12,pgrp=2938,timeout=300,minproto=5,maxproto=5,direct 0 0


The attached demo program creates a container and attempts to unmount the requested filesystem inside the container.

First trying to unmount '/net' fails:

# ./autofsdemo /net
We are the parent
We are the container!
Found mount point 1 /net -hosts autofs rw,relatime,fd=6,pgrp=2938,timeout=300,minproto=5,maxproto=5,indirect
Umount point 1 /net
Umount point 0 /net
Could not umount /net: Invalid argument


If we then trigger a mount within that location

# ls /net/marrow.example.com
bin  boot  cgroup  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var


And now try to unmount that volume inside the container

# ./autofsdemo /net/marrow.example.com
We are the parent
We are the container!
Found mount point 1 /net/marrow.example.com marrow.example.com:/ nfs rw,nosuid,nodev,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.1.2,mountvers=3,mountport=35386,mountproto=udp,local_lock=none,addr=10.1.1.2
Umount point 1 /net/marrow.example.com
Could not umount /net/marrow.example.com: Device or resource busy


Next up consider the other autofs location which is not triggered yet:

# ./autofsdemo /mnt/demo
We are the parent
We are the container!
Found mount point 1 /mnt/demo /etc/auto.marrow autofs rw,relatime,fd=12,pgrp=2938,timeout=300,minproto=5,maxproto=5,direct
Umount point 1 /mnt/demo
Could not umount /mnt/demo: Too many levels of symbolic links


Some odd has now happened though. The container has managed to trigger the mounting of this location in the host OS:

# grep /mnt /proc/mounts 
/etc/auto.marrow /mnt/demo autofs rw,relatime,fd=12,pgrp=2938,timeout=300,minproto=5,maxproto=5,direct 0 0
marrow.example.com:/var/lib/libvirt/images /mnt/demo nfs rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.1.2,mountvers=3,mountport=35386,mountproto=udp,local_lock=none,addr=10.1.1.2 0 0


Re-run the demo again, shows we can at least unmount the NFS volume this time, but still can't unmount the autofs point

# ./autofsdemo /mnt/demo
We are the parent
We are the container!
Found mount point 1 /mnt/demo /etc/auto.marrow autofs rw,relatime,fd=12,pgrp=2938,timeout=300,minproto=5,maxproto=5,direct
Found mount point 2 /mnt/demo marrow.example.com:/var/lib/libvirt/images nfs rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.1.2,mountvers=3,mountport=35386,mountproto=udp,local_lock=none,addr=10.1.1.2
Umount point 2 /mnt/demo
Umount point 1 /mnt/demo
Could not umount /mnt/demo: Too many levels of symbolic links



So there are multiple bugs here

 1. You can't unmount autofs filesystems of type 'indirect' - Invalid argument
 2. You can't unmount NFS filesystems underneath an 'indirect' autofs mount - Device or resource busy
 3. You can't unmount autofs filesystems of type 'direct' - Too many levels of symbolic links
 4. Attempting to umount direct mounts, actually triggers mounting of the NFS volume in the host OS

None of this would be fatal, if it were possible to use 'umount2($path, MNT_DETACH)' to hide the mount from the container, but that also fails.


Version-Release number of selected component (if applicable):
kernel 2.6.32-207.el6.x86_64
autofs-5.0.5-23.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. See above

Comment 3 RHEL Program Management 2011-10-18 18:40:49 UTC
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 4 Jeff Moyer 2011-11-28 14:33:19 UTC
Dan mentioned that this was fixed in the 3.1 kernel.

Comment 7 Ian Kent 2012-02-01 12:38:53 UTC
Created attachment 558812 [details]
Patch - vfs: automount should ignore LOOKUP_FOLLOW

Comment 8 Ian Kent 2012-02-01 12:40:06 UTC
Created attachment 558813 [details]
Patch - vfs pathname lookup: Add LOOKUP_AUTOMOUNT flag

Comment 9 Ian Kent 2012-02-01 12:41:34 UTC
Created attachment 558814 [details]
Patch - VFS: Fix the remaining automounter semantics regressions

Comment 10 Ian Kent 2012-02-01 12:43:01 UTC
Created attachment 558815 [details]
Patch - vfs: remove LOOKUP_NO_AUTOMOUNT flag

Comment 11 Ian Kent 2012-02-01 12:44:51 UTC
The above patches are essentially the changes that went into
kernel 3.1 that resolved the umount problem in Fedora.

Let's give them a try here in RHEL please.

Comment 12 RHEL Program Management 2012-02-01 12:49:35 UTC
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

Comment 13 Aristeu Rozanski 2012-02-24 21:55:44 UTC
Patch(es) available on kernel-2.6.32-238.el6

Comment 19 Ian Kent 2012-05-10 08:01:34 UTC
Created attachment 583470 [details]
Corrected problem demo program

I think this demo program has the corrections I described.
Please give it a try.

Comment 20 Ian Kent 2012-05-10 08:07:08 UTC
(In reply to comment #19)
> Created attachment 583470 [details]
> Corrected problem demo program
> 
> I think this demo program has the corrections I described.
> Please give it a try.

Note that the test program is not intelligent at all so it
can only check using simple mounts. So a hosts map with a
tree of mounts isn't appropriate and can't be used. But
what we want to test is if mounts inside the container can
be umounted at all.

Comment 21 yanfu,wang 2012-05-14 06:37:39 UTC
Verified on kernel 2.6.32-269.el6:
[root@sgi-xe320-01 ~]# rpm -qa|grep autofs
autofs-5.0.5-54.el6.x86_64
[root@sgi-xe320-01 ~]# uname -a
Linux sgi-xe320-01.rhts.eng.bos.redhat.com 2.6.32-269.el6.x86_64 #1 SMP Tue May 1 19:16:31 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

[root@sgi-xe320-01 ~]# cat /etc/auto.master 
/net	-hosts
/-        /etc/auto.marrow
[root@sgi-xe320-01 ~]# cat /etc/auto.marrow 
/mnt/demo rhel6-nfs:/export/home 

[root@sgi-xe320-01 ~]# grep -E ' (/net|/mnt)' /proc/mounts 
-hosts /net autofs rw,relatime,fd=13,pgrp=26906,timeout=300,minproto=5,maxproto=5,indirect 0 0
/etc/auto.marrow /mnt/demo autofs rw,relatime,fd=19,pgrp=26906,timeout=300,minproto=5,maxproto=5,direct 0 0
[root@sgi-xe320-01 ~]# ./nsdemo /net
We are the parent
We are the container!
Mount point 1 /net -hosts autofs rw,relatime,fd=13,pgrp=26906,timeout=300,minproto=5,maxproto=5,indirect
Umount point 1 /net
Done with umount

[root@sgi-xe320-01 ~]# ls -l /net/rhel6-nfs
total 0
dr-xr-xr-x. 3 root root 0 May 14 02:24 export
[root@sgi-xe320-01 ~]# ./nsdemo /net/rhel6-nfs
We are the parent
We are the container!
Done with umount

[root@sgi-xe320-01 ~]# cat /proc/mounts 
-hosts /net autofs rw,relatime,fd=13,pgrp=27074,timeout=300,minproto=5,maxproto=5,indirect 0 0
/etc/auto.marrow /mnt/demo autofs rw,relatime,fd=19,pgrp=27074,timeout=300,minproto=5,maxproto=5,direct 0 0
-hosts /net/rhel6-nfs/export/home autofs rw,relatime,fd=13,pgrp=27074,timeout=300,minproto=5,maxproto=5,offset 0 0
[root@sgi-xe320-01 ~]# ./nsdemo /mnt/demo
We are the parent
We are the container!
Mount point 1 /mnt/demo /etc/auto.marrow autofs rw,relatime,fd=19,pgrp=27074,timeout=300,minproto=5,maxproto=5,direct
Umount point 1 /mnt/demo
Done with umount

[root@sgi-xe320-01 ~]# mount
/dev/mapper/vg_sgixe32001-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_sgixe32001-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

[root@sgi-xe320-01 ~]# ls -l /mnt/demo
[root@sgi-xe320-01 ~]# cat /proc/mounts 
-hosts /net autofs rw,relatime,fd=13,pgrp=27074,timeout=300,minproto=5,maxproto=5,indirect 0 0
/etc/auto.marrow /mnt/demo autofs rw,relatime,fd=19,pgrp=27074,timeout=300,minproto=5,maxproto=5,direct 0 0
-hosts /net/rhel6-nfs/export/home autofs rw,relatime,fd=13,pgrp=27074,timeout=300,minproto=5,maxproto=5,offset 0 0
rhel6-nfs:/export/home /mnt/demo nfs4 rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2620:52:0:1040:230:48ff:fe7f:2c8a,minorversion=0,local_lock=none,addr=2620:52:0:1040:221:5eff:fe95:20f4 0 0
[root@sgi-xe320-01 ~]# ./nsdemo /mnt/demo
We are the parent
We are the container!
Mount point 1 /mnt/demo /etc/auto.marrow autofs rw,relatime,fd=19,pgrp=27074,timeout=300,minproto=5,maxproto=5,direct
Mount point 2 /mnt/demo rhel6-nfs:/export/home nfs4 rw,relatime,vers=4,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2620:52:0:1040:230:48ff:fe7f:2c8a,minorversion=0,local_lock=none,addr=2620:52:0:1040:221:5eff:fe95:20f4
Umount point 2 /mnt/demo
Umount point 1 /mnt/demo
Done with umount

Comment 23 errata-xmlrpc 2012-06-20 07:57:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html