Red Hat Bugzilla – Bug 808019
Condors MOUNT_UNDER_SCRATCH and autofs do no combine
Last modified: 2012-10-10 08:42:29 EDT
Description of problem: When MOUNT_UNDER_SCRATCH is used in one of condor's configuration values, directories to be automounted to run the user's job, do not automount, but error out with a 'Too many levels of symbolic links'. Version-Release number of selected component (if applicable): condor-7.7.5-0.2.fc16.x86_64 autofs-5.0.6-5.fc16.x86_64 How reproducible: Always Steps to Reproduce: 1. Set MOUNT_UNDER_SCRATCH to any existing directory. Do a condor_reconfig 2. Run a job, that needs to automount (over NFS here) anything apart from your home directory. 3. Save the job's error output Actual results: In the job's error log: execvp: Too many levels of symbolic links Expected results: The job should simply be run. Additional info: * It does not matter what directory is specified as MOUNT_UNDER_SCRATCH * The directory _is_ actually mounted, but new mounts are not seen by the user's job. Automounted directories that were mounted previously to job creation are visible. (So users report you have to run your job twice before it succeeds).
Hi Bert, Thanks for reporting the bug. Unfortunately, I think this is an issue with autofs. The MOUNT_UNDER_SCRATCH feature works by creating a separate mount namespace for each job, then bind mounting the sysadmin-specified directories (such as /tmp or /var/tmp) within the namespace. For more info, see http://osgtech.blogspot.com/2012/02/file-isolation-using-bind-mounts-and.html. Call the system's namespace A and the job's namespace B. Autofs appears to work by having a stub mount of type autofs; when it sees filesystem activity inside its mount, it replaces the stub mount with the real mount. The problem is that autofs sees activity in namespace B, but mounts the filesystem in namespace A - invisible to the job. There are two ways to fix this (that I can think of, at least): 1) Autofs can mark its mounts as "shared sub-tree". That means changes in namespace A automatically show up in namespace B. 2) Autofs can use the new "setns" call (available in Linux kernel 3.0) to temporarily associate itself with namespace B and do the mount only in namespace B. However, I don't know enough about the autofs to speculate which approach would be doable upstream. Matt - meta question: do you know how to CC the autofs maintainers on this ticket? Brian
(In reply to comment #1) > Hi Bert, > > Thanks for reporting the bug. > > Unfortunately, I think this is an issue with autofs. > > The MOUNT_UNDER_SCRATCH feature works by creating a separate mount namespace > for each job, then bind mounting the sysadmin-specified directories (such as > /tmp or /var/tmp) within the namespace. For more info, see > http://osgtech.blogspot.com/2012/02/file-isolation-using-bind-mounts-and.html. > Call the system's namespace A and the job's namespace B. > > Autofs appears to work by having a stub mount of type autofs; when it sees > filesystem activity inside its mount, it replaces the stub mount with the real > mount. Actually mounts over the top of the trigger. > > The problem is that autofs sees activity in namespace B, but mounts the > filesystem in namespace A - invisible to the job. > > There are two ways to fix this (that I can think of, at least): > 1) Autofs can mark its mounts as "shared sub-tree". That means changes in > namespace A automatically show up in namespace B. Not sure about that, I'll have to look around and see if I can evaluate the implications of that. > 2) Autofs can use the new "setns" call (available in Linux kernel 3.0) to > temporarily associate itself with namespace B and do the mount only in > namespace B. Sounds good but my initial impression is that autofs is not aware of the different namespace when it needs to be. Again I'll need to look around. > > However, I don't know enough about the autofs to speculate which approach would > be doable upstream. The question of namespace support within autofs is actually pretty difficult AFAICT from previous interactions with folks involved with the implementation. What's really needed is for me to consult with one of the namespace code maintainers, but I'm not sure who that should be. Ian
If the autofs folks are against namespace support (either due to a difficult implementation, or a technical disagreement), I think Condor could itself mark the mount directory as shared-subtree. Perhaps it's inevitable in order to properly support our local RHEL6 boxes. We already have to parse the relevant files as "/" itself is a shared-subtree in F16, and we want to have a per-job "/tmp". The approach is pretty distasteful as I am mucking around with mounts that don't belong to us, but I think should actually work.
A ticket has been filed upstream: https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2959 Patch is attached there.
(In reply to comment #4) > A ticket has been filed upstream: > > https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2959 > > Patch is attached there. Are you sure this will work around the problem? I thought the shared mount attribute was propagated to child mounts from the parent mount and systemd marks "/" as shared already IIRC.
I was able to reproduce (not out of the box though - I had to manually make the autofs mounts non-shared), and this patch fixed it. Your comment about "/" being shared already is correct (on RHEL6 and prior Fedoras, this is not true). As mentioned on the Condor ticket, I also noticed a buildsys issue that should have made the detection of "/" as shared broken - meaning, out-of-the-box on F16, the MOUNT_UNDER_SCRATCH feature should have been broken. So, I'm now curious about how the original bug reporter got their box into this state. It would be nice to see /proc/self/mountinfo from Bert's box.
(In reply to comment #6) > I was able to reproduce (not out of the box though - I had to manually make the > autofs mounts non-shared), and this patch fixed it. > > Your comment about "/" being shared already is correct (on RHEL6 and prior > Fedoras, this is not true). That's true, and I had to make a change to F16 autofs to make autofs work with the shared subtree root filesystem, which is not present in RHEL-6. So, keep that in mind too.
Created attachment 580747 [details] /proc/self/mountinfo on one of the compute nodes This is as seen from outside all things condor.
Hi Bert, Did you do anything special to make "/" not shared? From your attachment, it seems we have diagnosed the bug correctly. I think we're going to do a new build for F16/F17/rawhide today; I'll try to make sure this patch gets reviewed and added. Brian
Hi Brian, Nope, nothing special there, neither /etc/fstab nor automount configs. Might be a side effect of something, but what? Bert.
condor-7.9.0-0.1.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/condor-7.9.0-0.1.fc16
condor-7.9.0-0.1.fc17.2 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/condor-7.9.0-0.1.fc17.2
Package condor-7.9.0-0.1.fc17.2: * should fix your issue, * was pushed to the Fedora 17 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing condor-7.9.0-0.1.fc17.2' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-6880/condor-7.9.0-0.1.fc17.2 then log in and leave karma (feedback).
Hi all, I just tried condor-7.9.0-0.1.fc16 from updates-testing. The results are nearly same as with condor-7.7.5-0.2.fc16: (stderr) /users/visics/deknuydt/condor/testscript.sh: line 3: cd: /software/matlab/2012a/bin: Too many levels of symbolic links ls: cannot access /software/matlab/2012a/bin/matlab: Too many levels of symbolic links Which is the outcome of a simple: > cd /software/matlab/2012a/bin From inside the condor job, no automounts, except those that were already available Filesystem 1K-blocks Used Available Use% Mounted on rootfs 8362320 2402964 5539928 31% / [...] asgard:/visics/homedirs/visics/deknuydt 2113787904 1124507648 881906688 57% /users/visics/deknuydt /dev/mapper/vg_yingchang-LogVol02 698257668 10453424 652855536 2% /tmp /dev/mapper/vg_yingchang-LogVol02 698257668 10453424 652855536 2% /var/tmp From outside the condor job, you see the thing properly mounted: > df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 8362320 2402960 5539932 31% / [...] tmpfs 3108924 0 3108924 0% /tmp tmpfs 3108924 0 3108924 0% /var/tmp [...] asgard:/visics/homedirs/visics/deknuydt 2113787904 1124228096 882186240 57% /users/visics/deknuydt bocq:/softw/matlab 264224768 253955072 7584768 98% /software/matlab Do you get any wiser with the outputs of /proc/self/mountinfo?
Hi Bert, I see what's wrong. Look at my latest update here: https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2959,0 I fat-fingered the creation of a patch from my git branch, causing the fix to be #ifdef'd out. Working on it. Brian
Updated, will be in next spin.
Any movement to be expected here? Fedora 1{6,7} are still at 7.7.5, while more recent stuff is out upstream ...
condor-7.9.1-0.1.fc17.2 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/condor-7.9.1-0.1.fc17.2
Package condor-7.9.1-0.1.fc17.2: * should fix your issue, * was pushed to the Fedora 17 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing condor-7.9.1-0.1.fc17.2' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-12127/condor-7.9.1-0.1.fc17.2 then log in and leave karma (feedback).
condor-7.9.1-0.1.fc17.2 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.
I have a similar problem "Too many levels of symbolic links" (not using condor !) cf https://bugzilla.redhat.com/show_bug.cgi?id=833535 in comment #7 above, Ian kent said "I had to make a change to F16 autofs to make autofs work with the shared subtree root filesystem" I would like to try that, how do you configure autofs to work with shared subtree !? Thanks .
(In reply to comment #21) > I have a similar problem "Too many levels of symbolic links" (not using > condor !) > cf https://bugzilla.redhat.com/show_bug.cgi?id=833535 > > in comment #7 above, Ian kent said > "I had to make a change to F16 autofs to make > autofs work with the shared subtree root filesystem" > > I would like to try that, > how do you configure autofs to work with shared subtree !? Use autofs-5.0.6-3 or later.
I am already on : autofs-5.0.6-22.fc17.i686 , kernel-PAE-3.5.4-1.fc17.i686 , fedora17 but maybe there's an option, variable or whatever to set (in /etc/sysconfig/autofs ?) to force the use of shared subtree ? how can I check if I use shared subtree ? Thanks .
(In reply to comment #23) > I am already on : > autofs-5.0.6-22.fc17.i686 , kernel-PAE-3.5.4-1.fc17.i686 , fedora17 > > but maybe there's an option, variable or whatever to set (in > /etc/sysconfig/autofs ?) to force the use of shared subtree ? > how can I check if I use shared subtree ? There isn't, the issue was that systemd sets "/" as shared and that caused a problem with the construction of mount trees. When the containing file system is shared moving the mount (tree) to the final location after construction wouldn't work. But this mechanism isn't needed with the vfs-automount intrastrucure so it was removed, actually just disabled in fc17. Just what is causing the problem your seeing is puzzling. No matter how I look at it, it has to mean that there's a call to perform a mount that returns success but no mount is actually present. I haven't ruled out the possiblity this is reated in some way to the fact that "/" is shared. If you don't rely on this you could "mount --make-private /", stop autofs and make sure there's no autofs stuff mounted, then start it and see if the problem remains.
I might also add I have a patch that is yet to be posted that solves a problem with recurring mount requests but I can't see how that would cause this either, but maybe. I'll get to posting that soon as I can and then see about a test kernel. Probably the better place to continue this is bug 833535 rather than hijacking this bug.