Bug 1437754

I don't think that GVfs is a culprit. GVfs just monitors the mounts in this case. Those autofs mounts behave like a local filesystem and are mounted over autofs. Nautilus accesses to them over GLib/GIO API. GLib checks for .hidden file, when gathering file info. However, GLib doesn't call the mount operation on it, so I suppose it is some bug in autofs (if the error message means that somebody calls mount operation)...

Just an idea that GLib accesses this file at first, before any other operation with this filesystem. So autofs tries to mount this location first and fails and then it tries to mount its parent and succeeds. However, it still seems to me like a bug in autofs, or autofs configuration...

Please provide a full debug log covering the start up of autofs
up to and including duplicating the problem.

It sounds to me like Gvfs is trying to access the directories,
like .hidden, as /home/.hidden etc. which Gvfs really shouldn't
be doing, IMO, as they are not within the recently mounted mount
it has detected.

A debug log will show us if this is the case.

In this case (where the wildcard map entry is being used) the
kernel and autofs have no choice but to attempt a mount over
that directory (such as /home/.hidden) using the substitution
requested by the wildcard map entry.

autofs cannot know if this directory is a valid mount point
or not because it matches a map entry key, the wildcard.

OTOH, if this was an access to /home/<user directory>/.hidden
then no mount would be attempted.

This is because once the NFS mount has been mounted autofs
doesn't play a part in path resolution any more, instead
the VFS follows over that mount and continues into the NFS
file system, and autofs is none the wiser.

It may be possible for something unexpected to happen on
mount failure but I doubt that's the case.

Ian

(In reply to Ian Kent from comment #4)
> Please provide a full debug log covering the start up of autofs
> up to and including duplicating the problem.

Can you also please ask the customer to capture /proc/mounts and
/proc/self/mountinfo after starting autofs but before attempting
the automount and also capture each of these after performing the
automount attempt.

Oh, I see what is "wrong" - the problematic .hidden file is directly in autofs folder. So e.g. /home/.hidden is read by GLib if somebody (in this case probably Nautilus) tries to enumerate files in /home/ (i.e. g_file_enumerate_children), or tries to query info (i.e. g_file_query_info) about some file inside this dir (e.g. home dir mount). See the relevant GLib code [1]. It seems that autofs doesn't "support" .hidden files and thus it generates mount request for this location. Maybe autofs could ignore such requests, but this is probably bug in GLib, which should not try to read .hidden files in autofs folders...

Ian, does this makes sense to you? Isn't there anything what we can do with it on autofs side?

[1] https://git.gnome.org/browse/glib/tree/gio/glocalfileinfo.c#n1470

(In reply to Ondrej Holy from comment #8)
> Oh, I see what is "wrong" - the problematic .hidden file is directly in
> autofs folder. So e.g. /home/.hidden is read by GLib if somebody (in this
> case probably Nautilus) tries to enumerate files in /home/ (i.e.
> g_file_enumerate_children), or tries to query info (i.e. g_file_query_info)
> about some file inside this dir (e.g. home dir mount). See the relevant GLib
> code [1]. It seems that autofs doesn't "support" .hidden files and thus it
> generates mount request for this location. Maybe autofs could ignore such
> requests, but this is probably bug in GLib, which should not try to read
> .hidden files in autofs folders...

That is what's happening, certainly something is trying to access
a directory .hidden within the top level directory of the autofs
mount.

The autofs mount here is an indirect mount so directories within
the autofs mount are potential mount points. And .hidden is a
valid path component and is matched in the autofs mount map
because it uses the wildcard map entry, it matches any valid
path component.

So autofs (kernel or user space) can't know if an access should
be ignored. Users can't create directories within these autofs
mounts so if it gets accessed and matches a map key autofs is
obligated to attempt to mount it.

I don't know if it will be possible for GLib to check if it is
going to access a top level autofs (indirect) mount and not
access these directories if it is. That's because it would
need to know the file system of the mount point before
attempting an access.

I have seen that gfvs-udisk2-volume-monitor will use an excessive
amount of CPU under moderately heavy autofs mount activity (and
I'm guessing it uses GLib to get file system information) so
getting the needed file system information could make this even
worse if the file system type information isn't already known.

So I think this could be quite difficult to resolve.

> 
> Ian, does this makes sense to you? Isn't there anything what we can do with
> it on autofs side?

Yes, it does, it's fairly similar to a problem that I'm struggling
with now in Fedora 25. While the problem has been present for a
long time it has become quite serious in the recent Fedora release.

Consequently it will most likely be a problem in RHEL-8 since
we have quite a good number of customers that have busy automount
environments.

But this is not the time for that discussion.
 
> 
> [1] https://git.gnome.org/browse/glib/tree/gio/glocalfileinfo.c#n1470

I'll have a look at this and see if I can be at all helpful, ;)

Ian

So, it should be moved to Glib component if autofs can't do anything with it.

I have tried to use ioctl calls provided by autofs to fix it, however, I don't know how to link it with "linux/auto_dev-ioctl.h" header properly. Please see the upstream bug report:
https://bugzilla.gnome.org/show_bug.cgi?id=780967

(In reply to Ondrej Holy from comment #11)
> So, it should be moved to Glib component if autofs can't do anything with it.

Yep, it probably needs to be GLib but I plan on putting together
an example of how to get mounted status and whether it's an autofs
file system using one of the autofs ioctls (that's used a lot for
exactly this in the daemon).

> 
> I have tried to use ioctl calls provided by autofs to fix it, however, I
> don't know how to link it with "linux/auto_dev-ioctl.h" header properly.
> Please see the upstream bug report:
> https://bugzilla.gnome.org/show_bug.cgi?id=780967

I did think of making a shared library as a part of autofs
but then it looked like we need just one ioctl (from the dev
ioctls, most are admin only anyway) so I thought it would be
better to just write a code example using that ioctl and also
add some supporting code for parameter allocation etc.

Indeed, there may be some autofs kernel changes that come out
of this, not the least of which is getting the header file in
the right location for user space users, and making sure the
ioctls that shouldn't require admin privileges can be used
(although I thought I already did that long ago ...).

I started on the initial example so I can already see there
are things like that I need to work out.

Let me spend a little time on it and we'll talk more about
what is actually needed by GLib and how GLib should access
it as we go. Regardless of where the code lives, GLib or
autofs, it still needs to be written and that's what I'd
like to do to start with.

Ian

I think we can do this from userspace today with `fstatat(..., AT_NO_AUTOMOUNT)`, right?

(In reply to Colin Walters from comment #14)
> I think we can do this from userspace today with `fstatat(...,
> AT_NO_AUTOMOUNT)`, right?

That's an interesting observation.

Looking at the kernel code, I'm not sure that will work and based
on the man page description perhaps it should.

I'm pretty sure that there's one case (not the one we have here)
where this can't work the way we need it to but then I think the
autofs ioctl will also not help with that either.

This might have been missed in the initial implementation or it
might not be possible for the case we have here, I'll need to
look more closely at it.

(In reply to Ian Kent from comment #15)
> (In reply to Colin Walters from comment #14)
> > I think we can do this from userspace today with `fstatat(...,
> > AT_NO_AUTOMOUNT)`, right?
> 
> That's an interesting observation.
> 
> Looking at the kernel code, I'm not sure that will work and based
> on the man page description perhaps it should.

This doesn't work for the case here and should be fixed in the
VFS.

You would think fixing it would be straight forward but it isn't.
I'll have to think about how I can do it some more.

Carlos, how hard would it be to patch nautilus to not request the standard::is-hidden flag?

(For just /home)

(In reply to Colin Walters from comment #19)
> Carlos, how hard would it be to patch nautilus to not request the
> standard::is-hidden flag?

It's very hacky, but could be done.
However, how do we manage hidden/vs non hidden files in that case then? All hidden? All visible?

Created attachment 1312039 [details]
Tar of example autofs ioctl usage

Here is an example of usage of the autofs miscellaneous device
ioctl interface.

There are a couple of difficulties which are explained in an
included README with some comments in the example code also.

To be able to use this in RHEL-7, if it's decided that will
be useful, I will need to raise a bug to include the kernel
changes in RHEL-7.

All, please have a look and let me know what you think.

(In reply to Ian Kent from comment #30)
> (In reply to Laxminarayan from comment #29)
> > Hi Ian, 
> > 
> > Thank you very much for your explanation and for the sample code that you
> > have attached to understand how it works.

Just so it's clear on why autofs isn't able to recognize these
as invalid map entry keys and immediately fail, let me explain
one more time.

If a user decides to use the wildcard map key in an autofs
map then autofs must match any access to a directory that
results in a callback from the kernel that doesn't match
any other map key with the wildcard map entry and attempt
to mount it. It has no choice in this, it's duty bound to
do it because that's what it has been asked to do.

That accesses to directories that are clearly invalid to us
as human beings match is not something we can teach the
autofs program.

This is known and expected behaviour.

While at the same time it's often not feasible or desirable
to enumerate every possible auto-mount so it's a trade-off
that's at the discretion of the administrator that needs to
use the functionality.

It's this last point that is the motivation to at least
provide some way for applications to mitigate against
this for cases they are aware of.

It must be understood this is not a clear cut overall
resolution because there are still many possible access
patterns that will continue to be caught by a wildcard
map entry that applications have no awareness of.

Ian