Bug 537463

Created attachment 369466 [details]
autofs debug logs from the 'du' test

Curious, if I add a "/." to the end of the path, it works on the first try:

   [jbastian@termite ~]$ grep centipede /proc/mounts
   [jbastian@termite ~]$ du /data/centipede/.
   4448    /data/centipede/./data/wallpapers
   4456    /data/centipede/./data
   4464    /data/centipede/.

Side note: du works on the first try with Fedora 11 (autofs-5.0.4-45.x86_64 and kernel-2.6.30.9-96.fc11.x86_64)

There might be a problem with 'du' on RHEL-5.  With a bit of hacking, I managed to build RHEL-5's coreutils-5.97 on Fedora 11, and 'du' failed with the same 'No such file or directory' error on F-11.


I then built the latest coreutils from upstream on RHEL-5 and it worked fine:

     [jbastian@termite ~]$ grep centipede /proc/mounts
     [jbastian@termite ~]$ /tmp/coreutils-7.6/src/du /data/centipede
     4448    /data/centipede/data/wallpapers
     4456    /data/centipede/data
     4456    /data/centipede

I'll move this to the coreutils component.

Note: 'du -L' also works fine.

   [jbastian@termite ~]$ grep centipede /proc/mounts
   [jbastian@termite ~]$ du -L /data/centipede
   4448    /data/centipede/data/wallpapers
   4456    /data/centipede/data
   4456    /data/centipede

I've been comparing coreutils-5.97 and coreutil-6.5 (which works) and the -L flag changes the fts_open() flags from FTS_PHYSICAL to FTS_LOGICAL.

The lib/fts* files have had 100s of changes between 5.97 and 6.5 and it's difficult to narrow down which changes might have an effect here.

I'm not sure this will be useful but I had a quick look at
strace output from my F-10 system where this worked as required
(although I'm not sure this working as required is actually a
good thing, but that's another story) and RHEL-5.3 where it didn't.

Note that I observed that the -L appears to work but actually doesn't
quite get it right either.

On F-10 the interesting bit (du /test/foo):
newfstatat(AT_FDCWD, "/test/foo", {st_mode=02, st_size=17592186044416, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/test/foo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
fcntl(3, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(3, F_DUPFD, 3)                    = 4
getdents(3, /* 2 entries */, 4096)      = 48
getdents(3, /* 0 entries */, 4096)      = 0
close(3) 

and on RHEL (du /test/foo):
lstat("/test/foo", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open(".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY) = 3
fchdir(3)                               = 0
open("/test/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(4)

and on RHEL (du -L /test/foo):
stat("/test/foo", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/test/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
getdents(3, /* 2 entries */, 4096)      = 48
getdents(3, /* 0 entries */, 4096)      = 0
close(3)

The first thing that stands out is the use of O_NONBLOCK.
Although I only looked briefly I didn't see any select() or poll()
calls that would wait for this potentially lengthy callback to the
automount daemon to complete and I noticed what looks like incomplete information returned for /test/foo from the "du -L" command. Perhaps
the -L appears to work because it takes longer to complete but I can't
say for sure.

OTOH, in the F-10 trace the newfstatat() shouldn't cause the mount
to happen but the openat() will and it will block until the mount
completes. At least I think that is the case but I will need to
check the flags used in the kernel to say for sure. In any case
it behaves as required so I'm likely correct.

Anyway, I'll look a bit more when I get a chance.
Ian

(In reply to comment #6)
> 
> OTOH, in the F-10 trace the newfstatat() shouldn't cause the mount
> to happen but the openat() will and it will block until the mount
> completes. At least I think that is the case but I will need to
> check the flags used in the kernel to say for sure. In any case
> it behaves as required so I'm likely correct.

To clarify, openat() will block if it triggers a mount.

My question to self is will trigger the auto mount with the
provided flags and that appears to be the case (so probably
not much of a question really).

Ian

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

I am able to repeat the failure.  Bug 509923 and bug 501848 seem to be closely related.  I'll try to find some solution.

Created attachment 472895 [details]
possible solution

It looks like el5 clone of bug 501848 - the attached patch solves the problem for me.  Could anybody give it a try?

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

RHEL-5 findutils also detects the dev/ino change underneath it, but seems to be able to survive the event -- it just prints a warning:

# umount /autofs/boot; find /autofs
/autofs
/autofs/boot
find: WARNING: Hard link count is wrong for /autofs/boot: this may be a bug in your filesystem driver.  Automatically turning on find's -noleaf option.  Earlier results may have failed to include directories that should have been searched.
/autofs/boot/grub
...

This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.

(In reply to comment #19)
> It looks like el5 clone of bug 501848 - the attached patch solves the problem
> for me.  Could anybody give it a try?


Your patch fixed the problem for me!

And to my surprise, most of my original reproducer system was still set up from over a year ago so it was easy to verify this patch fixed the bug.

Jeff

Comment on attachment 472895 [details]
possible solution

Jeff, thank you for testing the patch.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If the ghost option is enabled for an automount point, the du command fails on an automounted directory if it's not mounted yet.  It works on the second attempt. Additional fts_stat call was added to reveal subsequent changes in hierarchy to du command, thus the du command succeeds even for the first attempt.

*** Bug 509923 has been marked as a duplicate of this bug. ***

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1074.html

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1074.html