Red Hat Bugzilla – Bug 537463
du using a ghosted automounted directory results in 'No such file or directory'
Last modified: 2011-07-21 08:11:40 EDT
Description of problem: If the ghost option is enabled for an automount point, the du command fails on an automounted directory if it's not mounted yet. It works on the second attempt. This problem was fixed for the find command in bug 448869, but it persists for du. Version-Release number of selected component (if applicable): autofs-5.0.1-0.rc2.131.el5_4.1 kernel-2.6.18-164.6.1.el5 kernel-xen-2.6.18-164.6.1.el5 How reproducible: every time Steps to Reproduce: 1. Configure an automount to use the ghost option [jbastian@termite ~]$ grep data /etc/auto.master /data /etc/auto.data --ghost [jbastian@termite ~]$ grep centipede /etc/auto.data centipede -intr,tcp centipede:/export [jbastian@termite ~]$ sudo service autofs restart Stopping automount: [ OK ] Starting automount: [ OK ] 2. Verify that the mount point is not yet mounted [jbastian@termite ~]$ grep centipede /proc/mounts [jbastian@termite ~]$ 3. Try running 'du' on the mount point [jbastian@termite ~]$ du /data/centipede du: `/data/centipede': No such file or directory 4. Note that it did get mounted, however [jbastian@termite ~]$ grep centipede /proc/mounts centipede:/export /data/centipede nfs rw,vers=3,rsize=32768,... 5. Subsequent attempts work fine [jbastian@termite ~]$ du /data/centipede 4448 /data/centipede/data/wallpapers 4456 /data/centipede/data 4464 /data/centipede Actual results: du failed on the first attempt (but it did mount the directory) Expected results: du succeeds Additional info:
Created attachment 369466 [details] autofs debug logs from the 'du' test
Curious, if I add a "/." to the end of the path, it works on the first try: [jbastian@termite ~]$ grep centipede /proc/mounts [jbastian@termite ~]$ du /data/centipede/. 4448 /data/centipede/./data/wallpapers 4456 /data/centipede/./data 4464 /data/centipede/.
Side note: du works on the first try with Fedora 11 (autofs-5.0.4-45.x86_64 and kernel-2.6.30.9-96.fc11.x86_64)
There might be a problem with 'du' on RHEL-5. With a bit of hacking, I managed to build RHEL-5's coreutils-5.97 on Fedora 11, and 'du' failed with the same 'No such file or directory' error on F-11. I then built the latest coreutils from upstream on RHEL-5 and it worked fine: [jbastian@termite ~]$ grep centipede /proc/mounts [jbastian@termite ~]$ /tmp/coreutils-7.6/src/du /data/centipede 4448 /data/centipede/data/wallpapers 4456 /data/centipede/data 4456 /data/centipede I'll move this to the coreutils component.
Note: 'du -L' also works fine. [jbastian@termite ~]$ grep centipede /proc/mounts [jbastian@termite ~]$ du -L /data/centipede 4448 /data/centipede/data/wallpapers 4456 /data/centipede/data 4456 /data/centipede I've been comparing coreutils-5.97 and coreutil-6.5 (which works) and the -L flag changes the fts_open() flags from FTS_PHYSICAL to FTS_LOGICAL. The lib/fts* files have had 100s of changes between 5.97 and 6.5 and it's difficult to narrow down which changes might have an effect here.
I'm not sure this will be useful but I had a quick look at strace output from my F-10 system where this worked as required (although I'm not sure this working as required is actually a good thing, but that's another story) and RHEL-5.3 where it didn't. Note that I observed that the -L appears to work but actually doesn't quite get it right either. On F-10 the interesting bit (du /test/foo): newfstatat(AT_FDCWD, "/test/foo", {st_mode=02, st_size=17592186044416, ...}, AT_SYMLINK_NOFOLLOW) = 0 openat(AT_FDCWD, "/test/foo", O_RDONLY) = 3 fstat(3, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 fcntl(3, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fcntl(3, F_SETFD, FD_CLOEXEC) = 0 fcntl(3, F_DUPFD, 3) = 4 getdents(3, /* 2 entries */, 4096) = 48 getdents(3, /* 0 entries */, 4096) = 0 close(3) and on RHEL (du /test/foo): lstat("/test/foo", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0 open(".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY) = 3 fchdir(3) = 0 open("/test/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4 fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 close(4) and on RHEL (du -L /test/foo): stat("/test/foo", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0 open("/test/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3 fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 getdents(3, /* 2 entries */, 4096) = 48 getdents(3, /* 0 entries */, 4096) = 0 close(3) The first thing that stands out is the use of O_NONBLOCK. Although I only looked briefly I didn't see any select() or poll() calls that would wait for this potentially lengthy callback to the automount daemon to complete and I noticed what looks like incomplete information returned for /test/foo from the "du -L" command. Perhaps the -L appears to work because it takes longer to complete but I can't say for sure. OTOH, in the F-10 trace the newfstatat() shouldn't cause the mount to happen but the openat() will and it will block until the mount completes. At least I think that is the case but I will need to check the flags used in the kernel to say for sure. In any case it behaves as required so I'm likely correct. Anyway, I'll look a bit more when I get a chance. Ian
(In reply to comment #6) > > OTOH, in the F-10 trace the newfstatat() shouldn't cause the mount > to happen but the openat() will and it will block until the mount > completes. At least I think that is the case but I will need to > check the flags used in the kernel to say for sure. In any case > it behaves as required so I'm likely correct. To clarify, openat() will block if it triggers a mount. My question to self is will trigger the auto mount with the provided flags and that appears to be the case (so probably not much of a question really). Ian
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
I am able to repeat the failure. Bug 509923 and bug 501848 seem to be closely related. I'll try to find some solution.
Created attachment 472895 [details] possible solution It looks like el5 clone of bug 501848 - the attached patch solves the problem for me. Could anybody give it a try?
RHEL-5 findutils also detects the dev/ino change underneath it, but seems to be able to survive the event -- it just prints a warning: # umount /autofs/boot; find /autofs /autofs /autofs/boot find: WARNING: Hard link count is wrong for /autofs/boot: this may be a bug in your filesystem driver. Automatically turning on find's -noleaf option. Earlier results may have failed to include directories that should have been searched. /autofs/boot/grub ...
This request was erroneously denied for the current release of Red Hat Enterprise Linux. The error has been fixed and this request has been re-proposed for the current release.
(In reply to comment #19) > It looks like el5 clone of bug 501848 - the attached patch solves the problem > for me. Could anybody give it a try? Your patch fixed the problem for me! And to my surprise, most of my original reproducer system was still set up from over a year ago so it was easy to verify this patch fixed the bug. Jeff
Comment on attachment 472895 [details] possible solution Jeff, thank you for testing the patch.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: If the ghost option is enabled for an automount point, the du command fails on an automounted directory if it's not mounted yet. It works on the second attempt. Additional fts_stat call was added to reveal subsequent changes in hierarchy to du command, thus the du command succeeds even for the first attempt.
*** Bug 509923 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2011-1074.html