Description of problem: I have 5 NFS volumes mounted from my NAS. Running 'mount' shows all 5, but running 'df' will only show 1 volume out of the 5. # mount | grep nas nas.example.com:/Photos on /mnt/Photos type nfs (rw,nosuid,nodev,noexec,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.64,mountvers=3,mountport=35825,mountproto=udp,local_lock=none,addr=192.168.1.64,user) nas.example.com:/Download on /mnt/Download type nfs (rw,nosuid,nodev,noexec,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.64,mountvers=3,mountport=35825,mountproto=udp,local_lock=none,addr=192.168.1.64,user) nas.example.com:/Multimedia on /mnt/Multimedia type nfs (rw,nosuid,nodev,noexec,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.64,mountvers=3,mountport=35825,mountproto=udp,local_lock=none,addr=192.168.1.64,user) nas.example.com:/VirtualMachines on /mnt/VirtualMachines type nfs (rw,nosuid,nodev,noexec,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.64,mountvers=3,mountport=35825,mountproto=udp,local_lock=none,addr=192.168.1.64,user) nas.example.com:/Backups on /mnt/Backups type nfs (rw,nosuid,nodev,noexec,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.64,mountvers=3,mountport=35825,mountproto=udp,local_lock=none,addr=192.168.1.64,user) # df | grep nas nas.example.com:/Photos 2879673344 2489566368 389582688 87% /mnt/Photos I believe this broken behaviour arrived in Fedora 18, though I no longer have any F17 install to confirm this categorically. Version-Release number of selected component (if applicable): coreutils-8.17-8.fc18.x86_64 How reproducible: Always Steps to Reproduce: 1. Export multiple NFS volumes from a remote server 2. Mount all volumes on the client 3. Run 'df' Actual results: df only shows usage for one random mount Expected results: df shows all usage for every single mount point that the 'mount' command reports Additional info:
Daniel that's probably fixed in 8.21 (in rawhide)
Ondrej, either the duplicate suppression patch should be backed out, or augmented with the alternative duplicate suppression method we came up with here: http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=bb116d35 thanks.
I've done a local RPM rebuild with the df dup patch removed and that solved this. Happy to try the upstream fix if someone wants to provide a backport of it to f18
I'm not convinced the current upstream duplicate suppression method suffices here either. Will need to look further into it. Perhaps remote file systems should be removed from duplicate suppression.
Indeed, the 'df' from 8.21 has the same problem as that seen in 8.17 dup patch with my NFS mounts.
So in 8.21 the deduping is done based on the "device ID". df is by default trying to impart the storage available, without displaying multiple entries to the same storage, which may be confusing, especially with --total. So I presume `stat -c %D /mnt/Photos /mnt/Downloads` give the same device ID for you. Hopefully that implies that they correspond to the same storage on the NFS server. df 8.21 should display /mnt/Photos by default as it is the shortest path, which is a heuristic used to display to most appropriate entry avoiding bind mounts, remounts, multiple mounts, pivot roots, ... If you want to display all nfs entries, you now need to be explicit: df -h -t nfs -a
I tend to NOTABUG this report - as this is one of the side effects of /proc/mounts -> /etc/mtab symlink... with this change, df shown a lot of duplicate entries. Users started to complain about it. -> Ok, deduplication was done - but it started to not show the entry for root filesystem. -> So the "shortest match" improvement was done. -> Still not perfect for nfs. Only improvement I can imagine is kind of "exception list" for "preferred filesystem types" - which would be hard to do properly anyway. Any thoughts?
I wanted to wait a little while for possible feedback from Daniel before closing, but yes this is an expected change in behaviour, the reasoning being detailed in comment 6. We have another bug #887763 to track issues with deduplication, though TBH the issues mentioned there are all outside of coreutils AFAICS. I do think we should consider adjusting the dedupe patch to what upstream currently uses, though that's not the issue being seen here. The orig bug for the dedup was bug #709351 thanks.
(In reply to comment #6) > So in 8.21 the deduping is done based on the "device ID". > df is by default trying to impart the storage available, > without displaying multiple entries to the same storage, > which may be confusing, especially with --total. > > So I presume `stat -c %D /mnt/Photos /mnt/Downloads` give > the same device ID for you. Hopefully that implies that they correspond > to the same storage on the NFS server. Yes, that is sort of correct. On one of my clients they all have the same device ID, one another F18 client some of them have different device IDs, so 'df' in fact shows 2 out of the 5 volumes > df 8.21 should display /mnt/Photos by default as it is the shortest path, > which is a heuristic used to display to most appropriate entry > avoiding bind mounts, remounts, multiple mounts, pivot roots, ... > > If you want to display all nfs entries, you now need to be explicit: > df -h -t nfs -a I think this is a really very unhelpful, unpleasant behaviour. As a user I want 'df' to report usage of each volume that I have explicitly mounted.
(In reply to comment #7) > I tend to NOTABUG this report - as this is one of the side effects of > /proc/mounts -> /etc/mtab symlink... with this change, df shown a lot of > duplicate entries. > Users started to complain about it. > -> > Ok, deduplication was done - but it started to not show the entry for root > filesystem. > -> > So the "shortest match" improvement was done. > -> > Still not perfect for nfs. If choosing between seeing some duplicate entries, vs missing out random volumes entirely, I absolutely prefer to see a few duplicates. Throwing away / hiding data is not good.
(In reply to comment #9) > (In reply to comment #6) > > So in 8.21 the deduping is done based on the "device ID". > > df is by default trying to impart the storage available, > > without displaying multiple entries to the same storage, > > which may be confusing, especially with --total. > > > > So I presume `stat -c %D /mnt/Photos /mnt/Downloads` give > > the same device ID for you. Hopefully that implies that they correspond > > to the same storage on the NFS server. > > Yes, that is sort of correct. On one of my clients they all have the same > device ID, one another F18 client some of them have different device IDs, so > 'df' in fact shows 2 out of the 5 volumes I didn't see that inconsistency in my testing last night. I wonder are there different options used or something on those mount points that would trigger different device IDs. > > df 8.21 should display /mnt/Photos by default as it is the shortest path, > > which is a heuristic used to display to most appropriate entry > > avoiding bind mounts, remounts, multiple mounts, pivot roots, ... > > > > If you want to display all nfs entries, you now need to be explicit: > > df -h -t nfs -a > > I think this is a really very unhelpful, unpleasant behaviour. As a user I > want 'df' to report usage of each volume that I have explicitly mounted. So df is in a quandary here to distinguish user vs system mounts. The recent changes beneath df in the area haven't been fully designed, and never really were TBH given both the old and new heuristics present in df. Maybe we could heap more heuristics into the mix, like giving entries in fstab precedence, though we probably don't want to open that can of worms. I do think that the current compromise to show one entry per unique device ID in the system isn't a bad scheme, and allows df to report "storage available" rather than "File systems mounted". Noting the Daniel's inconsistency above with distinct IDs from apparently the same storage on the NFS server, that inconsistency seems best addressed outside of df.
(In reply to comment #11) > > > df 8.21 should display /mnt/Photos by default as it is the shortest path, > > > which is a heuristic used to display to most appropriate entry > > > avoiding bind mounts, remounts, multiple mounts, pivot roots, ... > > > > > > If you want to display all nfs entries, you now need to be explicit: > > > df -h -t nfs -a > > > > I think this is a really very unhelpful, unpleasant behaviour. As a user I > > want 'df' to report usage of each volume that I have explicitly mounted. > > So df is in a quandary here to distinguish user vs system mounts. > The recent changes beneath df in the area haven't been fully > designed, and never really were TBH given both the old and > new heuristics present in df. Maybe we could heap more heuristics > into the mix, like giving entries in fstab precedence, though > we probably don't want to open that can of worms. > > I do think that the current compromise to show one entry > per unique device ID in the system isn't a bad scheme, and > allows df to report "storage available" rather than "File systems mounted". The issue I have is that when you show a list like # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1958380 0 1958380 0% /dev tmpfs 1970892 192 1970700 1% /dev/shm tmpfs 1970892 1180 1969712 1% /run tmpfs 1970892 0 1970892 0% /sys/fs/cgroup /dev/mapper/vg_t500wlan-lv_root 151476396 135012952 8762148 94% / tmpfs 1970892 564 1970328 1% /tmp /dev/sda1 194241 98663 85338 54% /boot nas.example.com:/Photos 2879673344 2494189312 384959744 87% /mnt/Photos This is implicitly telling the user that '/mnt/Download' (which is not shown) is inheriting "free space" from '/'. Unless the user actually knows ahead of time that /mnt/Download is actually a separate mount from the same NFS volume as /mnt/Photos, the user is simply being misled by 'df' into thinking that '/mnt/Downloads' is just part of the root filesystem mount volume & thus shares its free space. Seeing duplicates doesn't have the potential to cause harm since all info is still accurate even if duplicated. Leaving out mounts entirely does cause harm by giving the user a mis-leading view of space on various volumes. > Noting the Daniel's inconsistency above with distinct IDs from > apparently the same storage on the NFS server, that inconsistency > seems best addressed outside of df. Yep, lets ignore that inconsistency for sake of these discussions.
Just as a anti-argument to the "see few duplicates" - https://bugzilla.redhat.com/show_bug.cgi?id=709351#c16 ... with 300 bind mounts, df gets unusable completely without the deduplication. Of course, the heuristic may be improved, the shortest match was the easiest one, as Padraig said, we may probably skip all remote filesystems deduplication or something like that. Still - I think the current behaviour is better than having many duplicates - you can always get full listing if you want with df -a .
(In reply to comment #12) > (In reply to comment #11) > > The issue I have is that when you show a list like > > # df > Filesystem 1K-blocks Used Available Use% Mounted > on > devtmpfs 1958380 0 1958380 0% /dev > tmpfs 1970892 192 1970700 1% /dev/shm > tmpfs 1970892 1180 1969712 1% /run > tmpfs 1970892 0 1970892 0% > /sys/fs/cgroup > /dev/mapper/vg_t500wlan-lv_root 151476396 135012952 8762148 94% / > tmpfs 1970892 564 1970328 1% /tmp > /dev/sda1 194241 98663 85338 54% /boot > nas.example.com:/Photos 2879673344 2494189312 384959744 87% /mnt/Photos > > This is implicitly telling the user that '/mnt/Download' (which is not > shown) is inheriting "free space" from '/'. Yes I agree in part, and by that argument we should only be doing dedup for /'s device id. But it's a bit of a stretch to infer anything about a path not shown by df. By the same argument one might infer that /proc is inheriting "free space" from '/'? Is it too onerous to require `df -a` to display all mount points? That is a bit verbose I suppose, as is `findmnt`. Perhaps we should add a pseudo '[:dummy:]' param to -x so that one can enable the traditional behaviour of displaying all devices with storage even if duplicates like: alias df='df -a -x "[:dummy:]"'
(In reply to comment #14) > (In reply to comment #12) > > (In reply to comment #1) > > This is implicitly telling the user that '/mnt/Download' (which is not > > shown) is inheriting "free space" from '/'. > > Yes I agree in part, and by that argument we should only be > doing dedup for /'s device id. But it's a bit of a stretch to > infer anything about a path not shown by df. > By the same argument one might infer that /proc is inheriting > "free space" from '/'? > > Is it too onerous to require `df -a` to display all mount points? The issue is that users have to know there is something wrong, in order to decide that they need to issue 'df -a'. Since there is no indication that df is dropping valid volumes, they may never realize the need to run 'df -a' to see the correct data. Also if the user has a combination of many bind mounts, but also a number of NFS volumes at the same time, they're back where they started from a usability POV. There's no way to get the valid info for NFS, without also polluting their display with the bind mounts. I can't help thinking we're in a no-win situation here and that the kernel needs to provide more information in /proc/mounts to let userspace do the right job. ie provide some indication that the mount came from a bind mount, so userspace can then filter them ?
Yes in general the kernel could provide a lot more info to help here. Some notes for possible future dev: We added a find_bind_mount() function to stat(1) a while back which might be of use: http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/stat.c;h=2326698#l805 I see that findmnt now supports --df, and that it outputs duplicates by default.
*** Bug 1052369 has been marked as a duplicate of this bug. ***
(In reply to Daniel Berrange from comment #15) > I can't help thinking we're in a no-win situation here and that the kernel > needs to provide more information in /proc/mounts to let userspace do the > right job. ie provide some indication that the mount came from a bind mount, > so userspace can then filter them ? There is no "bind" flag stored in the kernel, so there is nothing to propagate to the userspace - there is no diff between bind mount and regular mount from the kernel POV. (In reply to Pádraig Brady from comment #16) > Yes in general the kernel could provide a lot more info to help here. > Some notes for possible future dev: > > We added a find_bind_mount() function to stat(1) a while back > which might be of use: > http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/stat.c; > h=2326698#l805 I don't see any advantage by using find_bind_mount() since kernel does not provide bind flag in /proc/self/mounts nor in /proc/self/mountinfo (as stated above). (In reply to Pádraig Brady from comment #11) > > > So I presume `stat -c %D /mnt/Photos /mnt/Downloads` give > > > the same device ID for you. Hopefully that implies that they correspond > > > to the same storage on the NFS server. > > > > Yes, that is sort of correct. On one of my clients they all have the same > > device ID, one another F18 client some of them have different device IDs, so > > 'df' in fact shows 2 out of the 5 volumes > > I didn't see that inconsistency in my testing last night. > I wonder are there different options used or something > on those mount points that would trigger different device IDs. I cannot reproduce this too. Does this issue persist? Could you please provide mount options?
ping?...
This bug is still a problem on F20 coreutils-8.21-21.fc20.x86_64 I'm not using any special mount flags # mount nas.home:/share/Photos /mnt/Photos/ # mount nas.home:/share/Backups /mnt/Backups/ # df | grep /mnt /dev/dm-3 240231096 182501608 45503332 81% /mnt/1c5321f5-14d6-40a2-8ca1-dd61524c9576 nas.home:/share/Photos 2882739712 2838224224 44515488 99% /mnt/Photos # umount /mnt/Photos # df | grep /mnt /dev/dm-3 240231096 182501608 45503332 81% /mnt/1c5321f5-14d6-40a2-8ca1-dd61524c9576 nas.home:/share/Backups 2882739712 2838224224 44515488 99% /mnt/Backups # mount nas.home:/share/Photos /mnt/Photos/ # df | grep /mnt /dev/dm-3 240231096 182501608 45503332 81% /mnt/1c5321f5-14d6-40a2-8ca1-dd61524c9576 nas.home:/share/Photos 2882739712 2838224224 44515488 99% /mnt/Photos I no longer have the machine that previously showed inconsistent device IDs for the same NFS server, so can't verify that qiwrd behaviour.
Created attachment 951624 [details] proposed upstream patch I'm coming around to the idea of not suppressing _separate_ exports as they're probably explicitly mounted and may have separate ACLs etc. Proposed patch against upstream trunk is attached.
Thanks Pádraig!
Just noting that this issue also affects RHEL 7, with coreutils 8.22. One scenario where the suppression of NFS mounts with duplicate device IDs is particularly harmful is with Netapp filers. One may have a single volume on the Netapp, which corresponds to the device ID, but have many exported qtrees defined within that volume, which all have their own space allocations which may differ from each other. For example: # stat -c %D /home/user1 /home/user2 /home/user3 2a 2a 2a # df -t nfs Filesystem 1K-blocks Used Available Use% Mounted on netapp1:/vol/home/user1 419430400 262429984 157000416 63% /home/user1 # df -a -t nfs Filesystem 1K-blocks Used Available Use% Mounted on netapp1:/vol/home/user1 524288000 221340192 302947808 43% /home/user2 netapp1:/vol/home/user2 419430400 262429984 157000416 63% /home/user1 netapp1:/vol/home/user3 325058560 134288864 190769696 42% /home/user3 Significant important information is thus being lost by the suppression of these (falsely presumed) duplicates.
Problem confirmed on CentOS 7. Most of my NFS mounts are from an Isilon system named "rio". Here's what I'm seeing on a CentOS 7 systems: [root@palazzo ~]# mount | grep -c rio 35 [root@palazzo ~]# df | grep -c rio 7 Interesting fact, don't know if it's coincidence or relevant: This is the same five to one ratio in the initial bug report.
Please do not mix Fedora and RHEL/CentOS reports. This is already reported against RHEL 7 several times (IOW, this bz was already cloned as #1197463 - and there are several others to some extent duplicates). Issue comes from /etc/mtab becoming symlink to /proc/mounts - which is much more verbose and contains duplicates. Having dozens mountpoints in df is not user friendly... and deduplication is tricky thing.
Closing RAWHIDE, as coreutils-8.24-1.fc23 includes fixes for NFS mounts mentioned above.