Red Hat Bugzilla – Bug 847960
Check for required disk space can be wildly off on NFS and ZFS
Last modified: 2017-08-22 07:41:19 EDT
Description of problem:
While I do have enough space on the filesystem that --installroot is on, I wouldn't have enough on the actual rootfs. Of course I won't really need that amount on the rootfs but on the --installroot instead.
Using yum to install Fedora into a directory on a NFS share, I do something along to lines of:
yum -y --installroot=/net/server/share/ --releasever=17 --nogpgcheck install <long list of packages>
What I get:
At least 102992MB more space needed on the / filesystem.
# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 117G 36G 80G 32% /
server:/share/ 200G 38G 163G 19% /net/server/share
So basically, while I do have 163GB available on the --installroot directory's filesystem to fit those 102992MB comfortably, I only have 80GB available on the rootfs which certainly wouldn't be enough space.
Version-Release number of selected component (if applicable):
Doing the same like:
chroot /net/server/share/ yum install <long list of packages>
shows the very same error. (@Core was installed previously either way).
So I'm either misinterpreting things and there really isn't enough space (though another system with the same list of packages is "only" <25GB in size) or the error probably isn't in --installroot (or not in yum at all?).
Really not sure what to make out of it. I'm pretty sure I see some kind of bug I just have no idea what's at fault here.
If on another system the same package set is < 25GB (which sounds reasonable) and on another one it claims > 100GB (which in itself is suspect, that'd be a HUGE amount of packages), something is indeed off.
One "obvious" question: is the /net/server/share actually read-write mounted, ie can you write anything at all onto it from that system, or can you install some small set of packages there? Rpm (which is where the disk calculations occur) doesn't have a proper error message for that but reports ro-disk as full, which can give strange looking results when trying to install into a read-only mount.
Also if the package set is really < 25GB as indicated by the other system, the set should fit quite easily into the ~80GB available on this "problematic" system's root fs. Might be worth trying, just to see whether its related to the nfs-share somehow or something else.
It is mounted read-writable. Right, I did have the same idea in the meantime and did the following two sets of installation successful:
chroot /net/server/share/ yum install <long list of packages | grep gnome>
chroot /net/server/share/ yum install <long list of packages | grep eclipse>
Since both have lots of dependencies, nearly half of the list of packages (660 of ~1470) is installed by now. Trying to install the rest, again like this:
chroot /net/server/share/ yum install <long list of packages>
At least 18806MB more space needed on the / filesystem.
...and I really do have 20GB available here.
My initial thought regarding the 25GB vs 100GB was that it needs more space during transaction as it download-unpacks-installed the data, i.e. needs several times the amount of space.
I'm trying to keep installing "small" subsets of packages, wondering whether I'm able to install everything or if there's maybe certain rpms triggering the behavior.
So installing 660 of ~1470 regular gnome, eclipse etc packages brought down the "missing" space from ~100GB to ~18GB. Rrrrrrright. This really doesn't add up :)
Can you try something like
# yum -y --installroot=/testroot --releasever=17 --nogpgcheck install <original long list of packages>
...so it tries to install into a chroot residing on the local disk? Just to see whether it's somehow related to the characteristics of the nfs-share or a more generic bug.
I would guess it's NFS specific since the problem occurs both with with yum --installroot=/net/server/share <...> and with chroot /net/server/share yum <...> :)
And I just confirmed this with what you suggested:
yum -y --installroot=/tmp/testroot/ --releasever 17 --nogpgcheck install <original long list of packages>
...works just fine.
Oh and what I said to try earlier works: installing the complete set in smaller chunks of <400 packages does the job. So I guess it really does fetch some wrong values or misinterpret/miscalculate them when installing to a NFS share.
Ok, thanks for testing + confirming those. I'd guess the nfs share has a "funny" blocksize that trips up something in rpm - what does 'stat -f /net/server/share/' say?
ID: 0 Namelen: 255 Type: nfs
Block size: 1048576 Fundamental block size: 1048576
Blocks: Total: 204800 Free: 160147 Available: 160147
Inodes: Total: 587007488 Free: 563058000
Oh, of course...
Rpm uses the fs block size and number of free blocks to calculate the disk space needs of a transaction. In case of NFS, the reported block size has nothing to do with the block size of the underlying filesystem (its just the read/write chunk size), and with lots of smaller-than-blocksize files, rpm ends up thinking there aren't enough free blocks on the disk. The bigger the difference between nfs block size and underlying block size, the larger the error. Here, the nfs block size is a whopping 1M whereas the underlying fs blocks are typically in 1-4K range, and the ridiculous "need 100G more" errors are a result of that difference.
To put it another way, rpm's disk space calculation is fundamentally broken on NFS, always been. In the days of NFSv2 with ~8K block size limit this wouldn't have been so pronounced, but as newer NFS versions have allowed bigger and bigger block sizes, its gotten worse and worse - it's almost strange that nobody has reported this before now.
How to fix / work around it in rpm is an entirely different question though... AFAIK there's no way to get details about the underlying fs from an NFS mount, so it comes down to guessing (something like "on nfs, assume underlying block size to 4K and never mind what statfs() says") and/or adding configuration tunables to allow adjusting.
If it can't be properly fixed I think disabling the disk space test and issuing an appropriate warning would be way better than just failing :) Only on NFS, of course. ;)
Unnecessarily failing early is the safer option than getting it wrong the other way around :)
The disk-size check can be manually overridden though: for rpm cli it's --ignoresize, for yum it'ss --setopt=diskspacecheck=0 from cli (or add diskspacecheck=0 to /etc/yum.conf to permanently disable, but that's not generally a good idea).
*** Bug 962769 has been marked as a duplicate of this bug. ***
Created attachment 747667 [details]
the full output of "yum install"
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '17'.
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 17's end of life.
Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 17 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged change the
'version' to a later Fedora version prior to Fedora 17's end of life.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Moving to rawhide to avoid timeouting.
*** Bug 1397569 has been marked as a duplicate of this bug. ***
In rawhide/F27 now as of rpm >= 4.13.90