Bug 847960 - Check for required disk space can be wildly off on NFS and ZFS
Check for required disk space can be wildly off on NFS and ZFS
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Fedora Packaging Toolset Team
Fedora Extras Quality Assurance
: FutureFeature
: 1397569 (view as bug list)
Depends On: 962769
  Show dependency treegraph
Reported: 2012-08-14 04:32 EDT by Sandro Mathys
Modified: 2017-08-22 07:41 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-22 07:41:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
the full output of "yum install" (58.70 KB, text/plain)
2013-05-14 08:28 EDT, IBM Bug Proxy
no flags Details

  None (edit)
Description Sandro Mathys 2012-08-14 04:32:30 EDT
Description of problem:
While I do have enough space on the filesystem that --installroot is on, I wouldn't have enough on the actual rootfs. Of course I won't really need that amount on the rootfs but on the --installroot instead.

Using yum to install Fedora into a directory on a NFS share, I do something along to lines of:
yum -y --installroot=/net/server/share/ --releasever=17 --nogpgcheck install <long list of packages>

What I get:
Error Summary
Disk Requirements:
  At least 102992MB more space needed on the / filesystem.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                117G   36G   80G  32% /
server:/share/        200G   38G  163G  19% /net/server/share

So basically, while I do have 163GB available on the --installroot directory's filesystem to fit those 102992MB comfortably, I only have 80GB available on the rootfs which certainly wouldn't be enough space.

Version-Release number of selected component (if applicable):
Comment 1 Sandro Mathys 2012-08-14 04:59:30 EDT
Doing the same like:
chroot /net/server/share/ yum install <long list of packages>
shows the very same error. (@Core was installed previously either way).

So I'm either misinterpreting things and there really isn't enough space (though another system with the same list of packages is "only" <25GB in size) or the error probably isn't in --installroot (or not in yum at all?).

Really not sure what to make out of it. I'm pretty sure I see some kind of bug I just have no idea what's at fault here.
Comment 2 Panu Matilainen 2012-08-14 06:34:24 EDT
If on another system the same package set is < 25GB (which sounds reasonable) and on another one it claims > 100GB (which in itself is suspect, that'd be a HUGE amount of packages), something is indeed off.

One "obvious" question: is the /net/server/share actually read-write mounted, ie can you write anything at all onto it from that system, or can you install some small set of packages there? Rpm (which is where the disk calculations occur) doesn't have a proper error message for that but reports ro-disk as full, which can give strange looking results when trying to install into a read-only mount.

Also if the package set is really < 25GB as indicated by the other system, the set should fit quite easily into the ~80GB available on this "problematic" system's root fs. Might be worth trying, just to see whether its related to the nfs-share somehow or something else.
Comment 3 Sandro Mathys 2012-08-14 06:46:32 EDT
It is mounted read-writable. Right, I did have the same idea in the meantime and did the following two sets of installation successful:

chroot /net/server/share/ yum install <long list of packages | grep gnome>
chroot /net/server/share/ yum install <long list of packages | grep eclipse>

Since both have lots of dependencies, nearly half of the list of packages (660 of ~1470) is installed by now. Trying to install the rest, again like this:
chroot /net/server/share/ yum install <long list of packages>

Results in:
Disk Requirements:
  At least 18806MB more space needed on the / filesystem.
...and I really do have 20GB available here.

My initial thought regarding the 25GB vs 100GB was that it needs more space during transaction as it download-unpacks-installed the data, i.e. needs several times the amount of space.

I'm trying to keep installing "small" subsets of packages, wondering whether I'm able to install everything or if there's maybe certain rpms triggering the behavior.
Comment 4 Panu Matilainen 2012-08-14 07:21:56 EDT
So installing 660 of ~1470 regular gnome, eclipse etc packages brought down the "missing" space from ~100GB to ~18GB. Rrrrrrright. This really doesn't add up :)

Can you try something like
# yum -y --installroot=/testroot --releasever=17 --nogpgcheck install <original long list of packages>

...so it tries to install into a chroot residing on the local disk? Just to see whether it's somehow related to the characteristics of the nfs-share or a more generic bug.
Comment 5 Sandro Mathys 2012-08-14 07:46:14 EDT
I would guess it's NFS specific since the problem occurs both with with yum --installroot=/net/server/share <...> and with chroot /net/server/share yum <...> :)

And I just confirmed this with what you suggested:
yum -y --installroot=/tmp/testroot/ --releasever 17 --nogpgcheck install <original long list of packages>
...works just fine.
Comment 6 Sandro Mathys 2012-08-14 07:50:33 EDT
Oh and what I said to try earlier works: installing the complete set in smaller chunks of <400 packages does the job. So I guess it really does fetch some wrong values or misinterpret/miscalculate them when installing to a NFS share.
Comment 7 Panu Matilainen 2012-08-14 07:58:19 EDT
Ok, thanks for testing + confirming those. I'd guess the nfs share has a "funny" blocksize that trips up something in rpm - what does 'stat -f /net/server/share/' say?
Comment 8 Sandro Mathys 2012-08-14 08:07:41 EDT
    ID: 0        Namelen: 255     Type: nfs
Block size: 1048576    Fundamental block size: 1048576
Blocks: Total: 204800     Free: 160147     Available: 160147
Inodes: Total: 587007488  Free: 563058000
Comment 9 Panu Matilainen 2012-08-14 09:06:38 EDT
Oh, of course...

Rpm uses the fs block size and number of free blocks to calculate the disk space needs of a transaction. In case of NFS, the reported block size has nothing to do with the block size of the underlying filesystem (its just the read/write chunk size), and with lots of smaller-than-blocksize files, rpm ends up thinking there aren't enough free blocks on the disk. The bigger the difference between nfs block size and underlying block size, the larger the error. Here, the nfs block size is a whopping 1M whereas the underlying fs blocks are typically in 1-4K range, and the ridiculous "need 100G more" errors are a result of that difference.

To put it another way, rpm's disk space calculation is fundamentally broken on NFS, always been. In the days of NFSv2 with ~8K block size limit this wouldn't have been so pronounced, but as newer NFS versions have allowed bigger and bigger block sizes, its gotten worse and worse - it's almost strange that nobody has reported this before now.

How to fix / work around it in rpm is an entirely different question though... AFAIK there's no way to get details about the underlying fs from an NFS mount, so it comes down to guessing (something like "on nfs, assume underlying block size to 4K and never mind what statfs() says") and/or adding configuration tunables to allow adjusting.
Comment 10 Sandro Mathys 2012-08-14 09:46:20 EDT
If it can't be properly fixed I think disabling the disk space test and issuing an appropriate warning would be way better than just failing :) Only on NFS, of course. ;)
Comment 11 Panu Matilainen 2012-08-14 10:48:24 EDT
Unnecessarily failing early is the safer option than getting it wrong the other way around :) 

The disk-size check can be manually overridden though: for rpm cli it's --ignoresize, for yum it'ss --setopt=diskspacecheck=0 from cli (or add diskspacecheck=0 to /etc/yum.conf to permanently disable, but that's not generally a good idea).
Comment 12 Zdeněk Pavlas 2013-05-14 08:14:51 EDT
*** Bug 962769 has been marked as a duplicate of this bug. ***
Comment 13 IBM Bug Proxy 2013-05-14 08:28:23 EDT
Created attachment 747667 [details]
the full output of "yum install"
Comment 14 Fedora End Of Life 2013-07-04 01:20:43 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 15 Panu Matilainen 2013-07-05 04:32:33 EDT
Moving to rawhide to avoid timeouting.
Comment 16 Panu Matilainen 2016-11-29 06:09:19 EST
*** Bug 1397569 has been marked as a duplicate of this bug. ***
Comment 18 Panu Matilainen 2017-08-22 07:41:19 EDT
In rawhide/F27 now as of rpm >= 4.13.90

Note You need to log in before you can comment on or make changes to this bug.