Red Hat Bugzilla – Bug 37211
Installer (incorrectly) reports "not enough file nodes"
Last modified: 2007-04-18 12:32:48 EDT
From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT) Trying to upgrade 7.0 -> 7.1; installation proceeds nicely to the part where it checks that there's enough disk space for the selected packages. I have 68MB free on "/", 580MB free in "/usr" and 3GB free on "/home" so I'm not worried. The installer comes back telling me I don't have enough file nodes on the "/" partition, I need a few hundred more. Reproducible: Sometimes Steps to Reproduce: 1. Upgrade of existing RH7.0 system to RH7.1. Actual Results: Upgrade fails at part of process where disk space checks are made. Expected Results: Installation should have proceeded normally. This problem has happened on three of the four systems I've tried to upgrade. The first upgrade, on a system with fairly limited available disk space, went smoothly. For the second (the first to fail), on my small firewall machine with very few packages and even less disk space, there were sufficiently few packages installed so that I could do the upgrade by just manually upgrading all of the installed packages, plus adding whatever new ones were required to satisfy dependencies. None of the new packages caused any problems when upgrading. The third upgrade was the machine with loads of available disk space, as mentioned earlier. Since there were a very large number of packages to install, I decided to leave the upgrade for a while after it failed. The last upgrade was a system with very limited disk space. Here I found an important clue. After the installer had failed I was trying a manual upgrade of the individual packages. The "dev" package refused to go in, rpm claiming that there were not enough file nodes, i.e. the same message as the installer. I cleared out plenty more disk space but still it complained. I had over 8000 free inodes on the root partition so I was convinced that it should go in. I tried using the "--ignoresize" option on rpm to force the upgrade to apply. It went in OK, leaving plenty of free inodes. Hence the problem would appear to be with rpm 4.0.2 (I'd already upgraded the rpm package) or with the "dev" package, or possibly some combination of the two. To test my theory, I returned to my third machine, which had not yet been upgraded. I manually upgraded the "dev" package and then ran the RH7.1 installer. The upgrade applied with no hitches. Hence I believe the problem lies with rpm 4.0.2 (which is linked with the installer I believe), or the "dev" package in RH7.1. I've put it in the "anaconda" category because it was during installation that I hit the problem, and that's probably where anyone else who hits it will probably look for existing bug reports.
Did you try supplying a "few hundred more" inodes? The disk accounting check in rpm checks both blocks and inodes, and AFAIK (and many, many installs show) that the algorithm is correct, but it's impossible to do the check precisely because of intermediate temp files and such ...
At the time I had the original problem I didn't know for sure that it was inodes that it was complaining about, so no, I didn't try freeing up more of them. I did, however, have 68MB of free space on my root partition, which was a plain vanilla ext2fs with a "clean" RH7 install on it, so I wasn't inclined to believe that I didn't actually have enough free inodes. Since the "dev" package on its own installed cleanly with no complaints from rpm, and subsequently the full upgrade completed with no complaints, I suspected a problem with the algorithm. I'm sure that you've done many more installs that I have, and presumably haven't come across anything like this before, but I *have* seen this on 3 of 4 attempted upgrades. Since I have now completed those upgrades, I'll only be able to provide more information if the problem crops up again with other machines (I still have a few to do). Is there any way I can get rpm to tell me how many free inodes it thinks a given package upgrade will need? I can then compare that with the output of "df -i", which presumably would be useful to you?
Several points: 1) the number of inodes available correlates with disk free space only because there are heuristics in mke2fs (and friends) that attempts to scale the two together. Truly, the two limits are different ... 2) rpm uses the results of statvfs() call as gospel. That may or may not be what "df -i" uses (an strace shows activity in /proc, not a stafs(2) call, but that may be the way that statfs is implemented in glibc.) Certainly the two results should be very similar, but that's not the same as identical. Write a short program to print out the return of statvfs() if you want to see see the values that rpm is using. 3) No matter what, rpm is not going to be able to better than nominal (i.e. 1% or so ) accuracy, as there's no attempt (nor should there be) to track each and every file created during an install. Even for disk blocks, rpm cannot Get It Right because of the root reserved limit (rpm adds a 5% reserved for root heuristic, not right in general). So, with file systems close to the edge, there may still be failures, possibly even intermittent non-reproducible failures. 4) Certain file system types do not return accurate information. Both vfat and reiserfs are known to fall in this category, there are other (ahem) "friends" of e2fs that will break. FWIW, rpm detects reiserfs/vfat breakage and disables inode checks, but there's bound to be more breakage ahead.
Regarding your points: 1. Understood. 2. I've written a program as suggested; the results from statvfs() line up exactly with "df" and "df -i", at least with the filesystems I've tried. 3. Can't rpm use the "f_bfree" part of the statvfs structure instead of the "f_bavail" part to determine available space if it wants to use the root reserved space? 4. Understood. So now I know how much space and inodes are actually available, how do I know how many rpm thinks it's going to need? For a system upgrade, does rpm add together the sum of the temporary inodes it'll need in each filesystem for each package, or does it just take the worst case (biggest) number from all of the packages? e.g. if (say) the "dev" package needs 5000 temporary inodes and the "dev86" package needs 500, does rpm consider that an upgrade of both packages requires 5000 or 5500? I ask this because of the case where a full upgrade failed but later passed after a manual upgrade of the "dev" package...
This is an rpm problem, fixed in rpm CVS, will be in rpm-4.0.3 final.