Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1102359

Summary: yum does not present useful errors when a filesystem runs out of inodes
Product: Red Hat Enterprise Linux 6 Reporter: Luke Meyer <lmeyer>
Component: rpmAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED WONTFIX QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.5CC: ffesti, james.antill, pmatilai
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-15 13:20:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1102123    

Description Luke Meyer 2014-05-28 20:09:05 UTC
Description of problem:
When yum is about to embark on a transaction that will fill a filesystem, it refuses if the filesystem does not have adequate space for the install, e.g.:

Transaction Check Error:
  installing package kernel-2.6.32-431.3.1.el6.x86_64 needs 129MB on the / filesystem

Error Summary
-------------
Disk Requirements:
  At least 129MB more space needed on the / filesystem.

It seems reasonable to extend the same test to check whether the filesystem has enough available inodes for the install. Or if not, to at least improve the error message when the install does fill the filesystem.

----------------------------------------------

Version-Release number of selected component (if applicable):
yum-3.2.29-43.el6_5.noarch


Steps to Reproduce:
1. Use up inodes on a target filesystem, e.g.:
  # mkdir -p /opt/manyfiles
  # for i in {1..10000000..1}; do touch /opt/manyfiles/file$i; done
[...]touch: cannot touch `/opt/manyfiles/file84030': No space left on device

2. Try to yum install something that goes in that FS, e.g. for /opt yum install ruby193-ruby


Actual results:
Total size: 7.3 M
Installed size: 17 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : scl-utils-20120927-11.el6_5.x86_64                                                                   1/9
Error unpacking rpm package scl-utils-20120927-11.el6_5.x86_64
error: unpacking of archive failed on file /opt/rh: cpio: mkdir
  Installing : ruby193-runtime-1.1-9.el6.x86_64                                                                     2/9
Error unpacking rpm package ruby193-runtime-1.1-9.el6.x86_64
error: unpacking of archive failed on file /opt/rh: cpio: mkdir
  Installing : ruby193-ruby-libs-1.9.3.484-49.el6.x86_64                                                            [...]
Failed:
  ruby193-ruby.x86_64 0:1.9.3.484-49.el6                    ruby193-ruby-irb.noarch 0:1.9.3.484-49.el6
  ruby193-ruby-libs.x86_64 0:1.9.3.484-49.el6               ruby193-rubygem-bigdecimal.x86_64 0:1.1.0-49.el6
  ruby193-rubygem-io-console.x86_64 0:0.3-49.el6            ruby193-rubygem-rdoc.x86_64 0:3.9.5-49.el6
  ruby193-rubygems.noarch 0:1.8.23-49.el6                   ruby193-runtime.x86_64 0:1.1-9.el6
  scl-utils.x86_64 0:20120927-11.el6_5

Complete!



Expected results:
Transaction Check Error:
  installing package ruby193-ruby-1.9.3.484-49.el6.x86_64 needs xxxxxxx more inodes on the /opt filesystem

OR

  Installing : scl-utils-20120927-11.el6_5.x86_64                                                                   1/9
Error unpacking rpm package scl-utils-20120927-11.el6_5.x86_64
error: unpacking of archive failed on file /opt/rh: cpio: mkdir DUE TO INADEQUATE INODES
[...]
Failed DUE TO INADEQUATE INODES:
  ruby193-ruby.x86_64 0:1.9.3.484-49.el6                    ruby193-ruby-irb.noarch 0:1.9.3.484-49.el6
[...]
(or at least bubbling up some kind of "No space left on device" would give a hint in the right direction)

Comment 2 James Antill 2014-05-29 15:49:12 UTC
Both checks should already be done by rpm (inodes and space). Maybe rpm has the accounting wrong somewhere (I wouldn't be shocked it nobody had tested inodes running out in a long time).

Comment 3 Luke Meyer 2014-05-29 18:05:55 UTC
Strangely, this bug is in response to a customer problem where further inquiry uncovered exactly the opposite problem: the transaction failed with a useful error message when they ran out of inodes, but failed without good messages similar to the above example when out of space.

I asked and we both had the same versions of yum and rpm:
# rpm -q yum rpm
yum-3.2.29-43.el6_5.noarch
rpm-4.8.0-37.el6.x86_64

I am not sure what to make of it.

Comment 4 Panu Matilainen 2014-06-02 10:39:19 UTC
Prior to rpm 4.11.x, the inode accounting is so broken "useless" doesn't begin to describe it: inodes are only accounted for .rpmnew|.rpmsave|.rpmorig files. So its technically possible to get the related error message, but you have to try fairly hard. And yes there are all sorts of situations where rpm gets the diskspace calculation wrong as well.

4.11.x is much better in this area but the accounting fixes rely on other fixes and changes that go too deep for backporting. I'd say CLOSED NEXTRELEASE.

Comment 5 Luke Meyer 2014-06-02 12:47:46 UTC
Thanks for the explanation. I can certainly see how it would be tricky to do the proper accounting for space and inodes, and no matter what you do it will be less than perfect. 

If it works exactly as you described, though, it sounds like RPM will essentially not check for inodes on most new RPM installs (where it would not be likely to have conflicts with existing config files). It wasn't hard at all for me to reproduce at least. It seems to me that could be improved, if imperfectly, without having to "backport the world".

Even if not, would it be possible to improve the reporting of what happens when we either run out of space or inodes during a transaction? "error: unpacking of archive failed on file /opt/rh: cpio: mkdir" isn't a very friendly error message to customers, being several steps removed from the command they entered (yum install x y z...) and buried in output that just essentially reports "Failure" at the end. Anyone can understand that things fail sometimes, but it's really frustrating when they don't at at least fail *helpfully*. Regardless of how well RPM detects the problem beforehand, the net result is that the user will have to fix their resource usage, so that's what should be communicated. I recognize getting this to bubble up helpful information may be non-trivial, but it may be worth considering.

I certainly can't determine product priorities, but assuming the NEXTRELEASE you're referring to is RHEL 7, it seems a shame to leave this so "known broken" for the duration of RHEL 6.

Comment 6 Panu Matilainen 2014-06-04 06:15:37 UTC
Yes, inodes are practically unchecked, and that's how its been since 2001. Prior to that, rpm grossly overestimated the number of inodes needed, and the current situation was considered "a fix" according to the commit log. Considering this is the first time I've ever come across anybody running out of inodes (except for the sole purpose of testing what happens in that case), it doesn't seem a very common situation.

I missed the reference to bug 1102123 - if there's now a case where these things commonly happen, sure we can see if it can be mitigated without backporting half the world.

Comment 7 Florian Festi 2014-12-15 13:20:13 UTC
Implementing a proper accounting for inodes might be done upstream at some point. But there are severe doubts if such code could be ported back to RHEL-6. Closing this as WONTFIX. 

Still thanks for bringing this to our attention. This issue is tracked upstream at http://rpm.org/ticket/882