RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1676685 - yum makes a wrong check of space available on /opt, when /opt is automounted; should not depend on /opt/rh
Summary: yum makes a wrong check of space available on /opt, when /opt is automounted;...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rpm
Version: 7.6
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pavlina Moravcova Varekova
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-12 21:05 UTC by Norman Gray
Modified: 2019-04-08 06:57 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-04 18:15:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
CentOS 10301 0 None None None 2019-02-12 21:05:39 UTC

Description Norman Gray 2019-02-12 21:05:39 UTC 
Description of problem:

Starting point: fresh install of CentOS 7.6-1810, plus configuration to make the machine an NFS client. Machine configured so that `/opt` is automounted, and `/opt/rh` maps to `localhost:/local/rh`, which does exist on the *local* machine.  This is mounted writable (ie `# echo foo >/opt/rh/stamp.txt`)

A `yum install rh-python36` fails with the error

    Transaction check error:
      installing package scl-utils-20130529-19.el7.x86_64 needs 4KB on the /opt filesystem

I note that the system reports `/opt` as having zero size available:

    # df /opt
    Filesystem 1K-blocks Used Available Use% Mounted on
    auto.opt 0 0 0 - /opt

(the /local filesystem has plenty of space)

I speculate therefore that at some point when checking the presence of, and space available under, `/opt/rh`, yum checks the space available on `/opt` (or something equivalent to this), and therefore erroneously gets a wrong size, causing it to fail.

I then set `diskspacecheck=0` in `yum.conf`, and the installation proceeds without error.

Note the *same* 'transaction check error' failure happens with `yum update` unless `diskspacecheck=0`, even though in that case nothing is actually written to `/opt/rh`.

So:

  * bug: when `/opt` is automouted, yum fails to detect that `/opt/rh` exists and is writable

  * enhancement: yum should not, I think, demand that `/opt/rh` exists, in the case where it is not going to install anything there. That is, given that I do not plan to install any Software Collections on a client machine, I should be able to mount `/opt` read-only, without `/opt/rh`. (I'm happy to make this a separate enhancement request if you feel that would be neater).

Version-Release number of selected component (if applicable):

    $ yum --version
    3.4.3
    $ cat /etc/redhat-release 
    CentOS Linux release 7.6.1810 (Core)

Steps to Reproduce:
1. See above

Actual results:

Software cannot be installed when `/opt/rh` is automounted.

Expected results:

Software should be installable in this case.


Additional info:

1. It would be desirable to be able to choose to relocate `/opt` at installation time (as opposed to when one is constructing the metapackage), but I appreciate this may be infeasible, and so I have not included this as an enhancement request.

2. The notes at <https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/2/html/Packaging_Guide/chap-Advanced_Topics.html#sect-Using_Software_Collections_over_NFS> remark that 'This enables you to mount the /opt file system hierarchy over NFS as read-only', so that this is at least intended as a supported configuration (though I can see that might be approaching the problem from the slightly different point of view of doing the yum install to /opt on the NFS server).

3. The FHS description at <http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#optAddonApplicationSoftwarePackages> is not particularly clear, but the remark 'Package files that are variable (change in normal operation) must be installed in /var/opt.' seems to imply that /opt is mountable read-only. The FHS doesn't say anything about NFS in the context of /opt The remark 'Distributions may install and otherwise manage software in /opt under an appropriately registered subdirectory' does not, I think, imply that /opt should be writable other than when deliberately installing things in /opt.

4. I mistakenly posted this in the CentOS bugparade first.
Comment 2 Panu Matilainen 2019-02-13 08:09:59 UTC 
Disk space calculation is done by rpm, not yum -> reassigning.

Disk space calculations on NFS are notoriously off in any case (see bug 847960) but that's not the issue here yet.
If df /opt reports 0 blocks available then that's what rpm will see and refuse to install. I guess /opt isn't actually mounted at that point, ie the automount doesn't trigger by df and equivalent?
Comment 4 Norman Gray 2019-02-13 10:16:56 UTC 
To be precise, `/opt` is mounted at the time the check is done, since it's the automounter process that's mounted there, but no, `/opt/rh` isn't mounted until specifically that directory is examined.

Invoking `df` on that directory does trigger the mount, though:

    % df /opt/rh
    Filesystem               1K-blocks     Used  Available Use% Mounted on
    /dev/mapper/centos-home 5799155712 58369960 5740785752   2% /opt/rh

It might therefore be that a test of specifically `/opt/rh` would fix the immediate problem.

I'll highlight the enhancement request, though.  The check of `/opt/rh` is done even when `yum` is about to update Core, when nothing (always?, typically?) is going to be written to `/opt/rh`.  Thus in the default configuration, the absence of `/opt/rh` will stop `yum update`.

(thanks for the reassignment to rpm from yum; I've continued to refer to yum above since that's the component I'm handling here)
Comment 6 Pavlina Moravcova Varekova 2019-03-11 11:33:19 UTC 
Please can you sent the relevant lines from fstab file?
Comment 7 Norman Gray 2019-03-11 16:16:32 UTC 
The filesystem /opt is automounted, so there is no relevant line in /etc/fstab
Comment 8 Pavlina Moravcova Varekova 2019-03-18 14:58:45 UTC 
I tested such scenario and rpm instalation was without any problem. Please could you write the relevant line from the mapping file?
Comment 9 Norman Gray 2019-03-18 20:27:11 UTC 
Hello Pavlina.

The relevant entry in the `auto.opt` automount map is:

    rh   -defaults,ro optmachine.example.org:/local/opt/by-os/$OSNAME/rh

and the entry in `auto.master` is

    /opt   auto.opt ro,intr,hard,nosuid,nodev,root_squash

Both of these are obtained from LDAP rather than NIS (though I'd be surprised if that were relevant).  The same thing happens when `optmachine.example.org` is the machine mounting this, and when it is not.

Note that if `diskspacecheck=0` is in `/etc/yum.conf`, then the installation will proceed without a problem.

This is using (`automount --version`) "Linux automount version 5.0.7-99.el7", from package autofs-5.0.7-99.el7.x86_64

Hmm.... What else could be relevant?

I have:

    client:etc# df /opt
    Filesystem     1K-blocks  Used Available Use% Mounted on
    auto.opt               0     0         0    - /opt
    client:etc# df /opt/rh
    Filesystem                                            1K-blocks     Used  Available Use% Mounted on
    optmachine.example.org:/local/opt/by-os/centos7/rh 5799155712 70018048 5729137664   2% /opt/rh

The `/etc/nsswitch.conf` has:

    automount:  files sss

`/etc/auto.master` includes:

    +auto.master

(I notice that on one machine I have the default other entries in this file -- eg `/net` -- commented out, and on another they're uncommented as by default; I don't think this makes a difference).

I don't think I do anything beyond these configurations, but I'd be happy to report any other configuration details you can think of, if you think they might be relevant.
Comment 10 Norman Gray 2019-03-19 10:38:18 UTC 
I've just reviewed the original problem report, and I should note that the setup I've described in the last comment is slightly different from that in the original report.

The initial report describes an NFS mount of localhost:/local/rh whereas the last comment describes a mount of optmachine.example.org (both when this is the same as localhost and when it is different).

The behaviour appears to be identical in each case, so I don't think this is relevant, but I wanted to acknowledge the marginal drift in the config being described.

Also, as reported in the comment above, the _current_ export of `/opt` to this machine is read-only (which of course won't work when actually trying to install an SCL package like rh-python36), but this is a later adjustment -- sorry.  When I reported this I did have the filesystem mounted read-write.

Also, I see that I mentioned in the original CentOS report (https://bugs.centos.org/view.php?id=10301) but apparently changed the text when I copied it here, that this behaviour also happens when doing `yum update`, ie, when one would _not_ expect anything to be installed in `/opt/rh`.  The process of `yum update`, without `diskspacecheck=0`, appears to include an attempted install, or at leats a check of space, on `/opt/rh`.  Or at least an unwarranted assumption that `/opt/rh` exists and is writable.

Sorry for these slight drifts.  If you want me to produce an absolutely from-scratch reproduction of the problem, I can surely do so, but I hope the notes here will make the problem reproducible, or otherwise lead someone to the underlying issue.

Thanks for your continued attention.
Comment 11 Pavlina Moravcova Varekova 2019-03-19 13:43:45 UTC 
Thank you. With the settings from Comment 9 ("auto.opt" + "auto.master") I reproduced the reported problem.
Comment 12 Pavlina Moravcova Varekova 2019-04-04 18:15:04 UTC 
I tested result of command:
rpm -U scl-utils-20130529-19.el7.x86_64

on three operating systems: CentOS-7.6 (rpm-4.11.3-35.el7.x86_64), RHEL 7.6 (rpm-4.11.3-35.el7.x86_64.rpm) and Fedora 27 (rpm-4.14.2.1-1.fc27.x86_64). In all cases I used the automount configuration from #c9.

When using rpm-4.11.3 the upgrade was not successful and it emits:
error: unpacking of archive failed on file /opt/rh: cpio: lsetfilecon failed - Read-only file system
In case of rpm-4.14.2 the upgrade was done without any problem.

When using rpm-4.11.3 with "rw" permissions in `auto.opt`: the upgrade was done without any problem.

The similar error like the one in #c1 occurs only if the updated rpm includes an additional file in directory /opt/rh and the directory (auto.opt) was without "ro" permissions. In that case it writes:
installing package dtest-2.0-1.noarch needs 16KB on the /opt/rh file system
(I modified a package for this test)

So it looks that in CentOS-8.0 the problem will be solved and for CentOS-7.6 I was not able to reproduce the exact problem. Because of the stage of CentOS-7.6 I will close the bug as NEXTRELEASE.
Comment 13 Norman Gray 2019-04-05 12:19:49 UTC 
Thanks for investigating.  I'll keep an eye out for this when I first try CentOS 8.  If it's still present then, I presume that reopening this would be the correct thing to do?
Comment 14 Pavlina Moravcova Varekova 2019-04-08 06:57:22 UTC 
Yes, in that case it would be correct.
Thank you for your very fast and precise replies on needinfo.
Note You need to log in before you can comment on or make changes to this bug.