Bug 1287438

Summary: installation fails due to insufficient space problem in tmpfs with 2GB RAM
Product: Red Hat Enterprise Linux 7 Reporter: Masahiro Matsuya <mmatsuya>
Component: anacondaAssignee: Brian Lane <bcl>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: cww, jhunt, mbanas, mhruscak, mkolman, mmatsuya, pholica
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: anaconda-21.48.22.84-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 23:19:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1203710, 1295926, 1313485    
Attachments:
Description Flags
anaconda.log
none
ifcfg.log
none
journal.log
none
ks-script-rRevgC.log
none
ks-script-SQSu2H.log
none
packaging.log
none
program.log
none
storage.log
none
syslog none

Description Masahiro Matsuya 2015-12-02 06:05:35 UTC
Description of problem:
A customer had a installation problem of RHEL7 due to insufficient space in /tmp/yum.cache. This is related with a fedora bug, https://bugzilla.redhat.com/show_bug.cgi?id=873831. 

tmpfs is created with 1/2 of system memory by default. But, it's not enough when many repositories are used. In this customer's case, the system memory was 4GB, and more than 2GB are needed for yum.cache, and it failed. 

Can we increase the size of tmpfs only during installation? For example, size=75% or size=90% as the mount option in tmpfs.mount.

If it's hard to change the default percentage, can we have a boot parameter to change the percentage?


Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7

How reproducible:
Always

Steps to Reproduce:
1. Create a kickstart file with many repositories including many packages.
2. Install RHEL7 with 1 GB memory (minimum size written in installation guide)

Actual results:
RHEL7 installation failed with insufficient space in /tmp

Expected results:
More space can be used for /tmp, not only half of the system memory.

Comment 2 Brian Lane 2016-05-10 00:42:42 UTC
Increasing the amount of RAM used for /tmp isn't going to be a solution, especially for smaller memory systems -- you'll just run into other problems with the oom killing off parts of the system or the installer.

Do you have an example kickstart and set of repos that demonstrates the problem?

You should also retry this with a current 7.3 nightly. In 7.2 we started using zram (compressed ram) for swap which should help the situation. It is used automatically for systems with up to 2GB of ram and can be turned on for systems with > 2GB RAM by passing inst.zram=on

Comment 7 Brian Lane 2016-05-10 19:16:22 UTC
Proposed patch moving /tmp/yum.cache to /mnt/sysimage/var/tmp/yum.cache after partitioning.

https://github.com/rhinstaller/anaconda/pull/626

This should help with memory usage during the installation, but I don't think there is much that can be done before that. The problem is we need all that metadata for the depsolve and there just isn't any other place to put it.

Comment 9 Pavel Holica 2016-08-10 14:19:29 UTC
I've found problem with the commit mentioned in comment 7.

In installed system, there is /mnt/sysimage/var/tmp/yum.cache directory, this is probably caused by following line (note root=iutil.getSysroot() and cache_dir=new_cache already containing /mnt/sysimage/var/tmp/yum.cache):
self._resetYum(root=iutil.getSysroot(), keep_cache=True, releasever=releasever, cache_dir=new_cache)

Moving this to ASSIGNED, since this should be easy to fix and this bug is targeted for RC.

Note:
The directory should be removed by following lines (already in commit) once the cache is in correct place.
# Cleanup temporary yum.cache on disk
if os.path.isdir(iutil.getSysroot()+"/var/tmp/yum.cache"):
    shutil.rmtree(iutil.getSysroot()+"/var/tmp/yum.cache")

Comment 10 Brian Lane 2016-08-10 18:33:10 UTC
Please attach the logs, there should be no way for that block of code to be skipped.

Comment 11 Brian Lane 2016-08-10 19:20:10 UTC
Note that I am unable to reproduce this failure. At the end of installation there is no yum.cache directory in /mnt/sysimage/var/tmp, and it is also not there on reboot. Please make sure you are using the correct version of anaconda. My test was done with anaconda-21.48.22.81-1 from the current nightly.

Comment 12 Pavel Holica 2016-08-11 06:01:55 UTC
Sorry, for confusion, I wasn't clear enough.
There is /mnt/sysimage/mnt/sysimage/var/tmp/yum.cache (from anaconda environment), so /mnt/sysimage/var/tmp/yum.cache in installed system.
It's empty directory, but it's still there.
Also, this seems to be 100% reproducible, I've tried interactive, kickstart and installations on different architectures, it was always there.

Attaching logs from installed system.

Comment 13 Pavel Holica 2016-08-11 06:02:16 UTC
Created attachment 1189905 [details]
anaconda.log

Comment 14 Pavel Holica 2016-08-11 06:02:20 UTC
Created attachment 1189906 [details]
ifcfg.log

Comment 15 Pavel Holica 2016-08-11 06:02:28 UTC
Created attachment 1189907 [details]
journal.log

Comment 16 Pavel Holica 2016-08-11 06:02:34 UTC
Created attachment 1189908 [details]
ks-script-rRevgC.log

Comment 17 Pavel Holica 2016-08-11 06:02:39 UTC
Created attachment 1189909 [details]
ks-script-SQSu2H.log

Comment 18 Pavel Holica 2016-08-11 06:02:45 UTC
Created attachment 1189910 [details]
packaging.log

Comment 19 Pavel Holica 2016-08-11 06:02:50 UTC
Created attachment 1189911 [details]
program.log

Comment 20 Pavel Holica 2016-08-11 06:02:55 UTC
Created attachment 1189912 [details]
storage.log

Comment 21 Pavel Holica 2016-08-11 06:03:01 UTC
Created attachment 1189913 [details]
syslog

Comment 25 Marek Hruscak 2016-09-06 09:23:42 UTC
Anaconda moves yum.cache used during installation to disk right after partitioning phase - this help to avoid depletion of /tmp filesystem.

Comment 27 errata-xmlrpc 2016-11-03 23:19:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2158.html