Bug 1410948 - Documented minimum RAM for RHEL 7 is too small
Summary: Documented minimum RAM for RHEL 7 is too small
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: doc-Installation_Guide
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Marc Muehlfeld
QA Contact: ecs-bugs
URL:
Whiteboard:
: 1595369 (view as bug list)
Depends On:
Blocks: ARMTracker 1425467
TreeView+ depends on / blocked
 
Reported: 2017-01-06 23:23 UTC by Gordon Messmer
Modified: 2019-03-06 00:40 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-07 13:59:30 UTC


Attachments (Terms of Use)

Description Gordon Messmer 2017-01-06 23:23:23 UTC
Document URL: 
https://access.redhat.com/articles/rhel-limits

Section Number and Name: 
"Required minimums"

Describe the issue: 
1GB of RAM is no longer sufficient to install some products, such as CentOS 7.3 (and presumably RHEL 7.3) on x86_64.

In earlier installers, the rootfs ram filesystem was not limited, so it provided as much space as total system memory - allocated memory.  If too much data was written to the filesystem (in /tmp, for example), the kernel would panic.  Newer installers limit the size of the filesystem to 50% of system memory.  On a system with only 1GB of memory, the root filesystem will be too small to run the installer.

Suggestions for improvement: 
For version 7 on x86_64, some value larger than 1GB should be listed as a minimum requirement.

Additional information:

Comment 4 Gordon Messmer 2017-01-09 19:04:28 UTC
Regarding needinfo:

As of 7.3, the installer runs in an in-memory "rootfs" that is limited to ~50% of system RAM (at least, as far as I can tell).  On a system with just 1GB of RAM, that'll be ~500MB of space.  It downloads the stage2 installer to that filesystem, and very nearly fills it.  As a result, installation will fail in many cases.  For example, I was able to use kickstart if I did not specify a network interface in the kickstart file, but if I did, the filesystem would not have space in /tmp for files that dhclient expected to write.  Installation would fail.  Similarly, if the kickstart file attempted to write a file that would be included with %include, it would also fail due to ENOSPC.

These problems didn't come up in 7.2.  As far as I can tell, it did not limit the size of the rootfs, and the stage2 installer was also smaller so it was less likely to trigger problems anyway.  Due to the larger stage2 and the smaller rootfs, 1GB of RAM is no longer enough RAM to install RHEL 7.

Comment 5 Clayton Spicer 2017-01-12 19:23:45 UTC
Thank you for the information Gordon, and for taking the time to file this bug - I'm currently investigating the RAM requirements of the installer, and will update the installation guide and the article to which you linked when I have more information.

Comment 11 Gordon Messmer 2017-04-10 20:07:17 UTC
If I may ask: where are the fixed versions available?

This document still says that 1GB is the minimum for x86_64:
https://access.redhat.com/articles/rhel-limits

This document also states that 1GB is sufficient for HTTP and FTP installs:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/sect-installation-planning-disk-space-memory-x86.html

Comment 12 Vit Ry 2017-05-15 12:56:04 UTC
+1 and bump

Comment 14 Martin Kolman 2017-05-15 14:32:05 UTC
A couple notes from my memory requirements testing on RHEL 7.4 a while back:

Theoretical numbers
===================
Basically, what constants are used in the code & how they should work.

Minimum RAM: 320 MB
Minimum RAM on PPC 64/PPC 64 LE: 768 MB

GUI extra RAM: + 90 MB
GUI extra RAM on PPC 64/PPC 64 LE: + 512 MB

Stage 2 squashfs in memory: + 350 MB


Stage 2 *is* in memory:
- during network installation from http or ftp source

Stage 2 *is not* in memory:
- during installation booted from a media
 (physical DVD, media image booted by hypervisor or firmware)
- during network installation from NFS


no SWAP: + 200 MB
- the no SWAP extra memory requirement is not strictly required and only triggers a warning
- it is also only checked before starting the installation, not upon entry to the installation environment like the others

Examples:

TUI non PPC 64 http/ftp netinst:
320 + 350 = 670 MB

GUI non PPC 64 http/ftp netinst:
320 + 350 + 90 = 760 MB

TUI non PPC 64 media install/NFS netinst:
320 MB

GUI non PPC 64 media install/NFS netinst:
320 + 90 = 410 MB

TUI on PPC 64 media install/NFS netinst:
768 MB

GUI on PPC 64 http/ftp netinst:
768 + 512 + 320 = 1630 MB


Actual experience (7.4)
=======================

- for netinst (stage 2 squashfs in RAM) 1024 MB is needed, less
  (512 & 768 MB tried) results in being dropped to Dracut emergency shell

- for installation from DVD 512 MB appears to be enough
  for GUI installation on default settings

- if you try to use a low amount of memory for GUI installation,
  Anaconda automatically falls back to TUI installation,
  provided there is enough memory for TUI installation

- the amount of memory seems to be rather approximate on multiple levels
-> setting 300 MB RAM in virt manager results in free -h reporting 278 total RAM size,
   probably due to part of the RAM being used for the GPU
-> in the same case Anaconda reports system memory as 384 MB, which is rather inexact
   and could be actually considered a bug (at least on VMs which can have arbitrary
   RAM size)
-> most likely caused by this:

# Because /proc/meminfo only gives us the MemTotal (total physical
# RAM minus the kernel binary code), we need to round this
# up. Assuming every machine has the total RAM MB number divisible
# by 128.

- Anaconda does not care any any way about the amount of GPU memory
  and in general it should not have to

- it seems that the stage2 image (or/and potentially) the initrd
  have grown since the +350 MB RAM requirements has been implemented
-> it would be good to check if the images could be made smaller again
-> or if that's not possible bump the amount of required RAM to > 3501 MB

- text-only stage2 image would be probably quite small, but AFAIK
  no such image is currently being generated


Installations tried (all with recent RHEL 7.4 nightly)
======================================================

VM with 256 MB of RAM booted from DVD image.
- fails to boot into installation environment due
  to a kernel stack trace

VM with 300 MB of RAM booted from DVD image.
- booted default menu entry
- Anaconda reported 384 MB of RAM
- automatically fell back to TUI
- did a default settings installation
- installation finished successfully
- installed system booted fine & it was possible to log in as root

VM with 384 MB of RAM booted from DVD image.
- same as above, but detected correctly as 384 MB RAM by Anaconda

VM with 512 MB of RAM booted from DVD image.
- booted default menu entry
- successfully started into graphical installation
- did a default settings installation
- installation finished successfully
- installed system booted fine & it was possible to log in as root

VM with 512 MB RAM http netinst.
- fails to boot into installation environment
-> curl in Dracut reports "no space on device"
-> loading the initrd & downloading the stage2 to RAM apparently
   exhaust available memory
- ends in Dracut emergency shell

VM with 640 MB RAM http netinst.
- 640 MB ought to be enough for anybody ;-)
- fails to boot into installation environment
- ends in Dracut emergency shell

VM with 768 MB RAM http netinst.
- fails to boot into installation environment
- ends in Dracut emergency shell

VM with 896 MB RAM http netinst.
- fails to boot into installation environment
- ends in Dracut emergency shell

VM with 1024 MB RAM http netinst.
- successfully started into graphical installation
- did a default settings installation
- installation finished successfully
- installed system booted fine & it was possible to log in as root

Comment 15 Martin Kolman 2017-05-15 14:33:19 UTC
Laos possibly related - I've looked at kernel, initrd & stage2 size, 7.0-7.4:

RHEL 7.0
========
kernel 4.7 MB
initrd 34 MB
stage2 260 MB
------
298.7 MB

RHEL 7.1
========
kernel 4.8 MB
initrd 35 MB
stage2 256 MB
------
295.8 MB

RHEL 7.2
========
kernel 4.9 MB
initrd 38 MB
stage2 261 MB
------
303.9 MB

RHEL 7.3
========
kernel 5.1 MB
initrd 43 MB
stage2 299 MB
------
347.1 MB

RHEL 7.4 nightly
================
kernel 5.5 MB
initrd 45 MB
stage2 311 MB
------
361 MB


So it looks like an overall growth of 61 MB 
since 7.0, which is quite significant,
but still does not really explain why
network installation needs >= 1024 MB or RAM
to even get past Dracut.

I guess other things could possibly be at play, such as:
- resurfacing of the download-stuff-multiple times
  bug in Dracut
- increased memory consumption/allocation by
  kernel/Dracut/systemd/journald/etc.

Comment 16 Petr Bokoc 2017-05-23 09:39:37 UTC
Hello, I'll provide answers inline.

(In reply to Gordon Messmer from comment #11)
> If I may ask: where are the fixed versions available?
> 
> This document still says that 1GB is the minimum for x86_64:
> https://access.redhat.com/articles/rhel-limits

Apologies, we sort of forgot about this - this BZ is filed against the RHEL7 Installation Guide, and that document is a separate issue, maintained usually by a completely different team. I'll contact them to get the table updated, but it might take a while, since the table would need fairly significant structural changes if we want to point out differences in memory consumption between local media/nfs and http/ftp and between GUI and TUI installs.

> This document also states that 1GB is sufficient for HTTP and FTP installs:
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/
> html/Installation_Guide/sect-installation-planning-disk-space-memory-x86.html

I unhid two comments from Martin Kolman, one of the Anaconda developers, above - see especially comment #14. It seems that while theoretically 760 MB should be enough for a FTP install with a GUI on non-PPC64 architectures, testing indicates that you actually need a full gigabyte, any lower than that and you end up in Dracut emergency shell.

Comment 17 Gordon Messmer 2017-05-23 17:43:52 UTC
Was that testing done with a kickstart file specified?  I'm able to do a manual install in 1G, but I need more in order for kickstart to succeed.

I'm not certain why more than 347MB is needed in the root fs, either (comment 14), but any of the suggestions in comment 15 sound plausible.

When I was testing 7.3, I mounted an installed system in order to have access to 'df' and 'mount'.  I saw:

:/# /mnt/usr/bin/df
Filesystem                   1K-blocks    Used Available Use% Mounted on
rootfs                          486220  486220         0 100% / 
:/# mount
rootfs on / type rootfs (rw,size=486220k,nr_inodes=121555) 

Compared to 7.2, where I saw:

cleanup:/# mount
rootfs on / type rootfs (rw) 

So, it seems like the older setup didn't impose a limit at all, aside from the actual memory limit.  Introducing the limitation appears to have caused the problem, in addition to the increased size of the stage1/stage2 installers.

Comment 22 Martin Kolman 2017-07-04 14:25:22 UTC
I did two kickstart installations with this kickstart on a recent 7.4 nightly:

install
lang en_us.UTF-8
keyboard us
timezone America/New_York
rootpw --plain foobar
user --name="user" --password="foobar" --plaintext
zerombr
clearpart --all
autopart
%packages
vim
%end

The first was a graphical network installation (PXE boot & stage2 + repo over http) on a x86_64 VM with 1024 MiB of RAM. The installation went fine and the resulting system was bootable & working fine as far as I could tell.

The second installation was a graphical installation from a DVD media image that contained package repositories. The x86_64 VM used had 512 MiB RAM. Again the installation went without a hitch and the installed system worked fine as well.

I did the same installations (netinst@1024 MB & media-install@512 MB) before that without kickstart, so it doesn't seem like kickstart usage increases RAM usage in a noticeable manner. 

Of course kickstart files are very flexible and can run arbitrary commands via the %pre/%pre-inst/%post scripts. So in case a user-provided kickstart script runs some software that is very memory hungry or runs a command that writes a lot of data to the ramdisk - then in this case a kickstart installation could require more RAM than an installation without kickstart, as without kickstart no custom user-provided scripts are run.

Comment 27 Marc Muehlfeld 2017-07-12 08:25:04 UTC
One of our Anaconda developers run several tests. He could not reproduce that using a Kickstart file requires additional RAM. All TUI/GUI installations required the same amount of RAM - regardless if it was a manual installation or using a kickstart file. Only the installation source (local media/NFS or HTTP/HTTPS/FTP) made a difference. However, it is possible that additional commands used in Kickstart files, which require additional RAM or write data to the RAM disk, may require additional memory.

Based on the results of the tests and the developer's recommendations, I updated the RHEL Installation Guide. The updated guide will be published on the same day as RHEL 7.4 GA - which is currently planned for August 01.

Additionally, I updated the minimum RAM requirements on https://access.redhat.com/articles/rhel-limits

Comment 29 Marc Muehlfeld 2017-08-07 13:59:30 UTC
The update is now available on the Customer Portal.

Comment 31 Samantha N. Bueno 2018-07-24 07:22:05 UTC
*** Bug 1595369 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.