Bug 696805 - installer does not report insufficient memory or swap
Summary: installer does not report insufficient memory or swap
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-04-14 21:12 UTC by Steve Tyler
Modified: 2011-05-14 17:09 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-05-13 20:14:10 UTC
Type: ---


Attachments (Terms of Use)
warning about not having a swap partition (65.35 KB, image/png)
2011-04-14 21:12 UTC, Steve Tyler
no flags Details
screenshot showing GUI while installer was hung after OOM (76.55 KB, image/png)
2011-04-14 21:14 UTC, Steve Tyler
no flags Details
screenshot showing out-of-memory message in /tmp/syslog (20.34 KB, image/png)
2011-04-14 21:20 UTC, Steve Tyler
no flags Details
At 1152 MB memory, screenshot showing kernel killed semodule on OOM (22.51 KB, image/png)
2011-05-05 19:46 UTC, Steve Tyler
no flags Details
diagram of memory and swap requirements for installation and booting (37.09 KB, image/jpeg)
2011-05-06 13:41 UTC, Steve Tyler
no flags Details

Description Steve Tyler 2011-04-14 21:12:35 UTC
Created attachment 492233 [details]
warning about not having a swap partition

Description of problem:
In a VM configured with 560MB memory, the installer allows the user to continue without configuring a swap partition under the custom layout menu.

This leads to an install failure due to out-of-memory.

Version-Release number of selected component (if applicable):
F15-Beta-Final net installer CD (32-bit).
anaconda 15.27

How reproducible:
Always.

Steps to Reproduce:
1. $ qemu-kvm -m 560 -cdrom Fedora-15-Beta-i386-netinst.iso -hda ../f15-test2.img -boot menu=on
2. Choose custom disk layout.
3. Format the entire disk as the root partition and do not specify swap.
  
Actual results:
A warning is displayed about not having a swap partition (attached screenshot).
The install fails with out-of-memory.

Expected results:
User is told how much memory and swap is required to complete the install.

Additional info:
Bug 639056 - vague warning when swap not configured and insufficient memory
Bug 682555 - Net-install dvd gets a kernel panic on install

Comment 1 Steve Tyler 2011-04-14 21:14:35 UTC
Created attachment 492234 [details]
screenshot showing GUI while installer was hung after OOM

NB: QEMU was consuming 100% CPU at this point.

Comment 2 Steve Tyler 2011-04-14 21:20:45 UTC
Created attachment 492235 [details]
screenshot showing out-of-memory message in /tmp/syslog

This is from a second run.
I started the package install, switched to the installer console, and ran "tail -f /tmp/syslog".

This shows that the OOM killer struck anaconda.
There was some free memory according to top.

In the first run, none of the installer consoles could be accessed.

Comment 3 Steve Tyler 2011-04-14 21:43:56 UTC
An install with 560MB memory and
the default full-disk partitioning (swap == 1140MB) succeeded, and
the VM could boot to the gnome desktop.

Comment 4 Steve Tyler 2011-04-14 23:21:08 UTC
Retitling, because I could work around Bug 696814 by increasing memory.

Comment 5 Steve Tyler 2011-04-15 00:29:52 UTC
For 32-bit install completion:

The install succeeds with 560MB and default swap space
(boots to gnome desktop, I excluded a few non-essential packages for speed).

The install with 560MB fails with OOM if there is no swap space.

Tested with:
$ qemu-kvm -m 560 -cdrom Fedora-15-Beta-i386-netinst.iso -hda ../f15-test1.img
-boot menu=on

For 64-bit install completion:

 592 fails with python traceback re swap (Bug 696814)
 608 hangs while examining disks
 624 install completes and can boot to gnome desktop

Tested with:
$ qemu-kvm -m 624 -cdrom Fedora-15-Beta-x86_64-netinst.iso -hda ../f15-test1.img -boot menu=on

Comment 6 Steve Tyler 2011-04-15 00:32:26 UTC
(In reply to comment #5)
> For 64-bit install completion:
...
>  624 install completes and can boot to gnome desktop

With default, full-disk partitioning.

Comment 7 Steve Tyler 2011-04-15 13:10:14 UTC
Memory required to complete install with no swap

F15-Beta-Final: Fedora-15-Beta-i386-DVD.iso
 560 fails with OOM at dependency check
 656 fails with OOM (OOM killer -> Xorg, anaconda)
 832 hangs at selinux-policy-targeted
 928 hangs at glibc-common or poppler-data
 976 succeeds (tested to gnome desktop)
1024 succeeds (tested to installer reboot panel)

The disk image was reformatted as a single ext4 root partition for each test.
Some non-essential packages were not installed to speed testing.

$ qemu-kvm -m 976 -cdrom Fedora-15-Beta-i386-DVD.iso -hda ../f15-test1.img -boot menu=on

Comment 8 Steve Tyler 2011-04-15 20:08:11 UTC
Memory required to complete install with no swap (64-bit case)

Fedora-15-Beta-x86_64-DVD.iso
1024 hangs at selinux-policy-targeted
1152 install completes
     during reboot:
     SELinux: Could not open policy file ... policy.24:
     No such file or directory
     Failed to load SELinux policy
1216 hangs at selinux-policy-targeted
1280 install completes, boots to gnome desktop

The disk image was reformatted as a single ext4 root partition for each test.
Some non-essential packages were not installed to speed testing.

$ qemu-kvm -m 1216 -cdrom Fedora-15-Beta-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on

Comment 9 Steve Tyler 2011-04-16 14:53:26 UTC
I having been using bisection to find intermediate test points.
For example, if 256 fails and 1024 succeeds, the next test would be at

>>> (256+1024)/2.0
640.0

(I use the Python CLI for the arithmetic and record results in a gedit window.)

It is just as important to find failure points as success points, so we can have some idea how much headroom a recommended minimum would provide. (It also helps to identify various failure modes.)

Failures are not entirely reproducible.
For example, in the i386, 928MB, no swap case the hang occurs in different packages in different runs.

The install may succeed, but the reboot may fail (x86_64, 1152, no swap).

I don't know how applicable VM results are to bare-metal installs.

Comment 10 Steve Tyler 2011-04-16 17:01:59 UTC
Here are the results of testing for the amount of memory required to boot to the first installer menu. These are useful for setting a definite lower bound on the amount of memory required to install.

F15-Beta-Final:

Fedora-15-Beta-i386-netinst.iso
544 panic (0x220)
560 boots (0x230)

Fedora-15-Beta-x86_64-netinst.iso
576 panic (0x240)
592 boots (0x250)

Columns are amount of memory (MB), test result, amount of memory in hex.

Example command:
$ qemu-kvm -m 512 Fedora-15-Beta-i386-netinst.iso

(Data copied from Bug 682555, Comment 42)

Comment 11 Steve Tyler 2011-04-20 17:50:36 UTC
(In reply to comment #9)
...
> I don't know how applicable VM results are to bare-metal installs.

Here's a partial answer:

memtest86+ in a VM reports the exact amount of memory allocated to the VM.

For example, in these two tests
memtest86+ reports 1234M and 1235M, respectively:
$ qemu-kvm -m 1234 -cdrom Fedora-15-Beta-i386-DVD.iso
$ qemu-kvm -m 1235 -cdrom Fedora-15-Beta-i386-DVD.iso

Slightly smaller numbers are reported by:
"head -1 /proc/meminfo" (verified in a VM running from the DVD in rescue mode)
"free" (not on the DVD, but checked on an installed system)

1234: 1209.66796875 (== 1238700/1024.0)
1235: 1210.6640625  (== 1239720/1024.0)

Comment 12 David Lehman 2011-04-26 15:30:12 UTC
Note: bug 699770

Comment 13 David Lehman 2011-04-26 15:36:36 UTC
On the matter of swap, you were warned. You can't ignore the warnings and then complain about what happens afterwards. We are looking into the matter of allowing installation to continue with insufficient memory.

Comment 14 Steve Tyler 2011-04-26 15:50:59 UTC
(In reply to comment #13)
> On the matter of swap, you were warned. You can't ignore the warnings and then
> complain about what happens afterwards. We are looking into the matter of
> allowing installation to continue with insufficient memory.

There is no indication in the message that the install could actually fail due to OOM.

The message should specifically state the memory and swap requirements to successfully complete the install.

The wording sounds less like a warning, and more like an advisory comment concerning "performance".

See also:
Bug 639056 - vague warning when swap not configured and insufficient memory

Comment 15 Steve Tyler 2011-04-26 16:09:23 UTC
(In reply to comment #14)
...
> There is no indication in the message that the install could actually fail due
> to OOM.
> 
> The message should specifically state the memory and swap requirements to
> successfully complete the install.

The code to do this is already there. The problem is the numbers are wrong.

http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=pyanaconda/storage/__init__.py

Comment 17 Steve Tyler 2011-04-26 17:35:57 UTC
New minimum for physical memory is 640MB. New minimum combined
swap and memory is 768MB.
http://article.gmane.org/gmane.linux.redhat.anaconda.devel/20036

Unless I am mistaken, the combined number is way too low:
Comment 7, Comment 8 which tested without swap.

Comment 18 David Lehman 2011-04-26 17:41:05 UTC
If you are willing to spend this much time and inspect the code and so on, do it in a way that does not require us to spend an equal, or greater, amount of time in addition. Write a patch. Test it. Propose it.

Comment 19 Steve Tyler 2011-04-26 19:08:04 UTC
Thanks for giving this bug some attention, Dave.

Comment 20 Steve Tyler 2011-05-05 02:52:36 UTC
> A nice theory, but it shows that you have no understanding of how
> package installation works. Or how yum is designed, for that matter.
http://article.gmane.org/gmane.linux.redhat.anaconda.devel/20069

<g> The theoretician could always try collecting some verifiable data by running "top" from the installer's shell console ...

I suggest sorting by "M" ...

Comment 21 Steve Tyler 2011-05-05 16:53:31 UTC
Thanks for the updated patch, Dave.

New minimum for physical memory is 640MB. New minimum memory for us to
not require swap space is 1152MB. I arrived at this number by doing an
x86_64 install of F15 Beta in kvm, using "Web Server" package set. I
tried 1024MB and it froze during selinux-policy-targeted installation.
http://news.gmane.org/gmane.linux.redhat.anaconda.devel

I'm still concerned about the 1152MB number. In Comment 8 I reported a boot failure after installing gnome desktop with 1152MB and a hang with 1216MB.

Were you able to boot with 1152MB after installing the "Web Server" package set?

Comment 22 Steve Tyler 2011-05-05 17:30:31 UTC
Boot failure at 1152 after installing web server group.

Fedora-15-Beta-x86_64-DVD.iso
No swap.
Full disk formatted as ext4, "/".
Web server group.

1152
SELinux:
Could not open policy file <= 
/etc/selinux/targeted/policy/policy.24: No such file or directory
Failed to load SELinux policy.

$ qemu-kvm -m 1152 -cdrom Fedora-15-Beta-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on

Comment 23 David Lehman 2011-05-05 17:39:27 UTC
There's nothing there to suggest that failure has anything to do with insufficient memory.

Comment 24 Steve Tyler 2011-05-05 17:56:32 UTC
(In reply to comment #23)
> There's nothing there to suggest that failure has anything to do with
> insufficient memory.

In Comment 8 I reported a successful install and boot at 1280.

Dan, we are trying to establish memory requirements for installation and booting. The error in Comment 22 occurs during boot after a successful install at 1152MB.

Comment 25 Steve Tyler 2011-05-05 18:27:44 UTC
(In reply to comment #23)
> There's nothing there to suggest that failure has anything to do with
> insufficient memory.

At 1152, I wanted to look at syslog before rebooting after the install completed, so I did the same thing again, and the install hung at selinux-policy-targeted:

Qemu cpu usage is at 100%.
Cannot switch to any installer console.

Comment 26 Chris Lumens 2011-05-05 18:38:24 UTC
anaconda will never be able to accurately predict the amount of memory needed for every single installation, given that package scriptlets are free to do anything they want and you are free to pick as many or as few packages as you would like.  There's no point in figuring out some hard limits that don't mean anything to most users.

We have updated the memory requirements to reflect what we believe to be the current state of things.  We have updated the documentation to reflect this information as well.  You receive a warning should you choose to not use swap, which is on top of having to go through custom partitioning and get rid of swap manually.  I believe we've made a best effort at making sure the vast majority of people do not hit an OOM situation during installation.  Please do not reopen this bug.

Comment 27 Steve Tyler 2011-05-05 18:57:54 UTC
(In reply to comment #26)
...
> You receive a warning should you choose to not use swap,
...

That's fine, but misses the point of testing without swap, which is to establish *combined* memory requirements (meaning physical memory plus swap). That seems so obvious that I didn't believe it needed to be pointed out.

Anyway, the takeaway lesson here is:
1. Fedora is a memory hog.
2. The development process lacks a memory profiling methodology.

Comment 28 Steve Tyler 2011-05-05 19:46:22 UTC
Created attachment 497210 [details]
At 1152 MB memory, screenshot showing kernel killed semodule on OOM

(In reply to comment #23)
> There's nothing there to suggest that failure has anything to do with
> insufficient memory.

The install appeared to succeed, because the reboot panel was displayed, but the reboot failed with:

SELinux:
Could not open policy file <= 
/etc/selinux/targeted/policy/policy.24: No such file or directory
Failed to load SELinux policy.

Comment 29 Steve Tyler 2011-05-06 13:41:31 UTC
Created attachment 497356 [details]
diagram of memory and swap requirements for installation and booting

This diagram shows what combinations of physical memory and swap will allow a successful installation and boot. Memory-requirement testing determines what numbers the two blue arrows point to.

Comment 30 Steve Tyler 2011-05-13 19:56:24 UTC
No warning and a reboot failure with:
768MB phys mem
256MB swap
F15-RC2 DVD x86_64

The install appeared to succeed, because the reboot panel was displayed, but
the reboot failed with:

SELinux:
Could not open policy file <= 
/etc/selinux/targeted/policy/policy.24: No such file or directory
Failed to load SELinux policy.

$ qemu-kvm -m 768 -cdrom Fedora-15-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on

Comment 31 Chris Lumens 2011-05-13 20:14:10 UTC
Please see comment #26.

Comment 32 Steve Tyler 2011-05-13 20:34:45 UTC
(In reply to comment #31)
> Please see comment #26.

Please see pseudo-code:

if (phys-mem + swap < 1280) {
    warn-user("need swap of at least %d MB\n", 1280 - phys-mem);
}

> anaconda will never be able to 
> accurately predict the amount of memory needed
> for every single installation,
> given that package scriptlets are free to do anything they want and
> you are free to pick as many or as few packages as you would like.

1. Predicting the future is obviously impossible,
   but *measuring* the memory+swap needed
   to install from the *DVD alone* is possible.
2. If a package scriptlet does "anything", that is a package bug.

Comment 33 David Lehman 2011-05-14 00:12:02 UTC
We will never have the time to perform exhaustive testing of every possible package set on every platform. The published memory limits are reasonable, and those who insist on toeing the line by using the absolute minimum should know the potential ramifications based on common sense.

Comment 34 Steve Tyler 2011-05-14 03:07:15 UTC
(In reply to comment #33)
> We will never have the time to perform exhaustive testing of every possible
> package set on every platform. The published memory limits are reasonable, and
> those who insist on toeing the line by using the absolute minimum should know
> the potential ramifications based on common sense.

Your hula-hoop-manager would know ... ;-)

Comment 35 Steve Tyler 2011-05-14 16:51:24 UTC
(In reply to comment #33)
> We will never have the time to perform exhaustive testing of every possible
> package set on every platform. The published memory limits are reasonable, and
> those who insist on toeing the line by using the absolute minimum should know
> the potential ramifications based on common sense.

Why couldn't anaconda do memory profiling itself?

It already supports automated installs ...
Couldn't it report installation statistics like maximum memory required by each package and the time to install each package?


Note You need to log in before you can comment on or make changes to this bug.