Created attachment 492233 [details] warning about not having a swap partition Description of problem: In a VM configured with 560MB memory, the installer allows the user to continue without configuring a swap partition under the custom layout menu. This leads to an install failure due to out-of-memory. Version-Release number of selected component (if applicable): F15-Beta-Final net installer CD (32-bit). anaconda 15.27 How reproducible: Always. Steps to Reproduce: 1. $ qemu-kvm -m 560 -cdrom Fedora-15-Beta-i386-netinst.iso -hda ../f15-test2.img -boot menu=on 2. Choose custom disk layout. 3. Format the entire disk as the root partition and do not specify swap. Actual results: A warning is displayed about not having a swap partition (attached screenshot). The install fails with out-of-memory. Expected results: User is told how much memory and swap is required to complete the install. Additional info: Bug 639056 - vague warning when swap not configured and insufficient memory Bug 682555 - Net-install dvd gets a kernel panic on install
Created attachment 492234 [details] screenshot showing GUI while installer was hung after OOM NB: QEMU was consuming 100% CPU at this point.
Created attachment 492235 [details] screenshot showing out-of-memory message in /tmp/syslog This is from a second run. I started the package install, switched to the installer console, and ran "tail -f /tmp/syslog". This shows that the OOM killer struck anaconda. There was some free memory according to top. In the first run, none of the installer consoles could be accessed.
An install with 560MB memory and the default full-disk partitioning (swap == 1140MB) succeeded, and the VM could boot to the gnome desktop.
Retitling, because I could work around Bug 696814 by increasing memory.
For 32-bit install completion: The install succeeds with 560MB and default swap space (boots to gnome desktop, I excluded a few non-essential packages for speed). The install with 560MB fails with OOM if there is no swap space. Tested with: $ qemu-kvm -m 560 -cdrom Fedora-15-Beta-i386-netinst.iso -hda ../f15-test1.img -boot menu=on For 64-bit install completion: 592 fails with python traceback re swap (Bug 696814) 608 hangs while examining disks 624 install completes and can boot to gnome desktop Tested with: $ qemu-kvm -m 624 -cdrom Fedora-15-Beta-x86_64-netinst.iso -hda ../f15-test1.img -boot menu=on
(In reply to comment #5) > For 64-bit install completion: ... > 624 install completes and can boot to gnome desktop With default, full-disk partitioning.
Memory required to complete install with no swap F15-Beta-Final: Fedora-15-Beta-i386-DVD.iso 560 fails with OOM at dependency check 656 fails with OOM (OOM killer -> Xorg, anaconda) 832 hangs at selinux-policy-targeted 928 hangs at glibc-common or poppler-data 976 succeeds (tested to gnome desktop) 1024 succeeds (tested to installer reboot panel) The disk image was reformatted as a single ext4 root partition for each test. Some non-essential packages were not installed to speed testing. $ qemu-kvm -m 976 -cdrom Fedora-15-Beta-i386-DVD.iso -hda ../f15-test1.img -boot menu=on
Memory required to complete install with no swap (64-bit case) Fedora-15-Beta-x86_64-DVD.iso 1024 hangs at selinux-policy-targeted 1152 install completes during reboot: SELinux: Could not open policy file ... policy.24: No such file or directory Failed to load SELinux policy 1216 hangs at selinux-policy-targeted 1280 install completes, boots to gnome desktop The disk image was reformatted as a single ext4 root partition for each test. Some non-essential packages were not installed to speed testing. $ qemu-kvm -m 1216 -cdrom Fedora-15-Beta-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on
I having been using bisection to find intermediate test points. For example, if 256 fails and 1024 succeeds, the next test would be at >>> (256+1024)/2.0 640.0 (I use the Python CLI for the arithmetic and record results in a gedit window.) It is just as important to find failure points as success points, so we can have some idea how much headroom a recommended minimum would provide. (It also helps to identify various failure modes.) Failures are not entirely reproducible. For example, in the i386, 928MB, no swap case the hang occurs in different packages in different runs. The install may succeed, but the reboot may fail (x86_64, 1152, no swap). I don't know how applicable VM results are to bare-metal installs.
Here are the results of testing for the amount of memory required to boot to the first installer menu. These are useful for setting a definite lower bound on the amount of memory required to install. F15-Beta-Final: Fedora-15-Beta-i386-netinst.iso 544 panic (0x220) 560 boots (0x230) Fedora-15-Beta-x86_64-netinst.iso 576 panic (0x240) 592 boots (0x250) Columns are amount of memory (MB), test result, amount of memory in hex. Example command: $ qemu-kvm -m 512 Fedora-15-Beta-i386-netinst.iso (Data copied from Bug 682555, Comment 42)
(In reply to comment #9) ... > I don't know how applicable VM results are to bare-metal installs. Here's a partial answer: memtest86+ in a VM reports the exact amount of memory allocated to the VM. For example, in these two tests memtest86+ reports 1234M and 1235M, respectively: $ qemu-kvm -m 1234 -cdrom Fedora-15-Beta-i386-DVD.iso $ qemu-kvm -m 1235 -cdrom Fedora-15-Beta-i386-DVD.iso Slightly smaller numbers are reported by: "head -1 /proc/meminfo" (verified in a VM running from the DVD in rescue mode) "free" (not on the DVD, but checked on an installed system) 1234: 1209.66796875 (== 1238700/1024.0) 1235: 1210.6640625 (== 1239720/1024.0)
Note: bug 699770
On the matter of swap, you were warned. You can't ignore the warnings and then complain about what happens afterwards. We are looking into the matter of allowing installation to continue with insufficient memory.
(In reply to comment #13) > On the matter of swap, you were warned. You can't ignore the warnings and then > complain about what happens afterwards. We are looking into the matter of > allowing installation to continue with insufficient memory. There is no indication in the message that the install could actually fail due to OOM. The message should specifically state the memory and swap requirements to successfully complete the install. The wording sounds less like a warning, and more like an advisory comment concerning "performance". See also: Bug 639056 - vague warning when swap not configured and insufficient memory
(In reply to comment #14) ... > There is no indication in the message that the install could actually fail due > to OOM. > > The message should specifically state the memory and swap requirements to > successfully complete the install. The code to do this is already there. The problem is the numbers are wrong. http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=pyanaconda/storage/__init__.py
http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=pyanaconda/isys/mem.h
New minimum for physical memory is 640MB. New minimum combined swap and memory is 768MB. http://article.gmane.org/gmane.linux.redhat.anaconda.devel/20036 Unless I am mistaken, the combined number is way too low: Comment 7, Comment 8 which tested without swap.
If you are willing to spend this much time and inspect the code and so on, do it in a way that does not require us to spend an equal, or greater, amount of time in addition. Write a patch. Test it. Propose it.
Thanks for giving this bug some attention, Dave.
> A nice theory, but it shows that you have no understanding of how > package installation works. Or how yum is designed, for that matter. http://article.gmane.org/gmane.linux.redhat.anaconda.devel/20069 <g> The theoretician could always try collecting some verifiable data by running "top" from the installer's shell console ... I suggest sorting by "M" ...
Thanks for the updated patch, Dave. New minimum for physical memory is 640MB. New minimum memory for us to not require swap space is 1152MB. I arrived at this number by doing an x86_64 install of F15 Beta in kvm, using "Web Server" package set. I tried 1024MB and it froze during selinux-policy-targeted installation. http://news.gmane.org/gmane.linux.redhat.anaconda.devel I'm still concerned about the 1152MB number. In Comment 8 I reported a boot failure after installing gnome desktop with 1152MB and a hang with 1216MB. Were you able to boot with 1152MB after installing the "Web Server" package set?
Boot failure at 1152 after installing web server group. Fedora-15-Beta-x86_64-DVD.iso No swap. Full disk formatted as ext4, "/". Web server group. 1152 SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.24: No such file or directory Failed to load SELinux policy. $ qemu-kvm -m 1152 -cdrom Fedora-15-Beta-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on
There's nothing there to suggest that failure has anything to do with insufficient memory.
(In reply to comment #23) > There's nothing there to suggest that failure has anything to do with > insufficient memory. In Comment 8 I reported a successful install and boot at 1280. Dan, we are trying to establish memory requirements for installation and booting. The error in Comment 22 occurs during boot after a successful install at 1152MB.
(In reply to comment #23) > There's nothing there to suggest that failure has anything to do with > insufficient memory. At 1152, I wanted to look at syslog before rebooting after the install completed, so I did the same thing again, and the install hung at selinux-policy-targeted: Qemu cpu usage is at 100%. Cannot switch to any installer console.
anaconda will never be able to accurately predict the amount of memory needed for every single installation, given that package scriptlets are free to do anything they want and you are free to pick as many or as few packages as you would like. There's no point in figuring out some hard limits that don't mean anything to most users. We have updated the memory requirements to reflect what we believe to be the current state of things. We have updated the documentation to reflect this information as well. You receive a warning should you choose to not use swap, which is on top of having to go through custom partitioning and get rid of swap manually. I believe we've made a best effort at making sure the vast majority of people do not hit an OOM situation during installation. Please do not reopen this bug.
(In reply to comment #26) ... > You receive a warning should you choose to not use swap, ... That's fine, but misses the point of testing without swap, which is to establish *combined* memory requirements (meaning physical memory plus swap). That seems so obvious that I didn't believe it needed to be pointed out. Anyway, the takeaway lesson here is: 1. Fedora is a memory hog. 2. The development process lacks a memory profiling methodology.
Created attachment 497210 [details] At 1152 MB memory, screenshot showing kernel killed semodule on OOM (In reply to comment #23) > There's nothing there to suggest that failure has anything to do with > insufficient memory. The install appeared to succeed, because the reboot panel was displayed, but the reboot failed with: SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.24: No such file or directory Failed to load SELinux policy.
Created attachment 497356 [details] diagram of memory and swap requirements for installation and booting This diagram shows what combinations of physical memory and swap will allow a successful installation and boot. Memory-requirement testing determines what numbers the two blue arrows point to.
No warning and a reboot failure with: 768MB phys mem 256MB swap F15-RC2 DVD x86_64 The install appeared to succeed, because the reboot panel was displayed, but the reboot failed with: SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.24: No such file or directory Failed to load SELinux policy. $ qemu-kvm -m 768 -cdrom Fedora-15-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on
Please see comment #26.
(In reply to comment #31) > Please see comment #26. Please see pseudo-code: if (phys-mem + swap < 1280) { warn-user("need swap of at least %d MB\n", 1280 - phys-mem); } > anaconda will never be able to > accurately predict the amount of memory needed > for every single installation, > given that package scriptlets are free to do anything they want and > you are free to pick as many or as few packages as you would like. 1. Predicting the future is obviously impossible, but *measuring* the memory+swap needed to install from the *DVD alone* is possible. 2. If a package scriptlet does "anything", that is a package bug.
We will never have the time to perform exhaustive testing of every possible package set on every platform. The published memory limits are reasonable, and those who insist on toeing the line by using the absolute minimum should know the potential ramifications based on common sense.
(In reply to comment #33) > We will never have the time to perform exhaustive testing of every possible > package set on every platform. The published memory limits are reasonable, and > those who insist on toeing the line by using the absolute minimum should know > the potential ramifications based on common sense. Your hula-hoop-manager would know ... ;-)
(In reply to comment #33) > We will never have the time to perform exhaustive testing of every possible > package set on every platform. The published memory limits are reasonable, and > those who insist on toeing the line by using the absolute minimum should know > the potential ramifications based on common sense. Why couldn't anaconda do memory profiling itself? It already supports automated installs ... Couldn't it report installation statistics like maximum memory required by each package and the time to install each package?