Description of problem: See summary Version-Release number of selected component (if applicable): F15 RC2 net-install dvd How reproducible: Steps to Reproduce: 1. download the cd 2. choose on Virtualbox the iso as dvd writer 3. boot the virtual box on Actual results: Installaion chrashes Expected results: Installtion is a sucess Additional info: Last 5 lines of the crash : do-one-initcall+0x8c/0x140 populate_rootfs+0x0/0x215 kernel-init+0x1e8/0x274 kernel-init+0x0/0x274 kernel_thread_helper+0x6/0x10 I have made a box of 512MB memory.
The minimum memory required is 1GB, due to anaconda changes. When you see "populate_rootfs" in the panic, it's caused by that.
wwoods is working on some potential fixes to lower memory requirements, but yeah. You need more memory.
768MB RAM should work - maybe 640MB if you're lucky (and you have a swap partition on the target system). The work to keep the anaconda runtime compressed in memory involves patches to dracut, lorax, and anaconda. These are still in progress - we'll see where it goes.
*** Bug 683403 has been marked as a duplicate of this bug. ***
*** Bug 684880 has been marked as a duplicate of this bug. ***
*** Bug 684295 has been marked as a duplicate of this bug. ***
*** Bug 689258 has been marked as a duplicate of this bug. ***
*** Bug 692788 has been marked as a duplicate of this bug. ***
Sorry but requiring 1GB for a Fedora install is embarrassing! it makes us look worse than Windows! F14 installed fine on 512MB - F15 failing like this is a major regression which really should be fixed before the final release!
I believe 768M was the rough requirement for F14 x86_64 as well, though 512M was enough for F14 i386. Not sure whether these numbers have actually increased from 14 to 15.
I was able to do an install of F14 x86_64 on a 512MB KVM guest on Friday without any problems, whereas the F15 one requires 640MB to get going. It seems like the initrd used is simply taking up too much space.
Yes, we know. The memory requirements have grown because of the way we're handling the anaconda runtime image in Fedora 15. But we're past the beta freeze, and it's too late to fix this now. We've got plans to fix it for Fedora 16 so you can install with <=512MB RAM. See the thread starting at http://www.redhat.com/archives/anaconda-devel-list/2011-March/msg00263.html for details. It *is* possible to repack the Fedora 15 images so they use less memory during install, but it's a grotesque hack and will *not* be supported. Might be helpful to people with small amounts of memory, though.
Isn't there are more graceful failure mode -- I have to power cycle my laptop after the kernel panics. With kernel 2.6.35.11-83.fc14.i686.PAE: $ free -mo total used free shared buffers cached Mem: 480 444 35 0 12 228 Swap: 509 0 509 NB: Video memory consumes some of the 512M on this system.
The F15 release notes say: Minimum RAM for text-mode: 256 MiB Minimum RAM for graphical: 384 MiB Recommended RAM for graphical: 512 MiB fedora-release-notes-14.96.1-2.fc15.noarch
(In reply to comment #15) > The F15 release notes say: > > Minimum RAM for text-mode: 256 MiB > Minimum RAM for graphical: 384 MiB > Recommended RAM for graphical: 512 MiB > > fedora-release-notes-14.96.1-2.fc15.noarch Using qemu, I have narrowed the memory requirements: mem boot? 512 no 544 no 560 yes 576 yes 640 yes 1024 yes Example command: $ qemu-kvm -m 512 Fedora-15-Beta-i386-netinst.iso NB: 544 == 0x220, 560 == 0x230.
Created attachment 491091 [details] screenshot showing kernel messages after out-of-memory panic This is what users will see if they do not have enough memory, instead of an informative error message. QEMU shows 100% CPU utilization at this point. $ qemu-kvm -m 512 Fedora-15-Beta-i386-netinst.iso F15-Beta-RC2
(In reply to comment #17) > Created attachment 491091 [details] > screenshot showing kernel messages after out-of-memory panic > > This is what users will see if they do not have enough memory, > instead of an informative error message. Bug 680542 - better error message when out of memory for initrd.img
A workaround is to install from the Live CD, which successfully boots on a system where the net installer CD panics. Confirmed with F15-Beta-RC2: Fedora-15-Beta-i686-Live-Desktop.iso on a CDRW.
With the F14 net installer CD in qemu: mem boot? 96 no with panic (0x60) 112 yes with message (0x70) 128 yes with message (0x80) 144 yes (0x90) The message is "You do not have enough RAM to install Fedora on this machine." It appears to be from anaconda. So the memory requirements to boot to the installer have increased from 112 MB to 560 MB in going from F14 to F15. That is a five-fold increase. Example command: $ qemu-kvm -m 112 Fedora-14-i386-netinst.iso Hex equivalents are in parentheses.
(In reply to comment #20) > So the memory requirements to boot to the installer have increased > from 112 MB to 560 MB in going from F14 to F15. > That is a five-fold increase. Technically correct, but mostly irrelevant. While F15 requires more memory to *boot* the installer than F14 did, it should actually require slightly *less* memory to *complete* the installation than F14. The published minimum memory requirements for F14 are the same as F15. Low-memory F14 installs would crash/hang mysteriously because they ran out of memory in the middle of the install, leaving the system half-installed and corrupt. F15 just won't boot. One could argue this is actually better behavior.
(In reply to comment #18) > (In reply to comment #17) > > Created attachment 491091 [details] > > screenshot showing kernel messages after out-of-memory panic > > > > This is what users will see if they do not have enough memory, > > instead of an informative error message. > > Bug 680542 - better error message when out of memory for initrd.img How would this work? As you can see from the screenshot, the crash happens during kernel_init, and the trace is too long to fit on the screen. Even if we printed a more informative error message it'd scroll off the screen because of the trace, and after the crash we can't do anything because.. the kernel has crashed.
(In reply to comment #22) ... > > Bug 680542 - better error message when out of memory for initrd.img > > How would this work? As you can see from the screenshot, the crash happens > during kernel_init, and the trace is too long to fit on the screen. Even if we > printed a more informative error message it'd scroll off the screen because of > the trace, and after the crash we can't do anything because.. the kernel has > crashed. The kernel may be crashing, but it is a controlled crash: [ 2.547107] [<ffffffff81467a3a>] ? panic+0x91/0x19c [ 2.547674] [<ffffffff810daf85>] ? out_of_memory+0x2e2/0x378 [ 2.548344] [<ffffffff810def45>] ? __alloc_pages_nodemask+0x5f3/0x772 I'm not a kernel developer, but my idea would be for the kernel to dump the backtrace first, and follow that with an informative error message and put the system in a state where it can easily be rebooted. BTW, Bug 680542 is assigned to the kernel and, if I am not mistaken, there is not much the installer itself can do when the kernel runs out of memory while unpacking initrd.img.
Installing F14 and running preupgrade seemed like another possible workaround. After doing a clean F14 install, running preupgrade, and rebooting I got the out-of-memory kernel panic. I had to power cycle my laptop to recover, and was able to boot back into the F14 installation. Another side-effect of this bug is that the disc cannot be used as a rescue disc. Workarounds are to use an F14 disc or to boot from another partition.
Adam, could we get your perspective on this bug and on Bug 680542.
Preupgrade is the same thing as netinst. The only difference is where the kernel/initrd are loaded from (CD or /boot).
steve: not sure there's any perspective to provide. this is an anaconda design issue, not a QA issue. QA tests that things work as intended, we mostly don't dictate what the intention ought to be in the first place. we should fix the release notes if it's not too late; the measurements are useful but it would be better to measure how much memory is required to *complete* an install, as Will notes.
(In reply to comment #27) > steve: not sure there's any perspective to provide. this is an anaconda design > issue, not a QA issue. QA tests that things work as intended, we mostly don't > dictate what the intention ought to be in the first place. So this bug doesn't fall under any of the release criteria, then? > we should fix the release notes if it's not too late; the measurements are > useful but it would be better to measure how much memory is required to > *complete* an install, as Will notes. OK. Thanks for the suggestion.
(In reply to comment #26) > Preupgrade is the same thing as netinst. The only difference is where the > kernel/initrd are loaded from (CD or /boot). Thanks for clarifying this. Can you suggest any other workarounds to try? So far I have: 1. Run F14 or earlier until F16. 2. Increase memory (whether physical or in a VM). 3. Install from the F15 Live CD.
"So this bug doesn't fall under any of the release criteria, then?" I can't think of any, no. As long as anaconda team considers this just the consequence of the design of anaconda and the kernel and dracut and so on, and not some kind of actual bug, then it's just part of the design of the distro. There are always going to be *some* kind of minimum requirements. The exact behaviour when the requirements aren't met isn't subject to any criteria, either.
(In reply to comment #30) > There are always going to be *some* kind of minimum requirements. OK. This could mention minimum requirements, though ... 4. The installer must boot (if appropriate) and run on all primary architectures from default live image, DVD, and boot.iso install media http://fedoraproject.org/wiki/Fedora_15_Alpha_Release_Criteria
That's an Alpha requirement. Nobody brought this up as an alpha blocker. In fact, the relevant anaconda changes happened almost six months ago. If it had been mentioned as a possible problem any time in the five months between the code change and the Beta freeze, we could have worked on getting a fix ready for F15. Making major changes to the install images in the last month of a 6-month cycle is a recipe for disaster, especially when we've already delayed the release a couple of times. This is almost certainly not going to be fixed in F15. HOWEVER. As mentioned, we have some new experimental code to make images that use less RAM. We were planning to work on that for F16, but it could also be used to build F15 images. We could release that code and let people make UNSUPPORTED EXPERIMENTAL USE-AT-YOUR-OWN-RISK low-RAM F15 images, which would also get us some early testing of this code for F16. It's very unlikely that we'd put these images on the media or in the trees or officially support them in anyway. But it might help if you absolutely must install Fedora 15 on a system that absolutely cannot have more than 512MB RAM.
As Adam says, "we should fix the release notes if it's not too late". Not sure who "we" is, but I would suggest that the anaconda team take responsibility for describing the problem, the workarounds, the new minimum requirements, and future plans in the release notes. "Common bugs" might be another place ... As for how to release "new experimental code" -- You could look into how systemd was handled after it was pulled from F14. (I didn't pay much attention, but I believe it was released in rawhide.)
steve: that's not really what I meant - I just meant we should correct the requirements stated in the release notes to match what's actually required. there's been a bug open for that for ages, though, and doc team doesn't seem to care much about it.
(In reply to comment #34) > steve: that's not really what I meant - I just meant we should correct the > requirements stated in the release notes to match what's actually required. > there's been a bug open for that for ages, though, and doc team doesn't seem to > care much about it. Thanks for pointing that out: Bug 691238 - FC14 needs >320 MB for text mode install Bug 541502 - Minimum Memory about 256M required for Hard Drive installation The second bug got me to mem.h in the anaconda source, which contains what appear to be defined constants for minimum required memory: http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=pyanaconda/isys/mem.h Do these "match what's actually required"? ;-) #ifndef _MEM_H_ #define _MEM_H_ #if defined(__powerpc64__) || defined(__sparc__) #define MIN_RAM 1024*1024 // 1 GB #define GUI_INSTALL_EXTRA_RAM 512*1024 // 512 MB #else #define MIN_RAM 256 * 1024 // 256 MB #define GUI_INSTALL_EXTRA_RAM 128 * 1024 // 128 MB #endif #define URL_INSTALL_EXTRA_RAM 192 * 1024 // 192 MB #define MIN_GUI_RAM MIN_RAM + GUI_INSTALL_EXTRA_RAM #define EARLY_SWAP_RAM 512 * 1024 // 512 MB int totalMemory(void); #endif /* _MEM_H_ */
This appears to be the main page for draft release notes: http://fedoraproject.org/wiki/Category:Documentation_beats Hardware Overview (lists minimum and recommended RAM): http://fedoraproject.org/wiki/Documentation_Beats_Hardware_Overview Documentation contacts for Installation Notes: http://fedoraproject.org/wiki/Category:Documentation_beats#Installation_Notes
While perusing initrd, I discovered a "Python unit testing framework". Is that really necessary on an installer disc? drwxr-xr-x 3 root root 0 Apr 8 14:34 usr/lib64/python2.7/unittest
python-libs, which it comes from, is.
The anaconda runtime is ~90MB compressed and ~360MB unpacked; deleting the unittest module would save 700kb - slightly more than a tenth of a percent. Not worth the effort. We've been working for months now to find new ways to save *significant* amounts of space in the initramfs. For example, see this patchset from February: https://www.redhat.com/archives/anaconda-devel-list/2011-February/msg00097.html https://www.redhat.com/archives/anaconda-devel-list/2011-February/msg00099.html which removes /usr/lib/locale/* and generates the needed locale files at runtime - saving us 20MB of space. The only sane way to fix this problem is by: A) Not storing the anaconda runtime image in RAM if we can avoid it (e.g. leave it on the disk for CD/DVD/USB-based installs), and B) Keeping the runtime image compressed in-RAM otherwise (like for pxeboot installs). Compressed images are about 150-250MB smaller, depending on the filesystem/compression used. And that's what we're planning to do for F16, but it's really too late to make this kind of change safely for F15.
Will and Adam, all good points. Moving unittest would essentially mean repackaging, and Python developers are very hard to persuade, in my experience ... :-) After looking at the unpacked initrd with the superb Baobab disk usage analyzer, it is obvious that focusing on the locale files is a really good idea. BTW, I have previously done memory profiling during installs. There is an interesting graph attached to this bug that shows memory usage during package installation: Bug 633807 - semodule memory usage causes install failures on minimally configured VMs time series plot of memory and swap usage during minimal install https://bugzilla.redhat.com/attachment.cgi?id=448057 Also, install failures due to OOM: Bug 629253 - anaconda: system freezes on installing selinux https://bugzilla.redhat.com/attachment.cgi?id=448057 ISTM, memory profiling of installs during the development cycle would a good way to identify the large memory consumers and to establish minimum memory requirements.
testing with virt-manager, a 64-bit VM and the network install image, I get 592MB as the exact minimum needed...32MB bigger than Steve's number. wonder why that is. btw, I'm told that fixing the numbers in mem.h is no use, because we don't hit that code before the kernel panic. practically speaking we can't really get detection code early enough; all we can do is document the minimum correctly. the release notes are frozen, now, but the freeze is about translations and numbers don't need translating, so we may be able to get a change in through the freeze, I'll see what I can do about that once we're sure about the numbers.
Thanks for looking into this, Adam. Confirming your results ... Here is a summary, (re)tested with qemu-kvm, F15-Beta-Final: Fedora-15-Beta-x86_64-netinst.iso 576 panic (0x240) 592 boots (0x250) Fedora-15-Beta-i386-netinst.iso 544 panic (0x220) 560 boots (0x230) FYI, I was hoping to test Chris' patch to anaconda to more completely report memory requirements when there is not enough. As you note, it is not possible to reach that code: Bug 639056 - vague warning when swap not configured and insufficient memory Do a better job of explaining how much memory is required to install (#639056). http://git.fedorahosted.org/git/?p=anaconda.git;a=commit;h=40eb625a3c7744d047486f819ebe2993f555e270 > ... practically speaking we can't really get detection code early enough ... That's why the kernel would need to provide a more informative message on OOM: Bug 680542 - better error message when out of memory for initrd.img With F14, you can trigger the kernel OOM panic at 64MB: $ qemu-kvm -m 64 Fedora-14-i386-netinst.iso $ qemu-kvm -m 64 Fedora-14-x86_64-netinst.iso
Just to clarify. These numbers are just to boot the installer, right? How much is required to successfully complete the install?
(In reply to comment #43) > Just to clarify. These numbers are just to boot the installer, right? How > much is required to successfully complete the install? Thanks for asking. Yes, they are to get to the first installer menu. As it happens, I just finished testing for 32-bit install completion. The install succeeds with 560MB and default swap space (boots to gnome desktop, I excluded a few non-essential packages for speed). The install with 560MB fails with OOM if there is no swap space: Bug 696805 - installer does not require swap with minimally configured memory in a VM Tested with: $ qemu-kvm -m 560 -cdrom Fedora-15-Beta-i386-netinst.iso -hda ../f15-test1.img -boot menu=on Adam, should we open another bug for these results?
(In reply to comment #44) > Adam, should we open another bug for these results? Bug 696805 - installer does not report insufficient memory or swap
Better than nothing "fix" would be to make virt-manager (for f14 & f15) default to e.g. 768 MB instead of 512 when creating new VM's. No real downside, as people can choose a lower amount, and way less people would run into this in the first place...
That's a good suggestion. Pulling in virt-manager maintainer. Cole, can we make virt-manager default to a higher value for RAM in new VMs in F15, since 512MB isn't enough to do an F15 install?
Minimal amounts of memory for installing and booting to the gnome desktop in a VM: With default swap: (Bug 696805, Comment 5) 32-bit: 560MB 64-bit: 624MB Without swap: 32-bit: 976 MB (Bug 696805, Comment 7) 64-bit: 1280 MB (Bug 696805, Comment 8)
Yeah I think we should just bump the virt-manager default to 1GB for now. Long term plan has always been to have ram default depend on the OS being installed, but it's a bit late for that for f15. I've filed a virt-manager bug to track this: https://bugzilla.redhat.com/show_bug.cgi?id=700480
Steve Tyler wrote: > Installing F14 and running preupgrade seemed like another possible workaround. > > After doing a clean F14 install, running preupgrade, and rebooting I got the > out-of-memory kernel panic. I had to power cycle my laptop to recover, and was > able to boot back into the F14 installation. Preupgrade doesn't help, but doing a yum upgrade inside the running system should work. https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum apt-rpm might be another solution, I used this in the past, but I'm not sure how well it works now.
(In reply to comment #50) ... > Preupgrade doesn't help, but doing a yum upgrade inside the running system > should work. > https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum Thanks for suggesting this. I managed to complete an F14->15 upgrade this way, but it was not smooth sailing. In particular, I ran afoul of gnome-panel dependency problems. This update fixes them: https://admin.fedoraproject.org/updates/gnome-panel-3.0.0.1-4.fc15,gnome-python2-desktop-2.32.0-4.fc15,pygtk2-2.24.0-3.fc15 Tested in a VM with 512MB and default, full-disk F14 installer partitioning. > apt-rpm might be another solution, I used this in the past, but I'm not sure > how well it works now. OK.
Not sure if this has been covered, but even a text-based install fails on i686 with 512 MiB of RAM (tested in VirtualBox).
Should have specified F15. It hangs almost immediately after boot.
(In reply to comment #53) > Should have specified F15. It hangs almost immediately after boot. Do you see something like the screenshot in Bug 680542, Comment 12?
Yes, I have one additional line "hrtimer: interrupt took 4924368 ns", but basically that's where it hangs. I was hoping to do a bare-metal clean install on an i686 machine with 512 MiB of RAM, but it's currently running F14 so can wait until F16 is released and hopefully has a lower memory requirement.
(In reply to comment #55) > Yes, I have one additional line "hrtimer: interrupt took 4924368 ns", but > basically that's where it hangs. I was hoping to do a bare-metal clean install > on an i686 machine with 512 MiB of RAM, but it's currently running F14 so can > wait until F16 is released and hopefully has a lower memory requirement. OK, that's this bug. Will a Live CD boot? I installed that way with 496MB RAM. (496=512-16 for shared graphics memory on a laptop.)
In a VM, Fedora-15-i686-Live-Desktop.iso says "Fedora requires 640 MB to install, but you only have 512 MB on this machine." Anyway, it's not important (the hardware is flaky so an initial minimal install would be preferred, and I can wait till F16), and I'd rather not add more comments to this bug for this issue, since there are 26 people on the CC list.
OK, the F15-Beta Live CD can install with 512 MB. So, the F15-Final Live CD cannot be used to work around this bug ... :-(
(In reply to comment #58) > OK, the F15-Beta Live CD can install with 512 MB. So, the F15-Final Live CD > cannot be used to work around this bug ... :-( Is there a work around for installing F15 on a low memory machine? Seems like there should be some officially supported method for doing this even if it is a hack.
After mentioning this bug on #fedora-websites, the minimum RAM listed at http://fedoraproject.org was increased from 512 to 768. Since 640 and 768 are both nice round numbers, it should probably be one of those - would 640 be safe for all types of install, or should it stay at 768?
Installing F14 and https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum is probably your best bet. It's not "officially supported" because upgrading using yum isn't, but it's the only way I know to bypass Anaconda. (Well, you can also use apt-rpm or smart, but those are even less supported than yum. ;-) )
The release notes unfortunately never got the memo about the increased memory requirements: http://docs.fedoraproject.org/en-US/Fedora/15/html/Release_Notes/sect-Release_Notes-Welcome_to_Fedora_15.html#sect-Release_Notes-Hardware_Overview
(In reply to comment #60) > After mentioning this bug on #fedora-websites, the minimum RAM listed at > http://fedoraproject.org was increased from 512 to 768. Since 640 and 768 are > both nice round numbers, it should probably be one of those - would 640 be safe > for all types of install, or should it stay at 768? 768 was recommended here: Bug 699770 - Required memory numbers for installation need updating 640 is in the release notes git: http://git.fedorahosted.org/git/?p=docs/release-notes.git;a=blob;f=en-US/Hardware_Overview.xml 384 is what the online release notes say: http://docs.fedoraproject.org/en-US/Fedora/15/html/Release_Notes/sect-Release_Notes-Welcome_to_Fedora_15.html#sect-Release_Notes-Hardware_Overview (In reply to comment #62) > The release notes unfortunately never got the memo about the increased memory requirements: ... Noted in Bug 699770, Comment 5.
(In reply to comment #61) > Installing F14 and https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum is > probably your best bet. It's not "officially supported" because upgrading using > yum isn't, but it's the only way I know to bypass Anaconda. (Well, you can also > use apt-rpm or smart, but those are even less supported than yum. ;-) ) I successfully completed an upgrade from F14 to F15 this way with 496 MB in a VM after a clean, default, full-disk F14 install with updates. There is one dependency problem worked around with: # yum erase libpanelappletmm (This will remove gnote too, but it can be reinstalled after the upgrade.) $ qemu-kvm -m 496 -cdrom ../../F14/Fedora-14-x86_64-DVD.iso -hda ../f15-test1.img -boot menu=on
(In reply to comment #59) > (In reply to comment #58) > > OK, the F15-Beta Live CD can install with 512 MB. So, the F15-Final Live CD > > cannot be used to work around this bug ... :-( > > Is there a work around for installing F15 on a low memory machine? Seems like > there should be some officially supported method for doing this even if it is a > hack. This hack seems to work with 496 MB in a VM. If the logical volume manager is started, you will get a traceback, so it might be a good idea to pre-partition the target drive with real partitions and use Custom Layout to specify formatting and mount points. Swap space is required. [liveuser@foo ~]$ diff -u /usr/sbin/anaconda.BAK /usr/sbin/anaconda --- /usr/sbin/anaconda.BAK 2011-05-06 11:53:34.000000000 -0700 +++ /usr/sbin/anaconda 2011-05-25 14:08:17.343340262 -0700 @@ -352,7 +352,8 @@ extra_ram = 0 reason = reason_strict total_ram = int(isys.total_memory() / 1024) - needed_ram = int((isys.MIN_RAM + extra_ram) / 1024) + #needed_ram = int((isys.MIN_RAM + extra_ram) / 1024) + needed_ram = 0 if needed_ram > total_ram: from snack import SnackScreen, ButtonChoiceWindow [liveuser@foo ~]$ Tested with: $ qemu-kvm -m 496 -cdrom Fedora-15-x86_64-Live-Desktop.iso -hda ../f15-test2.img -boot menu=on [root@foo ~]# fdisk -l /dev/sda Disk /dev/sda: 12.9 GB, 12884901888 bytes 255 heads, 63 sectors/track, 1566 cylinders, total 25165824 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000750ce Device Boot Start End Blocks Id System /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 23068671 11021312 83 Linux /dev/sda3 23068672 25165823 1048576 82 Linux swap / Solaris [root@foo ~]#
I've found a simple workaround that allows to install Fedora 15 from Live media on as low as 256MB of RAM. This 640MB requirement is totally artificial and false for Live installs: https://bugzilla.redhat.com/show_bug.cgi?id=708966
Since it's based on the same images, it's almost moot to say it, but PXE booting fails on 512MB as well.
*** Bug 751034 has been marked as a duplicate of this bug. ***