Bug 530541 - Free space check on /boot not thorough enough
Free space check on /boot not thorough enough
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: preupgrade (Show other bugs)
11
All Linux
low Severity medium
: ---
: ---
Assigned To: Seth Vidal
Fedora Extras Quality Assurance
https://fedoraproject.org/wiki/Common...
: CommonBugs
: 534052 534055 538584 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-10-23 07:44 EDT by Alexander Boström
Modified: 2014-01-21 18:11 EST (History)
13 users (show)

See Also:
Fixed In Version: 1.1.3-1.fc11
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-11-20 00:11:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
patched updates.img (1.41 MB, application/octet-stream)
2009-11-12 14:50 EST, Jerry Vonau
no flags Details
revised patch (4.36 KB, patch)
2009-11-12 15:11 EST, Jerry Vonau
no flags Details | Diff
revised updates.img (1.41 MB, application/octet-stream)
2009-11-12 15:14 EST, Jerry Vonau
no flags Details
typo in other patch (4.37 KB, patch)
2009-11-12 15:39 EST, Jerry Vonau
no flags Details | Diff
updates.img with typo fixed (1.41 MB, application/octet-stream)
2009-11-12 15:43 EST, Jerry Vonau
no flags Details

  None (edit)
Description Alexander Boström 2009-10-23 07:44:09 EDT
Description of problem:
The free space check on /boot is not thorough enough, causing the user to get stuck with too little disk space on /boot to complete the upgrade.

Version-Release number of selected component (if applicable):
preupgrade-1.1.0-1.fc11.noarch


I try to preupgrade to (pre-)F12. After the preupgrade assistant finishes, I have:

/dev/sda1             190M  177M  3.7M  98% /boot

After a reboot, Anaconda tells me it can't finish the upgrade because there's not enough space on /boot. I would have expected preupgrade to fail instead of allowing the reboot to Anaconda.

I suppose it's hard to tell beforehand how much space the rpm transaction will need, but perhaps a hard-coded minimum size check would be in order, then. (About twice the size of the current kernel and initrd together?)
Comment 1 seth vidal 2009-10-23 09:34:22 EDT
So from what I just looked at - we need for anaconda to put a kernel an initrd and a systemmap file in there. So files are roughly: 3M, 4M and 1.5M respectively.

So, maybe add 10M to the freespace check as a margin of error?

Will, does that sound reasonable?
Comment 2 Will Woods 2009-10-23 09:47:29 EDT
Sure, that's fine, although we may need to doublecheck since dracut initrds can be larger than our usual initrds sometimes.

This was supposed to be handled by the rpm transaction test, but the rpm transaction test takes several minutes to run and is (in the testing we performed) no more accurate than our rough estimates.
Comment 3 Will Woods 2009-11-10 10:14:02 EST
*** Bug 534052 has been marked as a duplicate of this bug. ***
Comment 4 seth vidal 2009-11-10 11:01:06 EST
and when rpm runs it makes some tmp files which suck up some space, too and are not always added to the calculation - I've learned that one the hard way.

sigh - fun.
Comment 5 James Laska 2009-11-10 19:01:48 EST
*** Bug 534055 has been marked as a duplicate of this bug. ***
Comment 6 James Laska 2009-11-10 19:03:52 EST
Since changing the size of a previously installed F-11 /boot is not an option, Wwoods outlined several approaches to mitigate this problem for users upgrading to F-12 from F-11.

[Nov 10 15:06:51] <      wwoods> | - write an anaconda patch to make /boot bigger and format it without reserved space
[Nov 10 15:07:23] <      wwoods> | - add instructions for doing tune2fs and removing old kernels to a wiki page (common bugs etc)
[Nov 10 15:07:30] <      wwoods> | - maybe link to that wiki page from the error dialog
[Nov 10 15:08:46] <      wwoods> | - figure out why the heck RPM / mkinitrd / whatever need so much extra overhead on /boot

I'm going to add the CommonBugs keyword to this issue which will add it to our list of bugs to document for the Common_F12_Bugs wiki.
Comment 7 Liam Li 2009-11-10 23:07:29 EST
I produced this on ppc,when upgrade from f11->f12rc4,will output error of insufficient disk space ,need 8M space in /mnt/sysimage/boot/
Comment 8 Kevin DeKorte 2009-11-11 09:25:28 EST
I ran into this as well with a preupgrade, telling me I needed 1M more space. I deleted an old kernel and tried again and got the same error. The initrd file in /boot/upgrade was quite large. I ended up using yum upgrade to upgrade since I could not resize /boot since my other partitions where LVMs and I could not figure out how to resize and move them.
Comment 9 Will Woods 2009-11-11 10:56:16 EST
/boot is typically partitioned with 5% (10MB) reserved for the root user, which anaconda/RPM seems to ignore. So 'tune2fs -r 0 /dev/your-boot-partition' would have freed up 10MB and let the upgrade complete.

We may add a dialog to offer to do this for you, but I'm wary of twiddling the filesystem bits this way. You gotta be careful with this kind of thing.
Comment 10 Jerry Vonau 2009-11-12 14:01:15 EST
(In reply to comment #6)
> Since changing the size of a previously installed F-11 /boot is not an option,
> Wwoods outlined several approaches to mitigate this problem for users upgrading
> to F-12 from F-11.
<>
> [Nov 10 15:08:46] <      wwoods> | - figure out why the heck RPM / mkinitrd /
> whatever need so much extra overhead on /boot


Think what you need to have happen is that the /upgrade/install.img in /boot should be moved off of /boot and into /mnt/sysimage/tmp just before yum/rpm is called. That should free up the needed space. I've have a untested patch.. Just looking for feedback at this point... I'll whip up an updates.img for testing and post it here later if there is interest. 

cat freeram1246.diff
diff -up ./backend.py.orig ./backend.py
--- ./backend.py.orig	2009-11-12 10:28:33.000000000 -0600
+++ ./backend.py	2009-11-12 12:46:09.000000000 -0600
@@ -154,28 +154,51 @@ class AnacondaBackend:
             return
 
         if self._loopbackFile and os.path.exists(self._loopbackFile):
+            log.info("and exists %s" % self._loopbackFile )
             return
-
-        # If we've booted off the first CD/DVD (so, not the boot.iso) then
-        # copy the install.img to the filesystem and switch loopback devices
-        # to there.  Otherwise we won't be able to unmount and swap media.
-        if not anaconda.mediaDevice or not os.path.exists(installimg):
+ 
+         # If they booted with a boot.iso, just continue using that install.img.
+        if os.path.exists("/mnt/stage2/images/install.img"):
+            log.info("Don't need to transfer stage2 image")
             return
+ 
+         # If we've booted off the first CD/DVD (so, not the boot.iso) then
+         # copy the install.img to the filesystem and switch loopback devices
+         # to there.  Otherwise we won't be able to unmount and swap media.
+        if not anaconda.mediaDevice or not os.path.exists(installimg):
+ 
+        log.info("mediaDevice is %s" % anaconda.mediaDevice )
+        log.info("Using %s as stage2 image" % installimg)
 
-        free = anaconda.id.storage.fsFreeSpace
         self._loopbackFile = "%s%s/rhinstall-install.img" % (anaconda.rootPath,
-                                                             free[0][0])
+                                                               "/tmp")
+        log.info("New image name %s" % self._loopbackFile )
+
+        if os.path.exists("/mnt/sysimage/boot/upgrade/install.img"):
+            log.info("OVERRIDE using stage2 image from /mnt/sysimage/boot/upgrade")
+            stage2img = "/mnt/sysimage/boot/upgrade/install.img"
+            stage2boot = 1
+  
+        # This s/b the one that was copied into /tmp .... 
+        # Why was this copied to ram when you have passed stage2= 
+        # lets free the RAM
+        if os.path.exists("/tmp/install.img"):
+            log.info("OVERRIDE Using /tmp/install.img as stage2 image")
+            stage2img = "/tmp/install.img"
+            stage2ram = 1
+        else:
+            stage2img = installimg
 
         try:
             win = anaconda.intf.waitWindow(_("Copying File"),
-                    _("Transferring install image to hard drive"))
-            shutil.copyfile(installimg, self._loopbackFile)
+                   _("Transferring install image to hard drive..."))
+            shutil.copyfile(stage2img, self._loopbackFile)
             win.pop()
         except Exception, e:
             if win:
                 win.pop()
 
-            log.critical("error transferring install.img: %s" %(e,))
+            log.critical("error transferring stage2img: %s" %(e,))
 
             if isinstance(e, IOError) and e.errno == 5:
                 msg = _("An error occurred transferring the install image "
@@ -195,7 +218,16 @@ class AnacondaBackend:
             return 1
 
         isys.lochangefd("/dev/loop0", self._loopbackFile)
-        isys.umount("/mnt/stage2")
+
+        if stage2ram or stage2boot
+            try:
+                os.unlink(stage2img)
+            except:
+                pass 
+        try:
+            isys.umount("/mnt/stage2")
+        except:
+            pass
 
     def removeInstallImage(self):
         if self._loopbackFile:
diff -up ./yuminstall.py.orig ./yuminstall.py
--- ./yuminstall.py.orig	2009-11-12 10:28:33.000000000 -0600
+++ ./yuminstall.py	2009-11-12 12:04:54.000000000 -0600
@@ -631,6 +631,9 @@ class AnacondaYum(YumSorter):
 
 
     def doConfigSetup(self, fn='/tmp/anaconda-yum.conf', root='/'):
+        stage2img = "%s/images/install.img" % self.tree
+        self.anaconda.backend.mountInstallImage(self.anaconda, stage2img)
+
         if hasattr(self, "preconf"):
             self.preconf.fn = fn
             self.preconf.root = root
@@ -858,12 +861,6 @@ class AnacondaYum(YumSorter):
         mkeys = self.tsInfo.reqmedia.keys()
         mkeys.sort(mediasort)
 
-        if len(mkeys) > 1:
-            stage2img = "%s/images/install.img" % self.tree
-            if self.anaconda.backend.mountInstallImage(self.anaconda, stage2img):
-                self.anaconda.id.storage.umountFilesystems()
-                return DISPATCH_BACK
-
         for i in mkeys:
             self.tsInfo.curmedia = i
             if i > 0:
Comment 11 Will Woods 2009-11-12 14:12:49 EST
Not a great idea, since AFAIK anaconda is being run from *inside* install.img. 

Moving the file that contains your root filesystem to a different device while the system is running might cause some problems. Not sure how/if the loop driver handles that.
Comment 12 Jerry Vonau 2009-11-12 14:42:54 EST
(In reply to comment #11)
> Not a great idea, since AFAIK anaconda is being run from *inside* install.img. 
> 

isys.lochangefd("/dev/loop0", self._loopbackFile) anaconda does that now after copying install.img from a cdrom/dvd, so it does work.

> Moving the file that contains your root filesystem to a different device while
> the system is running might cause some problems. Not sure how/if the loop
> driver handles that. 

That is what lochangefd is for, problem there is that install.img becomes
/mnt/sysimage/boot/rhinstall-install.img. 

free = anaconda.id.storage.fsFreeSpace
self._loopbackFile = "%s%s/rhinstall-install.img" %(anaconda.rootPath,
                                                        free[0][0])

I'm just trying to move rhinstall-install.img from being placed in /boot by anaconda to /tmp and then freeing up the space in /boot if needed. As a bonus anaconda's /tmp wouldn't be holding the install.img if it was present, freeing up the ram before calling yum. 

just my 2 cents
Comment 13 Jerry Vonau 2009-11-12 14:50:30 EST
Created attachment 369306 [details]
patched updates.img

updates.img containing the above patched files. place in /boot/upgrade to be picked up automatically by anaconda.
Comment 14 Jerry Vonau 2009-11-12 15:11:49 EST
Created attachment 369310 [details]
revised patch

a bit of a thinko... with preupgrade, the install method would be using hdinstall.c, install.img would already be in ram. revised patch
Comment 15 Jerry Vonau 2009-11-12 15:14:30 EST
Created attachment 369311 [details]
revised updates.img
Comment 16 Jerry Vonau 2009-11-12 15:39:32 EST
Created attachment 369316 [details]
typo in other patch
Comment 17 Jerry Vonau 2009-11-12 15:43:29 EST
Created attachment 369317 [details]
updates.img with typo fixed
Comment 18 Alexander Boström 2009-11-12 16:27:20 EST
Hi. Suggestions on how to make preupgrade better able to handle a small /boot are welcome, but I think they should go in separate bugs. This is about how to make the calculations on free and required space more accurate or at least more conservative. Thanks. :)
Comment 19 Jerry Vonau 2009-11-12 16:46:15 EST
Well if the 100+ meg of install.img is no longer in /boot when yum starts to install the rpms, this problem goes away, and the initial calculation of space needed by preupgrade is then valid IMHO :)
Comment 20 Alexander Boström 2009-11-12 16:50:14 EST
Sorry for being a bug nazi. I made bug 537243.
Comment 21 Alexander Boström 2009-11-12 16:51:46 EST
(In reply to comment #19)
> is then valid IMHO :)  

Well... :)
Comment 22 Fedora Update System 2009-11-13 17:11:41 EST
preupgrade-1.1.3-1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/preupgrade-1.1.3-1.fc12
Comment 23 Fedora Update System 2009-11-13 17:13:01 EST
preupgrade-1.1.3-1.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/preupgrade-1.1.3-1.fc10
Comment 24 Fedora Update System 2009-11-13 17:13:04 EST
preupgrade-1.1.3-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/preupgrade-1.1.3-1.fc11
Comment 25 Fedora Update System 2009-11-16 02:31:43 EST
preupgrade-1.1.3-1.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update preupgrade'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-11547
Comment 26 James Laska 2009-11-16 13:05:51 EST
The Common_F12_Bugs page has been updated with guidance on this issue. See https://fedoraproject.org/wiki/Common_F12_bugs#preupgrade-boot and https://fedoraproject.org/wiki/Common_F12_bugs#preupgrade-free-space.  Feedback on the content encouraged.
Comment 27 Will Woods 2009-11-19 11:22:21 EST
*** Bug 538584 has been marked as a duplicate of this bug. ***
Comment 28 Fedora Update System 2009-11-20 00:11:05 EST
preupgrade-1.1.3-1.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 29 Fedora Update System 2009-11-20 00:16:18 EST
preupgrade-1.1.3-1.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 30 Fedora Update System 2009-11-20 00:32:29 EST
preupgrade-1.1.3-1.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 31 david 2009-11-26 03:10:07 EST
this bug has persisted after installing preupgrade-1.1.3-1.fc11.
I did a clean install for fc11 - so have the default /boot size.

I also ran into Bug 538118 and had to wipe my preupgrade cache files to start again.  But can't see any traces in /boot of the earlier run that could be clogging it up.

Preupgrade warned me to clean out old kernels - which I did and then allowed me to finish the process. But after rebooting it came up with the error of not enough space in the /boot partition.
Comment 32 david 2009-11-26 03:13:10 EST
df -h /boot shows the following:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             194M  158M   27M  86% /boot
Comment 33 James Laska 2009-11-29 17:52:54 EST
(In reply to comment #32)
> df -h /boot shows the following:
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             194M  158M   27M  86% /boot  

Greetings David!  The bugfix for this issue also came with several recommended manual steps to alleviate the disk space shortage in /boot.  Please take a look at https://fedoraproject.org/wiki/Common_F12_bugs#preupgrade-boot for additional suggestions on addressing the low disk space on /boot when using preupgrade.  Thanks!
Comment 34 david 2009-11-30 00:06:03 EST
Thanks - I did find that resource and removing the reserve faced fixed the problem.
My point, though, preupgrade still allowed me to go through the whole process without warning me I had insufficient space. So it is not allowing enough margin for larger install requirements.
Btw, this was only the second of many problems with the install using preupgrade, that eventually led me to abandon it altogether and do a fresh install from media.
Comment 35 Robert Binkhorst 2009-12-08 14:14:08 EST
Hi!

I'm using the latest preupgrade for F11 (preupgrade-1.1.3-1.fc11.noarch), it gave me the warning I didn't have enough disk space, I cleaned it up until preupgrade was happy, and I started the installation. Anaconda then failed on me with not enough disk space, it claimed I needed to free up 0M. I'll remove some more stuff, but it seems the amount of free space required might need an extra buffer or something..
Comment 36 Bruno Medeiros 2009-12-21 08:56:57 EST
Hi, people!

Last week I tried to upgrade my two Fedora 10 servers to Fedora 12. And on the two servers I have problems with /boot free space.

The most bizarre problem was that message telling me to free up 0M. After follow the steps on http://fedoraproject.org/wiki/PreUpgrade#Not_enough_space_in_.2Fboot , I finally could upgrade without problems.

I think the 'free up 0M' message is given because preupgrade (or anaconda) does not consider the root reserved space on this counts, assuming that it will have this space in case it will not. 

The 'tune2fs' step solved the problem for me. Maybe it could be integrated to the preupgrade install scripts...
Comment 37 Klaus Pedersen 2010-06-26 09:13:26 EDT
I was hit by this problem when (trying to) update from F12 to F13.

The /boot partition is the (F12 default) 194MB (which is not enough for the F13 install.img).

Preupgrade is version 1.1.7 - far never than version 1.1.3 that was supposed to fix the problem.

Prerequisite:
 * preupgrade is not running.

Steps:
1) run preupgrade in a terminal
2) watch output in the window.

At some point there is a lot of chatter in the terminal window :
"urlgrabber" keeps casting  IOErrors (no space on device) but preupgrade continues regardless.

The last thing in the log is "raise KeyboardInterrupt" - does that make sense?! 

I had a 100MB filler file to force preupgrade to download "stage2" from the network. That didn't work - instead i was left with a (truncated) 60MB install.img in /boot/upgrade - and with no changes to grub.conf. (I guess because /boot is full).

Nevertheless - preupgrade proudly proclaims that it is ready to upgrade as soon as the "Reboot" button is pressed!


Result:
1) Rebooting just start F12 again.
2) Rebooting and setting "root/kernel/initrd" hangs (i assume that the truncated  install.img is blindly loaded and executed).
3) removing install.img, and doing step 2) leaves you with a not very helpful installer. I can select that the install.img is on a "URL" but what does the input field expect?


I'm out of ideas :-(

Note You need to log in before you can comment on or make changes to this bug.