Bug 573451

I get a warning to check the space on the device, but it will abort if I do the check and then close.  It doesn't offer to download stage2 later as it is supposed to do.

IMO this is a showstopper for F13. We should add it to F13Target so the QA folks can look at this problem.

I think i have found (part of) the problem: On the first attempt preupgrade did actually warn me about the low diskspace in /boot and told me that I need to download the installer image after booting through a wired connection. But when you plug in the ethernet cable before starting preupgrade, this warning is skipped and preupgrade tries to download the image regardless of the available space.

*** Bug 581948 has been marked as a duplicate of this bug. ***

I think I've found the cause - yum doesn't return the expected exception when downloading the stage2 file fails and thus preupgrade can't write the warning and continue. It's line 469 in preupgrade/__init__.py

Created attachment 407066 [details]
workaround

and probably has something to do with yum's interrupt_handler

Now it outputs this:

....
Fetched treeinfo from http://mirror.danny.cz/fedora/linux/development/13/x86_64/os//.treeinfo
treeinfo timestamp: Thu Apr 15 21:31:04 2010
MEMORY                                                                                                                         | 1.2 kB     00:00     
vmlinuz                                                                                                                        | 3.6 MB     00:00     
initrd.img                                                                                                                     |  29 MB     00:02     
Traceback (most recent call last):                           29% [==============-                                   ] 7.9 MB/s |  42 MB     00:12 ETA 
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1089, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
install.img                                                  31% [===============-                                  ] 6.7 MB/s |  44 MB     00:14 ETA 
 Current download cancelled, interrupt (ctrl-c) again within two seconds
to exit.

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1089, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Error: failure downloading install.img

The main installer image could not be saved to your hard drive. The installer can download this file once it starts, but this requires a wired network connection during installation.

If you do not have a wired network connection available, you should quit now.
...

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

(In reply to comment #2)
> IMO this is a showstopper for F13. We should add it to F13Target so the QA
> folks can look at this problem.    


Only if you insist upon using preupgrade.

Warning messages from yum would have been useful, but would not have changed the outcome.

The overall strategy needs to be revisited. Either install.img needs to be downloaded once the installer starts, or needs to be stored somewhere other than the /boot partition which is usually small. There are disadvantages and/or messy engineering whichever way you decide to jump.

The very minimum requirement is for preupgrade to work with default disk layout + default number of old kernels in /boot.

(In reply to comment #9)
> (In reply to comment #2)
> > IMO this is a showstopper for F13. We should add it to F13Target so the QA
> > folks can look at this problem.    

Not so.

Size of /boot increased to 500M in F13 initial build.

This is now merely a one-off irritation.

(In reply to comment #9)
> Only if you insist upon using preupgrade.

preupgrade is the officially recommended way to upgrade. Upgrades via DVD don't work if you have third party repos enabled and upgrades via yum are not supported by us.

(In reply to comment #10)
> Size of /boot increased to 500M in F13 initial build.

This doesn't help people upgrading from previous releases because they still have smaller /boot partitions that are not/cannot be increased during the update.

(In reply to comment #11)
> (In reply to comment #9)
> > Only if you insist upon using preupgrade.
> 
> preupgrade is the officially recommended way to upgrade. Upgrades via DVD don't
> work if you have third party repos enabled and upgrades via yum are not
> supported by us.
> 
> (In reply to comment #10)
> > Size of /boot increased to 500M in F13 initial build.
> 
> This doesn't help people upgrading from previous releases because they still
> have smaller /boot partitions that are not/cannot be increased during the
> update.    

The topic of this discussion was about this being a show-stopper for F13, which in my view it is not.

Remedial action has already been taken (increase /boot to 500M).

All that remains is the minor inconvenience of changing size of /boot yourself before attempting the upgrade to F13beta.

(In reply to comment #12)

> The topic of this discussion was about this being a show-stopper for F13, which
> in my view it is not.
> 
> Remedial action has already been taken (increase /boot to 500M).
> 
> All that remains is the minor inconvenience of changing size of /boot yourself
> before attempting the upgrade to F13beta.    

That's the problem, it's is not a "minor inconvenience", it's almost impossible to change /boot on the fly.  

Previous defaults for /boot in <= F-12 were 100 MB, at one point this was sufficient, but with only a single kernel and the minimal images, it's clearly not sufficient any more. 

Fixing /boot to 500 MB will only help for future upgrades from F-13 -> F-14.

In any case, this bug was about fixing the warning about the low disk space, which should be done in any case, since 500 MB is not sufficient in the future, so a low space warning should be emitted in any case that condition occurs.

(In reply to comment #12)
> (In reply to comment #11)
> 
> Remedial action has already been taken (increase /boot to 500M).

Not for users upgrading...

> All that remains is the minor inconvenience of changing size of /boot yourself
> before attempting the upgrade to F13beta.    

The default partitioning layout is to have all partitions except /boot in LVM. This means that users will have to resize the LVM in order to increase boot, which is definitely not a trivial task and more than just a "minor inconvenience" as it bears the risk of loosing personal data.

Anyway, I'm not intending to continue this discussion because I think we all agree that preupgrade should 
a) not crash and 
b) download the images after reboot as it did in previous releases.

Is there a workaround? For example downgrade preupgrade to fc11 version.

Possible explanation of the root cause is in comment #5 and a workaround is in attachment. The proper fix should get the size of the stage2 and skip downloading it when there is not enough space. With the workaround you must free some space manually before preupgrade would finish otherwise the changes to grub.conf couldn't be stored because there is 0 B free during the run of preupgrade, the space occupied by the partial stage2 is freed after it finishes.

(In reply to comment #15)
> Is there a workaround? For example downgrade preupgrade to fc11 version.    

The pragmatic solution/workaround (matter of taste) runs like this:

1) Download F13beta DVD.

2) Install replacement system disk.

3) Run F13beta DVD installation to the point where disk partitions have been created.

4) Reboot using Fedora rescue mode.

5) Remove any F13 data in partitions created at (2) above.

6) Recover contents of system disk from F12 backup.

7) Reboot F12 clone thus created.

8) Run preupgrade to reach F13beta.

I have further analysed the issue and here is the result - preupgrade really tries to determine whether there is enough place to store all downloaded files in /boot (see the _retrieve_file() function). It uses URLGrabber for getting the HTTP header for the file, storing and parsing the header and then checks the value of "content-length" member. And the problem is that the header parsing doesn't work, it returns an empty array. For the conversion see function _return_hdr_obj() in URLGrabber class. The conversion itself is done by mimetools.Message() and here is the problem. the urlgrabber module in F-11 is very different from the F-12 one.

Created attachment 409806 [details]
reproducer for the buggy urlgrabber

this code shows that URLGrabber is not able to return the parsed headers

(In reply to comment #12)
> The topic of this discussion was about this being a show-stopper for F13, which
> in my view it is not.
> 
> Remedial action has already been taken (increase /boot to 500M).
> 
> All that remains is the minor inconvenience of changing size of /boot yourself
> before attempting the upgrade to F13beta.    

Please keep in mind that people with software RAIDs (in my case, 200MB RAID1 /boot, and 1-2TB RAID5 LVM) will not longer be able to use preupgrade - ever, as playing with the size of RAID partitions is far from being ideal.

Why no simple default to placing the files the root directory, and the initrd/kernel in /boot (like any normal boot...)

- Gilboa

try out this urlgrabber pkg

http://kojipkgs.fedoraproject.org/packages/python-urlgrabber/3.9.1/4.1.fc12/noarch/python-urlgrabber-3.9.1-4.1.fc12.noarch.rpm

it has a fix from upstream that I think solves this problem.

Thanks python-urlgrabber-3.9.1-4.1.fc12 fixes the problem for me. :)

python-urlgrabber-3.9.1-4.1.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/python-urlgrabber-3.9.1-4.1.fc12

python-urlgrabber-3.9.1-4.1.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update python-urlgrabber'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/python-urlgrabber-3.9.1-4.1.fc12

I have just tested python-urlgrabber-3.9.1-4.1.fc12 and ran Preupgrade F12 > F13 on a small two disk software raid (all ext4 no LVM) system and /boot is small - 200meg, and the upgrade was flawless.

python-urlgrabber-3.9.1-4.1.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.

On another, x86_64 system I currently have

python-urlgrabber-3.9.1-4.2.fc12.noarch

installed.

The original problem is fully reproducible.

I updated all *url* packages to ones from fc13. The error is still reproducible:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1094, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 259, in on_assistant_apply
    self._do_main()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 278, in _do_main
    self.main_preupgrade()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 500, in main_preupgrade
    stage2file = self.pu.retrieve_non_critical_files()
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 571, in retrieve_non_critical_files
    self._retrieve_file(self.mainimage, targetdir, reserve_space=extra_space)
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 479, in _retrieve_file
    self.instrepo._getFile(relative=fileinfo, local=local)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 808, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 982, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 968, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1063, in __init__
    self._do_open()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1351, in _do_open
    self._do_grab()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1481, in _do_grab
    self._do_perform()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1276, in _do_perform
    raise KeyboardInterrupt
KeyboardInterrupt

Hmm, at least it is possible to preupgrade from F13 to F14
with a small /boot: I only have 100M /boot.

I think I removed one kernel in advance and then one more
when preupgrade complained about insufficient /boot space,
and the upgrade succeeded.

Should the bug be moved to F12?

i386 has been fixed by python-urlgrabber-3.9.1-4.1.fc12 and newer but x86_64 hasn't been tested by myself until recently.
F12 is not going to be supported with the release of F14 that is why I used F13 preupgrade and url download code and changed bug from F12 to F13 and platform from i386 to x86_64. My /boot currently has 147M free.

I see.  FWIW I have done two preupgrades from F13 to F14 on x86_64
one machine with 100M and one with 200M /boot.  Both were fine.
So just wondering if there is an additional condition needed to
reproduce this?

It appears that urlgrabber has problems getting file size via web proxy:

McAfee web Gateway 6.8.4 Build 4798 - [0]

wget has no problems getting file size via the same proxy though.

I added

print tmp.hdr
print "mysize=%s" % mysize
print "fileinfo=%s" % fileinfo

after
mysize = tmp.hdr.dict.get('content-length','-1') on line 452
in preupgrade/__init__.py file and don't see Content-Lenght header:

Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Thu, 04 Nov 2010 19:52:34 GMT
Last-Modified: Thu, 21 Oct 2010 18:30:17 GMT
Proxy-Connection: keep-alive
Server: nginx/0.7.67
Transfer-Encoding: chunked

mysize=-1
fileinfo=images/install.img

It appears that this happens in preupgrade if for whatever reason it can't detect the file size before downloading (in my case it seems it's because it's an FTP instead of HTTP server). However, this isn't the real problem - there will always be some cases where you can't know the file size in advance. The real problem is that python-urlgrabber is dumbly raising a KeyboardInterrupt on running out of disk space when writing, which preupgrade doesn't expect and therefore doesn't handle (see below). It seems like python-urlgrabber just doesn't handle errors writing to disk properly.

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1094, in _retrieve
    self.fo.write(buf)
IOError: [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 259, in on_assistant_apply
    self._do_main()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 278, in _do_main
    self.main_preupgrade()
  File "/usr/share/preupgrade/preupgrade-gtk.py", line 500, in main_preupgrade
    stage2file = self.pu.retrieve_non_critical_files()
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 572, in retrieve_non_critical_files
    self._retrieve_file(self.mainimage, targetdir, reserve_space=extra_space)
  File "/usr/lib/python2.6/site-packages/preupgrade/__init__.py", line 480, in _retrieve_file
    self.instrepo._getFile(relative=fileinfo, local=local)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 808, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 982, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 968, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1063, in __init__
    self._do_open()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1351, in _do_open
    self._do_grab()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1481, in _do_grab
    self._do_perform()
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1276, in _do_perform
    raise KeyboardInterrupt
KeyboardInterrupt

FYI the code in question:

                # this is probably wrong but ultimately this is what happens
                # we have a legit http code and a pycurl 'writer failed' code
                # which almost always means something aborted it from outside
                # since we cannot know what it is -I'm banking on it being
                # a ctrl-c. XXXX - if there's a way of going back two raises to 
                # figure out what aborted the pycurl process FIXME
                raise KeyboardInterrupt

(In reply to comment #32)

Since preugrade makes important decisions based on the file size would it be possible to download file to a temporary location first and then measure its size if Content-Lenght is not available?

(In reply to comment #29)
> Hmm, at least it is possible to preupgrade from F13 to F14
> with a small /boot: I only have 100M /boot.
> 
> I think I removed one kernel in advance and then one more
> when preupgrade complained about insufficient /boot space,
> and the upgrade succeeded.
> 
> Should the bug be moved to F12?

Results seem to vary. I have been unsuccessfully been trying to upgrade my laptop from F13 to F14 with preugrade.  When I installed F13 originally, I think indeed the suggested default was 500MB.   However, this seemed way to large to me for a couple of kernels, as in the past I never needed more than 150MB.  So I compromised and used 250MB.   Now, with only one kernel installed, I consistently run into the problem of boot being full when trying to do preupgrade.   Ironically, my Fedora 12 system did not have this problem, even though it had an even smaller boot partition.   The reason being is preupgrade on Fedora 12 correctly detected the problem and resolved it.

I personally don't know of any safe method to resize my /boot partition short of a complete backup to another drive and restore.   So this is pretty much a show stopper.   Really the fundamental problem to me seems to be not that the disk space is improperly detected, but that such a huge amount of space is needed for pregrade in boot.   You can put a complete live version of Fedora on a CD, which is only slightly larger than 500MB.  So why should so much space be required just to bootstrap the update?  Especially since all the packages are stored in the root file system, not boot...

Bill

I've used preupgrade F13->F14, on two systems with too small boot-partitions. 
- Ran preupgrade and the boot-partition was overfilled.
  On the first system, I followed the instruction and rebooted, it came up with 
  F13. Found that /boot, was overfilled. 
  On the second system, I noticed that /boot was full.
- I removed the partly downloaded /boot/upgrade/install.img (stage2).
- Ran preupgrade, once again, it warned me about to small boot-partition 
  and the rest of the upgrade was OK. 


BR,
Henrik

BTW.  Contrary to what I said in comment 35, in my case it turned out trivial to resize /boot.   The reason being is I NEVER use the default partitioning scheme suggested by the installer.   Instead what I do is create many partitions to span my lvm across.   So if I need to resize something I simply need to use the lvm commands to move data off the neighbouring partition.   Perhaps this strategy should be used by default in the installer as well.  Instead of creating one huge lvm partitions, always create 20.   Then even if root spans 100% of the disk space, the 5% reserved for root is enough to be able to resize things and move the partitions around later.

I upgraded the third system, with too small boot-partition, yesterday. 
This time preupgrade checked the remaining space on /boot, and the upgrade 
was OK.

(In reply to comment #36)
> I've used preupgrade F13->F14, on two systems with too small boot-partitions. 
> - Ran preupgrade and the boot-partition was overfilled.
>   On the first system, I followed the instruction and rebooted, it came up with 
>   F13. Found that /boot, was overfilled. 
>   On the second system, I noticed that /boot was full.
> - I removed the partly downloaded /boot/upgrade/install.img (stage2).
> - Ran preupgrade, once again, it warned me about to small boot-partition 
>   and the rest of the upgrade was OK. 
> 
> 
> BR,
> Henrik

This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.