Red Hat Bugzilla – Bug 1024223
fedup 19->20 failed with DVD iso upgrade
Last modified: 2014-01-23 15:01:01 EST
Description of problem:
I tried to upgrade one desktop from f19 to f20, by using:
#fedup --iso Fedora-20-Beta-TC6-x86_64-DVD.iso
But it failed, from the screen boot messages, it seems it can't find any rpm packages.
Or the iso media can't be not mounted?
Version-Release number of selected component (if applicable):
Use fedup --iso to upgrade the f19 machine, then the error messages are printed on the screen after rebooted and chose "System Upgrade (fedup)".
Steps to Reproduce:
1. Prepare a f19 machine
2. fedup --iso Fedora-20-Beta-TC6-x86_64-DVD.iso
Reboot, but system are not upgraded.
System are upgraded.
Created attachment 816993 [details]
fedup log file
I can confirm this as failing with both the beta DVD mentioned above and the official Alpha DVD iso. There was an error during boot about failing to mount install media (?) and a whole lot of quickly scrolling errors after the boot (seemingly about not finding individual packages but they scrolled too fast to read). Neither of this errors seem to show up in either system or fedup logs.
I see the same problem. I tried the latest Fedora-20-TC2-x86_64-DVD.iso. The update fails as described above.
This log message is strange:
fedup.sysprep:setup_media_mount() setting up mount for /dev/loop0 at /system-upgrade/media
Obviously, fedup needs to be able to find the .iso file after it reboots, so using --iso will fail if the image is somewhere that isn't automatically mounted on boot (like a USB stick). And "/dev/loop0" doesn't usually exist..
Where is your .iso file located?
I don't know where the user who uploaded the log kept the ISO, but in my case it was in the home folder of my standard account. That should be available to fefup.
Same here. The system eventually prompted for the root password for maintenance, after which I can see the .iso just fine where it was expected to be, but cannot mount it.
Running the mount command manually under strace shows it get ENOENT for /dev/loop0. After mknoding that, it fails with ENXIO (No such device or address). There's no loop.ko anywhere under /lib/modules/3.11.6-301.fc20.x86_64 and no mention of it in dmesg.
It appears the kernel (3.11.6-301.fc20.x86_64) lacks support for loopback mounts, either built-in or as a module.
IIRC, I put the DVD iso under /root/ directory.
No improvement with Fedora-20-TC3-x86_64-DVD.iso . I am still unable to upgrade using the --iso option.
I can confirm this. I added a new disk to my VM (/dev/vdb), copied TC3 DVD.iso onto it and ran fedup --iso. Then rebooted, and there was a quick flash of thousands of "can't find" messages, then it auto-rebooted again. The upgrade.log is empty, so I can' provide any details (but that probably has a different cause - bug 1016522 - and it has been reported for a long time).
Lucas: "It appears the kernel (3.11.6-301.fc20.x86_64) lacks support for loopback mounts, either built-in or as a module."
That's certainly not the problem. I do loopback mounts from a booted system all the time - 'mount -o loop some.iso /mnt/temp'. Works fine.
Discussed at 2013-12-02 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-12-02/f20-blocker-review-%234.2013-12-02-17.02.log.txt . Accepted as a blocker per Beta criterion "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed" - https://fedoraproject.org/wiki/Fedora_20_Beta_Release_Criteria#Upgrade_requirements .
Note that this is somewhat debatable; the criteria do not expressly define whether *all* fedup mechanisms are required to work, or just *at least one*. We decided to try and maintain some standards and go with the former interpretation, though, especially since the 'repo' method more or less requires an internet connection; the ISO method would likely be the method anyone without an internet connection would expect to use.
1st thank you for making this a blocker and trying to fix the issue. I read the meeting protocol and would like to give some feedback regarding that discussion.
1. The --iso option worked without problems for upgrades from F18 to F19.
2. The --iso upgrade is very easy and hassle free. Copy iso file to harddisk, execute command, done. Compare this to all the steps listed in the wiki for an yum upgrade (http://fedoraproject.org/wiki/Upgrading_Fedora_using_yum). Please consider less experienced users without internet connection.
3. And last but least a real world use case: I maintain my parents in law computer. They live in rural Malaysia. The internet there is not reliable and too slow to execute fedup --network successfully. So when I visit I bring the iso file and the rpm's from the updates repo for the new version. Within an hour they are running the latest version of Fedora without the need to connect to the internet at all.
Matthias: thanks for the notes. The appropriate comparison for point #2 isn't a yum upgrade, though, but fedup's other mode (fetching from a mirror), which is just as easy, but obviously only works if you have a network connection or a local repository. The implied question at the meeting was basically "could we get away with shipping if only fedup's repo mode worked and not its iso mode".
The local repo could work if the --iso option can't be fixed in time.
- Fedup wiki should then be updated to spell out that --iso does not work for F19->F20.
- The wiki should then also come with a step by step explanation where and what to download, how to setup the local repo and how to run the fedup command.
- One potential risk for your mirror load if you don't spell out what to download (I use F19 as example, guess 20 will be comparable):
fedora/releases/19/Fedora/x86_64/os/Packages/ has less than 4GB but
fedora/releases/19/Everything/x86_64/os/Packages/ has approx 35GB of packages.
In my testing it works from an actual DVD (well OK, a virtual DVD, but still /dev/sr0 and not an on-disk .iso), so the problem is probably either:
a) you have the .iso on a filesystem that doesn't get mounted at boot time
(this should be detected by the next fedup version; see
b) fedup doesn't set up the .mount unit properly
(this used to work fine; I'm testing it now)
In the meantime: as a workaround, you could probably dd the .iso to a USB stick and use "fedup --device" instead.
Ah. So the problem is that the "loop" driver is now a module, and nothing in dracut explicitly requires it, so it's not in upgrade.img/initrd-fedup.img, so we can't loop-mount anything.
This seems pretty goofy. We've got tcm_loop.ko, so you can make a block device into an iSCSI target, but no loop.ko.
Given that loop.ko is ~32kb uncompressed, I think it's safe/reasonable to include it by default in dracut. Failing that, fedup-dracut (and a few other dracut modules) will need to be modified to do "instmods loop". Which might not be a bad idea either way.
Fixing this will therefore require a new build of fedup-dracut (and/or dracut, depending on how Harald & co. want to fix things), and then new installer images.
Will is aware, but just in case, Harald - go/no-go is Thursday; we have a small chance of making it if everything's fixed early (US time) tomorrow, but anything after that is a slip. so it'd be much appreciated if you could agree on an approach and get something in Bodhi ASAP. thanks.
Here's the fix I'm testing. My upgrade is running, so I consider that success:
I'll try to put together a new build of fedup and fedup-dracut ASAP.
wwoods: any news on those builds? I don't see any in koji/bodhi. thanks!
fedup-dracut-0.8.0-1.fc20 has been submitted as an update for Fedora 20.
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-dracut-0.8.0-1.fc20'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
Will, how exactly can I use fedup --iso with the latest TC5 DVD image, and still use the latest fedup-dracut? I'm not sure how to test this. Is there a way how to test this faster than wait for TC6? Thanks.
The simplest way is to set up a F20 system and run 'makefeduprepo':
sudo /usr/share/doc/fedup-dracut/makefeduprepo ~/instrepo
( cd ~/instrepo; ./serve )
This gives you a working --instrepo at http://[IP]:8000/.
Now (in theory) on the F19 system you can run:
fedup --iso ISO_IMAGE --instrepo http://[F20_IP]:8000/
Unfortunately there's a small bug in fedup-0.8.0 where it will use the boot images from the ISO instead of instrepo. This is fixed in git:
So you can apply that patch, or just run fedup without --instrepo and manually replace /boot/vmlinuz-fedup and /boot/initramfs-fedup.img with vmlinuz and upgrade.img.
I installed fedup-dracut-0.8.0-1.fc20 on F20 machine and created the repo. Then I ran fedup --iso on F19 machine, and overwrote /boot/vmlinuz-fedup and /boot/initramfs-fedup.img with the generated files. On F19 I used fedup-0.7.3-4.fc19, because that's the latest version available.
First, I had the ISO file on a partition that is not mounted at boot. After reboot, fedup flashed the screen, I saw some systemd messages and then the machine rebooted and I was back in grub - everything happened under 10 seconds. The "System Upgrade" boot option was still present, wasn't removed. I tried it one more time, same result. I saw no obvious error message, but it's scrolling too fast. It would be a good idea to pause for a minute after fedup ends (regardless whether it's successful or not).
Then I booted F19 system again, this time I mounted the partition permanently (created entry in fstab, verified after reboot). I ran fedup --iso again, again overwrote the /boot files. The same thing happened - a very short boot, some systemd messages, reboot under 10 seconds. The "System Upgrade" menu option remained present.
Also, the System Upgrade menu option was selected _by default_, therefore if somebody does this on a remote machine, it will get stuck in an infinite reboot cycle.
There was no /var/log/upgrade.log file created, attaching at least /var/log/fedup.log.
Created attachment 834730 [details]
fedup.log for comment 24
looks like will made a newer build:
i'll see if I can reproduce the problem and verify the fix soon.
Looks good here. I was able to reproduce the initial breakage by using fedup from updates-testing and upgrade.img from the mirrors (i.e. just running it 'as normal', as the reporter did), then reproduce the breakage kparal saw when using a fixed kernel/initramfs built by fedup-dracut 0.8.0-1 but still the older fedup, and finally got a successful upgrade by using both the fixed kernel/initramfs and fedup 0.8.0-3.
Yes, it works for me as well with fedup-0.8.0-3.fc19.
Will, please create an update.
Also note that there's a newer fedup-dracut that fixes the breakage when using 0.7.x fedup with 0.8.x fedup-dracut:
Updates will happen shortly.
Here's the update:
Tested it with F19, fedup updated from bodhi, and it works OK.
fedup-dracut-0.8.0-2.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 1044128 has been marked as a duplicate of this bug. ***