Bug 1024223 - fedup 19->20 failed with DVD iso upgrade
Summary: fedup 19->20 failed with DVD iso upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup-dracut
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1044128 (view as bug list)
Depends On:
Blocks: F20FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2013-10-29 07:28 UTC by Peng Wu
Modified: 2014-01-23 20:01 UTC (History)
13 users (show)

Fixed In Version: fedup-dracut-0.8.0-2.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-13 05:35:56 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
fedup log file (598.75 KB, text/x-log)
2013-10-29 07:37 UTC, Peng Wu
no flags Details
fedup.log for comment 24 (1.02 MB, text/plain)
2013-12-10 12:38 UTC, Kamil Páral
no flags Details

Description Peng Wu 2013-10-29 07:28:10 UTC
Description of problem:
I tried to upgrade one desktop from f19 to f20, by using:
#fedup --iso Fedora-20-Beta-TC6-x86_64-DVD.iso

But it failed, from the screen boot messages, it seems it can't find any rpm packages.
Or the iso media can't be not mounted?


Version-Release number of selected component (if applicable):
fedup-0.7.3-4.fc19.noarch

How reproducible:
Use fedup --iso to upgrade the f19 machine, then the error messages are printed on the screen after rebooted and chose "System Upgrade (fedup)".

Steps to Reproduce:
1. Prepare a f19 machine
2. fedup --iso Fedora-20-Beta-TC6-x86_64-DVD.iso
3. reboot

Actual results:
Reboot, but system are not upgraded.

Expected results:
System are upgraded.

Additional info:

Comment 1 Peng Wu 2013-10-29 07:37:02 UTC
Created attachment 816993 [details]
fedup log file

Comment 2 igor.redhat@gmail.com 2013-10-30 22:26:33 UTC
I can confirm this as failing with both the beta DVD mentioned above and the official Alpha DVD iso. There was an error during boot about failing to mount install media (?) and a whole lot of quickly scrolling errors after the boot (seemingly about not finding individual packages but they scrolled too fast to read). Neither of this errors seem to show up in either system or fedup logs.

Comment 3 Matthias 2013-11-22 03:40:32 UTC
I see the same problem. I tried the latest Fedora-20-TC2-x86_64-DVD.iso. The update fails as described above.

Comment 4 Will Woods 2013-11-22 23:26:50 UTC
This log message is strange:

  fedup.sysprep:setup_media_mount() setting up mount for /dev/loop0 at /system-upgrade/media

Obviously, fedup needs to be able to find the .iso file after it reboots, so using --iso will fail if the image is somewhere that isn't automatically mounted on boot (like a USB stick). And "/dev/loop0" doesn't usually exist..

Where is your .iso file located?

Comment 5 Matthias 2013-11-23 00:44:23 UTC
I don't know where the user who uploaded the log kept the ISO, but in my case it was in the home folder of my standard account. That should be available to fefup.

Comment 6 Lucas Maneos 2013-11-23 14:18:52 UTC
Same here.  The system eventually prompted for the root password for maintenance, after which I can see the .iso just fine where it was expected to be, but cannot mount it.

Running the mount command manually under strace shows it get ENOENT for /dev/loop0.  After mknoding that, it fails with ENXIO (No such device or address).  There's no loop.ko anywhere under /lib/modules/3.11.6-301.fc20.x86_64 and no mention of it in dmesg.

It appears the kernel (3.11.6-301.fc20.x86_64) lacks support for loopback mounts, either built-in or as a module.

Comment 7 Peng Wu 2013-11-25 02:28:16 UTC
IIRC, I put the DVD iso under /root/ directory.

Comment 8 Matthias 2013-11-29 00:51:10 UTC
No improvement with Fedora-20-TC3-x86_64-DVD.iso . I am still unable to upgrade using the --iso option.

Comment 9 Kamil Páral 2013-12-02 09:55:08 UTC
I can confirm this. I added a new disk to my VM (/dev/vdb), copied TC3 DVD.iso onto it and ran fedup --iso. Then rebooted, and there was a quick flash of thousands of "can't find" messages, then it auto-rebooted again. The upgrade.log is empty, so I can' provide any details (but that probably has a different cause - bug 1016522 - and it has been reported for a long time).

Comment 10 Adam Williamson 2013-12-02 17:59:36 UTC
Lucas: "It appears the kernel (3.11.6-301.fc20.x86_64) lacks support for loopback mounts, either built-in or as a module."

That's certainly not the problem. I do loopback mounts from a booted system all the time - 'mount -o loop some.iso /mnt/temp'. Works fine.

Comment 11 Adam Williamson 2013-12-02 18:13:44 UTC
Discussed at 2013-12-02 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-12-02/f20-blocker-review-%234.2013-12-02-17.02.log.txt . Accepted as a blocker per Beta criterion "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed" - https://fedoraproject.org/wiki/Fedora_20_Beta_Release_Criteria#Upgrade_requirements .

Note that this is somewhat debatable; the criteria do not expressly define whether *all* fedup mechanisms are required to work, or just *at least one*. We decided to try and maintain some standards and go with the former interpretation, though, especially since the 'repo' method more or less requires an internet connection; the ISO method would likely be the method anyone without an internet connection would expect to use.

Comment 12 Matthias 2013-12-03 01:34:16 UTC
1st thank you for making this a blocker and trying to fix the issue. I read the meeting protocol and would like to give some feedback regarding that discussion.

1. The --iso option worked without problems for upgrades from F18 to F19.

2. The --iso upgrade is very easy and hassle free. Copy iso file to harddisk, execute command, done. Compare this to all the steps listed in the wiki for an yum upgrade (http://fedoraproject.org/wiki/Upgrading_Fedora_using_yum). Please consider less experienced users without internet connection.

3. And last but least a real world use case: I maintain my parents in law computer. They live in rural Malaysia. The internet there is not reliable and too slow to execute fedup --network successfully. So when I visit I bring the iso file and the rpm's from the updates repo for the new version. Within an hour they are running the latest version of Fedora without the need to connect to the internet at all.

Comment 13 Adam Williamson 2013-12-03 01:52:47 UTC
Matthias: thanks for the notes. The appropriate comparison for point #2 isn't a yum upgrade, though, but fedup's other mode (fetching from a mirror), which is just as easy, but obviously only works if you have a network connection or a local repository. The implied question at the meeting was basically "could we get away with shipping if only fedup's repo mode worked and not its iso mode".

Comment 14 Matthias 2013-12-03 02:22:07 UTC
The local repo could work if the --iso option can't be fixed in time. 

- Fedup wiki should then be updated to spell out that --iso does not work for F19->F20.

- The wiki should then also come with a step by step explanation where and what to download, how to setup the local repo and how to run the fedup command.

- One potential risk for your mirror load if you don't spell out what to download (I use F19 as example, guess 20 will be comparable): 
fedora/releases/19/Fedora/x86_64/os/Packages/ has less than 4GB but
fedora/releases/19/Everything/x86_64/os/Packages/ has approx 35GB of packages.

Comment 15 Will Woods 2013-12-03 18:38:22 UTC
In my testing it works from an actual DVD (well OK, a virtual DVD, but still /dev/sr0 and not an on-disk .iso), so the problem is probably either:

a) you have the .iso on a filesystem that doesn't get mounted at boot time
  (this should be detected by the next fedup version; see
   https://github.com/wgwoods/fedup/commit/33c8612)

b) fedup doesn't set up the .mount unit properly
  (this used to work fine; I'm testing it now)

In the meantime: as a workaround, you could probably dd the .iso to a USB stick and use "fedup --device" instead.

Comment 16 Will Woods 2013-12-03 22:06:02 UTC
Ah. So the problem is that the "loop" driver is now a module, and nothing in dracut explicitly requires it, so it's not in upgrade.img/initrd-fedup.img, so we can't loop-mount anything.

This seems pretty goofy. We've got tcm_loop.ko, so you can make a block device into an iSCSI target, but no loop.ko.

Given that loop.ko is ~32kb uncompressed, I think it's safe/reasonable to include it by default in dracut. Failing that, fedup-dracut (and a few other dracut modules) will need to be modified to do "instmods loop". Which might not be a bad idea either way.

Fixing this will therefore require a new build of fedup-dracut (and/or dracut, depending on how Harald & co. want to fix things), and then new installer images.

Comment 17 Adam Williamson 2013-12-03 22:42:52 UTC
Will is aware, but just in case, Harald - go/no-go is Thursday; we have a small chance of making it if everything's fixed early (US time) tomorrow, but anything after that is a slip. so it'd be much appreciated if you could agree on an approach and get something in Bodhi ASAP. thanks.

Comment 18 Will Woods 2013-12-04 00:40:59 UTC
Here's the fix I'm testing. My upgrade is running, so I consider that success:

  https://github.com/wgwoods/fedup-dracut/commit/d723d40

I'll try to put together a new build of fedup and fedup-dracut ASAP.

Comment 19 Adam Williamson 2013-12-05 21:59:42 UTC
wwoods: any news on those builds? I don't see any in koji/bodhi. thanks!

Comment 20 Fedora Update System 2013-12-06 22:19:17 UTC
fedup-dracut-0.8.0-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/fedup-dracut-0.8.0-1.fc20

Comment 21 Fedora Update System 2013-12-07 18:46:37 UTC
Package fedup-dracut-0.8.0-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-dracut-0.8.0-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-22956/fedup-dracut-0.8.0-1.fc20
then log in and leave karma (feedback).

Comment 22 Kamil Páral 2013-12-09 13:48:27 UTC
Will, how exactly can I use fedup --iso with the latest TC5 DVD image, and still use the latest fedup-dracut? I'm not sure how to test this. Is there a way how to test this faster than wait for TC6? Thanks.

Comment 23 Will Woods 2013-12-09 16:09:38 UTC
The simplest way is to set up a F20 system and run 'makefeduprepo':

  sudo /usr/share/doc/fedup-dracut/makefeduprepo ~/instrepo
  ( cd ~/instrepo; ./serve )

This gives you a working --instrepo at http://[IP]:8000/.

Now (in theory) on the F19 system you can run:

  fedup --iso ISO_IMAGE --instrepo http://[F20_IP]:8000/

Unfortunately there's a small bug in fedup-0.8.0 where it will use the boot images from the ISO instead of instrepo. This is fixed in git:

  https://github.com/wgwoods/fedup/commit/26291b1

So you can apply that patch, or just run fedup without --instrepo and manually replace /boot/vmlinuz-fedup and /boot/initramfs-fedup.img with vmlinuz and upgrade.img.

Comment 24 Kamil Páral 2013-12-10 12:37:46 UTC
I installed fedup-dracut-0.8.0-1.fc20 on F20 machine and created the repo. Then I ran fedup --iso on F19 machine, and overwrote /boot/vmlinuz-fedup and /boot/initramfs-fedup.img with the generated files. On F19 I used fedup-0.7.3-4.fc19, because that's the latest version available.

First, I had the ISO file on a partition that is not mounted at boot. After reboot, fedup flashed the screen, I saw some systemd messages and then the machine rebooted and I was back in grub - everything happened under 10 seconds. The "System Upgrade" boot option was still present, wasn't removed. I tried it one more time, same result. I saw no obvious error message, but it's scrolling too fast. It would be a good idea to pause for a minute after fedup ends (regardless whether it's successful or not).

Then I booted F19 system again, this time I mounted the partition permanently (created entry in fstab, verified after reboot). I ran fedup --iso again, again overwrote the /boot files. The same thing happened - a very short boot, some systemd messages, reboot under 10 seconds. The "System Upgrade" menu option remained present.

Also, the System Upgrade menu option was selected _by default_, therefore if somebody does this on a remote machine, it will get stuck in an infinite reboot cycle.

There was no /var/log/upgrade.log file created, attaching at least /var/log/fedup.log.

Comment 25 Kamil Páral 2013-12-10 12:38:16 UTC
Created attachment 834730 [details]
fedup.log for comment 24

Comment 27 Adam Williamson 2013-12-11 00:13:32 UTC
looks like will made a newer build:

http://koji.fedoraproject.org/koji/buildinfo?buildID=484014

i'll see if I can reproduce the problem and verify the fix soon.

Comment 28 Adam Williamson 2013-12-11 01:55:19 UTC
Looks good here. I was able to reproduce the initial breakage by using fedup from updates-testing and upgrade.img from the mirrors (i.e. just running it 'as normal', as the reporter did), then reproduce the breakage kparal saw when using a fixed kernel/initramfs built by fedup-dracut 0.8.0-1 but still the older fedup, and finally got a successful upgrade by using both the fixed kernel/initramfs and fedup 0.8.0-3.

Comment 29 Kamil Páral 2013-12-11 13:42:48 UTC
Yes, it works for me as well with fedup-0.8.0-3.fc19.

Will, please create an update.

Comment 30 Will Woods 2013-12-11 21:45:19 UTC
Also note that there's a newer fedup-dracut that fixes the breakage when using 0.7.x fedup with 0.8.x fedup-dracut:

  http://koji.fedoraproject.org/koji/buildinfo?buildID=484003

Updates will happen shortly.

Comment 31 Kamil Páral 2013-12-12 08:41:07 UTC
Here's the update:
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc19

Comment 32 Jan Sedlák 2013-12-12 12:31:15 UTC
Tested it with F19, fedup updated from bodhi, and it works OK.

Comment 33 Fedora Update System 2013-12-13 05:35:56 UTC
fedup-dracut-0.8.0-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 34 Will Woods 2014-01-23 20:01:01 UTC
*** Bug 1044128 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.