Bug 980543 - Fedup progress bar "stuck" at ~60 % (18->19)
Fedup progress bar "stuck" at ~60 % (18->19)
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: fedup-dracut (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
:
: 896057 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-02 12:25 EDT by Jan "Yenya" Kasprzak
Modified: 2013-12-23 22:55 EST (History)
6 users (show)

See Also:
Fixed In Version: fedup-dracut-0.8.0-2.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-13 00:35:44 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jan "Yenya" Kasprzak 2013-07-02 12:25:22 EDT
Description of problem:
When upgrading my F18 laptop with encrypted root fs, the upgrade proces is stuck - the progress bar under the blinking Fedora logo is at about 60 %.

Version-Release number of selected component (if applicable):
Fedora 18 (installed)

How reproducible:

Steps to Reproduce:
1. fedup-cli --network 19
2. reboot, select "fedup" from the grub2 menu
3. enter the luks passphrase when prompted for it.

Actual results:
the progress bar is stuck at about 60 %, no apparent activity (HDD LED) for at least 15 minutes.

Expected results:
Fedora 19, of course :-)

Additional info:
After pressing Escape, the contents of the screen is the following (rewriting manually, sorry for possible typos):

       Expecting device dev-mapper-luks\x2...
       Starting Cryptography Setup for luks-...
[ OK ] Started Show Plymouth Boot Screen.
       Starting Forward Password Requests to Plymouth...
[ OK ] Started Forward Password Requestss to Plymouth.
[ OK ] Reached target Paths.
dracut-intiqueue[221]: Failed to issue method call: Access denied
[ OK ] Started cryptography setup for luks-...
[ OK ] Reached target Encrypted Volumes.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ OK ] Found device /dev/mapper/luks-...
[ OK ] Started dracut initqueue hook.
       Starting dracut pre-mount hook...
[ OK ] Started dracut pre-mount hook.
       Mounting /sysroot...
[ OK ] Mounted /sysroot.
[ OK ] Reached target Initrd Root File System.
       Starting Reload Configuration from the Real Root...
[ OK ] Started Reload Configuration from the Real Root.
[ OK ] Reached target Initrd File systems.
[ OK ] Reached Target Initrd Default Target.

Welcome to Fedora 18 (Spherical Cow)!

(and then once again the same block of text text ending with the "Welcome to Fedora 18" line)

I have power-cycled the computer (Ctrl-alt-del was not working), rebooted back to F18, logged in as root, re-ran "fedup-cli --network 19" (this time, no further packages have been downloaded), and rebooted to the fedup GRUB2 entry. The situation was the same, except this time, the progress bar was stuck at about 30 %. The messages after pressing escape were the same.
Comment 1 Jan "Yenya" Kasprzak 2013-07-04 05:50:31 EDT
After upgrading fedup and the whole F18 system today, the upgrade process continued, but the upgraded system did not boot correctly - after pressing Escape on the "fedora logo with progressbar" screen, it displayed boot messages ending with "Reached Target Shutdown" at the bottom. Selecting "Rescue Fedora 19" entry from the GRUB menu leads to automatic reboot after half a minute or so. I have edited the parameters of the boot entry to get a shell, ran "grub2-mkconfig -o /boot/grub2/grub.cfg", and the new grub2 config works.

So there is probably another problem with fedup creating an incorrect grub2.cfg file.
Comment 2 Joe Zeff 2013-07-31 02:57:23 EDT
I had a similar issue going from 17 to 19.  The upgrade hung, forcing me to use the reset button.  Grub had a kernel for F 19, but it didn't boot; using the F17 kernel got me to a login screen, but neither the mouse nor the keyboard worked.  (A separate issue that I'm still working on.)  I was able to edit the grub entry for the F17 kernel to boot to a CLI, where I found over 1,000 duplicate packages and yum thought it was still working on F 17, even though all of the files were right.  (Again, a side-issue, mentioned just for completeness.)  Almost 24 hours later, I'm still climbing out from under with only a CLI and am posting this from my laptop.  More information available if needed.
Comment 3 Will Woods 2013-07-31 12:41:45 EDT
Thus far every "hang" during upgrade has turned out, on further investigation, to be a long-running package scriptlet and a premature reset.

Usually this happens while the selinux-policy package(s) are being installed. Those packages %post scripts take a minimum of 3 minutes and sometimes upwards of 10 minutes, during which time there's no progress being reported back to fedup. See bug 972358 for details.

Seriously, though: as long as the Fedora logo is still blinking, your system hasn't hung. DON'T RESET YOUR COMPUTER.

If your system actually locks up, the logo will stop blinking.
If the upgrade crashes, it will reboot the system.
If you think the upgrade is stuck, *wait at least 30 minutes*. Go for a walk or something, maybe?

I think I'm going to need to put a shell on VT2 so power-users don't get antsy when they can't see whether or not the upgrade is running..
Comment 4 Joe Zeff 2013-07-31 14:45:03 EDT
My desktop has nVidia graphics; I didn't get a Fedora logo.  And, at this point, what can I do about it?  There are dupes that I can't remove because dependency resolution loops over something that can't, currently, be removed and I can't even get into a GUI.  And, please note, there wasn't any disk activity when this happened.
Comment 5 Jan "Yenya" Kasprzak 2013-08-06 10:28:10 EDT
Re: comment #3

The progress bar stuck for >15 minutes (as it was in my case) provides no visual feedback, even though the progress bar is blinking. There should be at least one line of text saying what is going on. If anaconda/fedup devs want to stick with their aesthetics criteria, the line of text can for example be hidden by default and displayed only when no progress (as measured by the progress bar) is made in the last minute or so. Alternatively, pressing escape should display more complete info (the last few steps of the upgrade process executed, maybe ps axf).

A virtual console with the shell prompt would of course be good as well.
Comment 6 Joe Zeff 2013-08-09 15:39:00 EDT
Back in the old days of MS-DOS, or even CP/M, there was a common way to show progress on a text-only screen: at fairly frequent intervals, the program would output something like this:

| (pause) ^H/ (pause) ^H- (pause) ^H\ (pause) - (pause) ^H then repeat the sequence over and over.  As ^H was used as a backspace, the effect was a spinning propeller that showed you that the program was still working, because if it were stuck, the spin would pause, and if it were completely hung, it would simply stop.  Maybe something like this could be added to be shown only if you used Escape to look at the output messages, or along with the progress bar that I get because I don't get the Fedora logo on the computer with nVidia.  And, it's not so much a lack of screen output, but that combined with no sign of disk IO so that there's no way of telling if the box is working or hung.
Comment 7 Anne Mahoney 2013-08-15 09:44:15 EDT
I'm having a similar problem going from 17 to 19.  The first "fedup --network" ran fine.  I re-booted and chose "system upgrade" from the menu.  The logo *was* there blinking happily for half an hour, but I saw no disk activity lights on the computer, and when I pressed ESC to see the log, the last two messages were:

Reached target Initrd File Systems
Reached target Initrd Default Target

-- and nothing more happened, for a full half hour, neither on screen nor on the front panel lights.  It's a Dell Optplex desktop, if that's relevant.
Comment 8 Will Woods 2013-08-16 14:52:20 EDT
(In reply to Anne Mahoney from comment #7)
> I'm having a similar problem going from 17 to 19.  The first "fedup
> --network" ran fine.  I re-booted and chose "system upgrade" from the menu. 
> The logo *was* there blinking happily for half an hour, but I saw no disk
> activity lights on the computer, and when I pressed ESC to see the log, the
> last two messages were:
> 
> Reached target Initrd File Systems
> Reached target Initrd Default Target
> 
> -- and nothing more happened, for a full half hour, neither on screen nor on
> the front panel lights.  It's a Dell Optplex desktop, if that's relevant.

That's a different bug - I'm assuming the progress bar under the logo was stuck at something like 0% or 1%, which usually means your system's root device couldn't be found/mounted. (Although usually that's supposed to result in a dracut emergency shell after a minute, so I'm not sure why yours hung for half an hour...)

Anyway - in the case where the upgrade hasn't actually started, it's safe(ish) to reset the system, since none of the disks are mounted anyway. 

But once the upgrade is underway, SERIOUSLY DO NOT RESET YOUR SYSTEM. The upgrade can take *hours*, and certain scripts can take a *really* long time to finish. 

The next fedup release will have a debug shell available on tty2 so you can check to see if things are actually still running.

Until then, you can add 'rd.upgrade.debugshell' to the boot arguments, which will put a debug shell on tty2. Note that *this* debug shell doesn't restart when systemd switches root devices, so all the commands will seem to be missing; exit the shell and it will restart in your current root device, and then ls/ps/etc. will work.
Comment 9 Will Woods 2013-10-09 18:22:31 EDT
*** Bug 896057 has been marked as a duplicate of this bug. ***
Comment 10 Will Woods 2013-12-06 16:32:22 EST
fedup-dracut-0.8.0 should provide a shell on tty2 for the duration of the upgrade so you can check to see what's going on. The "hang" itself is discussed in bug 972358.

I don't think there's anything else I can do about this, so I'm considering the shell on tty2 the "fix".
Comment 11 Fedora Update System 2013-12-06 17:19:12 EST
fedup-dracut-0.8.0-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/fedup-dracut-0.8.0-1.fc20
Comment 12 Fedora Update System 2013-12-07 13:46:31 EST
Package fedup-dracut-0.8.0-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-dracut-0.8.0-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-22956/fedup-dracut-0.8.0-1.fc20
then log in and leave karma (feedback).
Comment 13 Fedora Update System 2013-12-12 11:31:56 EST
Package fedup-dracut-0.8.0-2.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-dracut-0.8.0-2.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-22956/fedup-dracut-0.8.0-2.fc20
then log in and leave karma (feedback).
Comment 14 Fedora Update System 2013-12-13 00:35:44 EST
fedup-dracut-0.8.0-2.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.