Bug 1056363 - Fedup 18-19/20 doesn't begin upgrade on reboot
Summary: Fedup 18-19/20 doesn't begin upgrade on reboot
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup
Version: 18
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1056362 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-22 04:06 UTC by Lucas
Modified: 2014-02-05 22:37 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2014-02-05 22:37:52 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
fstab (574 bytes, text/plain)
2014-01-22 04:08 UTC, Lucas
no flags Details
fedup.log (1.06 MB, text/plain)
2014-01-22 04:08 UTC, Lucas
no flags Details
new fedup.log (1.29 MB, text/plain)
2014-01-25 00:51 UTC, Lucas
no flags Details
no-upgrade.log (142.92 KB, text/x-vhdl)
2014-01-27 15:59 UTC, Lucas
no flags Details

Description Lucas 2014-01-22 04:06:48 UTC
Description of problem:
Trying to upgrade to Fedora 19 or 20 from Fedora 18 fails.  After running Fedup and rebooting the computer, Fedora 18 still loads. However, uname -a shows it running the latest fc20 kernel (impossible?).

Version-Release number of selected component (if applicable):
fedup 0.8.0

How reproducible:

Every time on this particular system. Haven't run on any other versions of Fedora 18.

Steps to Reproduce:
1. Run fedup --network 20
2. Reboot
3. Profit?

Actual results:
Fedora not upgrading


Expected results:
Fedora upgrading

Additional info:

lsblk:

NAME                              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                 8:0    0 149.1G  0 disk
ââsda1                              8:1    0   500M  0 part  /boot
ââsda2                              8:2    0 148.6G  0 part
ââfedora_homeserver-swap (dm-0) 253:0    0   3.8G  0 lvm   [SWAP]
ââfedora_homeserver-root (dm-1) 253:1    0    50G  0 lvm   /
ââfedora_homeserver-home (dm-2) 253:2    0  94.8G  0 lvm   /home

Comment 1 Lucas 2014-01-22 04:08:14 UTC
Created attachment 853581 [details]
fstab

fstab file

Comment 2 Lucas 2014-01-22 04:08:55 UTC
Created attachment 853582 [details]
fedup.log

Comment 3 Will Woods 2014-01-22 23:13:02 UTC
*** Bug 1056362 has been marked as a duplicate of this bug. ***

Comment 4 Will Woods 2014-01-22 23:24:58 UTC
Could you attach the output of:

ls -ld /system-upgrade /system-upgrade-root /run/system-upgrade
realpath /var/lib/system-upgrade /var/tmp/system-upgrade
rpm -q systemd

Comment 5 Lucas 2014-01-23 01:17:25 UTC
ls -ld /system-upgrade /system-upgrade-root /run/system-upgrade:

ls: cannot access /run/system-upgrade: No such file or directory
lrwxrwxrwx. 1 root root   25 Jan 21 17:34 /system-upgrade -> /share/lib/system-upgrade
drwxr-xr-x. 2 root root 4096 Jan 21 17:17 /system-upgrade-root


realpath /var/lib/system-upgrade /var/tmp/system-upgrade:

/var/lib/system-upgrade
/var/tmp/system-upgrade


rpm -q systemd
:
systemd-201-2.fc18.9.x86_64

Comment 6 Will Woods 2014-01-23 19:43:27 UTC
(In reply to Lucas from comment #5)
> /system-upgrade -> > /share/lib/system-upgrade

What's that about? Did you modify fedup to change the cachedir? Where's /share located?

Comment 7 Lucas 2014-01-23 20:30:58 UTC
I was trying the solution here: 
https://bugzilla.redhat.com/show_bug.cgi?id=1045168#c37

I've sinced cleaned fedup and reset that file back to the proper, /var/lib/system-upgrade

Neither way worked.

Comment 8 Brian Morrison 2014-01-24 12:08:12 UTC
You might find this helps you more:

https://bugzilla.redhat.com/show_bug.cgi?id=1045168#c45

Comment 9 Lucas 2014-01-24 14:35:45 UTC
Unfortunately,  I tried that before I posted this bug. That solution doesn't work for me.

Comment 10 Brian Morrison 2014-01-24 15:32:43 UTC
I had the problem that that link in comment 8 fixes on an F19->F20 upgrade.

Looks like it's different in the F18->F20 case. There have been a lot of changes to all this, it's hardly surprising that these problems appear.

Comment 11 Will Woods 2014-01-24 18:41:20 UTC
Yes. It depends heavily on the version of upgrade.img being used - i.e. what version you're upgrading *to*.

F18→F19 upgrades will fail to start unless you add "systemd.unit=system-upgrade.target" to the boot arguments. That's bug 1054988.

F18→F20 upgrades should generally be fine, except for other problems (that AFAICT aren't F18-specific).

Comment 12 Will Woods 2014-01-24 23:19:47 UTC
I just set up an F18 system with the same partition layout as reported above, and upgraded it to F20 using fedup. Everything worked fine.

I have to conclude that something about your system got messed up when you were modifying things to work around the other bugs.

Please try:

  yum reinstall fedup
  rm -rf /system-upgrade /system-upgrade-root
  mv /var/log/fedup.log /var/log/fedup.log.old

and re-run 'fedup --network 20'. If it fails, attach the new fedup.log.

Comment 13 Lucas 2014-01-25 00:50:55 UTC
Still not working. attaching new fedup.log

Comment 14 Lucas 2014-01-25 00:51:38 UTC
Created attachment 855257 [details]
new fedup.log

Comment 15 Lucas 2014-01-25 00:58:55 UTC
am I supposed to have a /run/system-upgrade file or directory? Because I don't even have a /run/ directory.

This is the beginning of my messages file when I boot the computer up to the system upgrade grub2 entry:


Jan 24 07:59:58 Homeserver rsyslogd: [origin software="rsyslogd" swVersion="7.2.6" x-pid="865" x-info="http://www.rsyslog.com"] start
Jan 24 07:59:58 Homeserver rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w"
Jan 24 07:59:58 Homeserver rpcbind: cannot open file = /var/lib/rpcbind/rpcbind.xdr for writing
Jan 24 07:59:58 Homeserver rpcbind: cannot save any registration
Jan 24 07:59:58 Homeserver kernel: [    0.000000] Initializing cgroup subsys cpuset
Jan 24 07:59:58 Homeserver kernel: [    0.000000] Initializing cgroup subsys cpu
Jan 24 07:59:58 Homeserver kernel: [    0.000000] Initializing cgroup subsys cpuacct
Jan 24 07:59:58 Homeserver rpcbind: cannot open file = /var/lib/rpcbind/portmap.xdr for writing
Jan 24 07:59:58 Homeserver rpcbind: cannot save any registration
Jan 24 07:59:58 Homeserver systemd-cgroups-agent[566]: Failed to get D-Bus connection: Failed to connect to socket /run/systemd/private: No such file or directory
Jan 24 07:59:58 Homeserver systemd-cgroups-agent[567]: Failed to get D-Bus connection: Failed to connect to socket /run/systemd/private: No such file or directory

Comment 16 Will Woods 2014-01-27 15:20:38 UTC
/run is a tmpfs that's created at boot time and shared between initrd and the normal system. It's been present in every Fedora system since F15.

If you're missing /run, something very weird (and unrelated to fedup) is happening on your system.

The log messages there aren't relevant to the upgrade failure. Could you boot the 'System Upgrade' entry, and once you can log in, do:

  journalctl -b > no-upgrade.log
  mount >> no-upgrade.log
  ls -l / >> no-upgrade.log

and attach that?

Comment 17 Lucas 2014-01-27 15:58:38 UTC
Okay, it's attached, but I now have a /run directory. There is still no system-upgrade file or folder in it though.  I don't have any idea what's going on...

Comment 18 Lucas 2014-01-27 15:59:12 UTC
Created attachment 856130 [details]
no-upgrade.log

Comment 19 Will Woods 2014-01-27 18:10:52 UTC
Strange. That log shows your system is running the F20 kernel and upgrade.img, but your system never seems to start the upgrade.

What is *supposed* to happen is this:

0) You boot the "System Upgrade" entry, with the special "upgrade.img" initrd.
1) The upgrade.img copies /sysroot/system-upgrade to /run/system-upgrade
   (plus some other stuff that's not relevant)
2) Your system starts normally
3) /lib/systemd/system-generators/system-upgrade-generator runs. If:
   a) /system-upgrade is a valid link to a directory, and 
   b) /system-upgrade-root is a directory, and
   c) /run/system-upgrade is a symbolic link
   then the default target is switched to "system-upgrade.target".

For some reason, your system isn't switching the target. So we need to figure out why.

On systems with a separate /var, 3a) fails because /var hasn't been mounted and therefore the symbolic link isn't valid yet. But that's not what's happening here, as far as I can tell. So one of the other steps is going wrong.

First, can you make sure system-upgrade-generator is intact? Run:

  rpm -q fedup && rpm -V fedup

The output should be just one line:

  fedup-0.8.0-3.fc18.noarch

Assuming that works, could you try booting the 'System Upgrade' entry again, but add 'rd.break' to the boot args? This will make upgrade.img drop into a shell immediately before it starts the real system.

At that point, please check to make sure that /run exists, /run/system-upgrade exists, and that /run/system-upgrade is a copy of the normal /system-upgrade.

You can do this like so:

  ls -l /sysroot/system-upgrade /run/system-upgrade

You can exit that shell and your system will continue booting. (Or just 'reboot' will probably reboot your system if you don't want to bother with the upgrade failing again)

Sorry for the trouble, but I want to figure this out - I can't reproduce the problem myself and nobody seems to have any idea what's special about their systems that makes them behave differently than everyone else's...

Comment 20 Lucas 2014-01-27 20:12:26 UTC
No problem on the trouble. It's annoying, but I understand how difficult troubleshooting is. I'm sorry that I can't provide more info on how my system differs from other systems. It's been a while but I'm pretty sure it was just a vanilla Fedora 18 installation. I don't normally do anything special when I install any Linux system.

Anyway, fedup-0.8.0-3.fc18.noarch is the only fedup package installed.

When I reboot to the emergency shell in the upgrade.img, both of those files are symlinks that point to /var/lib/system-upgrade

/run/ does exist, I have no idea what I was on when I said that it didn't, but I was probably incorrect.

Is there some way of editing system-upgrade-generator in order to make it log something? Or pause on an error?

Comment 21 Will Woods 2014-01-27 22:39:57 UTC
Generators run *very* early - before proper logging is available - but you could make it log to a file pretty easily.

Add this to the top of system-upgrade-generator (after the comments on the top three lines):

  exec &>/tmp/system-upgrade-generator.log; set -x

Boot the 'System Upgrade' menu item again, then save a copy of /tmp/system-upgrade-generator.log somewhere so you can attach it here.

Basically, there's three tests:

  # does the symlink point to a dir?
  [ -L /system-upgrade -a -d /system-upgrade ] 

  # is /system-upgrade-root a dir?
  [ -d /system-upgrade-root ]    

  # did we boot upgrade.img?
  [ -L /run/system-upgrade -o -f /run/initramfs/upgrade.conf ]

If any of those tests fail, the log will immediately end. If they all pass, the last log line will be:
  
  ln -sf <PATH>/system-upgrade.target <OTHERPATH>/default.target

So.. how far does it get?

Comment 22 Lucas 2014-01-28 15:59:56 UTC
Okay, so  I couldn't get that log file to show up anywhere. I don't know where it was going.  I did get it to upgrade, however.  It seems my issue was the following:

I needed to do as in https://bugzilla.redhat.com/show_bug.cgi?id=1045168#c45

but I needed to change the path to /usr/lib/... as opposed to /lib/...

that by itself didn't work so I needed to add "systemd.unit=system-upgrade.target" to the boot arguments as you suggested in a previous comment.  

So it was a combination of different bugs but I had to rework the solution a little.  

Still don't know about the log file though.

Thanks for your help, I have no idea how you would fix this or if it is even really a bug.

Comment 23 Will Woods 2014-01-28 18:13:49 UTC
(In reply to Lucas from comment #22)
> Okay, so  I couldn't get that log file to show up anywhere. I don't know
> where it was going.

If you couldn't get the log file to appear, that suggests that the generator was never being run.. which doesn't make sense, because you have the right versions of systemd and fedup, but:

> I needed to add "systemd.unit=system-upgrade.target" to the boot arguments
> as you suggested in a previous comment.  

That short-circuits the problem by forcing systemd to use "system-upgrade.target" as the default unit, rather than letting the generator do the switch. So that doesn't help me understand the problem any, sadly.

> I needed to do as in https://bugzilla.redhat.com/show_bug.cgi?id=1045168#c45
> 
> but I needed to change the path to /usr/lib/... as opposed to /lib/...

...wait, do you not have /lib as a symlink to usr/lib? What's "ls -l /" say?

Comment 24 Lucas 2014-01-28 19:42:52 UTC
Well, it says that I do now, but I'm wondering if I didn't before.

lrwxrwxrwx.   1 root    root        7 Jan 28 07:41 lib -> usr/lib
lrwxrwxrwx.   1 root    root        9 Jan 28 07:41 lib64 -> usr/lib64


weird.

Comment 25 Fedora End Of Life 2014-02-05 22:37:52 UTC
Fedora 18 changed to end-of-life (EOL) status on 2014-01-14. Fedora 18 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.