Bug 1023618 - System Upgrade using FedUp Fails After Reboot on Xen VM
System Upgrade using FedUp Fails After Reboot on Xen VM
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: fedup (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-26 01:32 EDT by wdouglascampbell
Modified: 2013-12-22 00:35 EST (History)
3 users (show)

See Also:
Fixed In Version: fedup-0.8.0-3.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 02:22:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/fedup.log (2.01 MB, text/plain)
2013-10-28 10:26 EDT, wdouglascampbell
no flags Details

  None (edit)
Description wdouglascampbell 2013-10-26 01:32:09 EDT
After running fedup-cli to attempt upgrade from Fedora 17 to 19, I reboot and select System Upgrade from menu.

At this point I receive the following error message:

Error: (2, 'Invalid kernel', 'elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images')


Version-Release number of selected component (if applicable):

0.7.3-5.fc17

How reproducible:

Every time on this virtual machine.  Have not tried others.


Steps to Reproduce:
1. Run Fedora 17.
2. Log in as root
3. Type:  fedup-cli --network 19
4. When prompted to reboot, reboot and then select System Upgrade
5. Receive error message

Actual results:

Failure.

Expected results:

Boot normally and upgrade to Fedora 19

Additional info:

I am running Fedora 16 for Dom0 and Xen version 4.1.2-2.fc16
Comment 1 Will Woods 2013-10-28 08:21:54 EDT
Can you please attach /var/log/upgrade.log and /var/log/fedup.log from the guest system?
Comment 2 wdouglascampbell 2013-10-28 10:26:55 EDT
Created attachment 816820 [details]
/var/log/fedup.log

There is no /var/log/upgrade.log.  I have attached the /var/log/fedup.log.  Thanks!
Comment 3 wdouglascampbell 2013-11-05 23:14:31 EST
Is there anything else I can provide to help in resolving this?  Thanks!
Comment 4 Will Woods 2013-11-06 13:30:18 EST
There's no upgrade.log? Did the system upgrade or not? Were any packages changed?

What was the error message? Did you get an emergency shell? If so, can you attach the output of 'journalctl -b'? Is there a file named /run/initramfs/rdsosreport.txt? If so, can you attach that?

What do you see if you remove 'rhgb quiet' from the boot arguments?
Comment 5 wdouglascampbell 2013-11-06 14:33:40 EST
No.  The system did not upgrade.  Sorry if I was not clear in my original post.

When I attempt to boot the virtual machine and have it run the System Upgrade, it dies with the following error message on the Xen Host:

Error: (2, 'Invalid kernel', 'elf_xen_note_check: ERROR: Will only load images built for the generic loader or Linux images')

and then I am back at the Xen Host shell prompt.

I went ahead and checked but there is not a file /run/initramfs/rdsosreport.txt in the guest when I boot it back into its current Fedora 17 kernel.

I also removed 'quiet rhgb' from the boot arguments for the System Upgrade but there was no additional information presented on the display.
Comment 6 Will Woods 2013-11-06 15:24:56 EST
Does /sys/hypervisor/type exist in the guest? How about /sys/bus/xen?

If /proc/xen/capabilities exists, what are its contents?
Comment 7 Will Woods 2013-11-06 16:06:18 EST
Does this command correctly identify the guest as a guest?

  [ -f /proc/xen/capabilities -a ! -f /proc/xen/xsd_kva ] && echo "xen guest"

Does it correctly *not* identify the xen host as a guest?

Also, what does this command print?

  grep -i 'xen' /proc/cpuinfo
Comment 8 wdouglascampbell 2013-11-06 16:25:54 EST
/sys/hypervisor/type exists in the guest and has as its contents:

xen

there is a directory /sys/bus/xen and its contents is:

drwxr-xr-x. 2 root root    0 Nov  7 03:14 devices
drwxr-xr-x. 7 root root    0 Nov  7 03:14 drivers
-rw-r--r--. 1 root root 4096 Nov  7 05:17 drivers_autoprobe
--w-------. 1 root root 4096 Nov  7 05:17 drivers_probe
--w-------. 1 root root 4096 Nov  7 03:14 uevent

the directory /proc/xen exists but it has no contents.

This command:

[ -f /proc/xen/capabilities -a ! -f /proc/xen/xsd_kva ] && echo "xen guest"

does not output anything

This command:

grep -i 'xen' /proc/cpuinfo

also does not output anything.

cat /proc/cpuinfo in the guest produces:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU           L5335  @ 2.00GHz
stepping        : 11
microcode       : 0xb4
cpu MHz         : 1995.062
cache size      : 4096 KB
fdiv_bug        : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht nx constant_tsc pni ssse3 hypervisor dtherm
bogomips        : 3990.12
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:


uname -r  produces

3.9.10-100.fc17.i686.PAE


I hope there is something there that helps.  Let me know what else I can provide.  Thanks!
Comment 9 Will Woods 2013-11-06 18:57:10 EST
Turns out that figuring out if your system is a xen guest is much trickier than you'd think! Last round of questions, I hope:

* What's `systemd-detect-virt --vm` say?
* Does the host have /sys/hypervisor/type?
* What's in /sys/hypervisor/properties/features on the host/guest?
* What's in /dev/xen (if anything) on the host/guest?
Comment 10 wdouglascampbell 2013-11-06 20:22:11 EST
systemd-detect-virt --vm on the guest returns

none


the host has /sys/hypervisor/type and it contains

xen


/sys/hypervisor/properties/features on the host contains

000000f0


/sys/hypervisor/properties/features on the guest contains

000000f0


/dev/xen on the host contains

crw------- 1 root root 10, 59 Jun 13 10:54 evtchn
crw------- 1 root root 10, 58 Jun 13 10:54 gntdev


/dev/xen on the guest contains

crw-------. 1 root root 10, 62 Nov  7 03:14 xenbus
Comment 11 wdouglascampbell 2013-11-16 14:27:07 EST
Any other information I can provide to help figure this out?  Thanks!
Comment 12 wdouglascampbell 2013-11-28 14:06:39 EST
Okay.  I think I understand the problem.  fedup is using a non-PAE kernel which xen is not going to like.  So, based on a comment someone had with a somewhat but not exactly related issue, I replaced the vmlinuz-fedup and initramfs-fedup.img images that fedup installed with the following:

wget http://mirrors.kernel.org/fedora/releases/19/Fedora/i386/os/images/pxeboot/vmlinuz-PAE -O /boot/vmlinuz-fedup

wget http://mirrors.kernel.org/fedora/releases/19/Fedora/i386/os/images/pxeboot/upgrade-PAE.img -O /boot/initramfs-fedup.img


Right now, I have booted the system and it has started the upgrade process.  It appears to now be working as expected.

So, if this indeed is fixing my issue, it means that fedup is not checking my environment to provide a valid kernel (in my case with PAE enabled).

Does my explanation seem correct?
Comment 13 wdouglascampbell 2013-11-29 02:49:57 EST
Just to confirm.  I have now successfully upgraded to Fedora 19 using fedup with the above change in the vmlinuz and initramfs.
Comment 14 Will Woods 2013-12-02 14:42:10 EST
Your explanation is correct. To pick the kernel, fedup examines the .treeinfo file, e.g.: 

  http://mirrors.kernel.org/fedora/releases/19/Fedora/i386/os/.treeinfo

Right now it always uses the images from the [images-$arch] section (where $arch is the arch listed in [general]. This works fine for x86_64 xen, but not i386.

Anyway, this should be fixed in git master:

  https://github.com/wgwoods/fedup/commit/6cc2a2a

So this bug should be fixed in the next fedup release.
Comment 15 wdouglascampbell 2013-12-03 15:13:53 EST
Awesome.  Thanks Will for your help in resolving this.  It is much appreciated!
Comment 16 Fedora Update System 2013-12-11 17:12:52 EST
fedup-0.8.0-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc19
Comment 17 Fedora Update System 2013-12-11 17:21:00 EST
fedup-0.8.0-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc20
Comment 18 Fedora Update System 2013-12-11 17:22:52 EST
fedup-0.8.0-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.8.0-3.fc18
Comment 19 Fedora Update System 2013-12-13 00:10:22 EST
Package fedup-0.8.0-3.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.8.0-3.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-23316/fedup-0.8.0-3.fc19
then log in and leave karma (feedback).
Comment 20 Fedora Update System 2013-12-19 02:22:50 EST
fedup-0.8.0-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Fedora Update System 2013-12-19 20:45:04 EST
fedup-0.8.0-3.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 22 Fedora Update System 2013-12-22 00:35:12 EST
fedup-0.8.0-3.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.