Bug 251669 - Updating dom0 userland from 5.0 to 5.1 beta problems
Summary: Updating dom0 userland from 5.0 to 5.1 beta problems
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen
Version: 5.1
Hardware: All
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Don Domingo
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 222082 RHEL5u3_relnotes
TreeView+ depends on / blocked
 
Reported: 2007-08-10 11:08 UTC by Matthias Saou
Modified: 2010-03-14 21:31 UTC (History)
5 users (show)

Fixed In Version: 5.1
Doc Type: Bug Fix
Doc Text:
(all architectures) If you are using the Virtualized kernel when upgrading from Red Hat Enterprise Linux 5 to 5.2, you must reboot after completing the upgrade. You should then boot the system using the updated Virtualized kernel. The hypervisors of Red Hat Enterprise Linux 5 and 5.2 are not ABI-compatible. If you do not boot the system after upgrading using the updated Virtualized kernel, the upgraded Virtualization RPMs will not match the running kernel.
Clone Of:
Environment:
Last Closed: 2008-02-01 01:52:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Matthias Saou 2007-08-10 11:08:26 UTC
I've updated some RHEL5 x86_64 systems to 5.1 beta, without rebooting. They're
still running kernel 2.6.18-8.1.8.el5xen.

After the updates have been applied, xend doesn't seem to have been restarted
automatically, but some things already fail :
- xentop reports "Failed to retrieve statistics from libxenstat"
- xm commands seem to still work (console, shutdown, create)

After restarting xend, everything I normally use starts to fail :
- xm commands report "Error: Unable to connect to xend: Connection refused. Is
  xend running?"
- xentop still reports the same error

The xend python processes are running, but xend.log contains :

[...]
[2007-08-10 12:59:42 xend 3241] INFO (SrvDaemon:190) Xend stopped due to signal 15.
[2007-08-10 12:59:46 xend 15807] INFO (SrvDaemon:283) Xend Daemon started
[2007-08-10 12:59:46 xend 15807] INFO (SrvDaemon:287) Xend changeset: unavailable .
[2007-08-10 12:59:46 xend 15807] ERROR (SrvDaemon:297) Exception starting xend
((13, 'Permission denied'))
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line
291, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvServer.py", line
108, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40,
in __init__
    self.get(name)
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 39, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 655, in
instance
    inst.init()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 76, in init
    self._add_domain(
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 139, in
xen_domains
    domlist = xc.domain_getinfo()
Error: (13, 'Permission denied')

And xend-debug.log contains something other than "Nothing to flush." for the
first time ever :
domctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')

Rebooting dom0 to a 5.1 beta kernel (2.6.18-36xen here) fixed the issues,
unfortunately once I get to the situation above, I can no longer "save" the
running domUs nor have xendomains do it automatically, there is no other
solution than to shutdown cleanly domUs by logging into them individually, or
have them destroyed upon dom0's shutdown.

There is no mention of this in the 5.1 beta RELEASE-NOTES-U1-en file, so I would
think it either needs to be fixed/prevented somehow, or at least be mentioned in
the final release notes.

Comment 1 Matthias Saou 2007-08-10 11:21:46 UTC
FWIW, rebooting to the old 2.6.18-8.1.8.el5xen kernel doesn't work, only
rebooting to the newest kernels. Something needs to be done to :
- Enforce the userland/kernel sync
- Make error messages clearer about the incompatibility
- Document as required in the 5.1 release notes

As the xen version stayed 3.0.3, I wouldn't have expected this to happen...

Comment 2 Daniel Berrangé 2007-08-10 13:48:03 UTC
Updating from RHEL-5.0  to 5.1 requires a reboot if running Xen. This is because
the hypervisor in 5.1 is not ABI compatible with the hypervisor in 5.0. This in
turn requires that your running kernel match the corresponding Xen RPM. 

Its impossible to enforce this in the RPM dependancies because we you can
obviously reboot between 5.0 and 5.1 kernels at will. So this needs to be in the
release notes.


Comment 3 Stephen Tweedie 2007-08-10 13:56:40 UTC
Reassigning for release-noting.

Comment 5 Jan Pazdziora 2007-11-15 09:53:52 UTC
I did reboot.

However, the grub.conf had default=1, pointing to the old
xen.gz-2.6.18-8.1.4.el5, not to the new xen.gz-2.6.18-53.el5.

So the release note should also note that the user should make sure that the new
kernel was really booted.

Comment 6 Matthias Saou 2007-11-15 10:43:15 UTC
This "bites" quite often for systems installed as normal servers, then changed
to use a Xen kernel. The /etc/sysconfig/kernel file still references the plain
"kernel" instead of "kernel-xen" to be the default new kernel.

Comment 7 Don Domingo 2007-11-15 22:31:01 UTC
Thanks Jan, Matthias. Editing as follows:
<quote>
If you are using the Virtualized kernel when upgrading from Red Hat Enterprise
Linux 5 to 5.1, you must reboot after completing the upgrade. You should then
boot the system using the updated Virtualized kernel.

The hypervisors of Red Hat Enterprise Linux 5 and 5.1 are not ABI-compatible. If
you do not boot the system after upgrading using the updated Virtualized kernel,
the upgraded Virtualization RPMs will not match the running kernel.
</quote>

Comment 8 Don Domingo 2008-02-01 01:52:00 UTC
closing thsi bug, since RHEL5.1 release notes already released.

Comment 9 Ryan Lerch 2008-08-07 01:46:41 UTC
Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.

Comment 10 Ryan Lerch 2008-08-07 01:46:41 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.


Note You need to log in before you can comment on or make changes to this bug.