Bug 251669

Summary: Updating dom0 userland from 5.0 to 5.1 beta problems
Product: Red Hat Enterprise Linux 5 Reporter: Matthias Saou <matthias>
Component: xenAssignee: Don Domingo <ddomingo>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 5.1CC: ddomingo, jpazdziora, kajtzu, k.georgiou, xen-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 5.1 Doc Type: Bug Fix
Doc Text:
(all architectures) If you are using the Virtualized kernel when upgrading from Red Hat Enterprise Linux 5 to 5.2, you must reboot after completing the upgrade. You should then boot the system using the updated Virtualized kernel. The hypervisors of Red Hat Enterprise Linux 5 and 5.2 are not ABI-compatible. If you do not boot the system after upgrading using the updated Virtualized kernel, the upgraded Virtualization RPMs will not match the running kernel.
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-02-01 01:52:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 222082, 454962    

Description Matthias Saou 2007-08-10 11:08:26 UTC
I've updated some RHEL5 x86_64 systems to 5.1 beta, without rebooting. They're
still running kernel 2.6.18-8.1.8.el5xen.

After the updates have been applied, xend doesn't seem to have been restarted
automatically, but some things already fail :
- xentop reports "Failed to retrieve statistics from libxenstat"
- xm commands seem to still work (console, shutdown, create)

After restarting xend, everything I normally use starts to fail :
- xm commands report "Error: Unable to connect to xend: Connection refused. Is
  xend running?"
- xentop still reports the same error

The xend python processes are running, but xend.log contains :

[...]
[2007-08-10 12:59:42 xend 3241] INFO (SrvDaemon:190) Xend stopped due to signal 15.
[2007-08-10 12:59:46 xend 15807] INFO (SrvDaemon:283) Xend Daemon started
[2007-08-10 12:59:46 xend 15807] INFO (SrvDaemon:287) Xend changeset: unavailable .
[2007-08-10 12:59:46 xend 15807] ERROR (SrvDaemon:297) Exception starting xend
((13, 'Permission denied'))
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line
291, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvServer.py", line
108, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40,
in __init__
    self.get(name)
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 39, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 655, in
instance
    inst.init()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 76, in init
    self._add_domain(
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 139, in
xen_domains
    domlist = xc.domain_getinfo()
Error: (13, 'Permission denied')

And xend-debug.log contains something other than "Nothing to flush." for the
first time ever :
domctl operation failed -- need to rebuild the user-space tool set?
Exception starting xend: (13, 'Permission denied')

Rebooting dom0 to a 5.1 beta kernel (2.6.18-36xen here) fixed the issues,
unfortunately once I get to the situation above, I can no longer "save" the
running domUs nor have xendomains do it automatically, there is no other
solution than to shutdown cleanly domUs by logging into them individually, or
have them destroyed upon dom0's shutdown.

There is no mention of this in the 5.1 beta RELEASE-NOTES-U1-en file, so I would
think it either needs to be fixed/prevented somehow, or at least be mentioned in
the final release notes.

Comment 1 Matthias Saou 2007-08-10 11:21:46 UTC
FWIW, rebooting to the old 2.6.18-8.1.8.el5xen kernel doesn't work, only
rebooting to the newest kernels. Something needs to be done to :
- Enforce the userland/kernel sync
- Make error messages clearer about the incompatibility
- Document as required in the 5.1 release notes

As the xen version stayed 3.0.3, I wouldn't have expected this to happen...

Comment 2 Daniel Berrangé 2007-08-10 13:48:03 UTC
Updating from RHEL-5.0  to 5.1 requires a reboot if running Xen. This is because
the hypervisor in 5.1 is not ABI compatible with the hypervisor in 5.0. This in
turn requires that your running kernel match the corresponding Xen RPM. 

Its impossible to enforce this in the RPM dependancies because we you can
obviously reboot between 5.0 and 5.1 kernels at will. So this needs to be in the
release notes.


Comment 3 Stephen Tweedie 2007-08-10 13:56:40 UTC
Reassigning for release-noting.

Comment 5 Jan Pazdziora 2007-11-15 09:53:52 UTC
I did reboot.

However, the grub.conf had default=1, pointing to the old
xen.gz-2.6.18-8.1.4.el5, not to the new xen.gz-2.6.18-53.el5.

So the release note should also note that the user should make sure that the new
kernel was really booted.

Comment 6 Matthias Saou 2007-11-15 10:43:15 UTC
This "bites" quite often for systems installed as normal servers, then changed
to use a Xen kernel. The /etc/sysconfig/kernel file still references the plain
"kernel" instead of "kernel-xen" to be the default new kernel.

Comment 7 Don Domingo 2007-11-15 22:31:01 UTC
Thanks Jan, Matthias. Editing as follows:
<quote>
If you are using the Virtualized kernel when upgrading from Red Hat Enterprise
Linux 5 to 5.1, you must reboot after completing the upgrade. You should then
boot the system using the updated Virtualized kernel.

The hypervisors of Red Hat Enterprise Linux 5 and 5.1 are not ABI-compatible. If
you do not boot the system after upgrading using the updated Virtualized kernel,
the upgraded Virtualization RPMs will not match the running kernel.
</quote>

Comment 8 Don Domingo 2008-02-01 01:52:00 UTC
closing thsi bug, since RHEL5.1 release notes already released.

Comment 9 Ryan Lerch 2008-08-07 01:46:41 UTC
Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.

Comment 10 Ryan Lerch 2008-08-07 01:46:41 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.