Bug 190899 - xen0 kernel update 2111 breaks xend
Summary: xen0 kernel update 2111 breaks xend
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: xen
Version: 5
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
Assignee: Xen Maintainance List
QA Contact: Brian Brock
URL:
Whiteboard:
: 190912 190944 191037 191038 191417 192039 192174 192549 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-05-06 02:56 UTC by Pat Murphy
Modified: 2007-11-30 22:11 UTC (History)
29 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2006-05-23 13:28:33 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Pat Murphy 2006-05-06 02:56:49 UTC
Description of problem:
After upgrading to 2.6.16-1.2111_FC5xen0 I can no longer issue xm commands or
start guest machines.

Version-Release number of selected component (if applicable):
kernel 2.6.16-1.2111_FC5xen0

How reproducible:
Upgrade to kernel 2.6.16-1.2111_FC5xen0 and reboot.  OS comes up, but xm does
not work and no guests will start.

Steps to Reproduce:
1.
2.
3.
  
Actual results:
xm commands return:
Error: Error connecting to xend: No such file or directory.  Is xend running?

xend.log shows:
[2006-05-05 17:52:11 xend] INFO (SrvDaemon:285) Xend Daemon started
[2006-05-05 17:52:11 xend] INFO (SrvDaemon:289) Xend changeset: unavailable .
[2006-05-05 17:52:11 xend] ERROR (SrvDaemon:299) Exception starting xend ((111,
'Connection refused'))
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line
293, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvServer.py", line
106, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40,
in __init__
    self.get(name)
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "/usr/lib64/python2.4/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.4/site-packages/xen/xend/server/SrvDomainDir.py",
line 39, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 550, in
instance
    inst.init()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 70, in init
    xstransact.Mkdir(VMROOT)
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 317, in Mkdir
    complete(path, lambda t: t.mkdir(*args))
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 323, in complete
    t = xstransact(path)
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 20, in __init__
    self.transaction = xshandle().transaction_start()
  File "/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xsutil.py", line
18, in xshandle
    xs_handle = xen.lowlevel.xs.xs()
RuntimeError: (111, 'Connection refused')


Expected results:
xm works and guest machines run.

Additional info:
If I boot the 2096 kernel the problem goes away.  I can boot guest machines that
have been upgraded to 2111 under the 2096 kernel.

Comment 1 Dave Jones 2006-05-06 03:47:30 UTC
are you also running the xen 3.0.2 rpm from updates-testing by any chance ?
If so, try dropping back to 3.0.1


Comment 2 Pat Murphy 2006-05-07 04:17:55 UTC
I am using the 3.0.1-4 version of xen and it does not work at this level.

Comment 3 Pontus Strömdahl 2006-05-07 06:43:22 UTC
Just installed fc5 and followed the quick start at 
http://fedoraproject.org/wiki/FedoraXenQuickstartFC5 with the exact same 
result when I tried the '/usr/sbin/xm list' command.

I'll try using the 2096 kernel.

Comment 4 Steve Snodgrass 2006-05-07 14:26:14 UTC
*** Bug 190944 has been marked as a duplicate of this bug. ***

Comment 5 Kenneth Ahn Jensen 2006-05-08 16:34:29 UTC
I just updated my FC5 and get the same results - same errormsgs.

[root@adm1-28 ~]# uname -r
2.6.16-1.2111_FC5xen0
[root@adm1-28 ~]# rpm -q xen
xen-3.0.1-4
[root@adm1-28 ~]# 

Cheers,
Kenneth

Comment 6 Rick Johnson 2006-05-08 18:37:42 UTC
Problem seemed to be introduced as of 2107. 

Xen version makes little difference (3.0.1-4 vs 3.0.2-x). Reverting back to 2096
seems to alleviate issue.

2.6.16-1.2111_FC5xenU kernels on unprivelaged domains seem to operate okay
(expected). 

Comment 7 Brian Stein 2006-05-08 18:46:16 UTC
*** Bug 191037 has been marked as a duplicate of this bug. ***

Comment 8 Brian Stein 2006-05-08 20:03:30 UTC
*** Bug 191038 has been marked as a duplicate of this bug. ***

Comment 9 Brian Stein 2006-05-08 20:07:06 UTC
*** Bug 190912 has been marked as a duplicate of this bug. ***

Comment 10 Roy Vestal 2006-05-09 17:18:24 UTC
I've been getting the same result. I've downgraded my kernel to
2.6.16-1.2096_FC5xen0 and get the same failures.

I've also tried the 2.6.15-1.2054_FC5xen0 with the same failure.

Comment 11 Roy Vestal 2006-05-09 18:20:10 UTC
I retried 2.6.16-1.2096_FC5xen0 and it worked...odd  :-/

Comment 12 Magnus Hyllander 2006-05-09 22:04:18 UTC
I don't agree with the problem being introduced in 2107. I used 2096 and 2107
without problems before trying 2111, and going back to 2107 still works fine for me.

Comment 13 Rick Johnson 2006-05-09 22:19:39 UTC
(In reply to comment #12)
> I don't agree with the problem being introduced in 2107. I used 2096 and 2107
> without problems before trying 2111, and going back to 2107 still works fine
for me.

You are correct, I just rebooted into 2107, and xend seemed to start correctly.
I am using xen 3.0.2-x from testing. First time I tested this was not the case -
unsure why.

Comment 14 Henning Schmiedehausen 2006-05-10 13:08:59 UTC
2111 works on SMP and breaks on UP. I have two XEN Boxes, one is Dual Processor
PIII, the other single CPU PIV (no HT). The SMP box works, the UP box fails.

See also Bug # 191237

Comment 15 Gianluca Cecchi 2006-05-10 14:19:36 UTC
In my case it is not so.
I have a Dell PE 6650 with 4 cpus.
2111 fails with both HT enabled (8 cpus) and disabled from bios (4) with the same
error as Pat Murphy's one.
I upgraded from 2054 to 2111 so that I have not the 2107 one to test.
Where can I pick it?
Gianluca
BTW: can I disable HT at runtime in dom0 boot string?

Comment 16 Rick Johnson 2006-05-10 14:32:13 UTC
My test case is x86_64 on dual opterons, and 2111 does *not* work for me.

Was the PE 6650 test x86 or x86_64?

Comment 17 Gianluca Cecchi 2006-05-10 14:41:43 UTC
the PE 6650 is x86 and does *not* work for me too

Comment 18 Christopher Johnson 2006-05-10 16:01:27 UTC
My test case is a dual Xeon HP DL380G4 (hyperthreading makes it smell like 4
processors).  Booting 2111 breaks xend.  Booting 2096 xend works.  So I don't
think the problem is architecture specific.

Comment 19 Gianluca Cecchi 2006-05-10 21:45:55 UTC
Same problem with a dual athlon MP server.
Also with xen 3.0.2 rpm from testing.
The system is configured with yum update run right today.
kernel 2111 (and devel one 2113) don't work, while 2096 is ok, in the sense that
xend starts.
So my working config is:
[root@fedora ~]# uname -r
2.6.16-1.2096_FC5xen0
[root@fedora ~]# rpm -q xen
xen-3.0.2-0.FC5.1

cat /proc/cpuinfo give  2 x:
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 8
model name      : AMD Athlon(tm) MP 2200+
stepping        : 1
cpu MHz         : 1800.111
cache size      : 256 KB

and 
[root@fedora ~]# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      256     2 r-----   352.1

I have no domains configured at this moment.....
Let me know if I can help debugging things...

Comment 20 Roy-Magne Mo 2006-05-12 13:54:11 UTC
2.6.16-1.2116_FC5xen0 from the kernel repository of davej on a dual PIII works
fine for me.



Comment 21 Juan Quintela 2006-05-12 15:32:45 UTC
kernel 2.6.16-1.2116_FC5 on updates-candidate fixes this (and other xen)
problems.  Will appear on updates RSN.

Comment 22 Michael Ahlers 2006-05-16 03:08:02 UTC
Someone mentioned using a newer kernel from davej helped.  For details, add http://people.redhat.com/
davej/kernels/Fedora/FC5/kernels.repo to /etc/yum.repos.d/ and update.

Comment 23 Gianluca Cecchi 2006-05-16 06:53:27 UTC
I think we should wait the next official FC5 kernel.
2116 is not there any more and now there is 2118.
I thought davej kernels were for rawhide, more than fc5 final...
Probably xen is yet a low priority technology for rh, until rh el 5 is out...
But the less deeply we will test it, the less it will be enterprise ready for rh
el 5 too.
just my opinion. YMMV.

Comment 24 Brian Stein 2006-05-17 12:15:10 UTC
*** Bug 192039 has been marked as a duplicate of this bug. ***

Comment 25 Brian Stein 2006-05-18 13:00:43 UTC
*** Bug 192174 has been marked as a duplicate of this bug. ***

Comment 26 Brian Stein 2006-05-18 13:03:59 UTC
2118 is available for testing in updates-testing and should resolve this issue.

Comment 27 Brian Stein 2006-05-18 16:50:07 UTC
*** Bug 191417 has been marked as a duplicate of this bug. ***

Comment 28 Michael Hampton 2006-05-19 07:03:37 UTC
2118 resolves the issue for me.

Comment 29 Gary Shi 2006-05-20 14:04:10 UTC
*** Bug 192549 has been marked as a duplicate of this bug. ***

Comment 30 Brian Stein 2006-05-22 12:22:04 UTC
*** Bug 192549 has been marked as a duplicate of this bug. ***

Comment 31 Brian Stein 2006-05-23 13:28:33 UTC
Thanks to everyone who reported / tested the newer kernels.  Keep it up!

Comment 32 Gonéri Le Bouder 2006-10-21 22:24:33 UTC
Hi,
I've the same issue with 2.6.18-1.2200.fc5xen0.

==> /var/log/xen/xend.log <==
[2006-10-22 00:04:06 xend 12249] INFO (SrvDaemon:283) Xend Daemon started
[2006-10-22 00:04:06 xend 12249] INFO (SrvDaemon:287) Xend changeset: 
unavailable .
[2006-10-22 00:04:06 xend 12249] ERROR (SrvDaemon:297) Exception starting xend 
((111, 'Connection refused'))
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDaemon.py", line 
291, in run
    servers = SrvServer.create()
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvServer.py", line 
108, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvRoot.py", line 40, 
in __init__
    self.get(name)
  File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 82, in get
    val = val.getobj()
  File "/usr/lib/python2.4/site-packages/xen/web/SrvDir.py", line 52, in 
getobj
    self.obj = klassobj()
  File "/usr/lib/python2.4/site-packages/xen/xend/server/SrvDomainDir.py", 
line 39, in __init__
    self.xd = XendDomain.instance()
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 636, in 
instance
    inst.init()
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 71, in 
init
    xstransact.Mkdir(VMROOT)
  File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 345, in Mkdir
    complete(path, lambda t: t.mkdir(*args))
  File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 351, in complete
    t = xstransact(path)
  File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xstransact.py", 
line 20, in __init__
    self.transaction = xshandle().transaction_start()
  File "/usr/lib/python2.4/site-packages/xen/xend/xenstore/xsutil.py", line 
18, in xshandle
    xs_handle = xen.lowlevel.xs.xs()
Error: (111, 'Connection refused')

Comment 33 Gonéri Le Bouder 2006-10-21 22:30:53 UTC
(In reply to comment #32)
> Hi,
> I've the same issue with 2.6.18-1.2200.fc5xen0.
This post seem to confirm the issue with this kernel:
http://article.gmane.org/gmane.linux.redhat.fedora.xen/1339

Regards

Comment 34 Rick Johnson 2006-10-21 22:37:27 UTC
Yesterday (10/20) attempted an upgrade to 2.6.18-1.2200.fc5xen0 on x86_64 and
experienced the same problem. Had to roll back to xen-3.0.2-0,
kernel-xen0-2.6.16-1.2133_FC5 and libvirt-0.1.1-1.FC5 before xend would properly
start. SELinux is disabled on this platform. 

Comment 35 Gonéri Le Bouder 2006-10-21 22:54:34 UTC
(In reply to comment #32)
> Hi,
> I've the same issue with 2.6.18-1.2200.fc5xen0.
This post seem to confirm the issue with this kernel:
http://article.gmane.org/gmane.linux.redhat.fedora.xen/1339

Regards

Comment 36 Yoshio YOSHIKAWA 2006-10-23 19:01:16 UTC
(In reply to comment #34)
> Yesterday (10/20) attempted an upgrade to 2.6.18-1.2200.fc5xen0 on x86_64 and
> experienced the same problem. Had to roll back to xen-3.0.2-0,
> kernel-xen0-2.6.16-1.2133_FC5 and libvirt-0.1.1-1.FC5 before xend would properly
> start. SELinux is disabled on this platform. 

I had the same issue with 2.6.18-1.2200.fc5xen0 too and it was 
enough to be rolled back to kernel-xen0-2.6.17-1.2187_FC5 via
yum on 2006/10/23.

# yum remove xen
# yum install xen-3.0.2-3.FC5 kernel-xen0-2.6.17-1.2187_FC5 libvirt-0.1.1-1.FC5
libvirt-python-0.1.1-1.FC5

Comment 37 Henning Schmiedehausen 2006-10-29 21:56:22 UTC
same problem on x86. See #189112 comment.


Note You need to log in before you can comment on or make changes to this bug.