Bug 893699 - xend is not started, causing libvirt to ERROR unable to connect to 'localhost:8000': Connection refused
xend is not started, causing libvirt to ERROR unable to connect to 'localh...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: xen (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Michael Young
Fedora Extras Quality Assurance
:
: 902883 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-01-09 13:19 EST by Konrad Rzeszutek Wilk
Modified: 2013-02-03 15:05 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-03 15:05:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Konrad Rzeszutek Wilk 2013-01-09 13:19:03 EST
Description of problem:

When trying to use virt-install to install a new guest:

# virt-install -l http://fedora.mirror.constant.com/linux/development/18/x86_64/os/ --ram 1024 --disk /dev/guests/F18-64 --name F18_64

I get:

ERROR    unable to connect to 'localhost:8000': Connection refused


If I start xend by hand:

# xend 

and retry it works just right.

Version-Release number of selected component (if applicable):

xen-4.2.0-7.fc18

How reproducible:

100%

Steps to Reproduce:
1. Install Fedora 18 on a machine.
2. Boot in the new OS
3. 'yum install libvirt xen virt-viewer'
4. Reboot, select 'Xen' option in the GRUB
5. As root do:
  virt-install -l http://fedora.mirror.constant.com/linux/development/18/x86_64/os/ --ram 1024 --disk /dev/guests/F18-64 --name F18_64

Actual results:
ERROR    unable to connect to 'localhost:8000': Connection refused


Expected results:
Starting install...
Retrieving file .treeinfo...                                                                              | 2.2 kB  00:00:00 !!! 
Retrieving file vmlinuz...                                                                                   | 9.3 MB  00:00:01 !!! 
Retrieving file initrd.img...                                                                                |  53 MB  00:00:06 !!! 
Creating domain...                                                                                           |    0 B  00:00:00     


Additional info:

It could also be that I failed to start the xend service somehow - but I can't seem to find in systemctl any mention of xend, just:

root@phenom konrad]# systemctl |grep xen
proc-xen.mount            loaded active mounted       Mount /proc/xen files
var-lib-xenstored.mount   loaded active mounted       mount xenstore file system
xenconsoled.service       loaded active running       Xenconsoled - handles logging from guest consoles and hypervisor
xendomains.service        loaded active exited        Xendomains - start and stop guests on boot and shutdown
xenstored.service         loaded active running       Xenstored - daemon managing xenstore file system
Comment 1 Konrad Rzeszutek Wilk 2013-01-09 13:21:40 EST
Per this thread:
http://lists.fedoraproject.org/pipermail/xen/2012-December/005949.html

it would seem that there are patches for libvirt to take advantage of 'xl' (which works great out of the box) but they haven't actually been put in libvirt (or maybe they have?)
Comment 2 Dario Faggioli 2013-01-09 13:36:42 EST
I am both seeing the same as Konrad, and also understood, per the linked thread, that we should have started using libvirt's libxl driver already, isn't this the case?

I'm also running a F18 beta, installed some time ago but updated frequently, including right now.

# rpm -qa | grep xen
xen-licenses-4.2.0-7.fc18.x86_64
libvirt-daemon-driver-xen-0.10.2.2-3.fc18.x86_64
xen-runtime-4.2.0-7.fc18.x86_64
xen-libs-4.2.0-7.fc18.x86_64
libvirt-daemon-xen-0.10.2.2-3.fc18.x86_64
xen-4.2.0-7.fc18.x86_64
xen-hypervisor-4.2.0-7.fc18.x86_64

# rpm -qa | grep libvirt
libvirt-daemon-driver-network-0.10.2.2-3.fc18.x86_64
libvirt-daemon-kvm-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-uml-0.10.2.2-3.fc18.x86_64
libvirt-glib-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-interface-0.10.2.2-3.fc18.x86_64
libvirt-daemon-qemu-0.10.2.2-3.fc18.x86_64
libvirt-0.10.2.2-3.fc18.x86_64
libvirt-client-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-nodedev-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-xen-0.10.2.2-3.fc18.x86_64
libvirt-gconfig-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-secret-0.10.2.2-3.fc18.x86_64
libvirt-gobject-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-qemu-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-lxc-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-nwfilter-0.10.2.2-3.fc18.x86_64
libvirt-daemon-config-network-0.10.2.2-3.fc18.x86_64
libvirt-daemon-xen-0.10.2.2-3.fc18.x86_64
libvirt-daemon-0.10.2.2-3.fc18.x86_64
libvirt-daemon-config-nwfilter-0.10.2.2-3.fc18.x86_64
libvirt-python-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-storage-0.10.2.2-3.fc18.x86_64

# yum search libxl
Loaded plugins: langpacks, presto, refresh-packagekit
Warning: No matches found for: libxl
No Matches found

# yum list *libxl*
Loaded plugins: langpacks, presto, refresh-packagekit
Error: No matching Packages to list
Comment 3 Konrad Rzeszutek Wilk 2013-01-09 13:54:12 EST
Note that in December when I had F17 dom0 installed, the F18 of a guest install did not hit this bug. Which is not surprising as at that time F17 was using Xen 4.1.x which could not do 'xl' properly.
Comment 4 Dario Faggioli 2013-01-09 13:57:20 EST
Let me also add that installing guests on F18-Beta Dom0 used to work last time I tried, a couple of months ago, right before the Virtualization Test Day (Nov 1st).

I'm sure it was xend that was being used, as libvirt's libxl driver was not ready at the time, and I explicitly installed the libvirt-daemon-xen package and removed/not installed the libvirt-daemon-driver-libxl package (which existed at the time).

And it all was out of the box, i.e., without the need of starting xend by hand, like it is now necessary.

Thanks,
Dario
Comment 5 Dario Faggioli 2013-01-10 05:21:56 EST
(In reply to comment #3)
> Note that in December when I had F17 dom0 installed, the F18 of a guest
> install did not hit this bug. Which is not surprising as at that time F17
> was using Xen 4.1.x which could not do 'xl' properly.

Mmm... I wonder whether it's my faul, but I just installed an F17 Dom0, `yum update'-ed it, `yum install xen'-ed on it and rebooted. I'm hitting right the same issue:

[root@Zhaman xen]# xm list
Error: Unable to connect to xend: Connection refused. Is xend running?
[root@Zhaman xen]# xend
[root@Zhaman xen]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 11989    16     r-----     77.2

[root@Zhaman xen]# rpm -qa xen
^C^[[A[root@Zhaman xen]# rpm -qa | grep xen
xen-runtime-4.1.4-1.fc17.x86_64
xen-libs-4.1.4-1.fc17.x86_64
xen-4.1.4-1.fc17.x86_64
xen-licenses-4.1.4-1.fc17.x86_64
netxen-firmware-4.0.534-5.fc17.noarch
xen-hypervisor-4.1.4-1.fc17.x86_64

I don't have anything related to libvirt installed yet, let me try adding them. I'll report back here what happens...

Regards,
Dario
Comment 6 Dario Faggioli 2013-01-10 09:49:55 EST
(In reply to comment #5)
> (In reply to comment #3)
> I don't have anything related to libvirt installed yet, let me try adding
> them. I'll report back here what happens...
> 
Ok, it looks like installing these packages managed in solving the issue! I did soemthing like this:

# yum install libvirt-daemon-xen python-virtinst libvirt-daemon-config-network libvirt-daemon-driver-network virt-manager

and I now (after rebooting) can see xend running:

# ps aux | grep xend
root      1221  0.4  0.1 918824 22996 ?        SLl  15:36   0:03 /usr/bin/python -Es /usr/sbin/xend status
root      3131  0.0  0.0 109404   868 pts/0    S+   15:48   0:00 grep --color=auto xend

without any need of manually starting it.

Is all this to be expected?

Thanks & Ragards,
Dario
Comment 7 Dario Faggioli 2013-01-10 12:05:50 EST
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #3)
> > I don't have anything related to libvirt installed yet, let me try adding
> > them. I'll report back here what happens...
> > 
> Ok, it looks like installing these packages managed in solving the issue! I
> did soemthing like this:
> 
> # yum install libvirt-daemon-xen python-virtinst
> libvirt-daemon-config-network libvirt-daemon-driver-network virt-manager
> 
> and I now (after rebooting) can see xend running:
>
> [...]
>
> Is all this to be expected?
>
Ok, thinking more about it, I think this I was saying above _is_ right. I mean, it is correct for xend to be started only when you install libvirt/virt-install/virt-manager (I don't exactly know who's responsible for this! :-D).

However, still on Fedora 17, I updated the virt-preview repository, and now I don't have xend automatically started any longer!

These means that this combination of packages:
libvirt-daemon-config-nwfilter-1.0.1-2.fc17.x86_64
libvirt-client-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-nwfilter-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-storage-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-network-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-secret-1.0.1-2.fc17.x86_64
libvirt-daemon-xen-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-nodedev-1.0.1-2.fc17.x86_64
libvirt-daemon-config-network-1.0.1-2.fc17.x86_64
libvirt-daemon-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-qemu-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-xen-1.0.1-2.fc17.x86_64
libvirt-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-interface-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-uml-1.0.1-2.fc17.x86_64
libvirt-daemon-driver-lxc-1.0.1-2.fc17.x86_64
libvirt-python-1.0.1-2.fc17.x86_64

Has the same effect of this one:
# rpm -qa | grep libvirt
libvirt-daemon-driver-network-0.10.2.2-3.fc18.x86_64
libvirt-daemon-kvm-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-uml-0.10.2.2-3.fc18.x86_64
libvirt-glib-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-interface-0.10.2.2-3.fc18.x86_64
libvirt-daemon-qemu-0.10.2.2-3.fc18.x86_64
libvirt-0.10.2.2-3.fc18.x86_64
libvirt-client-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-nodedev-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-xen-0.10.2.2-3.fc18.x86_64
libvirt-gconfig-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-secret-0.10.2.2-3.fc18.x86_64
libvirt-gobject-0.1.4-1.fc18.x86_64
libvirt-daemon-driver-qemu-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-lxc-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-nwfilter-0.10.2.2-3.fc18.x86_64
libvirt-daemon-config-network-0.10.2.2-3.fc18.x86_64
libvirt-daemon-xen-0.10.2.2-3.fc18.x86_64
libvirt-daemon-0.10.2.2-3.fc18.x86_64
libvirt-daemon-config-nwfilter-0.10.2.2-3.fc18.x86_64
libvirt-python-0.10.2.2-3.fc18.x86_64
libvirt-daemon-driver-storage-0.10.2.2-3.fc18.x86_64

which is not starting xend automatically.

If we assume that the responsible for this is libvirt-daemon-xen (is that right?), I think I can now say that libvirt-daemon-xen-0.9.11.8-2.fc17 does start xend automatically, while neither of libvirt-daemon-driver-xen-1.0.1-2.fc17.x86_64 or libvirt-daemon-xen-0.10.2.2-3.fc18.x86_64 does.

Thanks again,
Dario
Comment 8 Cole Robinson 2013-01-14 14:54:30 EST
Libvirt in rawhide should be able to talk to xl no problem. Libvirt in F18 has backported packages but I'm waiting for some other bug fixes to accumulate before I push a build. Likely next week.
Comment 9 Dario Faggioli 2013-01-15 12:22:00 EST
(In reply to comment #8)
> Libvirt in rawhide should be able to talk to xl no problem. Libvirt in F18
> has backported packages but I'm waiting for some other bug fixes to
> accumulate before I push a build. Likely next week.
>
Ok, very nice to read that, thanks! I'll take some time (tomorrow) to test rawhide.

However, besides having a working libxl driver, which is great, I think libvirt-daemon-driver-xen should still start xend automatically, shouldn't it? So that, if one wants to use xend, he can install such package, while if he wants libxl, he can go for libvirt-daemon-driver-libxl (or whatever it is called)... Or am I misunderstanding how the whole thing is meant to be?

Anyway, I'll test this too on rawhide and report back, either here or on the list.

Thanks a lot for looking at this!
Dario
Comment 10 Dario Faggioli 2013-01-17 05:27:21 EST
(In reply to comment #9)
> (In reply to comment #8)
> > Libvirt in rawhide should be able to talk to xl no problem. Libvirt in F18
> > has backported packages but I'm waiting for some other bug fixes to
> > accumulate before I push a build. Likely next week.
> >
>
> [...]
>
> Anyway, I'll test this too on rawhide and report back, either here or on the
> list.
>
Ok, here I am, a bit late as my testbox was otherwise engaged yesterday.

So, in rawhide, If in _do_ install libvirt-daemon-driver-xen and _do_not_ install libxl-daemon-driver-libxl, xend is automatically loaded and I can create guests with both virt-inst and virt-manager, which is good. :-)

However, if I do the opposite, i.e., I _do_not_ install (well, remove) libvirt-daemon-driver-xen and _do_install libvirt-daemon-driver-libxl, virt-inst and virt-manager stops working.

In fact, virt-inst does not even try:
# virt-install -l http://fedora.mirror.constant.com/linux/development/18/x86_64/os/ --ram 1024 --disk /dev/vms/F18_x64 --name F18_x64
WARNING  KVM acceleration not available, using 'qemu'

While this is what virt-manager tells me, after failing to connect to XEN:
Unable to connect to libvirt.

internal error libxenlight state driver is not active

Verify that:
 - A Xen host kernel was booted
 - The Xen service has been started

Libvirt URI is: xen:///

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1027, in _open_thread
    self.vmm = self._try_open()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1009, in _try_open
    flags)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: internal error libxenlight state driver is not active

Might be worth to say that xend seems to be still being run automatically:
# ps aux | grep xend
root      1770  0.4  0.2 838656 23672 ?        SLl  11:04   0:05 /usr/bin/python -Es /usr/sbin/xend status

and that I seem to be able to create guests manually with `xl create'.

Is all the above supposed to happen / am I doing something wrong?
Comment 11 Jim Fehlig 2013-01-17 10:49:10 EST
(In reply to comment #10)
> While this is what virt-manager tells me, after failing to connect to XEN:
> Unable to connect to libvirt.
> 
> internal error libxenlight state driver is not active

The libxl driver did not load.

> Might be worth to say that xend seems to be still being run automatically:
> # ps aux | grep xend
> root      1770  0.4  0.2 838656 23672 ?        SLl  11:04   0:05
> /usr/bin/python -Es /usr/sbin/xend status

Is it?  That looks to be a status check only.  But if xend is running, the libxl driver will not load.

Are there any messages in the libvirtd logs that indicate why the libxl driver refused/failed to load?
Comment 12 Jim Fehlig 2013-01-17 10:51:51 EST
Opps. looks like I cleared needinfo.  Adding Dario...
Comment 13 Michael Young 2013-01-17 11:17:41 EST
I have noticed before that libvirt does a status check by calling xend during start up. Unfortunately the process seems to stick around if xend isn't otherwise running, though I haven't got around to working out why and if it is worth fixing.
Comment 14 Xiao-Long Chen 2013-01-17 14:49:00 EST
I don't know if it helps at all, but I compiled the rawhide libvirt in Fedora 18 and have no problems with xl and libvirt. I didn't change any systemd services.

[chenxiaolong@cxl-xen ~]$ rpm -qa | grep -i -e xen -e libvirt | sort
libvirt-client-1.0.1-2.fc18.x86_64
libvirt-daemon-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-interface-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-libxl-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-network-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-nodedev-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-nwfilter-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-secret-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-storage-1.0.1-2.fc18.x86_64
libvirt-daemon-driver-xen-1.0.1-2.fc18.x86_64
libvirt-daemon-xen-1.0.1-2.fc18.x86_64
xen-4.2.0-7.fc18.x86_64
xen-hypervisor-4.2.0-7.fc18.x86_64
xen-libs-4.2.0-7.fc18.x86_64
xen-licenses-4.2.0-7.fc18.x86_64
xen-runtime-4.2.0-7.fc18.x86_64

[chenxiaolong@cxl-xen ~]$ ps aux | grep -i xen
root        21  0.0  0.0      0     0 ?        S    14:02   0:00 [xenwatch]
root        22  0.0  0.0      0     0 ?        S    14:02   0:00 [xenbus]
root       547  0.7  0.0  10900  1080 ?        SL   14:02   0:18 /usr/sbin/xenstored --pid-file /var/run/xenstored.pid
root       554  0.0  0.0  84616   764 ?        SLsl 14:02   0:00 /usr/sbin/xenconsoled --log=none --log-dir=/var/log/xen/console
root      1008  0.0  0.4  94684 12296 ?        Ss   14:02   0:00 /sbin/dhclient -H cxl-xen -1 -q -lf /var/lib/dhclient/dhclient--br0.lease -pf /var/run/dhclient-br0.pid br0
root      1199  2.0  0.7 1046832 19488 ?       SLl  14:02   0:50 /usr/bin/python -Es /usr/sbin/xend status
1000     14916  0.0  0.0 109180   896 pts/0    S+   14:44   0:00 grep --color=auto -i xen


No error messages for xl either:

[root@cxl-xen log]# grep -ri xenlight /var/log/
[root@cxl-xen log]#
Comment 15 Dario Faggioli 2013-01-18 14:32:48 EST
(In reply to comment #14)
> I don't know if it helps at all, but I compiled the rawhide libvirt in
> Fedora 18 and have no problems with xl and libvirt. I didn't change any
> systemd services.
> 
Ok, first of all, thanks for your input.

> [chenxiaolong@cxl-xen ~]$ ps aux | grep -i xen
> root        21  0.0  0.0      0     0 ?        S    14:02   0:00 [xenwatch]
> root        22  0.0  0.0      0     0 ?        S    14:02   0:00 [xenbus]
> root       547  0.7  0.0  10900  1080 ?        SL   14:02   0:18
> /usr/sbin/xenstored --pid-file /var/run/xenstored.pid
> root       554  0.0  0.0  84616   764 ?        SLsl 14:02   0:00
> /usr/sbin/xenconsoled --log=none --log-dir=/var/log/xen/console
> root      1008  0.0  0.4  94684 12296 ?        Ss   14:02   0:00
> /sbin/dhclient -H cxl-xen -1 -q -lf /var/lib/dhclient/dhclient--br0.lease
> -pf /var/run/dhclient-br0.pid br0
> root      1199  2.0  0.7 1046832 19488 ?       SLl  14:02   0:50
> /usr/bin/python -Es /usr/sbin/xend status
> 1000     14916  0.0  0.0 109180   896 pts/0    S+   14:44   0:00 grep
> --color=auto -i xen
>
Mmm... I do no seem to see any xend instance. Do virt-install and virt-manager work? Is this `ps' taken before or after trying to run the first virt-install (or to connect to xen:// for the first time in virt-manager)? IU'm asking because it seems to matter, at least on my system.

 
> No error messages for xl either:
> 
> [root@cxl-xen log]# grep -ri xenlight /var/log/
> [root@cxl-xen log]#
>
That is nice. However, it is still unknown whether or not you're actually using libxl at all. What makes you think you are?

Thanks a lot again, Dario
Comment 16 Dario Faggioli 2013-01-18 15:00:52 EST
(In reply to comment #11)
> (In reply to comment #10)
> > Might be worth to say that xend seems to be still being run automatically:
> > # ps aux | grep xend
> > root      1770  0.4  0.2 838656 23672 ?        SLl  11:04   0:05
> > /usr/bin/python -Es /usr/sbin/xend status
> 
> Is it?  That looks to be a status check only.  But if xend is running, the
> libxl driver will not load.
> 
Oh, I see what you mean, and I missed the 'status' part. Let's try again.

1) libvirt-daemon-driver-libxl INSTALLED
   libvirt-daemon-driver-xen   NOT INSTALLED

   # ps aux | grep xend
   root      1803  0.6  0.1 764920 22800 ?        SLl  20:39   0:03 /usr/bin/python -Es /usr/sbin/xend status

   virt-manager says "internal error libxenlight state driver is not active"

   # grep libvirt /var/log/messages
   Jan 18 20:39:18 localhost libvirtd[1555]: libvirt version: 1.0.1, package: 2.fc19 (Fedora Project, 2012-12-17-23:58:40, buildvm-05.phx2.fedoraproject.org)
   Jan 18 20:39:18 localhost libvirtd[1555]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_xen.so not accessible
   Jan 18 20:39:18 localhost libvirtd[1555]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
   Jan 18 20:39:18 localhost libvirtd[1555]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_uml.so not accessible
   Jan 18 20:39:24 localhost dnsmasq[1768]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
   Jan 18 20:39:24 localhost dnsmasq-dhcp[1768]: read /var/lib/libvirt/dnsmasq/default.hostsfile
   Jan 18 20:39:34 localhost kernel: [   79.715469] cgroup: libvirtd (1652) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
   Jan 18 20:39:34 localhost kernel: [   79.717445] cgroup: libvirtd (1652) created nested cgroup for controller "devices" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
   Jan 18 20:39:34 localhost kernel: [   79.719059] cgroup: libvirtd (1652) created nested cgroup for controller "blkio" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
   Jan 18 20:43:39 localhost libvirtd[1555]: internal error libxenlight state driver is not active

   It says it does not find libvirt_driver_xen.so, which is completely understandable, as I removed the package, and then just that it can't activate the module... Should I look somewhere esle?

   # xend
   # ps aux | grep xend
root      1803  0.4  0.1 764920 22800 ?        SLl  20:39   0:03 /usr/bin/python -Es /usr/sbin/xend status
   # xm list 
   Name                                        ID   Mem VCPUs      State   Time(s)
   Domain-0                                     0 11989    16     r-----    444.2

   I.e., not much different than before, and in fact, virt-manager still can't connect. But `xm list' works. Mmm...

   # killall xend
   # xm list
   Error: Unable to connect to xend: Connection refused. Is xend running?
   # xend
   # ps aux | grep xend
   root      3703 28.5  0.1 762824 22792 ?        SLl  20:54   0:02 /usr/bin/python -Es /sbin/xend
   root      3837  0.0  0.0 112620   972 pts/0    S+   20:54   0:00 grep --color=auto xend
   # xm list
   Name                                        ID   Mem VCPUs      State   Time(s)
   Domain-0                                     0 11989    16     r-----    491.5

   So, now the `ps' line for xend is different, but it looks it was running even when it seemed to only be a status check right (or xm wouldn't have worked).

   I'm a little bit in short of ideas here... What 
I'm sure about, is I can't seem to get the libxl module to activate, whether I have xend running or not!

1) libvirt-daemon-driver-libxl INSTALLED
   libvirt-daemon-driver-xen   NOT INSTALLED

   Everything seems to work, but I'm quite sure it is because in that case I have both xend running, and the correct libvirt module installed.

> Are there any messages in the libvirtd logs that indicate why the libxl
> driver refused/failed to load?

I'm not sure where to lock for this. /var/log/libvirt doesn't have anything, and /var/log/messages have only what I've shown before... Again, should I do something else?

Thanks and Regards, Dario
Comment 17 Xiao-Long Chen 2013-01-18 16:26:09 EST
(In reply to comment #15)
> (In reply to comment #14)
> > I don't know if it helps at all, but I compiled the rawhide libvirt in
> > Fedora 18 and have no problems with xl and libvirt. I didn't change any
> > systemd services.
> > 
> Ok, first of all, thanks for your input.

No problem :)

> 
> > [chenxiaolong@cxl-xen ~]$ ps aux | grep -i xen
> > root        21  0.0  0.0      0     0 ?        S    14:02   0:00 [xenwatch]
> > root        22  0.0  0.0      0     0 ?        S    14:02   0:00 [xenbus]
> > root       547  0.7  0.0  10900  1080 ?        SL   14:02   0:18
> > /usr/sbin/xenstored --pid-file /var/run/xenstored.pid
> > root       554  0.0  0.0  84616   764 ?        SLsl 14:02   0:00
> > /usr/sbin/xenconsoled --log=none --log-dir=/var/log/xen/console
> > root      1008  0.0  0.4  94684 12296 ?        Ss   14:02   0:00
> > /sbin/dhclient -H cxl-xen -1 -q -lf /var/lib/dhclient/dhclient--br0.lease
> > -pf /var/run/dhclient-br0.pid br0
> > root      1199  2.0  0.7 1046832 19488 ?       SLl  14:02   0:50
> > /usr/bin/python -Es /usr/sbin/xend status
> > 1000     14916  0.0  0.0 109180   896 pts/0    S+   14:44   0:00 grep
> > --color=auto -i xen
> >
> Mmm... I do no seem to see any xend instance. Do virt-install and
> virt-manager work? Is this `ps' taken before or after trying to run the
> first virt-install (or to connect to xen:// for the first time in
> virt-manager)? IU'm asking because it seems to matter, at least on my system.
> 

Both virt-install and virt-manager work. The 'ps' is taken after creating 4 virtual machines (1 hvm, 3 pvm). As far as I know, xl does not need xend to work properly. Only the old xm toolstack needs xend.

>  
> > No error messages for xl either:
> > 
> > [root@cxl-xen log]# grep -ri xenlight /var/log/
> > [root@cxl-xen log]#
> >
> That is nice. However, it is still unknown whether or not you're actually
> using libxl at all. What makes you think you are?
> 
> Thanks a lot again, Dario

I'm pretty sure I'm using xl because xend is not running (required for xm). For example:

[chenxiaolong@cxl-xen log]$ sudo xm list
Error: Unable to connect to xend: Connection refused. Is xend running?
[chenxiaolong@cxl-xen log]$ sudo xm shutdown FreeIPA
Error: Unable to connect to xend: Connection refused. Is xend running?
[chenxiaolong@cxl-xen log]$ sudo xm info
Error: Unable to connect to xend: Connection refused. Is xend running?

That is with a VM called FreeIPA running. My thinking is that if I can't run the commands, then neither can libvirt, so I must be using xl :)
Comment 18 Xiao-Long Chen 2013-01-18 16:41:20 EST
By the way, to get a log, change "log_level" to 1 in /etc/libvirt/libvirtd.service and then view the logs by running:

sudo journalctl -u libvirtd.service

(assuming libvirt was started by systemd, of course)
Comment 19 Xiao-Long Chen 2013-01-18 23:49:27 EST
Sorry for the noise in the previous comments. I was wrong about libxl working. I did quite a few tests with libxl and here are the results:

1. libvirt-daemon-driver-libxl always requires libvirt-daemon-driver-xen. This was my setup. As soon as I remove the xen driver, I get "internal error libxenlight state driver is not active" as other have gotten above.

2. That "/usr/bin/python -Es /usr/sbin/xend status" in ps is deceiving. Try starting libvirt, run "virsh -c xen:///", and notice how "xm info" works. Now kill that xend status process and watch how "xm info" fails. Running that command in a terminal returns the xend status. But when libvirt runs it, the actual xend daemon is started. *This explains why the status checking process never exits.*

3. Knowing that xl doesn't require xend, I replaced /usr/sbin/xend with:

#!/bin/bash
exit 0

virsh (with xen and libxl driver installed) fails with "unable to connect to 'localhost:8000': Connection refused", which is a xen message, not a libxl message. It means that libxl is falling back to the xen driver.
Comment 20 Jim Fehlig 2013-01-21 00:01:08 EST
(In reply to comment #19)
> 2. That "/usr/bin/python -Es /usr/sbin/xend status" in ps is deceiving. Try
> starting libvirt, run "virsh -c xen:///", and notice how "xm info" works.
> Now kill that xend status process and watch how "xm info" fails. Running
> that command in a terminal returns the xend status. But when libvirt runs
> it, the actual xend daemon is started. *This explains why the status
> checking process never exits.*

Odd, I don't see that on my SUSE-based systems.  Resolving this issue probably resolves the bug though.

> 3. Knowing that xl doesn't require xend, I replaced /usr/sbin/xend with:
> 
> #!/bin/bash
> exit 0

'exit 0' means xend is running, hence the libxl driver won't load.  'exit 3' (service not running) should cause the libxl driver to load.
Comment 21 Xiao-Long Chen 2013-01-21 01:51:36 EST
(In reply to comment #20)
> (In reply to comment #19)
> > 2. That "/usr/bin/python -Es /usr/sbin/xend status" in ps is deceiving. Try
> > starting libvirt, run "virsh -c xen:///", and notice how "xm info" works.
> > Now kill that xend status process and watch how "xm info" fails. Running
> > that command in a terminal returns the xend status. But when libvirt runs
> > it, the actual xend daemon is started. *This explains why the status
> > checking process never exits.*
> 
> Odd, I don't see that on my SUSE-based systems.  Resolving this issue
> probably resolves the bug though.
> 
> > 3. Knowing that xl doesn't require xend, I replaced /usr/sbin/xend with:
> > 
> > #!/bin/bash
> > exit 0
> 
> 'exit 0' means xend is running, hence the libxl driver won't load.  'exit 3'
> (service not running) should cause the libxl driver to load.

Thanks! I should stop working on things after midnight :) I'll test that again tomorrow.
Comment 22 Xiao-Long Chen 2013-01-21 18:37:23 EST
(In reply to comment #20)
> (In reply to comment #19)
> > 3. Knowing that xl doesn't require xend, I replaced /usr/sbin/xend with:
> > 
> > #!/bin/bash
> > exit 0
> 
> 'exit 0' means xend is running, hence the libxl driver won't load.  'exit 3'
> (service not running) should cause the libxl driver to load.

Okay, with "exit 3", I can force libvirt to use the libxl driver. Unfortunately, it's not very useful. "virsh -c xen:/// list" shows nothing, not even Domain0. I disabled SELinux temporarily to make sure that it wasn't preventing communication between libvirt and xl.
Comment 23 Jim Fehlig 2013-01-21 19:44:02 EST
Have you started any domains with the driver?  Like the qemu driver, the libxl driver ignores the host (domain 0), so unless you have persistent domains defined 'virsh list' will show nothing.  'virsh list --all' will show persistent domains that are inactive.

Does 'virsh version' report the libxl driver active?  E.g.

# virsh version
Compiled against library: libvirt 1.0.1
Using library: libvirt 1.0.1
Using API: xenlight 1.0.1
Running hypervisor: xenlight 4.3.0

BTW, the libxl driver currently does not support managing out-of-band domains, e.g. those created through libxl.
Comment 24 Jim Fehlig 2013-01-21 19:45:02 EST
(In reply to comment #23)
> BTW, the libxl driver currently does not support managing out-of-band
> domains, e.g. those created through libxl.

Opps, should have been "those created with xl".
Comment 25 Xiao-Long Chen 2013-01-21 21:16:36 EST
(In reply to comment #23)
> Have you started any domains with the driver?  Like the qemu driver, the
> libxl driver ignores the host (domain 0),

Thanks for the information! I didn't know that.

> so unless you have persistent
> domains defined 'virsh list' will show nothing.  'virsh list --all' will
> show persistent domains that are inactive.

If by persistent, you mean VMs created using libvirt (as opposed to "xl create"), then yes, none of my persistent VMs show up. 'virsh -c xen:/// list --all' shows nothing.

I've tried creating a PVM Debian DomU, but unfortunately, that fails too. Here's the libvirt log: http://paste.kde.org/653372/raw/ Whatever that error is, it completely killed the libvirt daemon: http://paste.kde.org/653384/raw/

> 
> Does 'virsh version' report the libxl driver active?  E.g.
> 
> # virsh version
> Compiled against library: libvirt 1.0.1
> Using library: libvirt 1.0.1
> Using API: xenlight 1.0.1
> Running hypervisor: xenlight 4.3.0
> 
> BTW, the libxl driver currently does not support managing out-of-band
> domains, e.g. those created through libxl.

[chenxiaolong@cxl-xen ~]$ virsh -c xen:/// version
Compiled against library: libvirt 1.0.1
Using library: libvirt 1.0.1
Using API: xenlight 1.0.1
Running hypervisor: xenlight 4.2.0

I see that you're running Xen 4.3.0. Would you like me to compile Xen from git/hg? Would libvirt need to be recompiled against the new Xen?
Comment 26 Jim Fehlig 2013-01-21 21:39:12 EST
(In reply to comment #25)
> If by persistent, you mean VMs created using libvirt (as opposed to "xl
> create"), then yes, none of my persistent VMs show up. 'virsh -c xen:///
> list --all' shows nothing.

By persistent, I mean domains that still exist when they are inactive.  Persistent domains are also known as define domains, and managed domains.  You create a persistent domain with 'virsh define', start it with 'virsh start', shut it down with 'virsh shutdown', and remove it entirely with 'virsh undefine'.  By contrast, transient (aka ephemeral, unmanaged) domains are no longer known to libvirt once they are shutdown.

> I've tried creating a PVM Debian DomU, but unfortunately, that fails too.
> Here's the libvirt log: http://paste.kde.org/653372/raw/ Whatever that error
> is, it completely killed the libvirt daemon: http://paste.kde.org/653384/raw/

Looks like you are hitting some of the issues (segfaults, asserts in libxl or pthreads) addressed by this patchset for the libxl driver

https://www.redhat.com/archives/libvir-list/2013-January/msg01463.html

and these patches for libxl

http://lists.xen.org/archives/html/xen-devel/2012-12/msg00684.html
http://lists.xen.org/archives/html/xen-devel/2012-12/msg00685.html

> I see that you're running Xen 4.3.0. Would you like me to compile Xen from
> git/hg? Would libvirt need to be recompiled against the new Xen?

I haven't tested this on a 4.2.x system yet, but need to do so before committing the libvirt patches, which are awaiting review btw :).
Comment 27 Xiao-Long Chen 2013-01-22 15:56:58 EST
(In reply to comment #26)
> By persistent, I mean domains that still exist when they are inactive. 
> Persistent domains are also known as define domains, and managed domains. 
> You create a persistent domain with 'virsh define', start it with 'virsh
> start', shut it down with 'virsh shutdown', and remove it entirely with
> 'virsh undefine'.  By contrast, transient (aka ephemeral, unmanaged) domains
> are no longer known to libvirt once they are shutdown.

Thanks for the description!

> 
> > I've tried creating a PVM Debian DomU, but unfortunately, that fails too.
> > Here's the libvirt log: http://paste.kde.org/653372/raw/ Whatever that error
> > is, it completely killed the libvirt daemon: http://paste.kde.org/653384/raw/
> 
> Looks like you are hitting some of the issues (segfaults, asserts in libxl
> or pthreads) addressed by this patchset for the libxl driver
> 
> https://www.redhat.com/archives/libvir-list/2013-January/msg01463.html
> 
> and these patches for libxl
> 
> http://lists.xen.org/archives/html/xen-devel/2012-12/msg00684.html
> http://lists.xen.org/archives/html/xen-devel/2012-12/msg00685.html
> 
> > I see that you're running Xen 4.3.0. Would you like me to compile Xen from
> > git/hg? Would libvirt need to be recompiled against the new Xen?
> 
> I haven't tested this on a 4.2.x system yet, but need to do so before
> committing the libvirt patches, which are awaiting review btw :).

I'll apply those patches and test it in Fedora.

Btw, someone found the cause of the "xend status" issue: http://lists.xen.org/archives/html/xen-devel/2013-01/msg01807.html
Comment 28 Xiao-Long Chen 2013-01-22 15:57:41 EST
(In reply to comment #27)
> Btw, someone found the cause of the "xend status" issue:
> http://lists.xen.org/archives/html/xen-devel/2013-01/msg01807.html

Oops, I didn't see that you sent the original message :)
Comment 29 Jim Fehlig 2013-01-22 21:58:47 EST
(In reply to comment #26)
> I haven't tested this on a 4.2.x system yet, but need to do so before
> committing the libvirt patches, which are awaiting review btw :).

I have now tested the patches on 4.2.1 and didn't notice any problems.  By patches, I mean the libvirt *and* libxl ones mentioned in #26.
Comment 30 Xiao-Long Chen 2013-01-23 00:25:37 EST
(In reply to comment #29)
> (In reply to comment #26)
> > I haven't tested this on a 4.2.x system yet, but need to do so before
> > committing the libvirt patches, which are awaiting review btw :).
> 
> I have now tested the patches on 4.2.1 and didn't notice any problems.  By
> patches, I mean the libvirt *and* libxl ones mentioned in #26.

Awesome. I'll try it on Fedora's Xen 4.2.1 and libvirt 1.0.1.
Comment 31 Dario Faggioli 2013-01-23 03:58:03 EST
(In reply to comment #29)
> (In reply to comment #26)
> > I haven't tested this on a 4.2.x system yet, but need to do so before
> > committing the libvirt patches, which are awaiting review btw :).
> 
> I have now tested the patches on 4.2.1 and didn't notice any problems.  By
> patches, I mean the libvirt *and* libxl ones mentioned in #26.
>
Very cool!

So, Jim, now it would be awesome if we could get all these patches into F18, as an update to the Xen and libvirt package, I guess. What do we need for that? My impression is that the libxl patches need to be backported to some 4.2.X Xen version... Have you already discussed this with the guys on xen-devel? Should you need any help (provided you think this is The Right Thing), do not hesitate to ask.

On the libvirt side, I think we just need to be sure you get the patch upstream, and then wait for an update of that package too. Do you mind letting me/us (either here or via e-mail) know when that happen (I mean, patches upstream)?

Thanks and Regards, Dario
Comment 32 Dario Faggioli 2013-01-23 06:12:31 EST
(In reply to comment #8)
> Libvirt in rawhide should be able to talk to xl no problem. Libvirt in F18
> has backported packages but I'm waiting for some other bug fixes to
> accumulate before I push a build. Likely next week.
>
Ok, so, despite the bug being filed because a xend issue, the discussion took a bunch of different directions, and it all was very useful, as it helped uncovering the causes of two other bugs! :-)

However, since it looks like having the libvirt libxl driver in a good enough shape in Fedora might require some more time, I wonder whether we at least can put the good old libvirt-daemon-driver-xen --the xend based one-- to work out of the box in F18, as it used to be. This way, provided we tell people _not_to_ install libvirt-daemon-driver-libxl (for now), they'll be able to use virt-install and virt-manager without issues.

So, wrt this, Cole, I can confirm that, on rawhide, having only libvirt-daemon-driver-xen-1.0.1-4.fc19.x86_64 works fine, xend is started and virt-{install,manager} are fully functional. OTOH, on F18, where the package in question is libvirt-daemon-driver-xen-0.10.2.2-3.fc18.x86_64, the issue is still there, xend does not start, and thus virt-* don't work.

Is the plan still to have the rawhide packages pushed back to F18, as an update? Please, don't get me wrong, I've zero intention to push, just trying to make sure I understand what will happen. :-)

Thanks and Regards, Dario
Comment 33 Xiao-Long Chen 2013-01-23 11:43:15 EST
(In reply to comment #32)
> Ok, so, despite the bug being filed because a xend issue, the discussion
> took a bunch of different directions, and it all was very useful, as it
> helped uncovering the causes of two other bugs! :-)

It did take a weird turn didn't it? I'm glad all of the bugs are being resolved :)

> However, since it looks like having the libvirt libxl driver in a good
> enough shape in Fedora might require some more time, I wonder whether we at
> least can put the good old libvirt-daemon-driver-xen --the xend based one--
> to work out of the box in F18, as it used to be. This way, provided we tell
> people _not_to_ install libvirt-daemon-driver-libxl (for now), they'll be
> able to use virt-install and virt-manager without issues.

Now that the "xend status" issue is fixed (not sure if it landed in the f18 updates repo yet), I assume that using xend is as simple as:

# systemctl enable xend.service

and using libxl is as simple as:

# systemctl disable xend.service

As far as I'm aware, Fedora just needs to enable the xend service by default and everything should work fine as before.

Cheers
Comment 34 Fedora Update System 2013-01-23 14:28:59 EST
xen-4.2.1-5.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/xen-4.2.1-5.fc18
Comment 35 Michael Young 2013-01-23 18:56:43 EST
*** Bug 902883 has been marked as a duplicate of this bug. ***
Comment 36 Cole Robinson 2013-01-24 18:17:34 EST
> 
> So, wrt this, Cole, I can confirm that, on rawhide, having only
> libvirt-daemon-driver-xen-1.0.1-4.fc19.x86_64 works fine, xend is started
> and virt-{install,manager} are fully functional. OTOH, on F18, where the
> package in question is libvirt-daemon-driver-xen-0.10.2.2-3.fc18.x86_64, the
> issue is still there, xend does not start, and thus virt-* don't work.
> 

Hmm, I don't know why there should be any difference in xend autostarting between rawhide and F18, nor why that would be related to libvirt. Can anyone elaborate?

> Is the plan still to have the rawhide packages pushed back to F18, as an
> update? Please, don't get me wrong, I've zero intention to push, just trying
> to make sure I understand what will happen. :-)
> 

For libvirt we generally do not rebase the latest rawhide version to F18. rawhide tracks latest libvirt releases, F18 stays on the same major release from Alpha onwards. However if the underlying issue is reasonable to backport we can pull it in to f18.


(In reply to comment #33)
> 
> As far as I'm aware, Fedora just needs to enable the xend service by default
> and everything should work fine as before.
> 

To do that, someone needs to file a bug against systemd to have xend start by default. Here's an example request:

https://bugzilla.redhat.com/show_bug.cgi?id=885406
Comment 37 Michael Young 2013-01-24 19:16:36 EST
(In reply to comment #36)
> Hmm, I don't know why there should be any difference in xend autostarting
> between rawhide and F18, nor why that would be related to libvirt. Can
> anyone elaborate?
xen on F18 and rawhide is the same (they use the same code), though rawhide gets updates first because of the way the Fedora update system works. So the difference might have been that the testing on rawhide was with xen-4.2.1-5 where xend status behaves as libvirt expects, but with xen-4.2.1-3 or xen-4.2.1-4 on F18 where it doesn't.
Comment 38 Fedora Update System 2013-02-01 23:32:01 EST
xen-4.2.1-5.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.