Bug 459912

The second the problem seems to be with dumpxml that it removes the <os /> parameters. While the inactive dump does add them.

Can you attach the complete xend.log file as an attachment - I need to see the logs from the time you first defined the inactive VM.

Oh, what Xen version is this with too...

My Xen log is huge, I have saved you the time: this is the new one:

[2008-08-24 21:32:54 3013] DEBUG (__init__:1094) XendDomainInfo.createDormant({'vcpus_params': {'cap': 0, 'weight': 256}, 'PV_args': ' root=/dev/xvda ro ip=85.17.131.54:1.2.3.4:85.17.131.254:255.255.255.0::eth0:off', 'features': '', 'cpus': [], 'devices': {'4e64b03a-4e4e-2861-bfd7-09500feaf960': ('vif', {'ip': '85.17.131.54', 'mac': '00:16:3E:00:00:2C', 'script': 'vif-bridge', 'uuid': '4e64b03a-4e4e-2861-bfd7-09500feaf960', 'bridge': 'xenbr0'})}, 'vcpu_avail': 1, 'VCPUs_live': 1, 'PV_bootloader': '', 'actions_after_crash': 'destroy', 'vbd_refs': [], 'PV_ramdisk': '', 'is_control_domain': False, 'name_label': 'klant1_libvirt', 'VCPUs_at_startup': 1, 'HVM_boot_params': {}, 'platform': {}, 'PV_kernel': '/usr/lib/xen/boot/linux-2.6.20-xen-r6', 'console_refs': [], 'on_xend_stop': 'ignore', 'memory_static_min': 0, 'HVM_boot_policy': '', 'VCPUs_max': 1, 'memory_static_max': 131072000, 'actions_after_shutdown': 'destroy', 'on_xend_start': 'ignore', 'memory_dynamic_max': 131072000, 'actions_after_suspend': '', 'is_a_template': False, 'memory_dynamic_min': 131072000, 'uuid': 'be2d1430-e12b-49dc-adcf-6a39cc30fbcd', 'shadow_memory': 0, 'PV_bootloader_args': '', 'other_config': {}, 'actions_after_reboot': 'restart', 'vif_refs': ['4e64b03a-4e4e-2861-bfd7-09500feaf960'], 'vtpm_refs': []})
[2008-08-24 21:32:54 3013] DEBUG (__init__:1094) Creating new managed domain: klant1_libvirt
[2008-08-24 21:33:01 3013] INFO (__init__:1094) Domain klant1_libvirt (be2d1430-e12b-49dc-adcf-6a39cc30fbcd) deleted.
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.create(['vm', ['name', 'klant1_libvirt'], ['memory', '125'], ['maxmem', '125'], ['vcpus', '1'], ['uuid', 'be2d1430-e12b-49dc-adcf-6a39cc30fbcd'], ['bootloader'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'destroy'], ['device', ['vif', ['mac', '00:16:3E:00:00:2C'], ['bridge', 'xenbr0'], ['script', 'vif-bridge'], ['ip', '85.17.131.54']]]])
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.constructDomain
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) Balloon: 7203852 KiB free; need 2048; done.
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) Adding Domain: 44
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.initDomain: 44 256
[2008-08-24 21:33:02 3013] ERROR (__init__:1094) Had a bootloader specified, but no disks are bootable
[2008-08-24 21:33:02 3013] ERROR (__init__:1094) VM start failed
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 420, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1705, in _initDomain
    self._configureBootloader()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 2035, in _configureBootloader
    raise VmError(msg)
VmError: Had a bootloader specified, but no disks are bootable
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.destroy: domid=44
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.destroyDomain(44)
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) No device model
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) Releasing devices
[2008-08-24 21:33:02 3013] ERROR (__init__:1094) Domain construction failed
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 87, in create
    vm.start()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 420, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1705, in _initDomain
    self._configureBootloader()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 2035, in _configureBootloader
    raise VmError(msg)
VmError: Had a bootloader specified, but no disks are bootable
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.destroy: domid=None
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) XendDomainInfo.destroyDomain(None)
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) No device model
[2008-08-24 21:33:02 3013] DEBUG (__init__:1094) Releasing devices
[2008-08-24 21:33:02 3013] ERROR (__init__:1094) Request create failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib64/python2.5/site-packages/xen/xend/server/SrvDomainDir.py", line 82, in op_create
    raise XendError("Error creating domain: " + str(ex))
XendError: Error creating domain: Had a bootloader specified, but no disks are bootable


<domain type='xen' id='-1'>
  <name>klant1_libvirt</name>
  <uuid>be2d1430-e12b-49dc-adcf-6a39cc30fbcd</uuid>
  <bootloader/>
  <os>
    <type>linux</type>
    <kernel>/usr/lib/xen/boot/linux-2.6.20-xen-r6</kernel>
    <cmdline> root=/dev/xvda ro ip=85.17.131.54:1.2.3.4:85.17.131.254:255.255.255.0::eth0:off</cmdline>
  </os>
  <memory>128000</memory>
  <vcpu>1</vcpu>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <interface type='bridge'>
      <source bridge='xenbr0'/>
      <target dev='vif-1.0'/>
      <mac address='00:16:3E:00:00:2C'/>
      <ip address='85.17.131.54'/>
      <script path='vif-bridge'/>
    </interface>
    <console type='pty'>
      <target port='0'/>
    </console>
  </devices>
</domain>

xen-3.2.1

Can you confirm that you really are using latest libvirt CVS checkout and xen 3.2.1 ?

I find it hard to believe, because the current libvirt CVS is broken with Xen >= 3.2, and thus won't even connect.

Once I fix the bug in CVS, then when I do try the sequence

 - Define with kernel+initrd
 - Start VM
 - Dumpxml
 - Destroy VM
 - Define with previously dumped XML
 - Start VM

then it correctly boots off the kernel+initrd everytime, even with the empty <bootloader/> tag present.

commit 1b85ae2769c4494d8261d07cca5e1bc8bf1eff47
Author: Daniel Veillard <veillard>
Date:   Tue Jul 22 17:49:53 2008 +0000

    documentation for java bindings
    * docs/java.html docs/java.html.in: added documentation for the
      java bindings
    Daniel

...

[ebuild   R   ] app-emulation/xen-3.2.1

...

xenapi libvirt # virsh list
 Id Name                 State
----------------------------------
  0 Domain-0             running
 36 klant1_gentoo        blocked

Created attachment 315265 [details]
Test script for validating libvirt/xen interaction

Ok, so I've written a script to validate the interaction  between libvirt and Xen. It downloads a F9 xen DomU initrd/kernel, creates a guest called 'demo' with a file backing it, and does a round-trip on the XML file, and saves various log data at every stage.

Please run this script without modification, and create an *attachment* to this bugzilla containing the output file 'work.log' that it generates in /root.

Just to be clear, save the 'test.sh' file in /root, and run it as root. It expects the desired 'virsh' to be in $PATH.

libvir: Xen Daemon error : POST operation failed: xend_post: error from xen daemon: (xend.err '[Errno 12] Cannot allocate memory')
error: Failed to start domain demo


...that is a big strange if I have:

total_memory           : 8191
free_memory            : 7035

http://xen.bot.nu/work.log

Hmm, another XenD memory allocation bug. It might have run out of heap, which is separate from the main 'free_memory' pool. Try shutting down any other VMs you have. Also if this is an x86_64 host, you might try changing the test.sh script to download the x86_64 kernel instead of i386, to remove 32-on-64 instability from the mix.

virsh create /mnt/netapp/users/klant1/imagetest/index.xml 
Domain klant1_imagetest created from /mnt/netapp/users/klant1/imagetest/index.xml


Now explain that...?

<domain type='xen' id='2'>
  <name>klant1_imagetest</name>
  <uuid>3b31af60-cdc1-ad42-3941-9f0642ca8e00</uuid>
  <os>
    <type>linux</type>
    <kernel>/usr/lib/xen/boot/linux-2.6.20-xen-r6</kernel>
    <cmdline> root=/dev/xvda ro ip=85.17.131.8:1.2.3.4:85.17.131.254:255.255.255.0::eth0:off</cmdline>
  </os>
  <memory>262144</memory>
  <vcpu>1</vcpu>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/mnt/test/mydisk.img'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <source bridge='xenbr0'/>
      <target dev='vif2.0'/>
      <mac address='00:16:e3:00:00:01'/>
      <script path='/etc/xen/scripts/vif-bridge'/>
    </interface>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
    </console>
  </devices>
</domain>

Some interesting things:

1) Amd64 kernels do not work
2) My own kernel does not work
3) Changing the memory to 262144 does not work
4) Still the vm above just starts...
5) After debugging I found the real issue... it was the vnc...

Ok, that's not critical then - just remove the <graphics> tag from my test.sh file & re-run it.

virsh start demo
libvir: Xen Daemon error : POST operation failed: xend_post: error from xen daemon: (xend.err 'cannot release un-aquired lock')


Now this is the following error... and I have never able to FIX that one...

http://xen.bot.nu/worknew.log


[2008-08-28 19:58:45 3013] INFO (__init__:1094) Domain demo (10000000-0000-0000-0000-000000000000) deleted.
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) XendDomainInfo.createDormant({'vcpus_params': {'cap': 0, 'weight': 256}, 'PV_args': '', 'features': '', 'cpus': [], 'devices': {'667a743e-c803-a305-4d34-e941985c7432': ('vbd', {'uuid': '667a743e-c803-a305-4d34-e941985c7432', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'xvda', 'uname': 'file:/var/lib/xen/images/demo.img', 'mode': 'w'})}, 'vcpu_avail': 1, 'VCPUs_live': 1, 'PV_bootloader': '', 'actions_after_crash': 'restart', 'vbd_refs': ['667a743e-c803-a305-4d34-e941985c7432'], 'PV_ramdisk': '', 'is_control_domain': False, 'name_label': 'demo', 'VCPUs_at_startup': 1, 'HVM_boot_params': {}, 'platform': {}, 'PV_kernel': '', 'console_refs': [], 'on_xend_stop': 'ignore', 'memory_static_min': 0, 'HVM_boot_policy': '', 'VCPUs_max': 1, 'memory_static_max': 265289728, 'actions_after_shutdown': 'destroy', 'on_xend_start': 'ignore', 'memory_dynamic_max': 265289728, 'actions_after_suspend': '', 'is_a_template': False, 'memory_dynamic_min': 265289728, 'uuid': '10000000-0000-0000-0000-000000000000', 'shadow_memory': 0, 'PV_bootloader_args': '', 'other_config': {}, 'actions_after_reboot': 'restart', 'vif_refs': [], 'vtpm_refs': []})
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) Creating new managed domain: demo
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) Starting domain demo False
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) XendDomainInfo.constructDomain
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) Balloon: 7203852 KiB free; need 2048; done.
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) Adding Domain: 57
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) XendDomainInfo.initDomain: 57 256
[2008-08-28 19:58:45 3013] ERROR (__init__:1094) VM start failed
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 420, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1705, in _initDomain
    self._configureBootloader()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 2064, in _configureBootloader
    bootloader_args, kernel, ramdisk, args)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendBootloader.py", line 83, in bootloader
    (child, m2) = pty.fork()
  File "/usr/lib64/python2.5/pty.py", line 108, in fork
    pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) XendDomainInfo.destroy: domid=57
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) XendDomainInfo.destroyDomain(57)
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) No device model
[2008-08-28 19:58:45 3013] DEBUG (__init__:1094) Releasing devices
[2008-08-28 19:58:45 3013] ERROR (__init__:1094) Request start failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/web/SrvBase.py", line 85, in perform
    return op_method(op, req)
  File "/usr/lib64/python2.5/site-packages/xen/xend/server/SrvDomain.py", line 77, in op_start
    return self.xd.domain_start(self.dom.getName(), paused)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomain.py", line 1030, in domain_start
    self.domains_lock.release()
  File "/usr/lib64/python2.5/threading.py", line 116, in release
    raise RuntimeError("cannot release un-aquired lock")
RuntimeError: cannot release un-aquired lock



I think this must give enough info on the matters ;)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      3013  0.0 47.9 511116 251232 ?       Sl   Jul26   4:49 python /usr/sbin/xend start

Things like that always make me think that I *really* want a memleak free version asap.

I'm able run the demo thing now up to this point:

libvir: Xen Daemon error : POST operation failed: xend_post: error from xen daemon: (xend.err "Boot loader didn't return any data!")
error: Failed to start domain demo

http://xen.bot.nu/worknew2.log

Yes, this is sufficient info now. The worknew2.log confirms that you are *not* in fact using libvirt from CVS

The XML it shows for the running VM is

  <bootloader/>
  <os>
    <type>linux</type>
    <kernel>/root/vmlinuz</kernel>
    <initrd>/root/initrd.img</initrd>
  </os>
  <memory>259072</memory>


Whereas latest CVS code has elements in a different order

  <memory>259072</memory>
  <currentMemory>259072</currentMemory>
  <vcpu>1</vcpu>
  <bootloader></bootloader>
  <os>
    <type>linux</type>
    <kernel>/root/vmlinuz</kernel>
    <initrd>/root/initrd.img</initrd>
  </os>

Can you update to latest CVS, and make sure you haven't got and old copies of virsh, or libvirt.so lieing around.

You'll likely need to apply this patch to latest CVS too

https://www.redhat.com/archives/libvir-list/2008-August/msg00725.html

(In reply to comment #18)
> Yes, this is sufficient info now. The worknew2.log confirms that you are *not*
> in fact using libvirt from CVS

> Can you update to latest CVS, and make sure you haven't got and old copies of
> virsh, or libvirt.so lieing around.
> 
> You'll likely need to apply this patch to latest CVS too
> 
> https://www.redhat.com/archives/libvir-list/2008-August/msg00725.html


... it seems the problem was in my git, what do you advice? Because updating my libvirt will break my Xen 3.2?

If you can't risk regressions, then try and reproduce the problem on a different  machine using latest libvirt CVS checkout. When I run the test.sh script on with latest libvirt, everything works correctly on xen 3.2.x. So unless you can reproduce your problems with current libvirt CVS, this bug can be closed NOTABUG.

xenapi cherokee # libvirtd -l
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory
libvir: Xen Daemon error : invalid connection pointer in xenDaemonNodeGetTopology
libvir: Xen error : out of memory

That's why you need this patch against libvirt CVS:

https://www.redhat.com/archives/libvir-list/2008-August/msg00725.html

http://xen.bot.nu/finallog.log

Is the output. A patch by Novell to fix the memory leak in xen-tools (metrics related) doesn't work. Does Red Hat currently have any patches to prevent Xen leaking?

Ok, so that logfile shows that the '<bootloader>', '<kernel>' and '<initrd>' tags are all being correctly handled, and kernel is being honoured by xend, even when <bootloader/> is set.

I'm not aware of any patches for Xen memory leaks, unless there's something upstream I missed.

Trivial fix... <bootloader> </bootloader> -> <bootloader />

Ok this 'upgrade' broke everything related to consoles...

<domain type='xen' id='66'>
  <name>klant1_monetdb</name>
  <uuid>a850ec08-7c7d-2133-1f0e-0b56a793f858</uuid>
  <memory>4194304</memory>
  <currentMemory>4194304</currentMemory>
  <vcpu>1</vcpu>
  <bootloader></bootloader>
  <os>
    <type>linux</type>
    <kernel>/usr/lib/xen/boot/linux-2.6.20-xen-r6</kernel>
    <cmdline> root=/dev/xvda ro ip=85.17.131.51:1.2.3.4:85.17.131.254:255.255.255.0::eth0:off</cmdline>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/mnt/images/klant1_monetdb_1'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/mnt/images/klant1_osm'/>
      <target dev='xvdb' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:00:00:29'/>
      <source bridge='xenbr0'/>
      <target dev='vif66.0'/>
    </interface>
    <console type='pty' tty='/dev/pts/12'>
      <source path='/dev/pts/12'/>
      <target port='0'/>
    </console>
  </devices>
</domain>

xenapi cherokee # xm console klant1_monetdb
xenconsole: Could not open tty `/dev/pts/12': No such file or directory


My other host with the 'newer' libvirt cannot get the console, when I switch back the older libvirt I am able to boot the systems... (but the bootloader issue is back)

One bug per ticket please. Don't re-open bugs for different problems.