Red Hat Bugzilla – Bug 228511
xen domain auto startup does not work reliable
Last modified: 2009-05-01 16:26:27 EDT
Description of problem:
When creating many domains and doing reboots, not all domains are started.
message from /var/log/xen/xend.log
[2007-02-13 08:41:29 xend 2209] INFO (image:138) buildDomain os=linux dom=7 vcpus=1
[2007-02-13 08:41:35 xend 2209] INFO (image:214) configuring linux guest
[2007-02-13 08:41:37 xend 2209] INFO (image:138) buildDomain os=linux dom=8 vcpus=1
[2007-02-13 08:41:39 xend 2209] INFO (XendDomain:370) Domain vm_5 (8) unpaused.
[2007-02-13 08:41:39 xend.XendDomainInfo 2209] WARNING (XendDomainInfo:875)
Domain has crashed: name=vm_4 id=7.
[2007-02-13 08:41:40 xend.XendDomainInfo 2209] ERROR (XendDomainInfo:1661) VM
vm_4 restarting too fast (13.252752 seconds since the last restart). Refusing
to restart to avoid loops.
Version-Release number of selected component (if applicable):
Version=5 beta 2
CPU=Intel(R) Pentium(R) 4 CPU 3.00GHz (no HT/SMP enabled)
everytime some VMs are missing.
Steps to Reproduce:
- ks install server using base + @virtualisation packages
- ks install 9 guests using virt-install
- sed -ie 's/XENDOMAINS_SAVE=.*/XENDOMAINS_SAVE=/' /etc/sysconfig/xendomains #
this does a shutdown instead of suspend
- ln -s /etc/xen/MY_VMS_* /etc/xen/auto
- do reboot
- after reboot verify that all VMs are started
I did 13 reboots. This is how often each VM came up automatically.
So vm_7 only started 5 times at 13 tries.
all machines are started at every reboot
fc6 with 2.6.19 kernel has similar behaviour.
The "Domain has crashed:" entries
Hmm, this is a little worrying - if it can't deal with multiple VMs starting in
very quick succession it sounds like there is some race condition/scalability
issue hiding in either HV or the XenD stack.
Can you reproduce this again & capture the output of 'xm dmesg' once booting has
completed - this will hopefull show if there are any hypervisor issues being
reported. Also can you attach the full /var/log/xen/xend.log,
/var/log/xen/xend-debug.log, /var/log/xen/xen-hotplug.log and finally if any are
HVM guests, also the qemu-dm-*.log files
Finally, can you attach the /etc/xen config file for at least one of the guests
- if they are all basically the same config one is sufficient - if every VM is
different upload a representative set.
Created attachment 151099 [details]
tgz logs and config
I am using only RHEL5 xen-guests, no HVM. (see first post)
[root@rhrc1s1 x]# crontab -l
01,31 * * * * /usr/sbin/xm list| logger -t XEN1
14,44 * * * * /usr/sbin/xm list| logger -t XEN2
15,45 * * * * /sbin/reboot
[root@rhrc1s1 x]# uname -a
Linux rhrc1s1 2.6.18-8.el5xen #1 SMP Fri Jan 26 14:42:21 EST 2007 i686 i686
xm dmesg >var/log/xen/xm.dmesg.out
The tgz file contains
We did some work in 5.1 to make this less likely to happen, but I'm not sure if
it is completely fixed. Is this still a problem?
Well, I have tried it using my SRPMS that can be found at http://people.redhat.com/minovotn/xen and I found no problem, I booted 9 domains total and all the domains booted correctly when testing on my box. The configuration was 4 PV and 5 FV machines...
I am unable to reproduce the problem with RH 5.3 after setting dom0_mem=512M in grub.conf. My tests ran 20 256 RH5.3 64 bit udoms.
Please set the state to fixed.
OK, thanks for the testing! Will close as FIXED.