Description of problem: When starting guests in sequence, virsh start sometimes fails. If I wait a bit and try again, more will start before it fails again, so eventually I can get all the guests started after a few tries. Note the calls to virsh are running sequence, not in parallel. $ for g in $(virsh list --inactive | grep -o 'kvm-[^ ]*'); do virsh start $g || break; done Domain kvm-tile33-aim1 started Domain kvm-tile33-idle1 started Domain kvm-tile33-mysql1 started Domain kvm-tile33-mysql2 started Domain kvm-tile33-postfix1 started Domain kvm-tile33-specweb1 started Domain kvm-tile33-webclient1 started Domain kvm-tile33-webserver1 started Domain kvm-tile34-aim1 started Domain kvm-tile34-idle1 started Domain kvm-tile34-mysql1 started Domain kvm-tile34-mysql2 started Domain kvm-tile34-postfix1 started Domain kvm-tile34-specweb1 started Domain kvm-tile34-webclient1 started Domain kvm-tile34-webserver1 started Domain kvm-tile35-aim1 started Domain kvm-tile35-idle1 started Domain kvm-tile35-mysql1 started Domain kvm-tile35-mysql2 started Domain kvm-tile35-postfix1 started Domain kvm-tile35-specweb1 started Domain kvm-tile35-webclient1 started Domain kvm-tile35-webserver1 started Domain kvm-tile36-aim1 started Domain kvm-tile36-idle1 started Domain kvm-tile36-mysql1 started Domain kvm-tile36-mysql2 started Domain kvm-tile36-postfix1 started Domain kvm-tile36-specweb1 started Domain kvm-tile36-webclient1 started Domain kvm-tile36-webserver1 started Domain kvm-tile37-aim1 started Domain kvm-tile37-idle1 started Domain kvm-tile37-mysql1 started Domain kvm-tile37-mysql2 started Domain kvm-tile37-postfix1 started Domain kvm-tile37-specweb1 started Domain kvm-tile37-webclient1 started Domain kvm-tile37-webserver1 started Domain kvm-tile38-aim1 started Domain kvm-tile38-idle1 started Domain kvm-tile38-mysql1 started Domain kvm-tile38-mysql2 started Domain kvm-tile38-postfix1 started Domain kvm-tile38-specweb1 started Domain kvm-tile38-webclient1 started Domain kvm-tile38-webserver1 started Domain kvm-tile39-aim1 started Domain kvm-tile39-idle1 started Domain kvm-tile39-mysql1 started Domain kvm-tile39-mysql2 started Domain kvm-tile39-postfix1 started Domain kvm-tile39-specweb1 started Domain kvm-tile39-webclient1 started Domain kvm-tile39-webserver1 started Domain kvm-tile40-aim1 started Domain kvm-tile40-idle1 started Domain kvm-tile40-mysql1 started Domain kvm-tile40-mysql2 started Domain kvm-tile40-postfix1 started Domain kvm-tile40-specweb1 started Domain kvm-tile40-webclient1 started Domain kvm-tile40-webserver1 started Domain kvm-tile41-aim1 started Domain kvm-tile41-idle1 started Domain kvm-tile41-mysql1 started Domain kvm-tile41-mysql2 started Domain kvm-tile41-postfix1 started Domain kvm-tile41-specweb1 started Domain kvm-tile41-webclient1 started Domain kvm-tile41-webserver1 started Domain kvm-tile42-aim1 started Domain kvm-tile42-idle1 started Domain kvm-tile42-mysql1 started Domain kvm-tile42-mysql2 started Domain kvm-tile42-postfix1 started Domain kvm-tile42-specweb1 started Domain kvm-tile42-webclient1 started Domain kvm-tile42-webserver1 started Domain kvm-tile43-aim1 started Domain kvm-tile43-idle1 started Domain kvm-tile43-mysql1 started Domain kvm-tile43-mysql2 started Domain kvm-tile43-postfix1 started Domain kvm-tile43-specweb1 started Domain kvm-tile43-webclient1 started Domain kvm-tile43-webserver1 started Domain kvm-tile44-aim1 started Domain kvm-tile44-idle1 started Domain kvm-tile44-mysql1 started Domain kvm-tile44-mysql2 started Domain kvm-tile44-postfix1 started Domain kvm-tile44-specweb1 started Domain kvm-tile44-webclient1 started Domain kvm-tile44-webserver1 started Domain kvm-tile45-aim1 started Domain kvm-tile45-idle1 started Domain kvm-tile45-mysql1 started Domain kvm-tile45-mysql2 started Domain kvm-tile45-postfix1 started Domain kvm-tile45-specweb1 started Domain kvm-tile45-webclient1 started Domain kvm-tile45-webserver1 started Domain kvm-tile46-aim1 started Domain kvm-tile46-idle1 started Domain kvm-tile46-mysql1 started Domain kvm-tile46-mysql2 started Domain kvm-tile46-postfix1 started Domain kvm-tile46-specweb1 started Domain kvm-tile46-webclient1 started Domain kvm-tile46-webserver1 started Domain kvm-tile47-aim1 started Domain kvm-tile47-idle1 started Domain kvm-tile47-mysql1 started Domain kvm-tile47-mysql2 started Domain kvm-tile47-postfix1 started Domain kvm-tile47-specweb1 started Domain kvm-tile47-webclient1 started Domain kvm-tile47-webserver1 started Domain kvm-tile48-aim1 started Domain kvm-tile48-idle1 started Domain kvm-tile48-mysql1 started Domain kvm-tile48-mysql2 started Domain kvm-tile48-postfix1 started Domain kvm-tile48-specweb1 started Domain kvm-tile48-webclient1 started Domain kvm-tile48-webserver1 started Domain kvm-tile49-aim1 started Domain kvm-tile49-idle1 started Domain kvm-tile49-mysql1 started Domain kvm-tile49-mysql2 started Domain kvm-tile49-postfix1 started Domain kvm-tile49-specweb1 started Domain kvm-tile49-webclient1 started Domain kvm-tile49-webserver1 started Domain kvm-tile50-aim1 started Domain kvm-tile50-idle1 started Domain kvm-tile50-mysql1 started Domain kvm-tile50-mysql2 started Domain kvm-tile50-postfix1 started Domain kvm-tile50-specweb1 started Domain kvm-tile50-webclient1 started Domain kvm-tile50-webserver1 started Domain kvm-tile51-aim1 started Domain kvm-tile51-idle1 started Domain kvm-tile51-mysql1 started Domain kvm-tile51-mysql2 started Domain kvm-tile51-postfix1 started Domain kvm-tile51-specweb1 started Domain kvm-tile51-webclient1 started Domain kvm-tile51-webserver1 started Domain kvm-tile52-aim1 started Domain kvm-tile52-idle1 started Domain kvm-tile52-mysql1 started Domain kvm-tile52-mysql2 started Domain kvm-tile52-postfix1 started Domain kvm-tile52-specweb1 started Domain kvm-tile52-webclient1 started Domain kvm-tile52-webserver1 started Domain kvm-tile53-aim1 started Domain kvm-tile53-idle1 started Domain kvm-tile53-mysql1 started Domain kvm-tile53-mysql2 started Domain kvm-tile53-postfix1 started Domain kvm-tile53-specweb1 started Domain kvm-tile53-webclient1 started Domain kvm-tile53-webserver1 started Domain kvm-tile54-aim1 started Domain kvm-tile54-idle1 started Domain kvm-tile54-mysql1 started Domain kvm-tile54-mysql2 started Domain kvm-tile54-postfix1 started Domain kvm-tile54-specweb1 started Domain kvm-tile54-webclient1 started Domain kvm-tile54-webserver1 started Domain kvm-tile55-aim1 started Domain kvm-tile55-idle1 started Domain kvm-tile55-mysql1 started Domain kvm-tile55-mysql2 started Domain kvm-tile55-postfix1 started Domain kvm-tile55-specweb1 started Domain kvm-tile55-webclient1 started Domain kvm-tile55-webserver1 started Domain kvm-tile56-aim1 started Domain kvm-tile56-idle1 started Domain kvm-tile56-mysql1 started Domain kvm-tile56-mysql2 started Domain kvm-tile56-postfix1 started Domain kvm-tile56-specweb1 started Domain kvm-tile56-webclient1 started Domain kvm-tile56-webserver1 started error: Failed to start domain kvm-tile57-aim1 error: internal error unable to start guest: char device redirected to /dev/pts/389 Version-Release number of selected component (if applicable): libvirt-0.6.3-13.el5.x86_64 libvirt-python-0.6.3-13.el5.x86_64
This is possibly a race condition in the libvirt QEMU startup handling code - IIRC there was one we fixed post-0.6.3 . Can you provide the file /var/log/libvirt/qemu/kvm-tile56-webserver1.log if it is available. (I'm assuming your host itself has sufficient RAM & CPU cycles to be able to cope with all these guests :-) Also could you tell me the total number of - PTYs present in /dev/pts - network interfaces in ifconfig - running guests at the time it fails
Created attachment 351611 [details] debug.txt.1 This isn't from the same run as above. The problem is easy to reproduce so this is from another run. I added this to the error path in my loop: ( PS4=$'\n$ '; set -x ls -l /dev/pts cat /var/log/libvirt/qemu/$g.log ifconfig -a /usr/bin/virsh list ) &> debug.txt
Created attachment 351612 [details] debug.txt.2
Created attachment 351616 [details] debug.txt.3
In each one of these, in the ifconfig output, there's a variation on: vnet212: error fetching interface information: Device not found
I dont believe the vnet error is serious - the QEMU VM just started up & quit, so there's a small race before the vnet device fully disappears from userspace. No signs of any actual QEMU errors in those logs, so I've got to suspect a libvirt race condition in starting up QEMU.
Fixed in libvirt-0.8.2-1.el5
verified on RHEL5.6 libvirt-0.8.2-6.el5.x86_64 kvm
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2011-0060.html