Bug 511241 - sequential virsh start chokes
sequential virsh start chokes
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: ---
Assigned To: Daniel Veillard
Virtualization Bugs
:
Depends On:
Blocks: 562684
  Show dependency treegraph
 
Reported: 2009-07-14 08:00 EDT by Aron Griffis
Modified: 2011-01-13 17:51 EST (History)
9 users (show)

See Also:
Fixed In Version: libvirt-0.8.2-1.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-01-13 17:51:53 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
debug.txt.1 (69.12 KB, text/plain)
2009-07-14 11:40 EDT, Aron Griffis
no flags Details
debug.txt.2 (94.02 KB, text/plain)
2009-07-14 11:42 EDT, Aron Griffis
no flags Details
debug.txt.3 (103.45 KB, text/plain)
2009-07-14 11:46 EDT, Aron Griffis
no flags Details

  None (edit)
Description Aron Griffis 2009-07-14 08:00:57 EDT
Description of problem:
When starting guests in sequence, virsh start sometimes fails.  If I wait a bit and try again, more will start before it fails again, so eventually I can get all the guests started after a few tries.  Note the calls to virsh are running sequence, not in parallel.

$ for g in $(virsh list --inactive | grep -o 'kvm-[^ ]*'); do virsh start $g || break; done
Domain kvm-tile33-aim1 started
Domain kvm-tile33-idle1 started
Domain kvm-tile33-mysql1 started
Domain kvm-tile33-mysql2 started
Domain kvm-tile33-postfix1 started
Domain kvm-tile33-specweb1 started
Domain kvm-tile33-webclient1 started
Domain kvm-tile33-webserver1 started
Domain kvm-tile34-aim1 started
Domain kvm-tile34-idle1 started
Domain kvm-tile34-mysql1 started
Domain kvm-tile34-mysql2 started
Domain kvm-tile34-postfix1 started
Domain kvm-tile34-specweb1 started
Domain kvm-tile34-webclient1 started
Domain kvm-tile34-webserver1 started
Domain kvm-tile35-aim1 started
Domain kvm-tile35-idle1 started
Domain kvm-tile35-mysql1 started
Domain kvm-tile35-mysql2 started
Domain kvm-tile35-postfix1 started
Domain kvm-tile35-specweb1 started
Domain kvm-tile35-webclient1 started
Domain kvm-tile35-webserver1 started
Domain kvm-tile36-aim1 started
Domain kvm-tile36-idle1 started
Domain kvm-tile36-mysql1 started
Domain kvm-tile36-mysql2 started
Domain kvm-tile36-postfix1 started
Domain kvm-tile36-specweb1 started
Domain kvm-tile36-webclient1 started
Domain kvm-tile36-webserver1 started
Domain kvm-tile37-aim1 started
Domain kvm-tile37-idle1 started
Domain kvm-tile37-mysql1 started
Domain kvm-tile37-mysql2 started
Domain kvm-tile37-postfix1 started
Domain kvm-tile37-specweb1 started
Domain kvm-tile37-webclient1 started
Domain kvm-tile37-webserver1 started
Domain kvm-tile38-aim1 started
Domain kvm-tile38-idle1 started
Domain kvm-tile38-mysql1 started
Domain kvm-tile38-mysql2 started
Domain kvm-tile38-postfix1 started
Domain kvm-tile38-specweb1 started
Domain kvm-tile38-webclient1 started
Domain kvm-tile38-webserver1 started
Domain kvm-tile39-aim1 started
Domain kvm-tile39-idle1 started
Domain kvm-tile39-mysql1 started
Domain kvm-tile39-mysql2 started
Domain kvm-tile39-postfix1 started
Domain kvm-tile39-specweb1 started
Domain kvm-tile39-webclient1 started
Domain kvm-tile39-webserver1 started
Domain kvm-tile40-aim1 started
Domain kvm-tile40-idle1 started
Domain kvm-tile40-mysql1 started
Domain kvm-tile40-mysql2 started
Domain kvm-tile40-postfix1 started
Domain kvm-tile40-specweb1 started
Domain kvm-tile40-webclient1 started
Domain kvm-tile40-webserver1 started
Domain kvm-tile41-aim1 started
Domain kvm-tile41-idle1 started
Domain kvm-tile41-mysql1 started
Domain kvm-tile41-mysql2 started
Domain kvm-tile41-postfix1 started
Domain kvm-tile41-specweb1 started
Domain kvm-tile41-webclient1 started
Domain kvm-tile41-webserver1 started
Domain kvm-tile42-aim1 started
Domain kvm-tile42-idle1 started
Domain kvm-tile42-mysql1 started
Domain kvm-tile42-mysql2 started
Domain kvm-tile42-postfix1 started
Domain kvm-tile42-specweb1 started
Domain kvm-tile42-webclient1 started
Domain kvm-tile42-webserver1 started
Domain kvm-tile43-aim1 started
Domain kvm-tile43-idle1 started
Domain kvm-tile43-mysql1 started
Domain kvm-tile43-mysql2 started
Domain kvm-tile43-postfix1 started
Domain kvm-tile43-specweb1 started
Domain kvm-tile43-webclient1 started
Domain kvm-tile43-webserver1 started
Domain kvm-tile44-aim1 started
Domain kvm-tile44-idle1 started
Domain kvm-tile44-mysql1 started
Domain kvm-tile44-mysql2 started
Domain kvm-tile44-postfix1 started
Domain kvm-tile44-specweb1 started
Domain kvm-tile44-webclient1 started
Domain kvm-tile44-webserver1 started
Domain kvm-tile45-aim1 started
Domain kvm-tile45-idle1 started
Domain kvm-tile45-mysql1 started
Domain kvm-tile45-mysql2 started
Domain kvm-tile45-postfix1 started
Domain kvm-tile45-specweb1 started
Domain kvm-tile45-webclient1 started
Domain kvm-tile45-webserver1 started
Domain kvm-tile46-aim1 started
Domain kvm-tile46-idle1 started
Domain kvm-tile46-mysql1 started
Domain kvm-tile46-mysql2 started
Domain kvm-tile46-postfix1 started
Domain kvm-tile46-specweb1 started
Domain kvm-tile46-webclient1 started
Domain kvm-tile46-webserver1 started
Domain kvm-tile47-aim1 started
Domain kvm-tile47-idle1 started
Domain kvm-tile47-mysql1 started
Domain kvm-tile47-mysql2 started
Domain kvm-tile47-postfix1 started
Domain kvm-tile47-specweb1 started
Domain kvm-tile47-webclient1 started
Domain kvm-tile47-webserver1 started
Domain kvm-tile48-aim1 started
Domain kvm-tile48-idle1 started
Domain kvm-tile48-mysql1 started
Domain kvm-tile48-mysql2 started
Domain kvm-tile48-postfix1 started
Domain kvm-tile48-specweb1 started
Domain kvm-tile48-webclient1 started
Domain kvm-tile48-webserver1 started
Domain kvm-tile49-aim1 started
Domain kvm-tile49-idle1 started
Domain kvm-tile49-mysql1 started
Domain kvm-tile49-mysql2 started
Domain kvm-tile49-postfix1 started
Domain kvm-tile49-specweb1 started
Domain kvm-tile49-webclient1 started
Domain kvm-tile49-webserver1 started
Domain kvm-tile50-aim1 started
Domain kvm-tile50-idle1 started
Domain kvm-tile50-mysql1 started
Domain kvm-tile50-mysql2 started
Domain kvm-tile50-postfix1 started
Domain kvm-tile50-specweb1 started
Domain kvm-tile50-webclient1 started
Domain kvm-tile50-webserver1 started
Domain kvm-tile51-aim1 started
Domain kvm-tile51-idle1 started
Domain kvm-tile51-mysql1 started
Domain kvm-tile51-mysql2 started
Domain kvm-tile51-postfix1 started
Domain kvm-tile51-specweb1 started
Domain kvm-tile51-webclient1 started
Domain kvm-tile51-webserver1 started
Domain kvm-tile52-aim1 started
Domain kvm-tile52-idle1 started
Domain kvm-tile52-mysql1 started
Domain kvm-tile52-mysql2 started
Domain kvm-tile52-postfix1 started
Domain kvm-tile52-specweb1 started
Domain kvm-tile52-webclient1 started
Domain kvm-tile52-webserver1 started
Domain kvm-tile53-aim1 started
Domain kvm-tile53-idle1 started
Domain kvm-tile53-mysql1 started
Domain kvm-tile53-mysql2 started
Domain kvm-tile53-postfix1 started
Domain kvm-tile53-specweb1 started
Domain kvm-tile53-webclient1 started
Domain kvm-tile53-webserver1 started
Domain kvm-tile54-aim1 started
Domain kvm-tile54-idle1 started
Domain kvm-tile54-mysql1 started
Domain kvm-tile54-mysql2 started
Domain kvm-tile54-postfix1 started
Domain kvm-tile54-specweb1 started
Domain kvm-tile54-webclient1 started
Domain kvm-tile54-webserver1 started
Domain kvm-tile55-aim1 started
Domain kvm-tile55-idle1 started
Domain kvm-tile55-mysql1 started
Domain kvm-tile55-mysql2 started
Domain kvm-tile55-postfix1 started
Domain kvm-tile55-specweb1 started
Domain kvm-tile55-webclient1 started
Domain kvm-tile55-webserver1 started
Domain kvm-tile56-aim1 started
Domain kvm-tile56-idle1 started
Domain kvm-tile56-mysql1 started
Domain kvm-tile56-mysql2 started
Domain kvm-tile56-postfix1 started
Domain kvm-tile56-specweb1 started
Domain kvm-tile56-webclient1 started
Domain kvm-tile56-webserver1 started
error: Failed to start domain kvm-tile57-aim1
error: internal error unable to start guest: char device redirected to
/dev/pts/389


Version-Release number of selected component (if applicable):
libvirt-0.6.3-13.el5.x86_64
libvirt-python-0.6.3-13.el5.x86_64
Comment 1 Daniel Berrange 2009-07-14 08:11:17 EDT
This is possibly a race condition in the libvirt QEMU startup handling code - IIRC there was one we fixed post-0.6.3 . Can you provide the file

 /var/log/libvirt/qemu/kvm-tile56-webserver1.log

if it is available.

(I'm assuming your host itself has sufficient RAM & CPU cycles to be able to cope with all these guests :-)

Also could you tell me the total number of

 - PTYs present in /dev/pts
 - network interfaces in ifconfig
 - running guests

at the time it fails
Comment 2 Aron Griffis 2009-07-14 11:40:37 EDT
Created attachment 351611 [details]
debug.txt.1

This isn't from the same run as above. The problem is easy to reproduce so this is from another run.  I added this to the error path in my loop:

    (   PS4=$'\n$ '; set -x
        ls -l /dev/pts
        cat /var/log/libvirt/qemu/$g.log 
        ifconfig -a
        /usr/bin/virsh list
    ) &> debug.txt
Comment 3 Aron Griffis 2009-07-14 11:42:09 EDT
Created attachment 351612 [details]
debug.txt.2
Comment 4 Aron Griffis 2009-07-14 11:46:23 EDT
Created attachment 351616 [details]
debug.txt.3
Comment 5 Aron Griffis 2009-07-14 11:49:23 EDT
In each one of these, in the ifconfig output, there's a variation on:

vnet212: error fetching interface information: Device not found
Comment 6 Daniel Berrange 2009-07-14 13:15:47 EDT
I dont believe the vnet error is serious - the QEMU VM just started up & quit, so there's a small race before the vnet device fully disappears from userspace.

No signs of any actual QEMU errors in those logs, so I've got to suspect a libvirt race condition in starting up QEMU.
Comment 11 Jiri Denemark 2010-09-02 07:59:54 EDT
Fixed in libvirt-0.8.2-1.el5
Comment 13 weizhang 2010-10-28 23:30:22 EDT
verified on RHEL5.6 libvirt-0.8.2-6.el5.x86_64 kvm
Comment 15 errata-xmlrpc 2011-01-13 17:51:53 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2011-0060.html

Note You need to log in before you can comment on or make changes to this bug.