Bug 867246

Summary: [LXC] A running guest will be stopped after restarting libvirtd service
Product: Red Hat Enterprise Linux 6 Reporter: Alex Jia <ajia>
Component: libvirtAssignee: Daniel Berrangé <berrange>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: acathrow, dallan, dyasny, dyuan, lsu, mzhan, rwu, xuzhang, ydu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.10.2-10.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 07:10:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 881827    

Description Alex Jia 2012-10-17 06:51:43 UTC
Description of problem:
A running lxc guest will be stopped after restarting libvirtd service.

Version-Release number of selected component (if applicable):
# rpm -q libvirt
libvirt-0.10.2-4.el6.x86_64


How reproducible:
sometimes

Steps to Reproduce:
1. virsh -c lxc:// start toy
2. virsh -c lxc:// list
3. service libvirtd restart
4. repeat step 2
  
Actual results:

# virsh -c lxc:// start toy
Domain toy started

# virsh -c lxc:// list
 Id    Name                           State
----------------------------------------------------
 23987 toy                            running

# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]

# virsh -c lxc:// list --all
 Id    Name                           State
----------------------------------------------------
 -     toy                            shut off


Expected results:
the guest should be running status.

Additional info:

# virsh -c lxc:// dumpxml toy
<domain type='lxc'>
  <name>toy</name>
  <uuid>bb428983-cb9f-4702-0f8d-7d4e143d9aad</uuid>
  <memory unit='KiB'>500000</memory>
  <currentMemory unit='KiB'>500000</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/bin/sh</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

Comment 2 Daniel Berrangé 2012-11-02 15:32:11 UTC
I can only reproduce this some of the time. When it does occur though I see the following in /var/log/libvirt/lxc/$GUEST.log

2012-11-02 15:30:24.416+0000: 6378: error : virNetSocketReadWire:1184 : End of file while reading data: Input/output error
2012-11-02 15:30:24.423+0000: 6378: error : virNetServerAddClient:270 : Too many active clients (1), dropping connection from 127.0.0.1;0

For some reason libvirt_lxc process is not handling the EOF correctly, causing it to refuse the libvirtd connection. Libvirtd then kills off the container.

Comment 7 Daniel Berrangé 2012-11-22 10:57:10 UTC
Upstream fix was

commit f33e43c2353cc10065b8a5f9965d5a57cfe8991f
Author: Daniel P. Berrange <berrange>
Date:   Fri Nov 16 09:11:23 2012 +0000

    Use virNetServerRun instead of custom main loop
    
    The LXC controller code currently directly invokes the
    libvirt main loop code. The problem is that this misses
    the cleanup of virNetServerClient connections that
    virNetServerRun takes care of.
    
    The result is that when libvirtd is stopped, the
    libvirt_lxc controller process gets stuck in a I/O loop.
    When libvirtd is then started again, it fails to connect
    to the controller and thus kills off the entire domain.
    
    Signed-off-by: Daniel P. Berrange <berrange>

Comment 10 yanbing du 2012-11-28 02:17:07 UTC
Verify this bug with libvirt-0.10.2-10.el6.x86_64.
Repeat restart libvirtd more than 50 times, the lxc guest still running.

Comment 11 errata-xmlrpc 2013-02-21 07:10:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html