Bug 857341

Summary: fail to start lxc domain
Product: Red Hat Enterprise Linux 6 Reporter: yanbing du <ydu>
Component: libvirtAssignee: Daniel Berrangé <berrange>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: acathrow, ajia, berrange, dyasny, dyuan, gsun, mzhan, rwu, yupzhang
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.10.2-1.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 07:23:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yanbing du 2012-09-14 07:16:30 UTC
Description of problem:
Fail to start LXC guest.

Version-Release number of selected component (if applicable):
libvirt-0.10.1-2.el6.x86_64
kernel-2.6.32-303.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Define a LXC guest
# cat toy.xml
<domain type='lxc'>
  <name>toy</name>
  <uuid>386f5b25-43ee-9d62-4ce2-62c3809e47c1</uuid>
  <memory>500000</memory>
  <currentMemory>500000</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/bin/sh</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <interface type='network'>
      <source network='default'/>
    </interface>
    <console type='pty'>
      <target port='0'/>
    </console>
  </devices>
</domain>

# virsh -c lxc:/// define toy.xml
Domain toy defined from toy.xml


2. Start the LXC guest
 # start toy
error: Failed to start domain toy
error: internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=386f5b25-43ee-9d62-4ce2-62c3809e47c1 LIBVIRT_LXC_UUID=386f5b25-43ee-9d62-4ce2-62c3809e47c1 LIBVIRT_LXC_NAME=toy /bin/sh
2012-09-14 07:06:00.613+0000: 1: info : libvirt version: 0.10.1, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-09-13-00:00:13, x86-009.build.bos.redhat.com)
2012-09-14 07:06:00.613+0000: 1: error : lxcContainerMountBasicFS:560 : Failed to mount /selinux on /selinux type selinuxfs: Device or resource busy
2012-09-14 07:06:00.620+0000: 6317: info : libvirt version: 0.10.1, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-09-13-00:00:13, x86-009.build.bos.redhat.com)
2012-09-14 07:06:00.620+0000: 6317: error : virLXCControllerRun:1420 : Unable to send container continue message: Broken pipe


  
Actual results:
Fail to start

Expected results:
libvirt-0.10.1-2.e

Additional info:
libvirtd.log
------
2012-09-14 07:06:00.646+0000: 6169: info : libvirt version: 0.10.1, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-09-13-00:00:13, x86-009.build.bos.redhat.com)
2012-09-14 07:06:00.646+0000: 6169: error : virNetSocketReadWire:1176 : Cannot recv data: Connection reset by peer
2012-09-14 07:06:00.745+0000: 6174: error : virLXCProcessStart:1145 : internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=386f5b25-43ee-9d62-4ce2-62c3809e47c1 LIBVIRT_LXC_UUID=386f5b25-43ee-9d62-4ce2-62c3809e47c1 LIBVIRT_LXC_NAME=toy /bin/sh
2012-09-14 07:06:00.613+0000: 1: info : libvirt version: 0.10.1, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-09-13-00:00:13, x86-009.build.bos.redhat.com)
2012-09-14 07:06:00.613+0000: 1: error : lxcContainerMountBasicFS:560 : Failed to mount /selinux on /selinux type selinuxfs: Device or resource busy
2012-09-14 07:06:00.620+0000: 6317: info : libvirt version: 0.10.1, package: 2.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2012-09-13-00:00:13, x86-009.build.bos.redhat.com)
2012-09-14 07:06:00.620+0000: 6317: error : virLXCControllerRun:1420 : Unable to send container continue message: Broken pipe

2012-09-14 07:06:00.755+0000: 6174: error : virNetDevSetOnline:538 : Cannot get interface flags on 'veth0': No such device
2012-09-14 07:06:00.835+0000: 6169: error : virLXCProcessStop:701 : internal error Invalid PID -1 for container
2012-09-14 07:06:00.835+0000: 6169: error : virLXCProcessStop:701 : internal error Invalid PID -1 for container

Comment 2 yanbing du 2012-09-14 08:06:18 UTC
As it works on libvirt-0.9.10-21.el6.x86_64, so mark this bug a regression bug.

Comment 4 Daniel Berrangé 2012-09-14 16:37:00 UTC
Ok this is a very peculiar error that I've never seen before

> 2012-09-14 07:06:00.613+0000: 1: error : lxcContainerMountBasicFS:560 : Failed > to mount /selinux on /selinux type selinuxfs: Device or resource busy

Comment 5 Daniel Berrangé 2012-09-17 16:27:33 UTC
I've been able to reproduce this locally on RHEL-6 box, but not on rawhide. So there must be some difference, perhaps in the libselinux behaviour

Comment 6 Daniel Berrangé 2012-09-21 10:18:34 UTC
Fixed upstream

commit c15d893252e8000d26a33813027edde38e1b6912
Author: Daniel P. Berrange <berrange>
Date:   Tue Sep 18 12:25:56 2012 +0100

    Ensure existing selinux mount is removed before mounting new one in LXC
    
    Some kernel versions (at least RHEL-6 2.6.32) do not let you over-mount
    an existing selinuxfs instance with a new one. Thus we must unmount the
    existing instance inside our namespace.

Comment 7 Jiri Denemark 2012-09-21 10:21:23 UTC
This issue is now fixed upstream by v0.10.2-rc2-2-gc15d893:

commit c15d893252e8000d26a33813027edde38e1b6912
Author: Daniel P. Berrange <berrange>
Date:   Tue Sep 18 12:25:56 2012 +0100

    Ensure existing selinux mount is removed before mounting new one
    in LXC
    
    Some kernel versions (at least RHEL-6 2.6.32) do not let you
    over-mount an existing selinuxfs instance with a new one. Thus we
    must unmount the existing instance inside our namespace.

Comment 8 Wayne Sun 2012-09-25 03:34:40 UTC
pkgs:
libvirt-0.10.2-1.el6.x86_64
kernel-2.6.32-279.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.313.el6.x86_64

steps:
Following steps in description:
# virsh -c lxc:/// list --all
 Id    Name                           State
----------------------------------------------------
 -     vm1                            shut off

# virsh -c lxc:///
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # start vm1
Domain vm1 started

virsh # list
 Id    Name                           State
----------------------------------------------------
 30672 vm1                            running

virsh # dumpxml vm1
<domain type='lxc' id='30672'>
  <name>vm1</name>
  <uuid>386f5b25-43ee-9d62-4ce2-58c3809e47c1</uuid>
  <memory unit='KiB'>500000</memory>
  <currentMemory unit='KiB'>500000</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/bin/sh</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <interface type='network'>
      <mac address='52:54:00:f2:2c:ac'/>
      <source network='default'/>
      <target dev='veth0'/>
    </interface>
    <console type='pty' tty='/dev/pts/8'>
      <source path='/dev/pts/8'/>
      <target type='lxc' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
  <seclabel type='none'/>
</domain>

virsh # destroy vm1
Domain vm1 destroyed


This is working now.

Comment 9 errata-xmlrpc 2013-02-21 07:23:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html