Bug 250206

Were you running libvirt 0.3.0 before or did you upgraded directly from
0.2.3 ? Can you make sure the libvirt daemon get properly restarted (you
may need to kill an old libvirt_qemu one).
Can you also provide the version of qemu, kernel to be sure we have a 
matching environment, and attach an XML description for the guest failing
to start (use virsh dumpxml guestname).

  thanks !

Daniel

Upgraded from libvirt 0.3.0 to this newer version. (I regularly run "yum update"
every few days)

Made sure after /etc/init.d/libvirtd stop, there were no "libvirtd --daemon"
process. Didn't help though.

# uname -a
Linux desktop 2.6.22.1-33.fc7 #1 SMP Mon Jul 23 16:59:15 EDT 2007 x86_64 x86_64
x86_64 GNU/Linux

# rpm -qa|egrep 'kernel|qemu|kvm'
kernel-2.6.22.1-33.fc7
kvm-24-1
qemu-0.9.0-2.fc7

The virsh tool fails with error messages:
# virsh dumpxml fedora7
error: failed to connect to the hypervisor
error: no valid connection

(I've never used virsh before, so I don't know whether it's a new regression.)

Thanks

virsh is the command associated to libvirt. When using QEmu you need to
pass an option, as root run by default it tries to use Xen:

virsh -c qemu://system/ dumpxml 

Daniel

(Sorry mate, still looks like no go)

# virsh -c qemu://system/ dumpxml fedora7
libvir: Remote error : Cannot access CA certificate '/etc/pki/CA/cacert.pem': No
such file or directory (2)
error: failed to connect to the hypervisor
error: no valid connection

I'm very happy to provide various xml configuration files for inspection, if
needed (the whole /etc/libvirt/ if needed)

Thanks

Actually, I managed to work out how to use virsh:
Here's the impacted system:
# virsh -c qemu:///system
virsh # dumpxml fedora1
<domain type='kvm'>
  <name>fedora1</name>
  <uuid>772e1c9c-aab9-1e65-e69f-0272445f5b0e</uuid>
  <memory>262144</memory>
  <currentMemory>262144</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/vm/guests/fedora1'/>
      <target dev='hda'/>
    </disk>
    <interface type='network'>
      <mac address='00:16:3e:2b:42:72'/>
      <source network='net1'/>
      <target dev='vnet%d'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' listen='127.0.0.1'/>
  </devices>
</domain>

Thanks

hum, <target dev='vnet%d'/> the %d looks suspicious to me.
Can you try to chnage this to a constant value ?

virsh -c qemu://system/ dumpxml fedora7 > /tmp/fedora7.xml

edit /tmp/fedora7.xml replacing vnet%d with vnet1 for example

virsh -c qemu://system/ undefine fedora7
virsh -c qemu://system/ define /tmp/fedora7.xml

does the guest starts then ?

Daniel

Followed the above instructions using virsh:
1. Dumped the fedora1's xml file
2. Undefined fedora1
3. Replaced vnet%d with vnet1 in the dumped xml file
4. Defined fedora1 with the edited xml file
5. Restarted /etc/init.d/libvirtd
(Observed no vnet0 interface getting created in ifconfig; bad sign)
6. Tried to bring up the guest fedora1; failed with this error:

virDomainCreate() failed Failed to add tap interface 'vnet1' to bridge 'vnet0':
No such device

Details:

Unable to start virtual machine '<class 'libvirt.libvirtError'>
virDomainCreate() failed Failed to add tap interface 'vnet1' to bridge 'vnet0' :
No such device
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/console.py", line 348, in control_vm_run
    self.vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 361, in startup
    self.vm.create()
  File "/usr/lib64/python2.5/site-packages/libvirt.py", line 228, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: virDomainCreate() failed Failed to add tap interface 'vnet1' to
bridge 'vnet0' : No such device
'

(I think whenever /etc/init.d/libvirtd is started, it used to create one vnetN
interface per virtual network; that's not happening now. That'd explain why the
above message complaining when trying to bridge vnet1 to vnet0 that vnet0 is
unavailable. My theory.)

Thanks

Hmm guys, the URI should be qemu:///system

Correct. In my case it is somewhat like this:

# virsh -c qemu:///system dumpxml fedora1 > /tmp/fedora1.xml
< alters /tmp/fedora1.xml >
# virsh -c qemu:///system undefine fedora1
# virsh -c qemu:///system define /tmp/fedora1.xml
 
(or by now I've learned the finer arts of virsh :-))

# virsh -c qemu:///system
virsh # help
...
virsh # dumpxml fedora1
...
virsh # undefine fedora1
...
virsh # define /tmp/fedora1.xml
...
virsh # quit

Thanks

I think these error messages in the /var/log/messages might explain the reason
behind the problem:
Aug  1 22:31:53 localhost setroubleshoot:      SELinux is preventing
/usr/sbin/brctl (brctl_t) "getattr" to /sys/class/net/vnet0/bridge/forward_delay
(sysfs_t).      For complete SELinux messages. run sealert -l
3717d1ca-db63-428b-80bb-474d33e499aa

$ sudo sealert -l 3717d1ca-db63-428b-80bb-474d33e499aa
Summary
    SELinux is preventing /usr/sbin/brctl (brctl_t) "getattr" to
    /sys/class/net/vnet0/bridge/forward_delay (sysfs_t).

Detailed Description
    SELinux denied access requested by /usr/sbin/brctl. It is not expected that
    this access is required by /usr/sbin/brctl and this access may signal an
    intrusion attempt. It is also possible that the specific version or
    configuration of the application is causing it to require additional access.

Allowing Access
    Sometimes labeling problems can cause SELinux denials.  You could try to
    restore the default system file context for
    /sys/class/net/vnet0/bridge/forward_delay, restorecon -v
    /sys/class/net/vnet0/bridge/forward_delay If this does not work, there is
    currently no automatic way to allow this access. Instead,  you can generate
    a local policy module to allow this access - see
    http://fedora.redhat.com/docs/selinux-faq-fc5/#id2961385 Or you can disable
    SELinux protection altogether. Disabling SELinux protection is not
    recommended. Please file a http://bugzilla.redhat.com/bugzilla/enter_bug.cgi
    against this package.

Additional Information

Source Context                user_u:system_r:brctl_t
Target Context                system_u:object_r:sysfs_t
Target Objects                /sys/class/net/vnet0/bridge/forward_delay [ file ]
Affected RPM Packages         bridge-utils-1.1-2 [application]
Policy RPM                    selinux-policy-2.6.4-29.fc7
Selinux Enabled               True
Policy Type                   targeted
MLS Enabled                   True
Enforcing Mode                Enforcing
Plugin Name                   plugins.catchall_file
Host Name                     desktop
Platform                      Linux desktop 2.6.22.1-33.fc7 #1 SMP Mon Jul 23
                              16:59:15 EDT 2007 x86_64 x86_64
Alert Count                   24
First Seen                    Tue Jul 31 10:51:54 2007
Last Seen                     Wed Aug  1 22:31:51 2007
Local ID                      3717d1ca-db63-428b-80bb-474d33e499aa
Line Numbers

Raw Audit Messages

avc: denied { getattr } for comm="brctl" dev=sysfs egid=0 euid=0
exe="/usr/sbin/brctl" exit=-13 fsgid=0 fsuid=0 gid=0 items=0
name="forward_delay" path="/sys/class/net/vnet0/bridge/forward_delay" pid=3226
scontext=user_u:system_r:brctl_t:s0 sgid=0 subj=user_u:system_r:brctl_t:s0
suid=0 tclass=file tcontext=system_u:object_r:sysfs_t:s0 tty=(none) uid=0


To prove the theory, I've booted with selinux=permissive, and indeed libvirtd
and cousins (even with the newest Fedora 7 libvirt updates & kernel updates)
work just fine :-).

(It appears it'd nothing to do with libvirt/virtual-machine-manager etc.; good
old SELinux was playing up I suspect :-). Perhaps the recent 2.6.22 kernel
updates or the SELinux policy updates broke libvirt. May I request the SELinux
policy be updated if this turns out to be a genuine SELinux layer problem?)

Thanks

Reproduced the same problem on a Fedora 7 x86 system also (with the same SELinux
AVC error messages in the system logs). Once again, fixed the problem by
disabling SELinux (with SELinux=permissive).

Thanks

I have the same problem on an F7 x86_64 system, without any AVC errors.  The
system has just been rebooted after installing the new kernel.

Based on comment #11, this bug looks related to an selinux-policy problem,
so reassigning to that component.

Daniel

Should be fixed in selinux-policy-2.6.4-30.fc7

That's good to hear. One another to the SELinux experts: once used
selinux=permissive, subsequent boots without that parameter makes the system
unbootable, as SELinux stops the system from booting (verified on both Laptop &
Destkop using both latest official kernel & SELinux policy for Fedora 7).

Is there a trick involved in getting it to boot with SELinux again? A URL,
pointer would be appreciable.

Thanks

What kind of avc messages are you seeing in /var/log/audit/audit.log?

(Many many lines of errors scroll past on the screen, so what's I'm posting here
is what remains on the screen.)

[many many lines of selinux error messages]
...
udevd-event[PID]: selinux_setfilecon: matchpathconf(/dev/looop3) failed
...
udevd-event[PID]: selinux_setscreatecon: matchpathcon(/dev/input/event5) failed
ditto for /dev/input/event4
ditto for /dev/input/event6
Setting up Logical Volume Management [OK}
Checking filesystems [FAILED]
*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
*** Warning -- SELinux is active
*** Disabling security enforcement for system recovery.
*** Run 'setenforce 1' to reenable.
Give root password for maintenance
(or type Control-D to continue):

Rebooting it by pressing Control-D & passing on selinux=permissive gets the
system booting again.

(Even as a system administrator of 250 Linux servers, I've no clue how to fix
this SELinux problem; SELinux is something to learn after my graduation at the
University in a week :-))

Thanks

This sounds like a badly mislabeled file system and udev is running with the
wrong context. You could try relabeling in permissive mode

touch /.autorelabel; reboot

Created attachment 161237 [details]
SELinux restorecon failure messges

(Stupid bugzilla didn't include my comments I've typed before attaching the
attachments. Ho hum.)

Here's some relevant remarks about the above attachment:
Actually, I had already tried that on the local file systems with no success. (I
think the auto-relabel procedure is what fails first, ref to error messages below)

With the new SELinux policy, things have improved slightly (well it doesn't fill
up kernel dmesg buffer such that you lose the many many lines of initial messages):
Linux version 2.6.22.1-41.fc7 (kojibuilder.phx.redhat.com)
(gcc version 4.1.2 20070502 (Red Hat 4.1.2-12)) #1 SMP Fri Jul 27 18:10:34 EDT 2007
...
SELinux:  Starting in permissive mode
...
audit(1187046658.940:3): policy loaded auid=4294967295
audit(1187046660.043:4): avc:  denied  { read } for  pid=504 comm="restorecon"
name="ld.so.cache" dev=dm-0 ino=427697
scontext=system_u:system_r:restorecon_t:s0 tcontext=system_u:object_r:file_t:s0
tclass=file
...

(The complete dmesg is attached.)

It seems like restorecon is unable to do what it wants to.

Thanks

When it should be alright now.  I will fix this in future releases. But you ran
restorecon in permissive mode so it should have relabeled everything.

I don't know how to explain the problem (lack of SELinux knowledge/experience).
I'll try to recap a few points now:
    I suspect /.autorelabel doesn't force a auto relabeling possibly as of
recent Fedora 7 updates.

    E.g., when I do "restorecon -R /" I see the hard drive spins madly; no such
thing happens with /.autorelabel, which leads me to believe my failure scenario
is way too earlier than autorelabel procedure can begin or the whole
.autorelabel procedure has stopped working perhaps for the root file system, I
don't know.

(My theory could be completely wrong as a person of no good SELinux skills.)

In any case I manually forced a 'restorecon /blahblah' for every one of my
mounted file systems (it's a pity initially I didn't know restorecon tool
properly; nor had an understanding of what's expected from .autorelabel) when
booted on selinux=permissive mode. Now things are working just fine without
selinux=permissive mode.

Thanks for your assistance.

Well the touch /.autorelabel would only relabel file systems that are in the
fstab, so if you mount file systems elsewhere this could be a problem.  But when
you touch /.autorelabel it should take 5-10 minutes or alot longer to label and
you should see something on the boot screen.

I can confirm that relabelling fixes this problem for me.  Odd that I didn't see
any AVC messages, but anyway, glad it's working.

Bulk closing all bugs in Fedora updates in the modified state.  If you bug is
not fixed, please reopen.