Description of problem: After upgrading from F11 to F12, all virtual guests fail to start. Error messages look like this: # virsh start v-build error: Failed to start domain v-build error: internal error unable to start guest: warning: could not configure /dev/net/tun: no virtual network emulation qemu: Could not initialize device 'tap' Version-Release number of selected component (if applicable): libvirt-0.7.1-15.fc12.x86_64 libvirt-client-0.7.1-15.fc12.x86_64 qemu-system-x86-0.11.0-12.fc12.x86_64 How reproducible: Always. Steps to Reproduce: 1. Configure a virtual machine with network interface enabled (using tun/tap) 2. Attempt to start this machine using libvirt (virsh + libvirtd) Actual results: Error: could not configure /dev/net/tun: no virtual network emulation Expected results: Machine starting without such errors :-) Additional info: strace shows that the actual problem happens within libvirtd, here: ... 10242 open("/dev/net/tun", O_RDWR) = 7 10242 ioctl(7, 0x800454cf, 0x7fff84d10f18) = 0 10242 ioctl(7, TUNSETIFF, 0x7fff84d11c70) = -1 EPERM (Operation not permitted) 10242 write(2, "warning: could not configure /dev/net/tun: no virtual network emulation\n", 72) = 72 10242 close(7) = 0 10242 write(2, "qemu: ", 6) = 6 10242 write(2, "Could not initialize device 'tap'\n", 34) = 34 10242 exit_group(1) = ? From strace and from the "/var/log/libvirt/qemu/v-build.log" log file I can see that the command that was used to start qemu-kvm was this: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -S -M pc -m 2048 -smp 4 -name v-build -uuid 0fae2214-3e12-47ac-ebdb-c11183651d48 -monitor unix:/var/lib/libvirt/qemu/v-build.monitor,server,nowait -boot c -drive file=/dev/mapper/virt-build_root,if=ide,index=0,boot=on -drive file=/dev/mapper/misc-build_work,if=ide,index=1 -drive file=/dev/mapper/virt-build_opt,if=ide,index=2 -drive file=/dev/mapper/misc-bd_eldk_old,if=ide,index=3 -net nic,macaddr=54:52:00:6b:9c:2e,vlan=0,name=nic.0 -net tap,ifname=vif0,script=/etc/libvirt/qemu/v-build.ifup,vlan=0,name=tap.0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus If I run this command manually from the command line (just omitting the "-S" option so the machine actually starts), then everything is fine - it boots without errors. /dev/net/tun exists and shouldbe accessible: # ls -l /dev/net/tun crw-rw-rw- 1 root root 10, 200 Jan 26 09:22 /dev/net/tun I have no idea why the TUNSETIFF ioctl is failing for libvirtd, but works when running qemu-kvm manually. I even disabled selinux: # selinuxenabled ; echo RC=$? RC=1 This did not change anuything.
You need CAP_NET_ADMIN to manipulate tap devices but these are being dropped because of VIR_EXEC_CLEAR_CAPS when virExec is called to run kvm. Building withouth libcap-ng should work around this.
How/Where does one indicate that the CAP_NET_ADMIN capability shouldn't be dropped? It really seems to be necessary when a domain specifies an ethernet interface.
That's what were using in Debian right now: http://git.debian.org/?p=pkg-libvirt/libvirt.git;a=blob;f=debian/patches/0006-Don-t-drop-caps-when-exec-ing-qemu.patch it gives the qemu process far more rights than it should have but I didn't get around to have a closer look yet.
I am also facing the same problem with my fedora 12 box which is recently upgraded live from FC11. All of my VMs got down and giving following error:- virsh start xxxx error: Failed to start domain xxxx error: internal error unable to start guest: warning: could not configure /dev/net/tun: no virtual network emulation qemu: Could not initialize device 'tap I have following in logs:- LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin /usr/bin/qemu-kvm -S -M pc -m 256 -smp 1 -name xxxx -uuid 40e6e2c7-c560-2a82-f2b6-bf635107286e -nographic -monitor unix:/var/lib/libvirt/qemu/xxxx.monitor,server,nowait -boot c -drive file=/dev/vmdata/vm6.root,if=ide,index=0,boot=on -drive file=/dev/vmdata/vm6.var,if=ide,index=1 -drive file=/dev/vmdata/vm6.swap,if=ide,index=3 -net nic,macaddr=52:54:00:cc:26:8b,vlan=0,name=nic.0 -net tap,ifname=tap6,script=/etc/KVM/xxxx_tap6.sh,vlan=0,name=tap.0 -serial pty -parallel none -usb warning: could not configure /dev/net/tun: no virtual network emulation qemu: Could not initialize device 'tap' Yes its true i can start VMs from command line without -S option but result VM doesn't appear in 'virsh list' Can you guys make it fix on urgent basis.
No reply from anywhere..... anyone looking into it ????
We are suffering badly from this problem. Is there any update on this issue ???
This sounds like its only hitting people who have updated from F11 to F12. Have you rebooted into the new kernel since upgrading? Is the machine fully updated with the latest F11 packages?
Also, please provide virsh dumpxml VMNAME for one of the affected VMs
Yes, I have done live yum Upgrade from FC11 to FC12. The machine is update-to-date and recently booted on latest kernel. I dont know how to update FC11 packages .... as they are supposed to be upgraded while live yum upgrade. ------------------------------ [root@server ~]# uname -a Linux server.awpdc.com 2.6.32.12-115.fc12.x86_64 #1 SMP Fri Apr 30 19:46:25 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux ------------------------------ All of VMs are infected due to this problem ------------------------------ [root@server ~]# virsh dumpxml server-b <domain type='kvm'> <name>server-b</name> <uuid>5dda1637-e995-2f60-b355-ce33e25bcf4d</uuid> <memory>524288</memory> <currentMemory>524288</currentMemory> <vcpu>1</vcpu> <os> <type arch='x86_64' machine='pc'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='block' device='disk'> <source dev='/dev/vmdata/vm3.root'/> <target dev='hda' bus='ide'/> </disk> <disk type='block' device='disk'> <source dev='/dev/vmdata/vm3.var'/> <target dev='hdb' bus='ide'/> </disk> <disk type='block' device='disk'> <source dev='/dev/vmdata/vm3.swap'/> <target dev='hdd' bus='ide'/> </disk> <interface type='ethernet'> <mac address='52:54:00:7c:a4:fd'/> <script path='/etc/KVM/server-b_tap3.sh'/> <target dev='tap3'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target port='0'/> </console> </devices> </domain> ------------------------------ ------------------------------ [root@server ~]# cat /etc/KVM/server-b_tap3.sh IP=xx.xx.xx.xx TAP=tap3 SIP=xx.xx.xx.xx tap=`ifconfig |grep $TAP` if [ "$tap" == '' ] ;then tap=`tunctl -b -u root -t $TAP` fi ifconfig $TAP $SIP netmask 255.255.255.255 up echo 1 > /proc/sys/net/ipv4/conf/$TAP/proxy_arp route=`ip route show |grep $IP` if [ "$route" == '' ] ;then ip route add $IP/32 dev $TAP fi ------------------------------ Plz let me know if you need anything else to diagnose the problem.
Okay, I was wrong, this doesn't have anything explicitly to do with F11->F12 upgrade. libvirt in F12 drops qemu emulator privs, which prevents it from running your ethernet script. Danpb provided some more info in this RHEL6 bug: https://bugzilla.redhat.com/show_bug.cgi?id=593903#c2 It probably makes sense to add an /etc/libvirt/qemu.conf option to not drop emulator privs using capng, so users can opt out until another solution is found (if there is one).
what exactly need to define in /etc/libvirt/qemu.conf to resolve this problem ??
There is nothing to change in qemu.conf yet, it was just an idea. There is no work around for this at the moment, your only options are one of: - Rebuild libvirt, but configure with --with-capng=no - Reconfigure your networking setup to not use pass a script
following worked for me. 1. I have run qemu with root user as follows and rebooted both qemu and libvirtd ---------------------------------- grep user /etc/libvirt/qemu.conf # The user ID for QEMU processes run by the system instance user = "root" ---------------------------------- 2. I run my networking scrip manually before started VM. All got fine after these 2 steps and now vm is booting normally. As per wiki docs of qemu - http://wiki.qemu.org/Documentation/Networking#Tap - for tap networking you need to run qemu with root privileges. I have also tried to recompile libvirt rpm --with-capng=no but resulting rpm didnt worked for me and behaviour is the same. Any thoughts ......??
See #c3 for a simple patch. This way you can leave --with-capng enabled (for all other HVs) and only disable it for qemu.
Libvirt in F12 and F13 now have a way to work around this without rebuilding libvirt. Basically it boils down to setting emulator_clear_caps=0 in /etc/libvirt/qemu.conf. More info here: https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#Errors_using_.3Cinterface_type.3D.27ethernet.27.2F.3E Not really sure what the proper way forward is, so that this works out of the box. It could involve libvirt running the networking script and passing the FD to qemu. Dropping severity since there's a workaround.
how much more time will require for the fix ??