Bug 559154 - libvirt: lowered privs break <interface type='ethernet'/>: could not configure /dev/net/tun: no virtual network emulation
libvirt: lowered privs break <interface type='ethernet'/>: could not configur...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
12
x86_64 Linux
low Severity medium
: ---
: ---
Assigned To: Daniel Veillard
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-01-27 05:00 EST by Wolfgang Denk
Modified: 2011-05-06 11:31 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-12 13:14:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Debian BTS 565767 None None None Never

  None (edit)
Description Wolfgang Denk 2010-01-27 05:00:49 EST
Description of problem:

After upgrading from F11 to F12, all virtual guests fail to start.
Error messages look like this:

# virsh start v-build
error: Failed to start domain v-build
error: internal error unable to start guest: warning: could not configure /dev/net/tun: no virtual network emulation
qemu: Could not initialize device 'tap'


Version-Release number of selected component (if applicable):

libvirt-0.7.1-15.fc12.x86_64
libvirt-client-0.7.1-15.fc12.x86_64
qemu-system-x86-0.11.0-12.fc12.x86_64


How reproducible:

Always.


Steps to Reproduce:
1. Configure a virtual machine with network interface enabled (using tun/tap)
2. Attempt to start this machine using libvirt (virsh + libvirtd)
  

Actual results:

Error: could not configure /dev/net/tun: no virtual network emulation


Expected results:

Machine starting without such errors :-)


Additional info:

strace shows that the actual problem happens within libvirtd, here:

...
10242 open("/dev/net/tun", O_RDWR)      = 7
10242 ioctl(7, 0x800454cf, 0x7fff84d10f18) = 0
10242 ioctl(7, TUNSETIFF, 0x7fff84d11c70) = -1 EPERM (Operation not permitted)
10242 write(2, "warning: could not configure /dev/net/tun: no virtual network emulation\n", 72) = 72
10242 close(7)                          = 0
10242 write(2, "qemu: ", 6)             = 6
10242 write(2, "Could not initialize device 'tap'\n", 34) = 34
10242 exit_group(1)                     = ?

From strace and from the "/var/log/libvirt/qemu/v-build.log" log file
I can see that the command that was used to start qemu-kvm was this:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -S -M pc -m 2048 -smp 4 -name v-build
-uuid 0fae2214-3e12-47ac-ebdb-c11183651d48
-monitor unix:/var/lib/libvirt/qemu/v-build.monitor,server,nowait
-boot c
-drive file=/dev/mapper/virt-build_root,if=ide,index=0,boot=on
-drive file=/dev/mapper/misc-build_work,if=ide,index=1
-drive file=/dev/mapper/virt-build_opt,if=ide,index=2
-drive file=/dev/mapper/misc-bd_eldk_old,if=ide,index=3
-net nic,macaddr=54:52:00:6b:9c:2e,vlan=0,name=nic.0
-net tap,ifname=vif0,script=/etc/libvirt/qemu/v-build.ifup,vlan=0,name=tap.0
-serial pty -parallel none -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus


If I run this command manually from the command line (just omitting
the "-S" option so the machine actually starts), then everything is
fine - it boots without errors.

/dev/net/tun exists and shouldbe accessible:

	# ls -l /dev/net/tun
	crw-rw-rw- 1 root root 10, 200 Jan 26 09:22 /dev/net/tun

I have no idea why the TUNSETIFF ioctl is failing for libvirtd, but
works when running qemu-kvm manually. I even disabled selinux:

	# selinuxenabled ; echo RC=$?
	RC=1

This did not change anuything.
Comment 1 Guido Günther 2010-02-01 01:55:43 EST
You need CAP_NET_ADMIN to manipulate tap devices but these are being dropped because of VIR_EXEC_CLEAR_CAPS when virExec is called to run kvm. Building withouth libcap-ng should work around this.
Comment 2 Mark Wielaard 2010-04-17 08:32:46 EDT
How/Where does one indicate that the CAP_NET_ADMIN capability shouldn't be dropped? It really seems to be necessary when a domain specifies an ethernet interface.
Comment 3 Guido Günther 2010-04-17 11:30:07 EDT
That's what were using in Debian right now:

http://git.debian.org/?p=pkg-libvirt/libvirt.git;a=blob;f=debian/patches/0006-Don-t-drop-caps-when-exec-ing-qemu.patch

it gives the qemu process far more rights than it should have but I didn't get around to have a closer look yet.
Comment 4 Mohammad Mateen 2010-04-27 15:40:28 EDT
I am also facing the same problem with my fedora 12 box which is recently upgraded live from FC11. All of my VMs got down and giving following error:-

virsh start xxxx
error: Failed to start domain xxxx
error: internal error unable to start guest: warning: could not configure /dev/net/tun: no virtual network emulation
qemu: Could not initialize device 'tap

I have following in logs:-


LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin /usr/bin/qemu-kvm -S -M pc -m 256 -smp 1 -name xxxx -uuid 40e6e2c7-c560-2a82-f2b6-bf635107286e -nographic -monitor unix:/var/lib/libvirt/qemu/xxxx.monitor,server,nowait -boot c -drive file=/dev/vmdata/vm6.root,if=ide,index=0,boot=on -drive file=/dev/vmdata/vm6.var,if=ide,index=1 -drive file=/dev/vmdata/vm6.swap,if=ide,index=3 -net nic,macaddr=52:54:00:cc:26:8b,vlan=0,name=nic.0 -net tap,ifname=tap6,script=/etc/KVM/xxxx_tap6.sh,vlan=0,name=tap.0 -serial pty -parallel none -usb 
warning: could not configure /dev/net/tun: no virtual network emulation
qemu: Could not initialize device 'tap'

Yes its true i can start VMs from command line without -S option but result VM doesn't appear in 'virsh list'

Can you guys make it fix on urgent basis.
Comment 5 Mohammad Mateen 2010-05-03 15:37:36 EDT
No reply from anywhere..... anyone looking into it ????
Comment 6 Mohammad Mateen 2010-05-24 08:02:49 EDT
We are suffering badly from this problem. Is there any update on this issue ???
Comment 7 Cole Robinson 2010-05-24 09:53:15 EDT
This sounds like its only hitting people who have updated from F11 to F12. Have you rebooted into the new kernel since upgrading? Is the machine fully updated with the latest F11 packages?
Comment 8 Cole Robinson 2010-05-24 09:55:07 EDT
Also, please provide virsh dumpxml VMNAME for one of the affected VMs
Comment 9 Mohammad Mateen 2010-05-24 10:17:23 EDT
Yes, I have done live yum Upgrade from FC11 to FC12. The machine is update-to-date and recently booted on latest kernel. I dont know how to update FC11 packages .... as they are supposed to be upgraded while live yum upgrade. 

------------------------------
[root@server ~]# uname -a
Linux server.awpdc.com 2.6.32.12-115.fc12.x86_64 #1 SMP Fri Apr 30 19:46:25 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
------------------------------

All of VMs are infected due to this problem

------------------------------
[root@server ~]# virsh dumpxml server-b
<domain type='kvm'>
  <name>server-b</name>
  <uuid>5dda1637-e995-2f60-b355-ce33e25bcf4d</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <source dev='/dev/vmdata/vm3.root'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <disk type='block' device='disk'>
      <source dev='/dev/vmdata/vm3.var'/>
      <target dev='hdb' bus='ide'/>
    </disk>
    <disk type='block' device='disk'>
      <source dev='/dev/vmdata/vm3.swap'/>
      <target dev='hdd' bus='ide'/>
    </disk>
    <interface type='ethernet'>
      <mac address='52:54:00:7c:a4:fd'/>
      <script path='/etc/KVM/server-b_tap3.sh'/>
      <target dev='tap3'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
  </devices>
</domain>
------------------------------

------------------------------
[root@server ~]# cat /etc/KVM/server-b_tap3.sh 
IP=xx.xx.xx.xx
TAP=tap3
SIP=xx.xx.xx.xx
tap=`ifconfig |grep $TAP`
if [ "$tap" == '' ] ;then
tap=`tunctl -b -u root -t $TAP`
fi
ifconfig $TAP $SIP netmask 255.255.255.255 up
echo 1 > /proc/sys/net/ipv4/conf/$TAP/proxy_arp
route=`ip route show |grep $IP`
if [ "$route" == '' ] ;then
ip route add $IP/32 dev $TAP
fi
------------------------------

Plz let me know if you need anything else to diagnose the problem.
Comment 10 Cole Robinson 2010-05-24 10:37:28 EDT
Okay, I was wrong, this doesn't have anything explicitly to do with F11->F12 upgrade. libvirt in F12 drops qemu emulator privs, which prevents it from running your ethernet script. Danpb provided some more info in this RHEL6 bug:

https://bugzilla.redhat.com/show_bug.cgi?id=593903#c2

It probably makes sense to add an /etc/libvirt/qemu.conf option to not drop emulator privs using capng, so users can opt out until another solution is found (if there is one).
Comment 11 Mohammad Mateen 2010-05-25 08:13:24 EDT
what exactly need to define in /etc/libvirt/qemu.conf to resolve this problem ??
Comment 12 Cole Robinson 2010-05-25 10:11:53 EDT
There is nothing to change in qemu.conf yet, it was just an idea. There is no work around for this at the moment, your only options are one of:

- Rebuild libvirt, but configure with --with-capng=no
- Reconfigure your networking setup to not use pass a script
Comment 13 Mohammad Mateen 2010-05-28 12:10:06 EDT
following worked for me.

1. I have run qemu with root user as follows and rebooted both qemu and libvirtd

----------------------------------
grep user /etc/libvirt/qemu.conf
# The user ID for QEMU processes run by the system instance
user = "root"
----------------------------------

2. I run my networking scrip manually before started VM.

All got fine after these 2 steps and now vm is booting normally.

As per wiki docs of qemu -  http://wiki.qemu.org/Documentation/Networking#Tap - for tap networking you need to run qemu with root privileges.

I have also tried to recompile libvirt rpm --with-capng=no but resulting rpm didnt worked for me and behaviour is the same.

Any thoughts ......??
Comment 14 Guido Günther 2010-05-31 18:05:00 EDT
See #c3 for a simple patch. This way you can leave --with-capng enabled (for all other HVs) and only disable it for qemu.
Comment 15 Cole Robinson 2010-07-12 12:56:00 EDT
Libvirt in F12 and F13 now have a way to work around this without rebuilding libvirt. Basically it boils down to setting emulator_clear_caps=0 in /etc/libvirt/qemu.conf. More info here:

https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#Errors_using_.3Cinterface_type.3D.27ethernet.27.2F.3E

Not really sure what the proper way forward is, so that this works out of the box. It could involve libvirt running the networking script and passing the FD to qemu.

Dropping severity since there's a workaround.
Comment 16 Muzi 2011-05-06 11:31:40 EDT
how much more time will require for the fix ??

Note You need to log in before you can comment on or make changes to this bug.