Bug 1125755

Debugging the bug, after ioctl(fd, SIOCBRADDBR, brname), a UP virbr3 was created, so virNetDevBridgeDelete(network->def->bridge) failed, brctl addbr aslo can hit it.

Breakpoint 1, virNetDevBridgeCreate (brname=0x7f55f4008490 "virbr3") at util/virnetdevbridge.c:233
233	{
(gdb) n
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
233	{
(gdb) 
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
233	{
(gdb) 
234	    int fd = -1;
(gdb) 
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
(gdb) 
246	    ret = 0;
(gdb) 
240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
(gdb) 
249	    VIR_FORCE_CLOSE(fd);
(gdb) 

Stopping at util/virnetdevbridge.c:249, check the virbr3 status:
[root@localhost libvirt-1.1.1]# ip link show virbr3
56: virbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT 
    link/ether 5a:d7:6f:2e:dd:0b brd ff:ff:ff:ff:ff:ff

But on my fedora 19, there is a down bridge was created after ioctl().

I found if networkmanager service is stop,this issue(In reply to Hu Jianwei from comment #2)
> Debugging the bug, after ioctl(fd, SIOCBRADDBR, brname), a UP virbr3 was
> created, so virNetDevBridgeDelete(network->def->bridge) failed, brctl addbr
> aslo can hit it.
> 
> Breakpoint 1, virNetDevBridgeCreate (brname=0x7f55f4008490 "virbr3") at
> util/virnetdevbridge.c:233
> 233	{
> (gdb) n
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 233	{
> (gdb) 
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 233	{
> (gdb) 
> 234	    int fd = -1;
> (gdb) 
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
> (gdb) 
> 246	    ret = 0;
> (gdb) 
> 240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
> (gdb) 
> 249	    VIR_FORCE_CLOSE(fd);
> (gdb) 
> 
> Stopping at util/virnetdevbridge.c:249, check the virbr3 status:
> [root@localhost libvirt-1.1.1]# ip link show virbr3
> 56: virbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
> UNKNOWN mode DEFAULT 
>     link/ether 5a:d7:6f:2e:dd:0b brd ff:ff:ff:ff:ff:ff
> 
> But on my fedora 19, there is a down bridge was created after ioctl().

I found if networkmanager service is stop. This issue will disappear.

So it seems networkmanager sets the bridge UP when it's created and that prevents libvirt from deleting the bridge. Comment 2 says the issue can be reproduced with just plain "brctl addbr", so the issue is not in the way libvirt creats bridges.

NetworkManager always sets IFF_UP for interfaces to get carrier detection.

But, I think libvirt should not rely on the bridge being DOWN when trying to remove it. It should either make it down first and then remove the bridge, or remove the interface with netlink.

# ip link set virbr13 down 
# brctl delbr virbr13

or simply

# ip link del virbr13

Over to libvirt due to comment 5.  The kernels' ioctl-based method does not allow bridges to be deleted when they are IFF_UP, but that is only due to historical reasons and to keep backwards compatibility.  Using netlink to delete the bridge does not care about IFF_UP.

Ah, thanks. Looks like we should rework the bridge code to use netlink then...

But why is NetworkManager playing around with IFF_UP on a transient bridge created by libvirt anyway? I thought it was supposed to keep its hands off libvirt's bridges (and anything else that wasn't a part of the configuration in /etc/sysconfig/network-scripts/ifcfg-*, or whereever is the new persistent config hotness).

The failure in the report is that libvirt created a bridge with an ioctl, then attempted to delete that same bridge with an ioctl.

I don't have a problem making libvirt use netlink to create and delete bridges (although that will mean that there is that much more code to support, because we will need to keep the code that uses ioctls for backwards compatibility with other/older platforms), but NetworkManager should very definitely *not* be doing anything with libvirt's bridges; (I just turned NM on on my system and noticed that libvirt's bridges are showing up in the list of network devices in NM's drop down. I thought we'd agreed a long time ago that this sort of thing would not be done. Is this an accidental regression? We really need to have QE testing that verifies this separation is maintained.)

I just tried the sequence of commands from the description and there was no bridge device left over. Based on that I'm assuming that NM no longer ifups the bridge device that libvirt creates. I still have made a patch that uses netlink RTM_DELLINK to delete bridge devices, as a defensive tactic, and will be sending that upstream as soon as I also code up the *creation* of bridges using RTM_NEWLINK.

The following were just pushed upstream, so now libvirt will create and delete bridge devices using netlink on any platform that supports it (which means essentially anything Linux that isn't ancient)

commit 0473b45cc16757ccc5392d2d1279d750af54bfb9
Author: Laine Stump <laine>
Date:   Tue Mar 17 14:27:21 2015 -0400

    util: netlink function to delete any network device
    

commit e849062a64f4baf62ea26b2ed17b1b9ab6d51786
Author: Laine Stump <laine>
Date:   Tue Mar 17 14:42:24 2015 -0400

    util: replace body of virNetDevMacVLanDelete() with virNetlinkDelLink()
    

commit 09778e0908822e9bf57a8b55e0150ad80e64e01f
Author: Laine Stump <laine>
Date:   Tue Mar 17 15:45:01 2015 -0400

    util: use netlink to delete bridge devices
    

commit fc7b23db02062f5649022db1fd238c4bcc28476a
Author: Laine Stump <laine>
Date:   Mon Mar 23 14:17:44 2015 -0400

    util: use netlink to create bridge devices

Verify it as follows. The result is expected. Move its status to VERIFIED.


# rpm -q libvirt kernel
libvirt-1.2.17-1.el7.x86_64

# uname -r
3.10.0-229.el7.x86_64

# service NetworkManager status
Redirecting to /bin/systemctl status  NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2015-07-13 14:02:00 CST; 13min ago
 Main PID: 18227 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           ├─18227 /usr/sbin/NetworkManager --no-daemon
           └─18254 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eno1.pid...

Jul 13 14:02:00 localhost.localdomain systemd[1]: Starting Network Manager...
Jul 13 14:02:00 localhost.localdomain systemd[1]: Started Network Manager.


# cat bridgetest.xml
<network>
  <name>bridgetest</name>
  <forward mode='nat'>
    <nat>
      <address start='1.2.3.4' end='1.2.3.10'/>
      <port start='0' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr13' stp='on' delay='0' />
  <ip address='192.168.13.13' netmask='255.255.255.0'>
</ip>
</network>


# virsh net-define bridgetest.xml
Network bridgetest defined from bridgetest.xml

# virsh net-start bridgetest
error: Failed to start network bridgetest
error: COMMAND_FAILED: '/sbin/iptables -w --table nat --insert POSTROUTING --source 192.168.13.0/24 -p udp ! --destination 192.168.13.0/24 --jump SNAT --to-source 1.2.3.4-1.2.3.10:0-65535' failed: iptables v1.4.21: Port `0-65535' not valid

Try `iptables -h' or 'iptables --help' for more information.

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0

# virsh net-undefine bridgetest
Network bridgetest has been undefined

# virsh net-create bridgetest.xml
error: Failed to create network from bridgetest.xml
error: COMMAND_FAILED: '/sbin/iptables -w --table nat --insert POSTROUTING --source 192.168.13.0/24 -p udp ! --destination 192.168.13.0/24 --jump SNAT --to-source 1.2.3.4-1.2.3.10:0-65535' failed: iptables v1.4.21: Port `0-65535' not valid

Try `iptables -h' or 'iptables --help' for more information.

[root@localhost images]# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
=====================================================================

Change the port to '1' ~ '65535'

# cat bridgetest.xml
<network>
  <name>bridgetest</name>
  <forward mode='nat'>
    <nat>
      <address start='1.2.3.4' end='1.2.3.10'/>
      <port start='1' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr13' stp='on' delay='0' />
  <ip address='192.168.13.13' netmask='255.255.255.0'>
</ip>
</network>

# virsh net-define bridgetest.xml
Network bridgetest defined from bridgetest.xml

# virsh net-start bridgetest
Network bridgetest started

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.525400334be4	yes		virbr13-nic

# virsh net-undefine bridgetest
Network bridgetest has been undefined

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.525400334be4	yes		virbr13-nic

# virsh net-destroy bridgetest
Network bridgetest destroyed

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     no            yes


# virsh net-create bridgetest.xml
Network bridgetest created from bridgetest.xml

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.52540017d871	yes		virbr13-nic

# virsh net-destroy bridgetest
Network bridgetest destroyed

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html