Bug 1125755 - Start a network with NAT port "0-65535" will fail but create a new bridge
Summary: Start a network with NAT port "0-65535" will fail but create a new bridge
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-01 02:49 UTC by Luyao Huang
Modified: 2017-10-06 17:57 UTC (History)
11 users (show)

Fixed In Version: libvirt-1.2.14-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 05:46:27 UTC
mleitner: needinfo-


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Luyao Huang 2014-08-01 02:49:30 UTC
description of problem:
Start network with NAT port "0-65535" will fail but create a new bridge

Version-Release number of selected component (if applicable):
libvirt-1.1.1-29.el7_0.1.x86_64


How reproducible:
100%

Steps to Reproduce:
1.prepare xml like this
# cat net.xml
<network>
  <name>net4</name>
  <uuid>d918f9ff-f9b2-4ab7-ae01-f1042cd303a6</uuid>
  <forward mode='nat'>
    <nat>
      <address start='1.2.3.4' end='1.2.3.10'/>
      <port start='0' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr13' stp='on' delay='0' />
  <mac address='52:54:00:c7:ce7'/>
  <ip address='192.168.13.13' netmask='255.255.255.0'>
</ip>
</network>

2.make sure there are no bridge named virbr13
# brctl show
bridge name        bridge id                STP enabled        interfaces
virbr0                8000.525400a70ba2        yes                top0
                                                        virbr0-nic
                                                        vnet0

3.# virsh net-define net.xml
Network net4 defined from net.xml

4.# virsh net-start net4
error: Failed to start network net4
error: failed to add iptables rule to enable UDP masquerading

5.# brctl show
bridge name        bridge id                STP enabled        interfaces
virbr0                8000.525400a70ba2        yes                top0
                                                        virbr0-nic
                                                        vnet0
virbr13                8000.000000000000        yes                




Actual results:
network created failed, but create a bridge


Expected results:
Do not create a bridge when network create fail


Additional info:

Comment 2 Hu Jianwei 2014-08-01 06:12:23 UTC
Debugging the bug, after ioctl(fd, SIOCBRADDBR, brname), a UP virbr3 was created, so virNetDevBridgeDelete(network->def->bridge) failed, brctl addbr aslo can hit it.

Breakpoint 1, virNetDevBridgeCreate (brname=0x7f55f4008490 "virbr3") at util/virnetdevbridge.c:233
233	{
(gdb) n
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
233	{
(gdb) 
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
233	{
(gdb) 
234	    int fd = -1;
(gdb) 
237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
(gdb) 
240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
(gdb) 
246	    ret = 0;
(gdb) 
240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
(gdb) 
249	    VIR_FORCE_CLOSE(fd);
(gdb) 

Stopping at util/virnetdevbridge.c:249, check the virbr3 status:
[root@localhost libvirt-1.1.1]# ip link show virbr3
56: virbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT 
    link/ether 5a:d7:6f:2e:dd:0b brd ff:ff:ff:ff:ff:ff

But on my fedora 19, there is a down bridge was created after ioctl().

Comment 3 Luyao Huang 2014-08-01 07:12:56 UTC
I found if networkmanager service is stop,this issue(In reply to Hu Jianwei from comment #2)
> Debugging the bug, after ioctl(fd, SIOCBRADDBR, brname), a UP virbr3 was
> created, so virNetDevBridgeDelete(network->def->bridge) failed, brctl addbr
> aslo can hit it.
> 
> Breakpoint 1, virNetDevBridgeCreate (brname=0x7f55f4008490 "virbr3") at
> util/virnetdevbridge.c:233
> 233	{
> (gdb) n
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 233	{
> (gdb) 
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 233	{
> (gdb) 
> 234	    int fd = -1;
> (gdb) 
> 237	    if ((fd = virNetDevSetupControl(NULL, NULL)) < 0)
> (gdb) 
> 240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
> (gdb) 
> 246	    ret = 0;
> (gdb) 
> 240	    if (ioctl(fd, SIOCBRADDBR, brname) < 0) {
> (gdb) 
> 249	    VIR_FORCE_CLOSE(fd);
> (gdb) 
> 
> Stopping at util/virnetdevbridge.c:249, check the virbr3 status:
> [root@localhost libvirt-1.1.1]# ip link show virbr3
> 56: virbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
> UNKNOWN mode DEFAULT 
>     link/ether 5a:d7:6f:2e:dd:0b brd ff:ff:ff:ff:ff:ff
> 
> But on my fedora 19, there is a down bridge was created after ioctl().

I found if networkmanager service is stop. This issue will disappear.

Comment 4 Jiri Denemark 2014-08-05 14:47:09 UTC
So it seems networkmanager sets the bridge UP when it's created and that prevents libvirt from deleting the bridge. Comment 2 says the issue can be reproduced with just plain "brctl addbr", so the issue is not in the way libvirt creats bridges.

Comment 5 Jirka Klimes 2014-08-25 13:29:39 UTC
NetworkManager always sets IFF_UP for interfaces to get carrier detection.

But, I think libvirt should not rely on the bridge being DOWN when trying to remove it. It should either make it down first and then remove the bridge, or remove the interface with netlink.

# ip link set virbr13 down 
# brctl delbr virbr13

or simply

# ip link del virbr13

Comment 8 Dan Williams 2014-11-26 01:15:14 UTC
Over to libvirt due to comment 5.  The kernels' ioctl-based method does not allow bridges to be deleted when they are IFF_UP, but that is only due to historical reasons and to keep backwards compatibility.  Using netlink to delete the bridge does not care about IFF_UP.

Comment 9 Jiri Denemark 2014-11-26 08:23:57 UTC
Ah, thanks. Looks like we should rework the bridge code to use netlink then...

Comment 10 Laine Stump 2015-03-03 16:40:50 UTC
But why is NetworkManager playing around with IFF_UP on a transient bridge created by libvirt anyway? I thought it was supposed to keep its hands off libvirt's bridges (and anything else that wasn't a part of the configuration in /etc/sysconfig/network-scripts/ifcfg-*, or whereever is the new persistent config hotness).

The failure in the report is that libvirt created a bridge with an ioctl, then attempted to delete that same bridge with an ioctl.

I don't have a problem making libvirt use netlink to create and delete bridges (although that will mean that there is that much more code to support, because we will need to keep the code that uses ioctls for backwards compatibility with other/older platforms), but NetworkManager should very definitely *not* be doing anything with libvirt's bridges; (I just turned NM on on my system and noticed that libvirt's bridges are showing up in the list of network devices in NM's drop down. I thought we'd agreed a long time ago that this sort of thing would not be done. Is this an accidental regression? We really need to have QE testing that verifies this separation is maintained.)

Comment 12 Laine Stump 2015-03-17 22:42:05 UTC
I just tried the sequence of commands from the description and there was no bridge device left over. Based on that I'm assuming that NM no longer ifups the bridge device that libvirt creates. I still have made a patch that uses netlink RTM_DELLINK to delete bridge devices, as a defensive tactic, and will be sending that upstream as soon as I also code up the *creation* of bridges using RTM_NEWLINK.

Comment 13 Laine Stump 2015-03-26 19:40:24 UTC
The following were just pushed upstream, so now libvirt will create and delete bridge devices using netlink on any platform that supports it (which means essentially anything Linux that isn't ancient)

commit 0473b45cc16757ccc5392d2d1279d750af54bfb9
Author: Laine Stump <laine@laine.org>
Date:   Tue Mar 17 14:27:21 2015 -0400

    util: netlink function to delete any network device
    

commit e849062a64f4baf62ea26b2ed17b1b9ab6d51786
Author: Laine Stump <laine@laine.org>
Date:   Tue Mar 17 14:42:24 2015 -0400

    util: replace body of virNetDevMacVLanDelete() with virNetlinkDelLink()
    

commit 09778e0908822e9bf57a8b55e0150ad80e64e01f
Author: Laine Stump <laine@laine.org>
Date:   Tue Mar 17 15:45:01 2015 -0400

    util: use netlink to delete bridge devices
    

commit fc7b23db02062f5649022db1fd238c4bcc28476a
Author: Laine Stump <laine@laine.org>
Date:   Mon Mar 23 14:17:44 2015 -0400

    util: use netlink to create bridge devices

Comment 15 hongming 2015-07-13 06:31:12 UTC
Verify it as follows. The result is expected. Move its status to VERIFIED.


# rpm -q libvirt kernel
libvirt-1.2.17-1.el7.x86_64

# uname -r
3.10.0-229.el7.x86_64

# service NetworkManager status
Redirecting to /bin/systemctl status  NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2015-07-13 14:02:00 CST; 13min ago
 Main PID: 18227 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           ├─18227 /usr/sbin/NetworkManager --no-daemon
           └─18254 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eno1.pid...

Jul 13 14:02:00 localhost.localdomain systemd[1]: Starting Network Manager...
Jul 13 14:02:00 localhost.localdomain systemd[1]: Started Network Manager.


# cat bridgetest.xml
<network>
  <name>bridgetest</name>
  <forward mode='nat'>
    <nat>
      <address start='1.2.3.4' end='1.2.3.10'/>
      <port start='0' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr13' stp='on' delay='0' />
  <ip address='192.168.13.13' netmask='255.255.255.0'>
</ip>
</network>


# virsh net-define bridgetest.xml
Network bridgetest defined from bridgetest.xml

# virsh net-start bridgetest
error: Failed to start network bridgetest
error: COMMAND_FAILED: '/sbin/iptables -w --table nat --insert POSTROUTING --source 192.168.13.0/24 -p udp ! --destination 192.168.13.0/24 --jump SNAT --to-source 1.2.3.4-1.2.3.10:0-65535' failed: iptables v1.4.21: Port `0-65535' not valid

Try `iptables -h' or 'iptables --help' for more information.

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0

# virsh net-undefine bridgetest
Network bridgetest has been undefined

# virsh net-create bridgetest.xml
error: Failed to create network from bridgetest.xml
error: COMMAND_FAILED: '/sbin/iptables -w --table nat --insert POSTROUTING --source 192.168.13.0/24 -p udp ! --destination 192.168.13.0/24 --jump SNAT --to-source 1.2.3.4-1.2.3.10:0-65535' failed: iptables v1.4.21: Port `0-65535' not valid

Try `iptables -h' or 'iptables --help' for more information.

[root@localhost images]# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
=====================================================================

Change the port to '1' ~ '65535'

# cat bridgetest.xml
<network>
  <name>bridgetest</name>
  <forward mode='nat'>
    <nat>
      <address start='1.2.3.4' end='1.2.3.10'/>
      <port start='1' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr13' stp='on' delay='0' />
  <ip address='192.168.13.13' netmask='255.255.255.0'>
</ip>
</network>

# virsh net-define bridgetest.xml
Network bridgetest defined from bridgetest.xml

# virsh net-start bridgetest
Network bridgetest started

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.525400334be4	yes		virbr13-nic

# virsh net-undefine bridgetest
Network bridgetest has been undefined

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.525400334be4	yes		virbr13-nic

# virsh net-destroy bridgetest
Network bridgetest destroyed

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
# virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     no            yes


# virsh net-create bridgetest.xml
Network bridgetest created from bridgetest.xml

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0
virbr13		8000.52540017d871	yes		virbr13-nic

# virsh net-destroy bridgetest
Network bridgetest destroyed

# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.525400a015f5	yes		virbr0-nic
							vnet0

Comment 17 errata-xmlrpc 2015-11-19 05:46:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html


Note You need to log in before you can comment on or make changes to this bug.