Bug 829037

Summary: 3.1 - Several bridgeless networks on same NIC/Bond should be avoided
Product: Red Hat Enterprise Linux 6 Reporter: Avi Tal <atal>
Component: vdsmAssignee: Igor Lvovsky <ilvovsky>
Status: CLOSED ERRATA QA Contact: Meni Yakove <myakove>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.4CC: abaron, bazulay, cpelland, danken, dyasny, iheim, jbiddle, lpeer, mavital, Rhev-m-bugs, srevivo, yeylon, ykaul
Target Milestone: rcFlags: ilvovsky: needinfo+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: vdsm-4.9.6-25.0 Doc Type: Bug Fix
Doc Text:
Previously, the VDSM was able to attach several bridgeless networks to the same interface. It did not clean up old bridgeless networks, which sometimes caused connection errors as the VDSM attempted to configure the obsolete network. Now, configurations are restricted to either a single bridged non-VLANed network or a single bridgeless non-VLANed network plus any amount of bridged/bridgeless VLANed networks. This prevents unnecessary connection errors caused by old or unused networks.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-04 18:58:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine log
none
vdsm log none

Description Avi Tal 2012-06-05 20:08:37 UTC
Created attachment 589630 [details]
engine log

Description of problem:
Bridgeless network are never being cleanup by VDSM.
using both SetupNetworks or the old Add network way to override an old bridgeless network over the same interface. VDSM will add the new network but won't remove the old bridgeless network. 
It will cause later problems when trying to reconfigure the old bridgeless network: ConfigNetworkError: (28, 'Bridge already exists').

I have added two bridgeless networks over the same interface (one after one. sw2 and sw3) then override this interface with bridged (sw4) network.

the results are:
[root@silver-vdsa ~]# virsh -r net-list --all
Name                 State      Autostart
-----------------------------------------
vdsm-rhevm           active     yes       
vdsm-sw2             active     yes       
vdsm-sw3             active     yes       
vdsm-sw4             active     yes       
default              inactive   no 

[root@silver-vdsa ~]# vdsClient 0 getVdsCaps |grep sw4 --color
        networks = {'rhevm': {'addr': '10.35.128.10', 'cfg': {'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '10.35.128.254', 'ports': ['eth0']}, 'sw3': {'interface': 'eth1', 'bridged': False}, 'sw2': {'interface': 'eth1', 'bridged': False}, 'sw4': {'addr': '', 'cfg': {'DEVICE': 'sw4', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'NM_CONTROLLED': 'no'}, 'mtu': '9000', 'netmask': '', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['eth1']}}



As you can see, VDSM configured sw2, sw3 and sw4 over the same interface.

The Blocker issue is when trying to reconfigure the old bridgeless network:
MainProcess|Thread-16544::ERROR::2012-06-05 22:31:55,420::supervdsmServer::60::SuperVdsm.ServerCallback::(wrapper) Error in editNetwork
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 58, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 102, in editNetwork
    return configNetwork.editNetwork(oldBridge, newBridge, **options)
  File "/usr/share/vdsm/configNetwork.py", line 783, in editNetwork
    addNetwork(newBridge, vlan=vlan, bonding=bonding, nics=nics, configWriter=configWriter, **options)
  File "/usr/share/vdsm/configNetwork.py", line 590, in addNetwork
    bridged=bridged, skipLibvirt=skipLibvirt)
  File "/usr/share/vdsm/configNetwork.py", line 503, in _addNetworkValidation
    raise ConfigNetworkError(ne.ERR_USED_BRIDGE, 'Bridge already exists')
ConfigNetworkError: (28, 'Bridge already exists')
Thread-16544::ERROR::2012-06-05 22:31:55,421::API::1209::vds::(editNetwork) Bridge already exists
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1207, in editNetwork
    supervdsm.getProxy().editNetwork(oldBridge, newBridge, options)
  File "/usr/share/vdsm/supervdsm.py", line 59, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 57, in <lambda>
    callMethod = lambda : getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, **kwargs)
  File "<string>", line 2, in editNetwork
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (28, 'Bridge already exists')


BTW, running vdsClient delNetwork <bridgeless net> directly on the host will clean the network from libvirt.


Steps to Reproduce:
1. create bridgeless network and attach to host eth1
2. create bridged network nad attach to host eth1
3. try to reconfigure the old bridgeless network on eth1
  
Actual results:
libvirt will still contain the old bridgeless network
and the operation will fail due to 'Bridge already exists'

Expected results:
override bridgeless network should remove the network before adding a newer one on the same interface.

Additional info:

Comment 1 Avi Tal 2012-06-05 20:13:31 UTC
Created attachment 589631 [details]
vdsm log

Comment 3 Dan Kenigsberg 2012-06-05 21:38:13 UTC
(In reply to comment #0)

> Steps to Reproduce:
> 1. create bridgeless network and attach to host eth1
> 2. create bridged network nad attach to host eth1
> 3. try to reconfigure the old bridgeless network on eth1
>   
> Actual results:
> libvirt will still contain the old bridgeless network
> and the operation will fail due to 'Bridge already exists'
> 
> Expected results:
> override bridgeless network should remove the network before adding a newer
> one on the same interface.

I don't understand why this is a vdsm bug.

You create sw3 (bridgeless). You create sw4 (bridged) on the same eth1 nic. That's good. Now you try to make sw3 a bridged net. That's illegal, because you cannot connect one nic to two bridges.

Note that the operation that fails is
editNetwork with ('sw4', 'sw3', '', '', ['eth1'], {'STP': 'no'})
and not "reconfigure the old bridgeless". Vdsm is asked to reuse the name of an existing (bridged) network. That's illegal.

I haven't reviewed engine.log, maybe there's an issue there.

addNetwork with ('sw1', '162', '', ['eth1'], {'STP': 'no'}) {} flowID [429984de]
setSafeNetworkConfig with () {} flowID [1c5ac982]
delNetwork with ('sw1', '162', '', ['eth1']) {} flowID [307a7306]
setSafeNetworkConfig with () {} flowID [2b96155b]
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw3': {'ni
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'ni
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'re
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'ni
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'ni
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'ni
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw4': {'ni
editNetwork with ('sw4', 'sw4', '', '', ['eth1'], {'STP': 'no'}) {} flowID [24439552]
setSafeNetworkConfig with () {} flowID [2fd4cc7]
editNetwork with ('sw4', 'sw3', '', '', ['eth1'], {'STP': 'no'}) {} flowID [144cd1bc]

Comment 4 lpeer 2012-06-06 11:14:55 UTC
After talking with Avi the bug is as follows:
1. Add bridgeless network (red) to a nic (eth0), submit change
2. Remove red from eth0, submit change
3. Add (again) red network to eth0, submit change

Avi please confirm that this is what you inteded in the bug above.

Comment 5 Avi Tal 2012-06-06 11:32:47 UTC
(with danken) there *is* a vdsm bug here:

setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw3': {'nic': 'eth1', 'bridged': 'false'}}, {'bond4': {'remove': 'true'}, 'bond0': {'remove': 'true'}, 'bond1': {'remo
setupNetworks with ({'rhevm': {'nic': 'eth0', 'BOOTPROTO': 'dhcp', 'bridged': 'true'}, 'sw2': {'nic': 'eth1', 'bridged': 'false', 'mtu': '9000'}}, {'bond4': {'remove': 'true'}, 'bond0': {'remove': 'true'}, 

second operation should have failed since you cannot have two bridgeless networks on the same interface.

Comment 7 Avi Tal 2012-06-06 11:57:24 UTC
(In reply to comment #4)
> After talking with Avi the bug is as follows:
> 1. Add bridgeless network (red) to a nic (eth0), submit change
> 2. Remove red from eth0, submit change
> 3. Add (again) red network to eth0, submit change
> 
> Avi please confirm that this is what you inteded in the bug above.

Separate BZ was opened for backend: BZ#823927

Comment 9 Igor Lvovsky 2012-06-14 09:37:33 UTC
We can't reproduce this problem from UI or REST with latest engine.
The only way to reproduce it, is via vdsClient.
So, it can't be TestBlocker.

Comment 10 Igor Lvovsky 2012-07-18 14:37:34 UTC
http://gerrit.ovirt.org/#/c/6384/

Comment 12 Meni Yakove 2012-08-05 08:01:22 UTC
Verified on vdsm-4.9.6-26.0.el6_3.x86_64.

Comment 15 errata-xmlrpc 2012-12-04 18:58:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html