Bug 980174 - adding a tagged vlan network to rhevm network interface will change rhevm network boot protocol from dhcp to none
adding a tagged vlan network to rhevm network interface will change rhevm net...
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
x86_64 Linux
urgent Severity urgent
: ---
: 3.3.0
Assigned To: Dan Kenigsberg
Martin Pavlik
: Reopened
: 997529 (view as bug list)
Depends On:
  Show dependency treegraph
Reported: 2013-07-01 11:17 EDT by Anil Vettathu
Modified: 2016-02-10 14:57 EST (History)
16 users (show)

See Also:
Fixed In Version: IS16
Doc Type: Bug Fix
Doc Text:
Occasionally, an incorrect or missing BOOTPROTO line in the ifcfg file of a vlan device would propagate to the underlying device. Consequently, adding a vlan network on top of a non-vlan network erased the dhcp setting on the latter. VDSM now maintains the defined boot protocol when adding a new network on top of an existing one.
Story Points: ---
Clone Of:
Last Closed: 2014-01-21 11:26:47 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
rhevm3.2 log collector (4.33 MB, application/x-xz)
2013-08-27 13:45 EDT, Anil Vettathu
no flags Details
RHEVM3.2 LC - RHEL6 Host (4.43 MB, application/x-xz)
2013-08-28 06:56 EDT, Anil Vettathu
no flags Details
vdsm.log (1005.79 KB, text/x-log)
2013-09-02 09:08 EDT, psebek
no flags Details
Reproducer using setupNetworks command (40.29 KB, text/plain)
2013-09-12 05:38 EDT, psebek
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 18911 None None None Never

  None (edit)
Description Anil Vettathu 2013-07-01 11:17:45 EDT
Description of problem:

Adding a tagged vlan network to rhevm network interface will change rhevm network boot protocol from dhcp to none

Version-Release number of selected component (if applicable):
RHEV 3.2

How reproducible:

Steps to Reproduce:
1. setup rhevm as non-vm network, with boot protocol dhcp. For testing I configured it on eth0 interface.
2. create a new tagged vlan logical network for vm network. ex:- vlan123 with tag 123
3. Apply vlan123 to eth0 interface on the host, with boot protocol "none"
4. Save network configuration

Actual results:
During saving, the rhevm network will also be changed from dhcp to none. This will make the rhevm network (ie eth0) with no IP. In the back end RHEV is removing bootproto line from ifcfg-eth0

Expected results:
adding vlan123 should not change the configuration of rhevm network, ie ifcfg-eth0 configuration file.

Additional info:
Comment 2 lpeer 2013-07-10 04:08:48 EDT

*** This bug has been marked as a duplicate of bug 980486 ***
Comment 3 lpeer 2013-07-11 02:52:10 EDT
Just to make sure, the scenario above was done via the REST API (or SDK or CLI)?
Comment 8 Moti Asayag 2013-08-27 07:59:08 EDT
Anil, can you provide engine.log and vdsm.log so we can see which parameters were sent via the setupNetworks to the host ? 

In addition, did you perform the test on Rhev-h or rhel ?
What are the engine and vdsm versions ?
How did you configured configured the management network as non-vm network on the host?

(The described sequence of operations is quite common and i wasn't able to reproduce it on ovirt-3.3)
Comment 9 Anil Vettathu 2013-08-27 12:59:29 EDT
Moti, I tried to the reproduce the issue on rhevm-3.3.0-0.16.master.el6ev.noarch & vdsm-4.12.0-72.git287bb7e.el6ev.x86_64. The issue was not reproducible.

The RHEV3.2 environment which I used for reproducing had been rebuilt. I will reproduce it again on RHEVM3.2 and provide the details.
Comment 10 Anil Vettathu 2013-08-27 13:45:09 EDT
Created attachment 791130 [details]
rhevm3.2 log collector
Comment 11 Anil Vettathu 2013-08-27 13:45:36 EDT
Moti, The issue is reproducible in rhevm-3.2.0-11.37.el6ev.noarch and vdsm-4.10.2-23.0.el6ev. I tested using RHEV-H not RHEL. I am also attaching the log collector output, which contains logs,db and sosreport from hypervisor.

Please note create rhevm network as non-vm network and then add the vlan logical network on top of the rhevm interface.

Let me know if you need more details.
Comment 12 Moti Asayag 2013-08-27 17:04:26 EDT
This is the churn from the logs:

engine.log indicates parameters were sent correctly to VDSM:
2013-08-27 08:04:18,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/ [274b135d] START, SetupNetworksVDSCommand(HostName = dhcp210-139.gsslab.pnq.redhat.com, HostId = 276cc728-eef4-4e2b-8b08-e363aa85265d, force=false, checkConnectivity=false, conectivityTimeout=120,
	networks=[vlan123 {id=06ed7636-fdb7-485c-b18f-c6eb460fe68b, description=null, subnet=null, gateway=null, type=null, vlanId=123, stp=false, dataCenterId=5849b030-626e-47cb-ad90-3ce782d831b3, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true}}],
	interfaces=[eth0 {id=c785b47d-5043-42f9-b851-88fe577b3639, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth0, macAddress=52:54:00:6e:94:37, networkName=rhevm, bondName=null, bootProtocol=DHCP, address=, subnet=, gateway=, mtu=1500, bridged=false, speed=0, type=2, networkImplementationDetails={inSync=true, managed=true}},
		eth2 {id=f3cc386a-9a4a-4a83-a175-4f72a6faebf1, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth2, macAddress=52:54:00:ab:d1:70, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth1 {id=fd510e34-d892-4598-9ada-0a5dce316070, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth1, macAddress=52:54:00:87:7e:da, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth4 {id=433fdb06-8198-4263-9188-563391f36256, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth4, macAddress=52:54:00:37:21:b2, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth3 {id=d56f1699-fe68-417a-86af-e5668755a586, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth3, macAddress=52:54:00:f2:e7:96, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth0.123 {id=null, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, macAddress=null, networkName=vlan123, vlanId=123, bonded=null, bondName=null, bondOptions=null, bootProtocol=NONE, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=null, type=0, networkImplementationDetails=null}],
	removedBonds=[]), log id: 2f223717
2013-08-27 08:06:00,794 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-49) [766fda37] Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2013-08-27 08:06:00,799 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [766fda37] ResourceManager::activateVds - failed to get VDS = 276cc728-eef4-4e2b-8b08-e363aa85265d capabilities with error: java.net.NoRouteToHostException: No route to host.

From vdsm.log:
Thread-58::DEBUG::2013-08-27 12:04:17,845::BindingXMLRPC::913::vds::(wrapper) client []::call setupNetworks with ({'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'false'}) {} flowID [274b135d]

Thread-59::DEBUG::2013-08-27 12:06:12,230::BindingXMLRPC::913::vds::(wrapper) client []::call getCapabilities with () {}
Thread-59::DEBUG::2013-08-27 12:06:12,293::BindingXMLRPC::920::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:213184f810f5'}], 'FC': []}, 'packages2': {'kernel': {'release': '358.14.1.el6.x86_64', 'buildtime': 1371484460.0, 'version': '2.6.32'}, 'spice-server': {'release': '12.el6_4.1', 'buildtime': 1368193709L, 'version': '0.12.0'}, 'vdsm': {'release': '23.0.el6ev', 'buildtime': 1371042053L, 'version': '4.10.2'}, 'qemu-kvm': {'release': '2.355.el6_4.5', 'buildtime': 1369322701L, 'version': ''}, 'libvirt': {'release': '18.el6_4.9', 'buildtime': 1371761993L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.355.el6_4.5', 'buildtime': 1369322701L, 'version': ''}}, 'cpuModel': 'Intel Core i7 9xx (Nehalem Class Core i7)', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '
5662659ddc999e23946a1f6cf7a73796'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'netmask': '', 'iface': u'eth0', 'addr': '', 'bridged': False, 'interface': u'eth0', 'gateway': '', 'mtu': '1500'}, 'vlan123': {'iface': 'vlan123', 'addr': '', 'cfg': {'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vlan123', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'gateway': '', 'ports': ['eth0.123']}}, 'bridges': {'vlan123': {'addr': '', 'cfg': {'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vlan123', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': ['eth0.123']}}, 'uuid': '17B65ED6-68F1-F609-F968-4711842F7ADC', '
lastClientIface': 'eth0', 'nics': {'eth4': {'addr': '', 'cfg': {'DEVICE': 'eth4', 'HWADDR': '52:54:00:37:21:b2', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:37:21:b2', 'speed': 0}, 'eth3': {'addr': '', 'cfg': {'DEVICE': 'eth3', 'HWADDR': '52:54:00:f2:e7:96', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:f2:e7:96', 'speed': 0}, 'eth2': {'addr': '', 'cfg': {'DEVICE': 'eth2', 'HWADDR': '52:54:00:ab:d1:70', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:ab:d1:70', 'speed': 0}, 'eth1': {'addr': '', 'cfg': {'DEVICE': 'eth1', 'HWADDR': '52:54:00:87:7e:da', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:87:7e:da', 'speed': 0}, 'eth0': {'addr': '', 'cfg': {'NM_CONTROLLED': 'no', 'HWADDR': '52:54:00:6e:94:37', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:6e:94:37', 'speed': 0}}, 'software_revision': '23.0', 'management_ip': '', '
clusterLevels': ['3.0', '3.1', '3.2'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pse36,clflush,mmx,fxsr,sse,sse2,ss,syscall,nx,rdtscp,lm,up,rep_good,unfair_spinlock,pni,vmx,ssse3,cx16,sse4_1,sse4_2,popcnt,hypervisor,lahf_lm,model_Nehalem,model_Conroe,model_Penryn', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:213184f810f5', 'netConfigDirty': 'True', 'supportedENGINEs': ['3.0', '3.1', '3.2'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_
version': '4.10', 'memSize': '868', 'cpuSpeed': '2393.984', 'version_name': 'Snow Man', 'vlans': {'eth0.123': {'cfg': {'BRIDGE': 'vlan123', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.123', 'ONBOOT': 'yes'}, 'netmask': '', 'iface': 'eth0', 'addr': '', 'mtu': '1500'}}, 'cpuCores': '1', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '1', 'emulatedMachines': [u'rhel6.4.0', u'pc', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20130709.0.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}, 'lastClient': ''}}

It seems that the setupNetworks invoked with the proper parameters, but by looking the output of the getVdsCapabilities no ports are reported for the 'rhevm' network. In addition there is a strange interface name reported ('interface': u'eth0' - with leading u).
Comment 13 Dan Kenigsberg 2013-08-27 18:46:27 EDT
It's fine that rhevm has no ports, as this is a bridgeless network. u'eth0' is fine, too, as the leading u only means that it's a unicode string coming from python-ethtool.

What *is* worrying is that rhevm completely missing the 'cfg' element, as if ifcfg-rhevm has disappeared. And indeed etc/sysconfig/network-scripts/ifcfg-
rhevm is missing from the log collection.

It's a vdsm bug, which could be related to the fact it's a rhev-h.

vdsm 4.10.2-23.0.el6ev
operatingSystem': {'release': '20130709.0.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}

Anil, could you try to reproduce this on RHEL-H, too?
Comment 16 Anil Vettathu 2013-08-28 06:56:07 EDT
Created attachment 791300 [details]
RHEVM3.2 LC - RHEL6 Host
Comment 17 psebek 2013-08-29 06:22:31 EDT
It seems that problem is in stating the network as not bridged (change network to non-vm network I suppose?). The CFG field dissapper from RHEVH:vdsm.log after command:

Thread-32::DEBUG::2013-08-27 12:00:04,256::BindingXMLRPC::913::vds::(wrapper) client []::call setupNetworks with ({'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}}, {}, {'connectivityCheck': 'false'}) {} flowID [5939762f]

In RHEL there are creating vlan123 and getting rhevm['bridged'] to false in one command:

Thread-86::DEBUG::2013-08-28 05:52:02,449::BindingXMLRPC::913::vds::(wrapper) client []::call setupNetworks with ({'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}, 'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'false'}) {} flowID [613f0a]

and after that there is also NO cfg in getCapabillities.
Comment 18 psebek 2013-09-02 09:08:03 EDT
Created attachment 792828 [details]

Vdsm log from given procedure with:
RHEL - 6Server -
RHEVM 3.2.3-0.42.el6ev
Comment 19 psebek 2013-09-02 09:09:13 EDT
I managed to reproduce the bug:

RHEL - 6Server -
RHEVM 3.2.3-0.42.el6ev

Procedure was:
Change RHEVM to non-vm network in rhevm.
Add tagged network vlan123.
Add Vlan123 alongside rhevm to eth0.

After that host became non responsive. File /etc/sysconfig/network-scripts/ifcfg-rhevm was gone. Bridge rhevm from the host was gone.
Comment 20 psebek 2013-09-09 10:56:08 EDT
I've made some progress examining this bug.

I've been trying it with vdsm-4.12.0

Same behaviour as in described in Actual results can be achieved with command from host:

# vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bootproto:dhcp,bridged:false}}'

this should create bridgeless network on empty nic eth1. It ends with return code 0 (SUCCESS), but created network has no 'cfg' attribute in getVdsCaps output and also there is no file /etc/sysconfig/network-scripts/ifcfg-testnetwork.
Comment 21 psebek 2013-09-10 04:09:58 EDT
Incriminated piece of supervdsm.log:

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,539::supervdsmServer::89::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'testnetwork': {'nic': 'eth1', 'bootproto': 'dhcp', 'bridged': 'false'}}, {}, {'connectivityCheck': 'False'}) {}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,540::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,551::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,553::libvirtconnection::124::libvirtconnection::(get) trying to connect libvirt
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,601::configNetwork::491::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'testnetwork': {'nic': 'eth1', 'bootproto': 'dhcp', 'bridged': 'false'}}, bondings:{}, options:{'connectivityCheck': 'False'}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,601::configNetwork::495::root::(setupNetworks) Validating configuration
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,602::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,604::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,609::configNetwork::498::setupNetworks::(setupNetworks) Applying...
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,610::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,613::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,618::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,620::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,625::configNetwork::539::setupNetworks::(setupNetworks) Adding network 'testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,625::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,628::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,633::configNetwork::176::root::(addNetwork) validating network...
MainProcess|Thread-15::INFO::2013-09-10 09:41:02,633::configNetwork::191::root::(addNetwork) Adding network testnetwork with vlan=None, bonding=None, nics=['eth1'], bondingOptions=None, mtu=None, bridged=False, options={'bootproto': 'dhcp', 'implicitBonding': True}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,633::utils::486::root::(execCmd) '/sbin/ip route show to table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,636::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::311::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth1
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::386::root::(_persistentBackup) backing up ifcfg-eth1: DEVICE=eth1

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::255::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/ifcfg-eth1 (until next 'set safe config')
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,702::utils::486::root::(execCmd) '/sbin/ifdown eth1' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,829::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,830::utils::486::root::(execCmd) '/sbin/ifup eth1' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,348::sourceRouteThread::18::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1378798866
MainThread::INFO::2013-09-10 09:41:06,350::sourceRoute::81::root::(configure) Configuring gateway - ip:, network:, subnet:, gateway:, table: 170033873, device: eth1
MainThread::DEBUG::2013-09-10 09:41:06,350::utils::486::root::(execCmd) '/sbin/ip route add via dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,367::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,367::utils::486::root::(execCmd) '/sbin/ip route add via dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,371::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,371::utils::486::root::(execCmd) '/sbin/ip rule add from table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,374::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,374::utils::486::root::(execCmd) '/sbin/ip rule add from all to dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,378::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,402::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,408::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::ifcfg::265::root::(_atomicNetworkBackup) Backed up testnetwork
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::ifcfg::274::root::(_persistentNetworkBackup) backing up network testnetwork: # original file did not exist

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,410::ifcfg::255::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/logicalnetworks/testnetwork (until next 'set safe config')
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,527::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) return setupNetworks with None
Comment 22 psebek 2013-09-10 07:00:46 EDT
Together with Dan we found out that I was going wrong way. My assumption was that for every network should be ifcfg file, but it is not true. For bridgless network there is no ifcfg file and bootproto should be taken from underlying interface. So it is correct behaviour what I write in commands #20 and #21. We should only focus on disappearing BOOTPROTO line from network interface as was stated in description of this bug.
Comment 23 psebek 2013-09-11 05:25:17 EDT
Anil, I was looking to logs you provided and in ifcfg-eth0 files is 
BOOTPROTO=dhcp line. Where are logs with missing bootproto? Or am I
looking to wrong files?

Also little note: I was not able to reproduce it with vdsm-4.12.0 and 
rhevm 3.2.3 using 'testnetwork' or 'rhevm'.
Comment 24 psebek 2013-09-12 05:38:56 EDT
Created attachment 796716 [details]
Reproducer using  setupNetworks command
Comment 25 psebek 2013-09-12 05:39:59 EDT
Ok I was able to reproduce this only using vdsm-4.12.0 (not engine) with these commands:
$ vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bridged:true,bootproto:dhcp}}'

$ vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bridged:false,bootproto:dhcp}}'

$ vdsClient -s 0 setupNetworks networks='{vlan123:{nic:eth1,bridged:true,vlan:123,bootproto:none}}'

$ vdsClient -s 0 setupNetworks networks='{vlan123:{nic:eth1,bridged:true,vlan:123,bootproto:none}}'

I added here output of my actions.  attachment 796716 [details]

And this is actually happening when are this actions done in the engine. In supervdsm.log of rhel that Anil provided (rhel/log-collector-data/dhcp209-216.gsslab.pnq.redhat.com/dhcp209-216-2013082806061377650210/var/log/vdsm/supervdsm.log) that it add vlan123 on line 55:

setupNetworks::(setupNetworks) Adding network 'vlan123'

then it removes that network on line 74:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'vlan123': {'remove': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

And add it again on line 113:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

First vlan is added using setupNetworks command from line 8:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}, 'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

But I don't understand why is called another setupNetworks with removal and setting up again. Maybe something from engine that does not count on setting both networks in two commands (just speculation)? 

So we have got two questions here:
1) Why is 'vlan123' added, removed and added again?
2) Why vdsm remove BOOTPROTO line when is edited 'vlan123' network?
Comment 26 psebek 2013-09-12 12:08:13 EDT
I believe I found solution for one use case. I added link to gerrit to external tracker. As you can see when the setup was called in one setupNetworks command the problem was with overwriting bootproto value with value from current ifcfg file. Ifcfg-eth1 was setted by vlan123 and therefore without bootproto (vlan123 is bridged so there should be no bootproto in ifcfg-eth1). New patch fixes this wrong behaviour.

But there still stays problem with double call of setupNetworks add vlan123. I hope it will be something related and I find solution quickly.
Comment 28 psebek 2013-09-17 11:54:08 EDT
Ok I found second half of the problem. When I double add the vlan123, the bridge vlan123 is removed before second adding obviously. Together with bridge vlan123 is removed also vlan eth1.123 and is called removal of nic eth1. During removal of nic is called method ConfigWriter._removeConfigValues and that effectively deletes line BOOTPROTO from cfg file. So I will have to add condition to not remove that line, or nic if the nic is still used by another network.
The patch will be ready by tomorrow.
Comment 29 Martin Pavlik 2013-09-18 09:04:08 EDT
*** Bug 997529 has been marked as a duplicate of this bug. ***
Comment 30 Martin Pavlik 2013-09-24 08:56:25 EDT
works in is16
Comment 32 Charlie 2013-11-27 19:29:03 EST
This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:


Thanks in advance.
Comment 33 errata-xmlrpc 2014-01-21 11:26:47 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.