Bug 980174

*** This bug has been marked as a duplicate of bug 980486 ***

Just to make sure, the scenario above was done via the REST API (or SDK or CLI)?

Anil, can you provide engine.log and vdsm.log so we can see which parameters were sent via the setupNetworks to the host ? 

In addition, did you perform the test on Rhev-h or rhel ?
What are the engine and vdsm versions ?
How did you configured configured the management network as non-vm network on the host?

(The described sequence of operations is quite common and i wasn't able to reproduce it on ovirt-3.3)

Moti, I tried to the reproduce the issue on rhevm-3.3.0-0.16.master.el6ev.noarch & vdsm-4.12.0-72.git287bb7e.el6ev.x86_64. The issue was not reproducible.

The RHEV3.2 environment which I used for reproducing had been rebuilt. I will reproduce it again on RHEVM3.2 and provide the details.

Created attachment 791130 [details]
rhevm3.2 log collector

Moti, The issue is reproducible in rhevm-3.2.0-11.37.el6ev.noarch and vdsm-4.10.2-23.0.el6ev. I tested using RHEV-H not RHEL. I am also attaching the log collector output, which contains logs,db and sosreport from hypervisor.

Please note create rhevm network as non-vm network and then add the vlan logical network on top of the rhevm interface.

Let me know if you need more details.

This is the churn from the logs:

engine.log indicates parameters were sent correctly to VDSM:
2013-08-27 08:04:18,238 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp-/127.0.0.1:8702-8) [274b135d] START, SetupNetworksVDSCommand(HostName = dhcp210-139.gsslab.pnq.redhat.com, HostId = 276cc728-eef4-4e2b-8b08-e363aa85265d, force=false, checkConnectivity=false, conectivityTimeout=120,
	networks=[vlan123 {id=06ed7636-fdb7-485c-b18f-c6eb460fe68b, description=null, subnet=null, gateway=null, type=null, vlanId=123, stp=false, dataCenterId=5849b030-626e-47cb-ad90-3ce782d831b3, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true}}],
	bonds=[],
	interfaces=[eth0 {id=c785b47d-5043-42f9-b851-88fe577b3639, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth0, macAddress=52:54:00:6e:94:37, networkName=rhevm, bondName=null, bootProtocol=DHCP, address=10.65.210.139, subnet=255.255.252.0, gateway=10.65.211.254, mtu=1500, bridged=false, speed=0, type=2, networkImplementationDetails={inSync=true, managed=true}},
		eth2 {id=f3cc386a-9a4a-4a83-a175-4f72a6faebf1, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth2, macAddress=52:54:00:ab:d1:70, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth1 {id=fd510e34-d892-4598-9ada-0a5dce316070, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth1, macAddress=52:54:00:87:7e:da, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth4 {id=433fdb06-8198-4263-9188-563391f36256, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth4, macAddress=52:54:00:37:21:b2, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth3 {id=d56f1699-fe68-417a-86af-e5668755a586, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, name=eth3, macAddress=52:54:00:f2:e7:96, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null},
		eth0.123 {id=null, vdsId=276cc728-eef4-4e2b-8b08-e363aa85265d, macAddress=null, networkName=vlan123, vlanId=123, bonded=null, bondName=null, bondOptions=null, bootProtocol=NONE, address=null, subnet=null, gateway=null, mtu=0, bridged=true, speed=null, type=0, networkImplementationDetails=null}],
	removedNetworks=[],
	removedBonds=[]), log id: 2f223717
...
2013-08-27 08:06:00,794 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-49) [766fda37] Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host
2013-08-27 08:06:00,799 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-49) [766fda37] ResourceManager::activateVds - failed to get VDS = 276cc728-eef4-4e2b-8b08-e363aa85265d capabilities with error: java.net.NoRouteToHostException: No route to host.

From vdsm.log:
Thread-58::DEBUG::2013-08-27 12:04:17,845::BindingXMLRPC::913::vds::(wrapper) client [10.65.210.228]::call setupNetworks with ({'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'false'}) {} flowID [274b135d]

Thread-59::DEBUG::2013-08-27 12:06:12,230::BindingXMLRPC::913::vds::(wrapper) client [10.65.210.228]::call getCapabilities with () {}
Thread-59::DEBUG::2013-08-27 12:06:12,293::BindingXMLRPC::920::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:213184f810f5'}], 'FC': []}, 'packages2': {'kernel': {'release': '358.14.1.el6.x86_64', 'buildtime': 1371484460.0, 'version': '2.6.32'}, 'spice-server': {'release': '12.el6_4.1', 'buildtime': 1368193709L, 'version': '0.12.0'}, 'vdsm': {'release': '23.0.el6ev', 'buildtime': 1371042053L, 'version': '4.10.2'}, 'qemu-kvm': {'release': '2.355.el6_4.5', 'buildtime': 1369322701L, 'version': '0.12.1.2'}, 'libvirt': {'release': '18.el6_4.9', 'buildtime': 1371761993L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.355.el6_4.5', 'buildtime': 1369322701L, 'version': '0.12.1.2'}}, 'cpuModel': 'Intel Core i7 9xx (Nehalem Class Core i7)', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '
5662659ddc999e23946a1f6cf7a73796'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'netmask': '255.255.252.0', 'iface': u'eth0', 'addr': '10.65.210.139', 'bridged': False, 'interface': u'eth0', 'gateway': '10.65.211.254', 'mtu': '1500'}, 'vlan123': {'iface': 'vlan123', 'addr': '', 'cfg': {'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vlan123', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['eth0.123']}}, 'bridges': {'vlan123': {'addr': '', 'cfg': {'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vlan123', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': ['eth0.123']}}, 'uuid': '17B65ED6-68F1-F609-F968-4711842F7ADC', '
lastClientIface': 'eth0', 'nics': {'eth4': {'addr': '', 'cfg': {'DEVICE': 'eth4', 'HWADDR': '52:54:00:37:21:b2', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:37:21:b2', 'speed': 0}, 'eth3': {'addr': '', 'cfg': {'DEVICE': 'eth3', 'HWADDR': '52:54:00:f2:e7:96', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:f2:e7:96', 'speed': 0}, 'eth2': {'addr': '', 'cfg': {'DEVICE': 'eth2', 'HWADDR': '52:54:00:ab:d1:70', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:ab:d1:70', 'speed': 0}, 'eth1': {'addr': '', 'cfg': {'DEVICE': 'eth1', 'HWADDR': '52:54:00:87:7e:da', 'ONBOOT': 'no'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:87:7e:da', 'speed': 0}, 'eth0': {'addr': '10.65.210.139', 'cfg': {'NM_CONTROLLED': 'no', 'HWADDR': '52:54:00:6e:94:37', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.252.0', 'hwaddr': '52:54:00:6e:94:37', 'speed': 0}}, 'software_revision': '23.0', 'management_ip': '', '
clusterLevels': ['3.0', '3.1', '3.2'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pse36,clflush,mmx,fxsr,sse,sse2,ss,syscall,nx,rdtscp,lm,up,rep_good,unfair_spinlock,pni,vmx,ssse3,cx16,sse4_1,sse4_2,popcnt,hypervisor,lahf_lm,model_Nehalem,model_Conroe,model_Penryn', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:213184f810f5', 'netConfigDirty': 'True', 'supportedENGINEs': ['3.0', '3.1', '3.2'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_
version': '4.10', 'memSize': '868', 'cpuSpeed': '2393.984', 'version_name': 'Snow Man', 'vlans': {'eth0.123': {'cfg': {'BRIDGE': 'vlan123', 'VLAN': 'yes', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'eth0.123', 'ONBOOT': 'yes'}, 'netmask': '', 'iface': 'eth0', 'addr': '', 'mtu': '1500'}}, 'cpuCores': '1', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '1', 'emulatedMachines': [u'rhel6.4.0', u'pc', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20130709.0.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}, 'lastClient': '10.65.210.228'}}

It seems that the setupNetworks invoked with the proper parameters, but by looking the output of the getVdsCapabilities no ports are reported for the 'rhevm' network. In addition there is a strange interface name reported ('interface': u'eth0' - with leading u).

It's fine that rhevm has no ports, as this is a bridgeless network. u'eth0' is fine, too, as the leading u only means that it's a unicode string coming from python-ethtool.

What *is* worrying is that rhevm completely missing the 'cfg' element, as if ifcfg-rhevm has disappeared. And indeed etc/sysconfig/network-scripts/ifcfg-
rhevm is missing from the log collection.

It's a vdsm bug, which could be related to the fact it's a rhev-h.

vdsm 4.10.2-23.0.el6ev
operatingSystem': {'release': '20130709.0.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}

Anil, could you try to reproduce this on RHEL-H, too?

Created attachment 791300 [details]
RHEVM3.2 LC - RHEL6 Host

It seems that problem is in stating the network as not bridged (change network to non-vm network I suppose?). The CFG field dissapper from RHEVH:vdsm.log after command:

Thread-32::DEBUG::2013-08-27 12:00:04,256::BindingXMLRPC::913::vds::(wrapper) client [10.65.210.228]::call setupNetworks with ({'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}}, {}, {'connectivityCheck': 'false'}) {} flowID [5939762f]

In RHEL there are creating vlan123 and getting rhevm['bridged'] to false in one command:

Thread-86::DEBUG::2013-08-28 05:52:02,449::BindingXMLRPC::913::vds::(wrapper) client [10.65.210.228]::call setupNetworks with ({'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}, 'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'false'}) {} flowID [613f0a]

and after that there is also NO cfg in getCapabillities.

Created attachment 792828 [details]
vdsm.log

Vdsm log from given procedure with:
RHEL - 6Server - 6.4.0.4.el6
vdsm-4.10.2-25.0.el6ev
RHEVM 3.2.3-0.42.el6ev

I managed to reproduce the bug:

RHEL - 6Server - 6.4.0.4.el6
vdsm-4.10.2-25.0.el6ev
RHEVM 3.2.3-0.42.el6ev

Procedure was:
Change RHEVM to non-vm network in rhevm.
Add tagged network vlan123.
Add Vlan123 alongside rhevm to eth0.

After that host became non responsive. File /etc/sysconfig/network-scripts/ifcfg-rhevm was gone. Bridge rhevm from the host was gone.

I've made some progress examining this bug.

I've been trying it with vdsm-4.12.0

Same behaviour as in described in Actual results can be achieved with command from host:

# vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bootproto:dhcp,bridged:false}}'

this should create bridgeless network on empty nic eth1. It ends with return code 0 (SUCCESS), but created network has no 'cfg' attribute in getVdsCaps output and also there is no file /etc/sysconfig/network-scripts/ifcfg-testnetwork.

Incriminated piece of supervdsm.log:

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,539::supervdsmServer::89::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'testnetwork': {'nic': 'eth1', 'bootproto': 'dhcp', 'bridged': 'false'}}, {}, {'connectivityCheck': 'False'}) {}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,540::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,551::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,553::libvirtconnection::124::libvirtconnection::(get) trying to connect libvirt
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,601::configNetwork::491::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'testnetwork': {'nic': 'eth1', 'bootproto': 'dhcp', 'bridged': 'false'}}, bondings:{}, options:{'connectivityCheck': 'False'}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,601::configNetwork::495::root::(setupNetworks) Validating configuration
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,602::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,604::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,609::configNetwork::498::setupNetworks::(setupNetworks) Applying...
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,610::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,613::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,618::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,620::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,625::configNetwork::539::setupNetworks::(setupNetworks) Adding network 'testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,625::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,628::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,633::configNetwork::176::root::(addNetwork) validating network...
MainProcess|Thread-15::INFO::2013-09-10 09:41:02,633::configNetwork::191::root::(addNetwork) Adding network testnetwork with vlan=None, bonding=None, nics=['eth1'], bondingOptions=None, mtu=None, bridged=False, options={'bootproto': 'dhcp', 'implicitBonding': True}
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,633::utils::486::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,636::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::311::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth1
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::386::root::(_persistentBackup) backing up ifcfg-eth1: DEVICE=eth1
ONBOOT=yes
HWADDR=52:54:00:30:eb:41
NM_CONTROLLED=no
STP=no

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,641::ifcfg::255::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/ifcfg-eth1 (until next 'set safe config')
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,702::utils::486::root::(execCmd) '/sbin/ifdown eth1' (cwd None)
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,829::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:02,830::utils::486::root::(execCmd) '/sbin/ifup eth1' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,348::sourceRouteThread::18::root::(process_IN_CLOSE_WRITE_filePath) Responding to DHCP response in /var/run/vdsm/sourceRoutes/1378798866
MainThread::INFO::2013-09-10 09:41:06,350::sourceRoute::81::root::(configure) Configuring gateway - ip: 10.34.130.209, network: 10.34.130.0/23, subnet: 255.255.254.0, gateway: 10.34.131.254, table: 170033873, device: eth1
MainThread::DEBUG::2013-09-10 09:41:06,350::utils::486::root::(execCmd) '/sbin/ip route add 0.0.0.0/0 via 10.34.131.254 dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,367::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,367::utils::486::root::(execCmd) '/sbin/ip route add 10.34.130.0/23 via 10.34.130.209 dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,371::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,371::utils::486::root::(execCmd) '/sbin/ip rule add from 10.34.130.0/23 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,374::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2013-09-10 09:41:06,374::utils::486::root::(execCmd) '/sbin/ip rule add from all to 10.34.130.0/23 dev eth1 table 170033873' (cwd None)
MainThread::DEBUG::2013-09-10 09:41:06,378::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,402::utils::505::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,408::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::ifcfg::265::root::(_atomicNetworkBackup) Backed up testnetwork
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-testnetwork'
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,409::ifcfg::274::root::(_persistentNetworkBackup) backing up network testnetwork: # original file did not exist

MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,410::ifcfg::255::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/logicalnetworks/testnetwork (until next 'set safe config')
MainProcess|Thread-15::DEBUG::2013-09-10 09:41:06,527::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) return setupNetworks with None

Together with Dan we found out that I was going wrong way. My assumption was that for every network should be ifcfg file, but it is not true. For bridgless network there is no ifcfg file and bootproto should be taken from underlying interface. So it is correct behaviour what I write in commands #20 and #21. We should only focus on disappearing BOOTPROTO line from network interface as was stated in description of this bug.

Anil, I was looking to logs you provided and in ifcfg-eth0 files is 
BOOTPROTO=dhcp line. Where are logs with missing bootproto? Or am I
looking to wrong files?

Also little note: I was not able to reproduce it with vdsm-4.12.0 and 
rhevm 3.2.3 using 'testnetwork' or 'rhevm'.

Created attachment 796716 [details]
Reproducer using  setupNetworks command

Ok I was able to reproduce this only using vdsm-4.12.0 (not engine) with these commands:
# ADD FIRST BRIDGED NETWORK
$ vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bridged:true,bootproto:dhcp}}'

# CHANGE IT TO THE BRIDGELESS NETWORK, REMOVE ifcfg-testnetwork, ADD LINE "BOOTPROTO=dhcp" TO ifcfg-eth1
$ vdsClient -s 0 setupNetworks networks='{testnetwork:{nic:eth1,bridged:false,bootproto:dhcp}}'

# ADD NETWORK VLAN123, EVERYTHING IS OK, BOOTPROTO LINE STILL PRESENT
$ vdsClient -s 0 setupNetworks networks='{vlan123:{nic:eth1,bridged:true,vlan:123,bootproto:none}}'

# CALL SAME COMMAND TO ADD VLAN123, NOW THE BOOTPROTO LINE IS GONE! 
$ vdsClient -s 0 setupNetworks networks='{vlan123:{nic:eth1,bridged:true,vlan:123,bootproto:none}}'

I added here output of my actions.  attachment 796716 [details]

And this is actually happening when are this actions done in the engine. In supervdsm.log of rhel that Anil provided (rhel/log-collector-data/dhcp209-216.gsslab.pnq.redhat.com/dhcp209-216-2013082806061377650210/var/log/vdsm/supervdsm.log) that it add vlan123 on line 55:

setupNetworks::(setupNetworks) Adding network 'vlan123'

then it removes that network on line 74:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'vlan123': {'remove': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

And add it again on line 113:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

First vlan is added using setupNetworks command from line 8:

setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'rhevm': {'nic': 'eth0', 'bootproto': 'dhcp', 'bridged': 'false'}, 'vlan123': {'nic': 'eth0', 'vlan': '123', 'STP': 'no', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'false'}

But I don't understand why is called another setupNetworks with removal and setting up again. Maybe something from engine that does not count on setting both networks in two commands (just speculation)? 

So we have got two questions here:
1) Why is 'vlan123' added, removed and added again?
2) Why vdsm remove BOOTPROTO line when is edited 'vlan123' network?

I believe I found solution for one use case. I added link to gerrit to external tracker. As you can see when the setup was called in one setupNetworks command the problem was with overwriting bootproto value with value from current ifcfg file. Ifcfg-eth1 was setted by vlan123 and therefore without bootproto (vlan123 is bridged so there should be no bootproto in ifcfg-eth1). New patch fixes this wrong behaviour.

But there still stays problem with double call of setupNetworks add vlan123. I hope it will be something related and I find solution quickly.

Ok I found second half of the problem. When I double add the vlan123, the bridge vlan123 is removed before second adding obviously. Together with bridge vlan123 is removed also vlan eth1.123 and is called removal of nic eth1. During removal of nic is called method ConfigWriter._removeConfigValues and that effectively deletes line BOOTPROTO from cfg file. So I will have to add condition to not remove that line, or nic if the nic is still used by another network.
The patch will be ready by tomorrow.

*** Bug 997529 has been marked as a duplicate of this bug. ***

works in is16

This bug is currently attached to errata RHBA-2013:15291. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to 
minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0040.html