Description of problem:
when using setupNetworks with 'connectivityCheck=true' and losing network connectivity, vdsm does not rollback network configuration to the previous state.


Version-Release number of selected component (if applicable):
vdsm-4.9.3.2-0.fc16.x86_64

Thread-38836::DEBUG::2012-02-06 17:38:52,789::clientIF::76::vds::(wrapper) [10.35.97.87]::call setupNetworks with ({'ovirtmgmt': {'remove': 'true'}, 'david': {'nic': 'eth1', 'netmask': '255.255.255.0', 'ipaddr': '1.1.1.1'}}, {'bond4': {'
remove': 'true'}, 'bond0': {'remove': 'true'}, 'bond1': {'remove': 'true'}, 'bond2': {'remove': 'true'}, 'bond3': {'remove': 'true'}}, {'connectivityCheck': 'true', 'connectivityTimeout': '30'}) {}
MainProcess|Thread-38836::INFO::2012-02-06 17:38:52,801::configNetwork::809::setupNetworks::(setupNetworks) Setting up network
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:52,802::configNetwork::810::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'remove': 'true'}, 'david': {'nic': 'eth1', 'netmask': '2
55.255.255.0', 'ipaddr': '1.1.1.1'}}, bondings:{'bond4': {'remove': 'true'}, 'bond0': {'remove': 'true'}, 'bond1': {'remove': 'true'}, 'bond2': {'remove': 'true'}, 'bond3': {'remove': 'true'}}, options:{'connectivityCheck': 'true', 'conn
ectivityTimeout': '30'}
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:52,802::configNetwork::814::root::(setupNetworks) Validating configuration
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:52,834::configNetwork::817::setupNetworks::(setupNetworks) Applying...
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:52,834::configNetwork::823::setupNetworks::(setupNetworks) Removing network 'ovirtmgmt'
MainProcess|Thread-38836::INFO::2012-02-06 17:38:52,846::configNetwork::536::root::(delNetwork) Removing bridge ovirtmgmt with vlan=None, bonding=None, nics=['eth0']. options={}
Thread-38837::DEBUG::2012-02-06 17:38:52,857::clientIF::76::vds::(wrapper) [10.35.97.87]::call getVdsCapabilities with () {}
Thread-38837::DEBUG::2012-02-06 17:38:52,874::utils::595::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None)
Thread-38837::DEBUG::2012-02-06 17:38:52,900::utils::595::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-38837::DEBUG::2012-02-06 17:38:52,901::utils::595::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None)
Thread-38837::DEBUG::2012-02-06 17:38:52,926::utils::595::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-38837::DEBUG::2012-02-06 17:38:52,927::utils::595::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None)
Thread-38837::DEBUG::2012-02-06 17:38:52,953::utils::595::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-38837::DEBUG::2012-02-06 17:38:52,954::utils::595::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None)
Thread-38837::DEBUG::2012-02-06 17:38:52,980::utils::595::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-38837::DEBUG::2012-02-06 17:38:52,981::utils::595::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None)
Thread-38837::DEBUG::2012-02-06 17:38:53,008::utils::595::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-38837::DEBUG::2012-02-06 17:38:53,009::clientIF::81::vds::(wrapper) return getVdsCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:f93ed9c053
9a'}], 'FC': []}, 'packages2': {'kernel': {'release': '3.fc16.x86_64', 'buildtime': 1327325777.0, 'version': '3.2.1'}, 'spice-server': {'release': '1.fc16', 'buildtime': '1327339129', 'version': '0.10.1'}, 'vdsm': {'release': '0.fc16', '
buildtime': '1327521056', 'version': '4.9.3.2'}, 'qemu-kvm': {'release': '2.fc17', 'buildtime': '1326486018', 'version': '1.0'}, 'libvirt': {'release': '4.fc16', 'buildtime': '1324326688', 'version': '0.9.6'}, 'qemu-img': {'release': '2.
fc17', 'buildtime': '1326486018', 'version': '1.0'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'addr': '10.35.77.4', 
'cfg': {'DEVICE': 'ovirtmgmt', 'BOOTPROTO': 'dhcp', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['eth0'], 'netmask': '255.255.255.0', 'stp': 'off', 'gateway': '10.35.77.254'}}, 'uuid': 'B494A402-4B76-3796-BDF1-B76758D61B1B_00:21:5E:3F:D
B:2C', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth1': {'hwaddr': '00:21:5E:3F:DB:2E', 'netmask': '', 'speed': 0, 'addr': ''}, 'eth0': {'hwaddr': '00:21:5E:3F:DB:2C', 'netmask': '', 'speed': 1000, 'addr': ''}}, 'software_revision': '0',
 'management_ip': '', 'clusterLevels': ['3.0'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,lm,constant_tsc,arch_perfmon,pebs,bts,rep_go
od,nopl,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,xsave,lahf_lm,dts,tpr_shadow,vnmi,flexpriority,model_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_coreduo,mod
el_core2duo,model_n270,model_Conroe,model_Penryn,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:f93ed9c0539a', 'memSize': '7987', 'reservedMem': '321', 'bondings': {'bond4': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'n
etmask': '', 'addr': '', 'slaves': []}, 'bond0': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond1': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond2': 
{'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond3': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}}, 'software_version': '4.9', 'cpuSpeed': '1992.000', 'cpuSo
ckets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [], 'operatingSystem': {'release': '1', 'version': '16', 'name': 'Fedora'}, 'lastClient': '10.35.97.87'}}
MainProcess|Thread-38836::WARNING::2012-02-06 17:38:53,353::configNetwork::78::root::(ifdown) 
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,364::configNetwork::162::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth0
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,366::configNetwork::208::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-eth0 (until next 'set safe config')
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,366::configNetwork::162::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,367::configNetwork::208::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt (until next 'set safe config')
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,380::configNetwork::585::root::(delNetwork) failed to remove libvirt network vdsm-ovirtmgmt
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,381::configNetwork::839::setupNetworks::(setupNetworks) Adding network 'david'
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,392::configNetwork::429::root::(addNetwork) validating bridge...
MainProcess|Thread-38836::INFO::2012-02-06 17:38:53,415::configNetwork::434::root::(addNetwork) Adding bridge david with vlan=None, bonding=None, nics=['eth1']. bondingOptions=None, options={}
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,416::configNetwork::208::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-david (until next 'set safe config')
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,381::configNetwork::839::setupNetworks::(setupNetworks) Adding network 'david'
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,392::configNetwork::429::root::(addNetwork) validating bridge...
MainProcess|Thread-38836::INFO::2012-02-06 17:38:53,415::configNetwork::434::root::(addNetwork) Adding bridge david with vlan=None, bonding=None, nics=['eth1']. bondingOptions=None, options={}
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,416::configNetwork::208::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-david (until next 'set safe config')
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,416::configNetwork::162::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth1
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:53,417::configNetwork::208::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-eth1 (until next 'set safe config')
MainProcess|Thread-38836::WARNING::2012-02-06 17:38:53,574::configNetwork::78::root::(ifdown) 
MainProcess|Thread-38836::DEBUG::2012-02-06 17:38:56,095::configNetwork::846::setupNetworks::(setupNetworks) Checking connectivity...
MainProcess|Thread-38836::INFO::2012-02-06 17:39:27,128::configNetwork::848::setupNetworks::(setupNetworks) Connectivity check failed, rolling back
MainProcess|Thread-38836::INFO::2012-02-06 17:39:27,139::configNetwork::536::root::(delNetwork) Removing bridge david with vlan=None, bonding=None, nics=['eth1']. options={}
MainProcess|Thread-38836::WARNING::2012-02-06 17:39:27,549::configNetwork::78::root::(ifdown) 
MainProcess|Thread-38836::DEBUG::2012-02-06 17:39:27,560::configNetwork::162::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth1
MainProcess|Thread-38836::DEBUG::2012-02-06 17:39:27,561::configNetwork::162::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-david
MainProcess|Thread-38836::INFO::2012-02-06 17:39:27,576::configNetwork::167::root::(restoreAtomicBackup) Rolling back configuration (restoring atomic backup)
MainProcess|Thread-38836::ERROR::2012-02-06 17:39:27,577::configNetwork::856::setupNetworks::(setupNetworks) (10, 'connectivity check failed')
Traceback (most recent call last):
  File "/usr/share/vdsm/configNetwork.py", line 852, in setupNetworks
    raise ConfigNetworkError(ne.ERR_LOST_CONNECTION, 'connectivity check failed')
ConfigNetworkError: (10, 'connectivity check failed')
MainProcess|Thread-38836::ERROR::2012-02-06 17:39:27,577::supervdsmServer::50::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 48, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 100, in setupNetworks
    return configNetwork.setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/configNetwork.py", line 852, in setupNetworks
    raise ConfigNetworkError(ne.ERR_LOST_CONNECTION, 'connectivity check failed')
ConfigNetworkError: (10, 'connectivity check failed')
Thread-38836::ERROR::2012-02-06 17:39:27,578::clientIF::1207::vds::(setupNetworks) connectivity check failed
Traceback (most recent call last):
  File "/usr/share/vdsm/clientIF.py", line 1205, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 49, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 47, in <lambda>
    callMethod = lambda : getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (10, 'connectivity check failed')
Thread-38836::DEBUG::2012-02-06 17:39:27,579::clientIF::81::vds::(wrapper) return setupNetworks with {'status': {'message': 'connectivity check failed', 'code': 10}}
Thread-1274::ERROR::2012-02-06 17:40:00,110::domainMonitor::120::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `e9f64fd3-9fa8-48cc-8b9a-c1f44febd705` monitoring information
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/domainMonitor.py", line 103, in _monitorDomain
    domain.selftest()
  File "/usr/share/vdsm/storage/nfsSD.py", line 131, in selftest
    fileSD.FileStorageDomain.selftest(self)
  File "/usr/share/vdsm/storage/fileSD.py", line 295, in selftest
    self.oop.os.statvfs(self.domaindir)
  File "/usr/share/vdsm/storage/processPool.py", line 53, in wrapper
    return self.runExternally(func, *args, **kwds)
  File "/usr/share/vdsm/storage/processPool.py", line 64, in runExternally
    return self._procPool.runExternally(*args, **kwargs)
  File "/usr/share/vdsm/storage/processPool.py", line 145, in runExternally
    raise Timeout("Operation Stuck")
Timeout: Operation Stuck


How reproducible:
100%

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info: