Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1549145

Summary:	Network actions fail due to VDSM NM driver async operations
Product:	[oVirt] vdsm	Reporter:	Edward Haas <edwardh>
Component:	Core	Assignee:	Edward Haas <edwardh>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Meni Yakove <myakove>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.20.19	CC:	bugs, danken, edwardh, mburman, phoracek
Target Milestone:	ovirt-4.2.2	Flags:	rule-engine: ovirt-4.2+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-29 11:09:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Network	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Edward Haas 2018-02-26 14:00:31 UTC

VDSM is using NM at the moment to mainly acquire devices that are managed by NM and perform a "release" in order to be used by VDSM.

During the integration and OST tests several failures have been detected which have been pin-pointed to the NM driver that VDSM uses.

It looks like the NM devices properties are not always available, either when the device disappears in between accessing them or when the state of the device does not yet exposes the related properties.

The driver should expose better errors and the use of it should take into account that the device state can change while checking its properties.

Comment 1 Edward Haas 2018-02-27 12:54:00 UTC

Another incident was detected:

MainProcess|jsonrpc/6::ERROR::2018-02-27 05:00:47,207::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 217, in setupNetworks
    _setup_networks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 238, in _setup_networks
    netswitch.configurator.setup(networks, bondings, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 140, in setup
    _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 154, in _setup_legacy
    _netinfo)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 346, in remove_networks
    keep_bridge=keep_bridge)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 168, in wrapped
    return func(network, configurator, net_info, bondattr, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 306, in _del_network
    net_ent_to_remove.remove()
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 146, in remove
    self.configurator.removeVlan(self)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 241, in removeVlan
    vlan.device.remove()
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 266, in remove
    self.configurator.removeBond(self)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 291, in removeBond
    bonding.configure()
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 244, in configure
    self.configurator.configureBond(self, **opts)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 130, in configureBond
    IfcfgAcquire.acquire_device(bond.name)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg_acquire.py", line 38, in acquire_device
    IfcfgAcquireNMonline.acquire_device(device)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg_acquire.py", line 55, in acquire_device
    dev.syncoper.waitfor_activated_state()
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 93, in waitfor_activated_state
    self.waitfor_state(types.NMDeviceState.ACTIVATED, timeout)
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 97, in waitfor_state
    actual_state = self._device.state
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 73, in state
    return self._property('State')
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/errors.py", line 69, in wrapped_func
    six.reraise(dst_exception, dst_exception(*value.args), tb)
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/errors.py", line 65, in wrapped_func
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/nm/nmdbus/device.py", line 85, in _property
    return self._properties.Get(self.IF_NAME, property_name)
  File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 70, in __call__
    return self._proxy_method(*args, **keywords)
  File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 145, in __call__
    **keywords)
  File "/usr/lib64/python2.7/site-packages/dbus/connection.py", line 651, in call_blocking
    message, timeout)
NMPropertiesNotFoundError: No such interface 'org.freedesktop.DBus.Properties' on object at path /org/freedesktop/NetworkManager/Devices/35

Comment 2 Michael Burman 2018-03-06 14:39:09 UTC

Edy can you please verify this report on your OST tests?
Thanks,

Comment 3 Edward Haas 2018-03-07 16:33:19 UTC

DS OST has not detected this problem again since, so I'm assuming it was fixed.
If this will happen again, we can consider reopening this BZ.

Comment 4 Sandro Bonazzola 2018-03-29 11:09:48 UTC

This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.