Bug 1020356

Summary: stray libvirt networks make vdsm fail to recognize management network on native interface with "Network defined withoutdevices" error of setupNetworks
Product: Red Hat Enterprise Virtualization Manager Reporter: David Jaša <djasa>
Component: vdsmAssignee: Antoni Segura Puimedon <asegurap>
Status: CLOSED ERRATA QA Contact: Martin Pavlik <mpavlik>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: adahms, bazulay, danken, djasa, ecohen, gklein, iheim, lpeer, mpavlik, myakove, nyechiel, sbonazzo, yeylon
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: network
Fixed In Version: ovirt-3.4.0-beta2 Doc Type: Bug Fix
Doc Text:
The Red Hat Enterprise Virtualization Manager now recognizes preconfigured networks in which the management network is bridgeless and untagged, making it unnecessary to drag and drop these networks to the bond interface in order to recognize them.
Story Points: ---
Clone Of:
: 1044024 (view as bug list) Environment:
Last Closed: 2014-06-09 13:25:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1044024, 1078909, 1142926    

Description David Jaša 2013-10-17 13:59:46 UTC
Description of problem:
I added to RHEV-M a host with pre-configured networks with management network on untagged interface. vdsm failed to recognize this interface as a management interface. In webadmin, when I launch "setup host networks", I can see "rhevm" as an unattached network". When I attach the management network to the interface in the webui, the action fails with "Illegal Network parameters".

vdsmd.log says:
Thread-41::DEBUG::2013-10-17 15:47:02,068::BindingXMLRPC::974::vds::(wrapper) client [10.34.73.33]::call setupNetworks with ({'rhevm': {'bonding': 'bond0', 'ipaddr': '10.34.73.74', 'gateway': '10.34.73.126', 'bridged': 'false', 'netmask': '255.255.255.192'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
Thread-42::DEBUG::2013-10-17 15:47:02,070::BindingXMLRPC::974::vds::(wrapper) client [10.34.73.33]::call ping with () {}
Thread-42::DEBUG::2013-10-17 15:47:02,071::BindingXMLRPC::981::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-41::ERROR::2013-10-17 15:47:02,264::API::1262::vds::(setupNetworks) Network defined withoutdevices.
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1260, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (21, 'Network defined withoutdevices.')
Thread-41::DEBUG::2013-10-17 15:47:02,266::BindingXMLRPC::981::vds::(wrapper) return setupNetworks with {'status': {'message': 'Network defined withoutdevices.', 'code': 21}}


My networks            VLAN-tagged     VM network    IP configuration
---------------------------------------------------------------------------
rhevm                  no              no            static (preconfigured)
vm_dynamic             180             yes           no
display                183             no            static (preconfigured)
display_wanem          184             no            static (preconfigured)


My network layout on host:

eth1                                   vm_dynamic
     \                              /
       >-- bond0 (10.34.73.74)  --< -  display
     /     bonding mode 4           \
eth2                                   display_wanem



Version-Release number of selected component (if applicable):
si18 / vdsm-4.13.0-0.2.beta1.el6ev.x86_64
si19 / vdsm-4.13.0-0.3.beta1.el6ev.x86_64

How reproducible:
always

Steps to Reproduce:
1. preconfigure a network on the host with management network bridgeless and untagged
2. add the host to RHEV
3.

Actual results:
network can not be made functional - the already working native network is not recognized as "rhevm" management network

Expected results:
rhev-m picks up such networking automatically

Additional info:
in 3.2, the network is not recognized right away either but when you drag and drop it to the bond interface, it gets successfully recognized --> Regression.

Comment 1 David Jaša 2013-10-17 14:26:15 UTC
I tried to work around the issue by:

1) marking the tagged networks as not required - and trying to refresh capabilities followed by manually attaching rhevm network to the bond
2) marking the tagged networks as not required and actually disabling them on the host (ifdown NETWORK) - refresh, attach
3) marking the tagged networks as not required, actually disabling them on the host and marking management network as a VM network - refresh, attach
4) everything as in 3) - remove host, add host

none of these do actually work, the error is the same all the time. VDSM log for 3) is:
Thread-21::DEBUG::2013-10-17 16:18:04,986::BindingXMLRPC::984::vds::(wrapper) client [10.34.73.33]::call setupNetworks with ({'rhevm': {'ipaddr': '10.34.73.74', 'netmask': '255.255.255.192', 'bonding': 'bond0', 'STP': 'no', 'bridged': 'true', 'gateway': '10.34.73.126'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
Thread-22::DEBUG::2013-10-17 16:18:04,988::BindingXMLRPC::984::vds::(wrapper) client [10.34.73.33]::call ping with () {}
Thread-22::DEBUG::2013-10-17 16:18:04,989::BindingXMLRPC::991::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-21::ERROR::2013-10-17 16:18:05,100::API::1262::vds::(setupNetworks) Network defined withoutdevices.
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1260, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (21, 'Network defined withoutdevices.')
Thread-21::DEBUG::2013-10-17 16:18:05,110::BindingXMLRPC::991::vds::(wrapper) return setupNetworks with {'status': {'message': 'Network defined withoutdevices.', 'code': 21}}

Comment 2 David Jaša 2013-10-17 15:54:05 UTC
I found out what happened - there were vdsm-* networks left in libvirt, vdsm-rhevm was then configured for bond0.182 device. This prevented picking up of actual network configuration or any configuration changes.

Comment 3 David Jaša 2013-10-18 09:25:55 UTC
Created attachment 813664 [details]
engine log

Comment 11 Dan Kenigsberg 2013-10-18 20:55:24 UTC
Could you reproduce the creation of these "stray vdsm-* networks"? The content of supervdsm.log is expected to be more interesting than that of vdsm.log.

Comment 12 David Jaša 2013-10-19 11:20:34 UTC
The stray networks were there from the previous setup that had management network on bond interface - from another host that has original setup:
# virsh -r net-dumpxml vdsm-rhevm
<network>
  <name>vdsm-rhevm</name>
  <uuid>UUID</uuid>
  <forward dev='bond0.182' mode='passthrough'>
    <interface dev='bond0.182'/>
  </forward>
</network>

This "vdsm-rhevm" libvirt network effectively prevented attaching of RHEV/vdsm "rhevm" network to interface bond0.

Comment 13 Dan Kenigsberg 2013-10-20 21:42:19 UTC
Could you attach the content of supervdsm.log?

Comment 19 Martin Pavlik 2014-02-10 16:06:00 UTC
works with 
oVirt Engine Version: 3.4.0-0.7.beta2.el6 
vdsm-4.14.1-3.el6.x86_64

Comment 22 errata-xmlrpc 2014-06-09 13:25:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0504.html