Bug 1115001 - Error code 23 when invoking Setup Networks
Summary: Error code 23 when invoking Setup Networks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-core
Version: 3.5
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: 3.5.0
Assignee: Alona Kaplan
QA Contact: GenadiC
URL:
Whiteboard: network
Depends On:
Blocks: 1073943
TreeView+ depends on / blocked
 
Reported: 2014-07-01 12:20 UTC by GenadiC
Modified: 2016-02-10 19:36 UTC (History)
9 users (show)

Fixed In Version: ovirt-3.5.0-beta2
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-17 12:28:15 UTC
oVirt Team: Network


Attachments (Terms of Use)
engine log (286.75 KB, text/x-log)
2014-07-01 12:20 UTC, GenadiC
no flags Details
engine, vdsm and supervdsm logs (1.22 MB, application/zip)
2014-07-13 07:16 UTC, GenadiC
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 30170 master MERGED engine: Error code 23 when invoking Setup Networks Never
oVirt gerrit 30213 ovirt-engine-3.5 MERGED engine: Error code 23 when invoking Setup Networks Never

Description GenadiC 2014-07-01 12:20:28 UTC
Created attachment 913737 [details]
engine log

Description of problem:
Tyring to update network attached to Host (with VLAN ) to another VLAN or to be non-tagged fails with error ERR_BAD_NIC and code 23. and the network is in unsync state

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create network with VLAN 0
2. Attach it to Host
3. Update network to be untagged or to have another VLAN

Actual results:
Network is unsynced and you get Error code 23

Expected results:
Network should be synced and no error should appear

Additional info:

Comment 1 Lior Vernia 2014-07-06 07:52:20 UTC
Happened to Genadi as described above, happened to me when I added dummy NICs on the host and then tried to bond them via the GUI (without attaching any network). Looks like a VDSM issue. Attaching excerpt from the VDSM log:



Thread-36::DEBUG::2014-07-03 19:17:44,372::BindingXMLRPC::1107::vds::(wrapper) client [10.35.202.248]::call setupNetworks with ({}, {'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100'}}, {'co
nnectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [36cf97b3]
Thread-13::DEBUG::2014-07-03 19:17:44,378::BindingXMLRPC::1107::vds::(wrapper) client [10.35.202.248]::call ping with () {} flowID [36cf97b3]
Thread-13::DEBUG::2014-07-03 19:17:44,379::BindingXMLRPC::1114::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-36::ERROR::2014-07-03 19:17:44,559::API::1448::vds::(_rollback) Unknown nics in: ['dummy1', 'dummy0']
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1446, in _rollback
    yield rollbackCtx
  File "/usr/share/vdsm/API.py", line 1334, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (23, "Unknown nics in: ['dummy1', 'dummy0']")
Thread-36::DEBUG::2014-07-03 19:17:44,581::BindingXMLRPC::1114::vds::(wrapper) return setupNetworks with {'status': {'message': "Unknown nics in: ['dummy1', 'dummy0']", 'code': 23}}



Later things started working again, not sure what has changed. Version of VDSM is 4.16.0 from the test day repository.

Comment 2 Dan Kenigsberg 2014-07-07 23:00:49 UTC
When /usr/share/vdsm/supervdsm.py is seen in the traceback, it means that the juicy stuff is in supervdsm.log. Could you attach that?

Comment 3 Lior Vernia 2014-07-08 12:53:01 UTC
Good to know! Attaching the relevant snippet.



MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,396::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({}, {'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100
'}}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,422::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,428::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::INFO::2014-07-03 19:17:44,433::netinfo::636::root::(_getNetInfo) Obtaining info for net vlan102.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 613, in _getNetInfo
    data.update({'ports': ports(iface), 'stp': bridge_stp_state(iface),
  File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 218, in ports
    return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/vlan102/brif'
MainProcess|Thread-36::INFO::2014-07-03 19:17:44,434::netinfo::636::root::(_getNetInfo) Obtaining info for net vlan101.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 613, in _getNetInfo
    data.update({'ports': ports(iface), 'stp': bridge_stp_state(iface),
  File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 218, in ports
    return os.listdir('/sys/class/net/' + bridge + '/brif')
OSError: [Errno 2] No such file or directory: '/sys/class/net/vlan101/brif'
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,435::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,440::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,448::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,455::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,455::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,462::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,464::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,472::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,474::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,479::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,481::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,486::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,487::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,492::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,494::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,499::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,500::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,505::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,507::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None)
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,520::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,527::api::614::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100'}}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 120}
MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,528::api::618::root::(setupNetworks) Validating configuration
MainProcess|Thread-36::ERROR::2014-07-03 19:17:44,532::supervdsmServer::103::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 101, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 221, in setupNetworks
    return setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/network/api.py", line 619, in setupNetworks
    _validateNetworkSetup(dict(networks), dict(bondings))
  File "/usr/share/vdsm/network/api.py", line 496, in _validateNetworkSetup
    "Unknown nics in: %r" % list(nics))
ConfigNetworkError: (23, "Unknown nics in: ['dummy1', 'dummy0']")

Comment 4 Antoni Segura Puimedon 2014-07-10 23:22:02 UTC
Can't reproduce following the original description:

    1. Add a network with vlan 0 and attach it to a host:
        vdsClient -s 0 addNetwork bridge=foo nics=eth1 vlan=0
    2. Update network to be untagged or to have another vlan:
        In [1]: from vdsm import vdscli
        In [2]: from vdsm import netinfo
        In [3]: c = vdscli.connect()
        In [4]: c.setupNetworks({'foo': {'nic': 'eth1', 'vlan': 11}}, {}, {'connectivityCheck': False})
        Out[4]: {'status': {'code': 0, 'message': 'Done'}}
        In [5]: n = netinfo.NetInfo(c.getVdsCapabilities()['info'])
        In [6]: n.networks
        Out[6]: 
        {'foo': {'addr': '',
          'bootproto4': 'none',
          'bridged': True,
          'cfg': {'BOOTPROTO': 'none',
           'DEFROUTE': 'no',
           'DELAY': '0',
           'DEVICE': 'foo',
           'HOTPLUG': 'no',
           'NM_CONTROLLED': 'no',
           'ONBOOT': 'no',
           'STP': 'off',
           'TYPE': 'Bridge'},
          'gateway': '',
          'iface': 'foo',
          'ipv4addrs': [],
          'ipv6addrs': ['fe80::21a:4aff:fe0c:8792/64'],
          'ipv6gateway': '::',
          'mtu': '1500',
          'netmask': '',
          'ports': ['eth1.11'],
          'stp': 'off'}}

As seen above, the network has been correctly updated to run on top of the same
device but on vlan 11 instead of 0.

It could have been that the dummy was not recognized properly, so let's try
that with the tip of the 3.5 branch:
        
    In [11]: c.setupNetworks({'foo': {'nic': 'dummy_14', 'vlan': 0}}, {}, {'connectivityCheck': False})
    Out[11]: {'status': {'code': 0, 'message': 'Done'}}
    In [12]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks
    Out[12]: 
    {'foo': {'addr': '',
      'bootproto4': 'none',
      'bridged': True,
      'cfg': {'BOOTPROTO': 'none',
       'DEFROUTE': 'no',
       'DELAY': '0',
       'DEVICE': 'foo',
       'HOTPLUG': 'no',
       'NM_CONTROLLED': 'no',
       'ONBOOT': 'no',
       'STP': 'off',
       'TYPE': 'Bridge'},
      'gateway': '',
      'iface': 'foo',
      'ipv4addrs': [],
      'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
      'ipv6gateway': '::',
      'mtu': '1500',
      'netmask': '',
      'ports': ['dummy_14.0'],
      'stp': 'off'}}

It was properly created. Let's now change from vlan tag 0 to 11

    In [13]:c.setupNetworks({'foo': {'nic': 'dummy_14', 'vlan': 11}}, {}, {'connectivityCheck': False})
    Out[13]: {'status': {'code': 0, 'message': 'Done'}}

    In [14]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks
    Out[14]: 
    {'foo': {'addr': '',
      'bootproto4': 'none',
      'bridged': True,
      'cfg': {'BOOTPROTO': 'none',
       'DEFROUTE': 'no',
       'DELAY': '0',
       'DEVICE': 'foo',
       'HOTPLUG': 'no',
       'NM_CONTROLLED': 'no',
       'ONBOOT': 'no',
       'STP': 'off',
       'TYPE': 'Bridge'},
      'gateway': '',
      'iface': 'foo',
      'ipv4addrs': [],
      'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
      'ipv6gateway': '::',
      'mtu': '1500',
      'netmask': '',
      'ports': ['dummy_14.11'],
      'stp': 'off'}}

One more try, now bonded:

    In [15]: c.setupNetworks({'foo': {'bonding': 'bond5', 'vlan': 0}}, {'bond5': {'nics': ['dummy_14', 'dummy_15']}}, {'connectivityCheck': False})
    Out[15]: {'status': {'code': 0, 'message': 'Done'}}

    In [16]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networksOut[16]: 
    {'foo': {'addr': '',
      'bootproto4': 'none',
      'bridged': True,
      'cfg': {'BOOTPROTO': 'none',
       'DEFROUTE': 'no',
       'DELAY': '0',
       'DEVICE': 'foo',
       'HOTPLUG': 'no',
       'NM_CONTROLLED': 'no',
       'ONBOOT': 'no',
       'STP': 'off',
       'TYPE': 'Bridge'},
      'gateway': '',
      'iface': 'foo',
      'ipv4addrs': [],
      'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
      'ipv6gateway': '::',
      'mtu': '1500',
      'netmask': '',
      'ports': ['bond5.0'],
      'stp': 'off'}}
    In [18]: netinfo.NetInfo(c.getVdsCapabilities()['info']).bondings['bond5']
    Out[18]: 
    {'addr': '',
     'cfg': {'BONDING_OPTS': 'mode=802.3ad miimon=150',
      'DEVICE': 'bond5',
      'HOTPLUG': 'no',
      'NM_CONTROLLED': 'no',
      'ONBOOT': 'no'},
     'hwaddr': '9e:a4:89:42:c5:68',
     'ipv4addrs': [],
     'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
     'mtu': '1500',
     'netmask': '',
     'opts': {'miimon': '150', 'mode': '4'},
     'slaves': ['dummy_14', 'dummy_15']}

Now update to vlan tag 11:

In [19]: c.setupNetworks({'foo': {'bonding': 'bond5', 'vlan': 11}}, {}, {'connectivityCheck': False})Out[19]: {'status': {'code': 0, 'message': 'Done'}}

In [20]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks
Out[20]: 
{'foo': {'addr': '',
  'bootproto4': 'none',
  'bridged': True,
  'cfg': {'BOOTPROTO': 'none',
   'DEFROUTE': 'no',
   'DELAY': '0',
   'DEVICE': 'foo',
   'HOTPLUG': 'no',
   'NM_CONTROLLED': 'no',
   'ONBOOT': 'no',
   'STP': 'off',
   'TYPE': 'Bridge'},
  'gateway': '',
  'iface': 'foo',
  'ipv4addrs': [],
  'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
  'ipv6gateway': '::',
  'mtu': '1500',
  'netmask': '',
  'ports': ['bond5.11'],
  'stp': 'off'}}

In [21]: netinfo.NetInfo(c.getVdsCapabilities()['info']).bondings['bond5']Out[21]: 
{'addr': '',
 'cfg': {'BONDING_OPTS': 'mode=802.3ad miimon=150',
  'DEVICE': 'bond5',
  'HOTPLUG': 'no',
  'NM_CONTROLLED': 'no',
  'ONBOOT': 'no'},
 'hwaddr': '9e:a4:89:42:c5:68',
 'ipv4addrs': [],
 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'],
 'mtu': '1500',
 'netmask': '',
 'opts': {'miimon': '150', 'mode': '4'},
 'slaves': ['dummy_14', 'dummy_15']}

Since reproduction is not possible, I'm closing the bug.

Comment 5 GenadiC 2014-07-13 07:16:38 UTC
Created attachment 917598 [details]
engine, vdsm and supervdsm logs

Comment 6 GenadiC 2014-07-13 07:18:19 UTC
Managed to reproduce it when playing with changing the network from tag to untagged and vice versa.
Eventually it happened when I changed the tagged network to be untagged.
All the logs are attached

Comment 7 GenadiC 2014-07-13 07:25:41 UTC
Reproduced with vdsm-4.16.0-3.git601f786.el6.x86_64

Comment 8 Dan Kenigsberg 2014-07-14 14:49:37 UTC
The bug is on Engine side:

Thread-709::DEBUG::2014-07-13 10:11:41,775::BindingXMLRPC::1107::vds::(wrapper) client [10.35.162.33]::call setupNetworks with ({'net11': {'nic': 'eth1.0', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [4bd4551f]

Vdsm is asked to create a network over the non-existing eth1.0 nic.

Comment 9 Lior Vernia 2014-07-14 15:01:49 UTC
It is quite possible that there's an engine regression related to to the arbitrary VLAN feature.

However, what about the case I attached? The same error occurred there, and there was no network involved, it were just dummy0 and dummy1 being bonded under bond0 (where dummy0 and dummy1 had recently been added using modprobe).

Comment 10 Dan Kenigsberg 2014-07-14 22:35:42 UTC
(In reply to Lior Vernia from comment #9)

> However, what about the case I attached? The same error occurred there, and
> there was no network involved, it were just dummy0 and dummy1 being bonded
> under bond0 (where dummy0 and dummy1 had recently been added using modprobe).

Have you modified the value for fake_nics in vdsm.conf? By default, Vdsm treats dummy devices as nics only if they are named dummy_* (note the underscore), and justly ignores dummy0 and dummy1 as non-nics.

Comment 11 Lior Vernia 2014-07-15 06:55:34 UTC
Yes, I have - they had to appear in the engine for me to try to bond them.

Comment 12 GenadiC 2014-07-27 07:23:23 UTC
Verified in 3.5.0-0.0.master.20140722232058.git8e1babc.el6

Comment 13 Sandro Bonazzola 2014-10-17 12:28:15 UTC
oVirt 3.5 has been released and should include the fix for this issue.


Note You need to log in before you can comment on or make changes to this bug.