Created attachment 913737 [details] engine log Description of problem: Tyring to update network attached to Host (with VLAN ) to another VLAN or to be non-tagged fails with error ERR_BAD_NIC and code 23. and the network is in unsync state Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create network with VLAN 0 2. Attach it to Host 3. Update network to be untagged or to have another VLAN Actual results: Network is unsynced and you get Error code 23 Expected results: Network should be synced and no error should appear Additional info:
Happened to Genadi as described above, happened to me when I added dummy NICs on the host and then tried to bond them via the GUI (without attaching any network). Looks like a VDSM issue. Attaching excerpt from the VDSM log: Thread-36::DEBUG::2014-07-03 19:17:44,372::BindingXMLRPC::1107::vds::(wrapper) client [10.35.202.248]::call setupNetworks with ({}, {'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100'}}, {'co nnectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [36cf97b3] Thread-13::DEBUG::2014-07-03 19:17:44,378::BindingXMLRPC::1107::vds::(wrapper) client [10.35.202.248]::call ping with () {} flowID [36cf97b3] Thread-13::DEBUG::2014-07-03 19:17:44,379::BindingXMLRPC::1114::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}} Thread-36::ERROR::2014-07-03 19:17:44,559::API::1448::vds::(_rollback) Unknown nics in: ['dummy1', 'dummy0'] Traceback (most recent call last): File "/usr/share/vdsm/API.py", line 1446, in _rollback yield rollbackCtx File "/usr/share/vdsm/API.py", line 1334, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) ConfigNetworkError: (23, "Unknown nics in: ['dummy1', 'dummy0']") Thread-36::DEBUG::2014-07-03 19:17:44,581::BindingXMLRPC::1114::vds::(wrapper) return setupNetworks with {'status': {'message': "Unknown nics in: ['dummy1', 'dummy0']", 'code': 23}} Later things started working again, not sure what has changed. Version of VDSM is 4.16.0 from the test day repository.
When /usr/share/vdsm/supervdsm.py is seen in the traceback, it means that the juicy stuff is in supervdsm.log. Could you attach that?
Good to know! Attaching the relevant snippet. MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,396::supervdsmServer::99::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({}, {'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100 '}}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,422::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,428::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::INFO::2014-07-03 19:17:44,433::netinfo::636::root::(_getNetInfo) Obtaining info for net vlan102. Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 613, in _getNetInfo data.update({'ports': ports(iface), 'stp': bridge_stp_state(iface), File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 218, in ports return os.listdir('/sys/class/net/' + bridge + '/brif') OSError: [Errno 2] No such file or directory: '/sys/class/net/vlan102/brif' MainProcess|Thread-36::INFO::2014-07-03 19:17:44,434::netinfo::636::root::(_getNetInfo) Obtaining info for net vlan101. Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 613, in _getNetInfo data.update({'ports': ports(iface), 'stp': bridge_stp_state(iface), File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 218, in ports return os.listdir('/sys/class/net/' + bridge + '/brif') OSError: [Errno 2] No such file or directory: '/sys/class/net/vlan101/brif' MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,435::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,440::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,448::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,455::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,455::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,462::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,464::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,472::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,474::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,479::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,481::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,486::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,487::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,492::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,494::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,499::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,500::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/7bcxNnBeWIaGDNl/bonding/mode (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,505::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,507::utils::738::root::(execCmd) /usr/bin/tee /sys/class/net/bonding_masters (cwd None) MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,520::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,527::api::614::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{}, bondings:{'bond0': {'nics': ['dummy1', 'dummy0'], 'options': 'mode=4 miimon=100'}}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 120} MainProcess|Thread-36::DEBUG::2014-07-03 19:17:44,528::api::618::root::(setupNetworks) Validating configuration MainProcess|Thread-36::ERROR::2014-07-03 19:17:44,532::supervdsmServer::103::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 101, in wrapper res = func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 221, in setupNetworks return setupNetworks(networks, bondings, **options) File "/usr/share/vdsm/network/api.py", line 619, in setupNetworks _validateNetworkSetup(dict(networks), dict(bondings)) File "/usr/share/vdsm/network/api.py", line 496, in _validateNetworkSetup "Unknown nics in: %r" % list(nics)) ConfigNetworkError: (23, "Unknown nics in: ['dummy1', 'dummy0']")
Can't reproduce following the original description: 1. Add a network with vlan 0 and attach it to a host: vdsClient -s 0 addNetwork bridge=foo nics=eth1 vlan=0 2. Update network to be untagged or to have another vlan: In [1]: from vdsm import vdscli In [2]: from vdsm import netinfo In [3]: c = vdscli.connect() In [4]: c.setupNetworks({'foo': {'nic': 'eth1', 'vlan': 11}}, {}, {'connectivityCheck': False}) Out[4]: {'status': {'code': 0, 'message': 'Done'}} In [5]: n = netinfo.NetInfo(c.getVdsCapabilities()['info']) In [6]: n.networks Out[6]: {'foo': {'addr': '', 'bootproto4': 'none', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'foo', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no', 'STP': 'off', 'TYPE': 'Bridge'}, 'gateway': '', 'iface': 'foo', 'ipv4addrs': [], 'ipv6addrs': ['fe80::21a:4aff:fe0c:8792/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ports': ['eth1.11'], 'stp': 'off'}} As seen above, the network has been correctly updated to run on top of the same device but on vlan 11 instead of 0. It could have been that the dummy was not recognized properly, so let's try that with the tip of the 3.5 branch: In [11]: c.setupNetworks({'foo': {'nic': 'dummy_14', 'vlan': 0}}, {}, {'connectivityCheck': False}) Out[11]: {'status': {'code': 0, 'message': 'Done'}} In [12]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks Out[12]: {'foo': {'addr': '', 'bootproto4': 'none', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'foo', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no', 'STP': 'off', 'TYPE': 'Bridge'}, 'gateway': '', 'iface': 'foo', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ports': ['dummy_14.0'], 'stp': 'off'}} It was properly created. Let's now change from vlan tag 0 to 11 In [13]:c.setupNetworks({'foo': {'nic': 'dummy_14', 'vlan': 11}}, {}, {'connectivityCheck': False}) Out[13]: {'status': {'code': 0, 'message': 'Done'}} In [14]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks Out[14]: {'foo': {'addr': '', 'bootproto4': 'none', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'foo', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no', 'STP': 'off', 'TYPE': 'Bridge'}, 'gateway': '', 'iface': 'foo', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ports': ['dummy_14.11'], 'stp': 'off'}} One more try, now bonded: In [15]: c.setupNetworks({'foo': {'bonding': 'bond5', 'vlan': 0}}, {'bond5': {'nics': ['dummy_14', 'dummy_15']}}, {'connectivityCheck': False}) Out[15]: {'status': {'code': 0, 'message': 'Done'}} In [16]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networksOut[16]: {'foo': {'addr': '', 'bootproto4': 'none', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'foo', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no', 'STP': 'off', 'TYPE': 'Bridge'}, 'gateway': '', 'iface': 'foo', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ports': ['bond5.0'], 'stp': 'off'}} In [18]: netinfo.NetInfo(c.getVdsCapabilities()['info']).bondings['bond5'] Out[18]: {'addr': '', 'cfg': {'BONDING_OPTS': 'mode=802.3ad miimon=150', 'DEVICE': 'bond5', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no'}, 'hwaddr': '9e:a4:89:42:c5:68', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'mtu': '1500', 'netmask': '', 'opts': {'miimon': '150', 'mode': '4'}, 'slaves': ['dummy_14', 'dummy_15']} Now update to vlan tag 11: In [19]: c.setupNetworks({'foo': {'bonding': 'bond5', 'vlan': 11}}, {}, {'connectivityCheck': False})Out[19]: {'status': {'code': 0, 'message': 'Done'}} In [20]: netinfo.NetInfo(c.getVdsCapabilities()['info']).networks Out[20]: {'foo': {'addr': '', 'bootproto4': 'none', 'bridged': True, 'cfg': {'BOOTPROTO': 'none', 'DEFROUTE': 'no', 'DELAY': '0', 'DEVICE': 'foo', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no', 'STP': 'off', 'TYPE': 'Bridge'}, 'gateway': '', 'iface': 'foo', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ports': ['bond5.11'], 'stp': 'off'}} In [21]: netinfo.NetInfo(c.getVdsCapabilities()['info']).bondings['bond5']Out[21]: {'addr': '', 'cfg': {'BONDING_OPTS': 'mode=802.3ad miimon=150', 'DEVICE': 'bond5', 'HOTPLUG': 'no', 'NM_CONTROLLED': 'no', 'ONBOOT': 'no'}, 'hwaddr': '9e:a4:89:42:c5:68', 'ipv4addrs': [], 'ipv6addrs': ['fe80::9ca4:89ff:fe42:c568/64'], 'mtu': '1500', 'netmask': '', 'opts': {'miimon': '150', 'mode': '4'}, 'slaves': ['dummy_14', 'dummy_15']} Since reproduction is not possible, I'm closing the bug.
Created attachment 917598 [details] engine, vdsm and supervdsm logs
Managed to reproduce it when playing with changing the network from tag to untagged and vice versa. Eventually it happened when I changed the tagged network to be untagged. All the logs are attached
Reproduced with vdsm-4.16.0-3.git601f786.el6.x86_64
The bug is on Engine side: Thread-709::DEBUG::2014-07-13 10:11:41,775::BindingXMLRPC::1107::vds::(wrapper) client [10.35.162.33]::call setupNetworks with ({'net11': {'nic': 'eth1.0', 'STP': 'no', 'bridged': 'true', 'mtu': '1500'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [4bd4551f] Vdsm is asked to create a network over the non-existing eth1.0 nic.
It is quite possible that there's an engine regression related to to the arbitrary VLAN feature. However, what about the case I attached? The same error occurred there, and there was no network involved, it were just dummy0 and dummy1 being bonded under bond0 (where dummy0 and dummy1 had recently been added using modprobe).
(In reply to Lior Vernia from comment #9) > However, what about the case I attached? The same error occurred there, and > there was no network involved, it were just dummy0 and dummy1 being bonded > under bond0 (where dummy0 and dummy1 had recently been added using modprobe). Have you modified the value for fake_nics in vdsm.conf? By default, Vdsm treats dummy devices as nics only if they are named dummy_* (note the underscore), and justly ignores dummy0 and dummy1 as non-nics.
Yes, I have - they had to appear in the engine for me to try to bond them.
Verified in 3.5.0-0.0.master.20140722232058.git8e1babc.el6
oVirt 3.5 has been released and should include the fix for this issue.