After changing MTU for one interface, one of the nodes in the cluster didn't apply the configuration correctly. vdsmd service wasn't started due to the missing dependency on vdms-network. When I manually run /usr/bin/vdsm-tool restore-nets I get the following backtrace: Traceback (most recent call last): File "/usr/share/vdsm/vdsm-restore-net-config", line 479, in <module> restore(args) File "/usr/share/vdsm/vdsm-restore-net-config", line 442, in restore unified_restoration() File "/usr/share/vdsm/vdsm-restore-net-config", line 134, in unified_restoration changed_config = _filter_changed_nets_bonds(available_config) File "/usr/share/vdsm/vdsm-restore-net-config", line 261, in _filter_changed_nets_bonds kernel_config = kernelconfig.KernelConfig(NetInfo(netswitch.netinfo())) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 298, in netinfo _netinfo = netinfo_get(compatibility=compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 109, in get return _get(vdsmnets) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 101, in _get report_network_qos(networking) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/qos.py", line 42, in report_network_qos iface, = host_ports ValueError: too many values to unpack Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 219, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 41, in restore_command exec_restore(cmd) File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 54, in exec_restore raise EnvironmentError('Failed to restore the persisted networks') EnvironmentError: Failed to restore the persisted networks No other node in the network reports this issue and they all the same configuration. /var/log/vdsm/upgrade.log reports: MainThread::ERROR::2016-12-12 16:21:11,272::upgrade::94::upgrade::(apply_upgrade) Failed to run upgrade-unified-persistence Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/tool/upgrade.py", line 92, in apply_upgrade upgrade.run(ns, args) File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 195, in run run() File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 46, in run networks, bondings = _getNetInfo() File "/usr/lib/python2.7/site-packages/vdsm/tool/unified_persistence.py", line 132, in _getNetInfo netinfo = NetInfo(netswitch.netinfo()) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 298, in netinfo _netinfo = netinfo_get(compatibility=compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 109, in get return _get(vdsmnets) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 101, in _get report_network_qos(networking) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/qos.py", line 42, in report_network_qos iface, = host_ports ValueError: too many values to unpack
Could you share your `brctl show`? your ifcfg-* and /var/lib/vdsm/persistence may be of use, too. I suspect that you have a Linux bridge that is connected to two NICs, which is not supported by Vdsm.
Wow, I'm impressed how precisely you managed to nail it down from this log! I've spent days trying to figure it out! Actually it seems the problem happened when I added VLAN tag to ovirtmgmt. The configuration ended up like this: ovirtmgmt 8000.f46d047ad7c9 no enp7s0 enp7s0.3 Manually removing enp7s0 from the bridge made it all work.
ACTUALLY, I should have been quicker to recognize that this is a duplicate of bug 1372798. Just wait for ovirt-4.0.6 (around the cornet) to see it fixed. *** This bug has been marked as a duplicate of bug 1372798 ***