Created attachment 1293088 [details] vdsm,supervdsm,messages and journalctl Description of problem: After a few Setupnetworks calls, SetupNetworks failed with: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 202, in setupNetworks _setup_networks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 223, in _setup_networks netswitch.configurator.setup(networks, bondings, options, in_rollback) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 137, in setup _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 151, in _setup_legacy _netinfo) File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 331, in remove_networks keep_bridge=keep_bridge) File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 160, in wrapped return func(network, configurator, net_info, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 291, in _del_network net_ent_to_remove.remove() File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 193, in remove self.configurator.removeBridge(self) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 213, in removeBridge bridge.port.remove() File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 262, in remove self.configurator.removeBond(self) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 275, in removeBond bonding.configure() File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 244, in configure self.configurator.configureBond(self, **opts) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 131, in configureBond _ifup(bond) File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 920, in _ifup _exec_ifup(iface, cgroup) File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/waitfor.py", line 41, in waitfor_linkup return File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 129, in __exit__ self.stop() File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 175, in stop os.write(self._pipetrick[1], b'c') OSError: [Errno 9] Bad file descriptor Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 185, in run ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 167, in _scan libnl.nl_recvmsgs_default(sock) File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/libnl.py", line 437, in nl_recvmsgs_default raise IOError(-err, nl_geterror(err)) IOError: [Errno 5] Out of memory Version-Release number of selected component (if applicable): vdsm-4.20.1-89.git2912779.el7.centos.x86_64 How reproducible: 100% Steps to Reproduce: 1.host_network_api automation test (Many SetupNetworks calls via rest)
Working vdsm version: vdsm-4.20.1-66.git228c7be.el7.centos.x86_64 Non working vdsm version: vdsm-4.20.1-89.git2912779.el7.centos.x86_64
I'm unsure if this will help, but the following patch: https://gerrit.ovirt.org/#/c/78928 Is increasing the buffer size for the netlink socket. Could you please try to see if this help with the reported problem?
I can't reproduce this one on latest master vdsm-4.20.1-120.git28558d7.el7.centos.x86_64 All our tests passed. Attach 100 networks to host NIC passed. I'm fine with closing this bug until next time.