Bug 1466628 - SetupNetworks failed after sometime with 'IOError: [Errno 5] Out of memory' and 'OSError: [Errno 9] Bad file descriptor'
SetupNetworks failed after sometime with 'IOError: [Errno 5] Out of memory' a...
Product: vdsm
Classification: oVirt
Component: SuperVDSM (Show other bugs)
All Linux
high Severity urgent (vote)
: ---
: ---
Assigned To: Edward Haas
Meni Yakove
: Automation, Regression
Depends On:
  Show dependency treegraph
Reported: 2017-06-30 01:49 EDT by Meni Yakove
Modified: 2017-07-05 03:44 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-07-05 03:44:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.2+
rule-engine: blocker+

Attachments (Terms of Use)
vdsm,supervdsm,messages and journalctl (954.05 KB, application/zip)
2017-06-30 01:49 EDT, Meni Yakove
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 78928 master MERGED net: Increase netlink socket buffer size to 512KB 2017-09-27 16:02 EDT

  None (edit)
Description Meni Yakove 2017-06-30 01:49:56 EDT
Created attachment 1293088 [details]
vdsm,supervdsm,messages and journalctl

Description of problem:
After a few Setupnetworks calls, SetupNetworks failed with:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 202, in setupNetworks
    _setup_networks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 223, in _setup_networks
    netswitch.configurator.setup(networks, bondings, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 137, in setup
    _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 151, in _setup_legacy
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 331, in remove_networks
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 160, in wrapped
    return func(network, configurator, net_info, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 291, in _del_network
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 193, in remove
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 213, in removeBridge
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 262, in remove
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 275, in removeBond
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 244, in configure
    self.configurator.configureBond(self, **opts)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 131, in configureBond
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 920, in _ifup
    _exec_ifup(iface, cgroup)
  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/waitfor.py", line 41, in waitfor_linkup
  File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 129, in __exit__
  File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 175, in stop
    os.write(self._pipetrick[1], b'c')
OSError: [Errno 9] Bad file descriptor

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 185, in run
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/monitor.py", line 167, in _scan
  File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/libnl.py", line 437, in nl_recvmsgs_default
    raise IOError(-err, nl_geterror(err))
IOError: [Errno 5] Out of memory

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.host_network_api automation test (Many SetupNetworks calls via rest)
Comment 1 Meni Yakove 2017-07-03 03:49:22 EDT
Working vdsm version: vdsm-4.20.1-66.git228c7be.el7.centos.x86_64
Non working vdsm version: vdsm-4.20.1-89.git2912779.el7.centos.x86_64
Comment 2 Edward Haas 2017-07-03 08:22:52 EDT
I'm unsure if this will help, but the following patch: https://gerrit.ovirt.org/#/c/78928
Is increasing the buffer size for the netlink socket.

Could you please try to see if this help with the reported problem?
Comment 3 Meni Yakove 2017-07-04 10:29:21 EDT
I can't reproduce this one on latest master vdsm-4.20.1-120.git28558d7.el7.centos.x86_64

All our tests passed.
Attach 100 networks to host NIC passed.

I'm fine with closing this bug until next time.

Note You need to log in before you can comment on or make changes to this bug.