Bug 1337276 - rhel-osp-director: On undercloud: neutron-openvswitch-agent: failed (memory issue)
Summary: rhel-osp-director: On undercloud: neutron-openvswitch-agent: failed (memory ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: James Slagle
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-18 16:53 UTC by Alexander Chuzhoy
Modified: 2016-08-11 11:30 UTC (History)
9 users (show)

Fixed In Version: instack-undercloud-4.0.0-2.el7ost.noarch
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-11 11:30:49 UTC
Target Upstream Version:


Attachments (Terms of Use)
install-undercloud.log (690.16 KB, text/plain)
2016-05-18 16:55 UTC, Alexander Chuzhoy
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1599 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 director Release Candidate Advisory 2016-08-11 15:25:37 UTC

Description Alexander Chuzhoy 2016-05-18 16:53:15 UTC
rhel-osp-director:  On undercloud: neutron-openvswitch-agent:              failed

Environment:

instack-undercloud-2.1.2-39.el7ost.noarch


Steps to reproduce:
1. Deploy undercloud.
2. Run "openstack-status|grep neutron-openvswitch-agent"

Result:

neutron-openvswitch-agent:              failed

Expected result:


neutron-openvswitch-agent:  active

Comment 2 Alexander Chuzhoy 2016-05-18 16:55:50 UTC
Created attachment 1159000 [details]
install-undercloud.log

Comment 3 Alexander Chuzhoy 2016-05-18 19:17:28 UTC
from /var/log/neutron/openvswitch-agent.log

2016-05-18 12:14:16.002 30518 CRITICAL neutron [req-0841b653-94a6-4c22-a9df-b43de9dd9850 - - - - -] OSError: [Errno 12] Cannot allocate memory
2016-05-18 12:14:16.002 30518 ERROR neutron Traceback (most recent call last):
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2016-05-18 12:14:16.002 30518 ERROR neutron     sys.exit(main())
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     agent_main.main()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     mod.main()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     ovs_neutron_agent.main(bridge_classes)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2101, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     agent.daemon_loop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2024, in daemon_loop
2016-05-18 12:14:16.002 30518 ERROR neutron     self.rpc_loop(polling_manager=pm)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2016-05-18 12:14:16.002 30518 ERROR neutron     self.gen.throw(type, value, traceback)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 42, in get_polling_manager
2016-05-18 12:14:16.002 30518 ERROR neutron     pm.stop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 61, in stop
2016-05-18 12:14:16.002 30518 ERROR neutron     self._monitor.stop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 129, in stop
2016-05-18 12:14:16.002 30518 ERROR neutron     self._kill(kill_signal)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 162, in _kill
2016-05-18 12:14:16.002 30518 ERROR neutron     pid = self.pid
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 158, in pid
2016-05-18 12:14:16.002 30518 ERROR neutron     run_as_root=self.run_as_root)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 257, in get_root_helper_child_pid
2016-05-18 12:14:16.002 30518 ERROR neutron     pid = find_child_pids(pid)[0]
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 188, in find_child_pids
2016-05-18 12:14:16.002 30518 ERROR neutron     log_fail_as_error=False)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 119, in execute
2016-05-18 12:14:16.002 30518 ERROR neutron     addl_env=addl_env)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 88, in create_process
2016-05-18 12:14:16.002 30518 ERROR neutron     stderr=subprocess.PIPE)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 214, in subprocess_popen
2016-05-18 12:14:16.002 30518 ERROR neutron     close_fds=close_fds, env=env)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in __init__
2016-05-18 12:14:16.002 30518 ERROR neutron     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
2016-05-18 12:14:16.002 30518 ERROR neutron     errread, errwrite)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
2016-05-18 12:14:16.002 30518 ERROR neutron     self.pid = os.fork()
2016-05-18 12:14:16.002 30518 ERROR neutron OSError: [Errno 12] Cannot allocate memory
2016-05-18 12:14:16.002 30518 ERROR neutron
2016-05-18 12:14:19.235 30518 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 6a2f2019df7c4c168b7377bebee5d60f exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454

Comment 5 Terry Wilson 2016-06-03 15:09:52 UTC
The traceback seems to show things happening in a fork() call, which I believe marks the memory copy-on-write and isn't just blindly duplicating all of the memory. So I wouldn't expect that particular call to be a *huge* amount of memory allocation.

Would it be possible for someone to do a deploy and watch the output of top -o%MEM or something similar to see where the memory is going? Unless something really weird is going on and ovs-neutron-agent is stuck in some kind of loop fork-bombing the system, I wouldn't expect it to be the culprit (especially since I tend to run it on VMs with only 2GB ram).

Comment 6 Alexander Chuzhoy 2016-06-06 19:30:24 UTC
The reported issue doesn't seem to reproduce:
instack-undercloud-4.0.0-2.el7ost.noarch

Comment 7 Alexander Chuzhoy 2016-06-07 14:27:17 UTC
Verified:

Environment:
instack-undercloud-4.0.0-2.el7ost.noarch

The issue doesn't reproduce.

Comment 9 errata-xmlrpc 2016-08-11 11:30:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html


Note You need to log in before you can comment on or make changes to this bug.