Bug 1337276

Summary:

rhel-osp-director: On undercloud: neutron-openvswitch-agent: failed (memory issue)

Product:

Red Hat OpenStack

Reporter:

Alexander Chuzhoy <sasha>

Component:

instack-undercloud

Assignee:

James Slagle <jslagle>

Status:

CLOSED ERRATA

QA Contact:

Alexander Chuzhoy <sasha>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

9.0 (Mitaka)

CC:

dbecker, jason.dobies, jcoufal, mburns, morazi, rhel-osp-director-maint, sasha, tvignaud, twilson

Target Milestone:

Keywords:

TestOnly

Target Release:

9.0 (Mitaka)

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

instack-undercloud-4.0.0-2.el7ost.noarch

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2016-08-11 11:30:49 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
install-undercloud.log	none

Description Alexander Chuzhoy 2016-05-18 16:53:15 UTC

rhel-osp-director:  On undercloud: neutron-openvswitch-agent:              failed

Environment:

instack-undercloud-2.1.2-39.el7ost.noarch


Steps to reproduce:
1. Deploy undercloud.
2. Run "openstack-status|grep neutron-openvswitch-agent"

Result:

neutron-openvswitch-agent:              failed

Expected result:


neutron-openvswitch-agent:  active

Comment 2 Alexander Chuzhoy 2016-05-18 16:55:50 UTC

Created attachment 1159000 [details]
install-undercloud.log

Comment 3 Alexander Chuzhoy 2016-05-18 19:17:28 UTC

from /var/log/neutron/openvswitch-agent.log

2016-05-18 12:14:16.002 30518 CRITICAL neutron [req-0841b653-94a6-4c22-a9df-b43de9dd9850 - - - - -] OSError: [Errno 12] Cannot allocate memory
2016-05-18 12:14:16.002 30518 ERROR neutron Traceback (most recent call last):
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2016-05-18 12:14:16.002 30518 ERROR neutron     sys.exit(main())
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     agent_main.main()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     mod.main()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     ovs_neutron_agent.main(bridge_classes)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2101, in main
2016-05-18 12:14:16.002 30518 ERROR neutron     agent.daemon_loop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2024, in daemon_loop
2016-05-18 12:14:16.002 30518 ERROR neutron     self.rpc_loop(polling_manager=pm)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2016-05-18 12:14:16.002 30518 ERROR neutron     self.gen.throw(type, value, traceback)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 42, in get_polling_manager
2016-05-18 12:14:16.002 30518 ERROR neutron     pm.stop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 61, in stop
2016-05-18 12:14:16.002 30518 ERROR neutron     self._monitor.stop()
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 129, in stop
2016-05-18 12:14:16.002 30518 ERROR neutron     self._kill(kill_signal)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 162, in _kill
2016-05-18 12:14:16.002 30518 ERROR neutron     pid = self.pid
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 158, in pid
2016-05-18 12:14:16.002 30518 ERROR neutron     run_as_root=self.run_as_root)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 257, in get_root_helper_child_pid
2016-05-18 12:14:16.002 30518 ERROR neutron     pid = find_child_pids(pid)[0]
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 188, in find_child_pids
2016-05-18 12:14:16.002 30518 ERROR neutron     log_fail_as_error=False)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 119, in execute
2016-05-18 12:14:16.002 30518 ERROR neutron     addl_env=addl_env)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 88, in create_process
2016-05-18 12:14:16.002 30518 ERROR neutron     stderr=subprocess.PIPE)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 214, in subprocess_popen
2016-05-18 12:14:16.002 30518 ERROR neutron     close_fds=close_fds, env=env)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in __init__
2016-05-18 12:14:16.002 30518 ERROR neutron     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
2016-05-18 12:14:16.002 30518 ERROR neutron     errread, errwrite)
2016-05-18 12:14:16.002 30518 ERROR neutron   File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child
2016-05-18 12:14:16.002 30518 ERROR neutron     self.pid = os.fork()
2016-05-18 12:14:16.002 30518 ERROR neutron OSError: [Errno 12] Cannot allocate memory
2016-05-18 12:14:16.002 30518 ERROR neutron
2016-05-18 12:14:19.235 30518 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 6a2f2019df7c4c168b7377bebee5d60f exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454

Comment 5 Terry Wilson 2016-06-03 15:09:52 UTC

The traceback seems to show things happening in a fork() call, which I believe marks the memory copy-on-write and isn't just blindly duplicating all of the memory. So I wouldn't expect that particular call to be a *huge* amount of memory allocation.

Would it be possible for someone to do a deploy and watch the output of top -o%MEM or something similar to see where the memory is going? Unless something really weird is going on and ovs-neutron-agent is stuck in some kind of loop fork-bombing the system, I wouldn't expect it to be the culprit (especially since I tend to run it on VMs with only 2GB ram).

Comment 6 Alexander Chuzhoy 2016-06-06 19:30:24 UTC

The reported issue doesn't seem to reproduce:
instack-undercloud-4.0.0-2.el7ost.noarch

Comment 7 Alexander Chuzhoy 2016-06-07 14:27:17 UTC

Verified:

Environment:
instack-undercloud-4.0.0-2.el7ost.noarch

The issue doesn't reproduce.

Comment 9 errata-xmlrpc 2016-08-11 11:30:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html