rhel-osp-director: On undercloud: neutron-openvswitch-agent: failed Environment: instack-undercloud-2.1.2-39.el7ost.noarch Steps to reproduce: 1. Deploy undercloud. 2. Run "openstack-status|grep neutron-openvswitch-agent" Result: neutron-openvswitch-agent: failed Expected result: neutron-openvswitch-agent: active
Created attachment 1159000 [details] install-undercloud.log
from /var/log/neutron/openvswitch-agent.log 2016-05-18 12:14:16.002 30518 CRITICAL neutron [req-0841b653-94a6-4c22-a9df-b43de9dd9850 - - - - -] OSError: [Errno 12] Cannot allocate memory 2016-05-18 12:14:16.002 30518 ERROR neutron Traceback (most recent call last): 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module> 2016-05-18 12:14:16.002 30518 ERROR neutron sys.exit(main()) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main 2016-05-18 12:14:16.002 30518 ERROR neutron agent_main.main() 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main 2016-05-18 12:14:16.002 30518 ERROR neutron mod.main() 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main 2016-05-18 12:14:16.002 30518 ERROR neutron ovs_neutron_agent.main(bridge_classes) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2101, in main 2016-05-18 12:14:16.002 30518 ERROR neutron agent.daemon_loop() 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2024, in daemon_loop 2016-05-18 12:14:16.002 30518 ERROR neutron self.rpc_loop(polling_manager=pm) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ 2016-05-18 12:14:16.002 30518 ERROR neutron self.gen.throw(type, value, traceback) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 42, in get_polling_manager 2016-05-18 12:14:16.002 30518 ERROR neutron pm.stop() 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 61, in stop 2016-05-18 12:14:16.002 30518 ERROR neutron self._monitor.stop() 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 129, in stop 2016-05-18 12:14:16.002 30518 ERROR neutron self._kill(kill_signal) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 162, in _kill 2016-05-18 12:14:16.002 30518 ERROR neutron pid = self.pid 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 158, in pid 2016-05-18 12:14:16.002 30518 ERROR neutron run_as_root=self.run_as_root) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 257, in get_root_helper_child_pid 2016-05-18 12:14:16.002 30518 ERROR neutron pid = find_child_pids(pid)[0] 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 188, in find_child_pids 2016-05-18 12:14:16.002 30518 ERROR neutron log_fail_as_error=False) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 119, in execute 2016-05-18 12:14:16.002 30518 ERROR neutron addl_env=addl_env) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 88, in create_process 2016-05-18 12:14:16.002 30518 ERROR neutron stderr=subprocess.PIPE) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 214, in subprocess_popen 2016-05-18 12:14:16.002 30518 ERROR neutron close_fds=close_fds, env=env) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 53, in __init__ 2016-05-18 12:14:16.002 30518 ERROR neutron subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__ 2016-05-18 12:14:16.002 30518 ERROR neutron errread, errwrite) 2016-05-18 12:14:16.002 30518 ERROR neutron File "/usr/lib64/python2.7/subprocess.py", line 1224, in _execute_child 2016-05-18 12:14:16.002 30518 ERROR neutron self.pid = os.fork() 2016-05-18 12:14:16.002 30518 ERROR neutron OSError: [Errno 12] Cannot allocate memory 2016-05-18 12:14:16.002 30518 ERROR neutron 2016-05-18 12:14:19.235 30518 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 6a2f2019df7c4c168b7377bebee5d60f exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
The traceback seems to show things happening in a fork() call, which I believe marks the memory copy-on-write and isn't just blindly duplicating all of the memory. So I wouldn't expect that particular call to be a *huge* amount of memory allocation. Would it be possible for someone to do a deploy and watch the output of top -o%MEM or something similar to see where the memory is going? Unless something really weird is going on and ovs-neutron-agent is stuck in some kind of loop fork-bombing the system, I wouldn't expect it to be the culprit (especially since I tend to run it on VMs with only 2GB ram).
The reported issue doesn't seem to reproduce: instack-undercloud-4.0.0-2.el7ost.noarch
Verified: Environment: instack-undercloud-4.0.0-2.el7ost.noarch The issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1599.html