1655207 – neutron-openvswitch-agent wont start and is caught in restart loop

Bug 1655207 - neutron-openvswitch-agent wont start and is caught in restart loop

Summary: neutron-openvswitch-agent wont start and is caught in restart loop

Keywords:
Status:	CLOSED DUPLICATE of bug 1806684
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Brent Eagles
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-12-01 06:18 UTC by Robin Cernin
Modified:	2022-03-13 16:20 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-02-24 18:24:08 UTC
Target Upstream Version:
Embargoed:
Flags:	skaplons: needinfo-

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-13742	0	None	None	None	2022-03-13 16:20:30 UTC

Internal Links: 1806684

Description Robin Cernin 2018-12-01 06:18:53 UTC

Description of problem:

neutron-openvswitch-agent wont start and is hang in the reboot loop

/etc/neutron/plugins/ml2/ml2_conf.ini
firewall_driver = openvswitch 

/etc/neutron/plugins/ml2/openvswitch_agent.ini
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

2018-11-30 19:01:20.456 413650 WARNING stevedore.named [req-fce281e1-0c38-4cc4-bd7c-3082c2a246d9 - - - - -] Could not load neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
2018-11-30 19:01:25.667 413650 INFO oslo_rootwrap.client [req-fce281e1-0c38-4cc4-bd7c-3082c2a246d9 - - - - -] Spawned new rootwrap daemon process with pid=413692
2018-11-30 19:01:27.639 413650 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-eb5ddd06-2be6-48d0-a8bc-f5d5b05cb0c4 - - - - -] Agent initialized successfully, now running... 
2018-11-30 19:01:32.647 413650 CRITICAL neutron [-] Timeout: 5 seconds
2018-11-30 19:01:32.647 413650 ERROR neutron Traceback (most recent call last):
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2018-11-30 19:01:32.647 413650 ERROR neutron     sys.exit(main())
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     agent_main.main()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 51, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     mod.main()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 375, in run_apps
2018-11-30 19:01:32.647 413650 ERROR neutron     hub.joinall(services)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 97, in joinall
2018-11-30 19:01:32.647 413650 ERROR neutron     t.wait()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2018-11-30 19:01:32.647 413650 ERROR neutron     return self._exit_event.wait()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2018-11-30 19:01:32.647 413650 ERROR neutron     current.throw(*self._exc)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     result = function(*args, **kwargs)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 59, in _launch
2018-11-30 19:01:32.647 413650 ERROR neutron     raise e
2018-11-30 19:01:32.647 413650 ERROR neutron Timeout: 5 seconds
2018-11-30 19:01:32.647 413650 ERROR neutron

Version-Release number of selected component (if applicable):

OSP 10

openstack-neutron-openvswitch-9.4.1-5.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. systemctl start neutron-openvswitch-agent
2.
3.

Actual results:


Expected results:


Additional info:

Comment 5 Robin Cernin 2018-12-01 06:49:41 UTC

More detailed traceback attached by the customer:

Added additional debugging to /usr/lib/python2.7/site-packages/ryu/lib/hub.py:57

            except BaseException as e:
                LOG.error('hub: uncaught exception: %s',
                          traceback.format_exc())

Additional traceback generated by properly logging exception:

2018-12-01 01:39:14.955 984341 ERROR ryu.lib.hub [req-fdf8e6c0-dab5-4c22-88af-59b0d651e4e2 - - - - -] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 43, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2185, in main
    agent.daemon_loop()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2101, in daemon_loop
    self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager
    pm.start()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 57, in start
    self._monitor.start(block=True)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovsdb_monitor.py", line 117, in start
    while not self.is_active():
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 100, in is_active
    self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 159, in pid
    run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 297, in get_root_helper_child_pid
    pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 179, in find_child_pids
    log_fail_as_error=False)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 128, in execute
    _stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate
    stdout, stderr = self._communicate_with_select(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1504, in _communicate_with_select
    rlist, wlist, xlist = select.select(read_set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in select
    return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
Timeout: 5 seconds

2018-12-01 01:39:14.956 984341 CRITICAL neutron [-] Timeout: 5 seconds

Note You need to log in before you can comment on or make changes to this bug.