Bug 1655207

Summary: neutron-openvswitch-agent wont start and is caught in restart loop
Product: Red Hat OpenStack Reporter: Robin Cernin <rcernin>
Component: openstack-neutronAssignee: Brent Eagles <beagles>
Status: CLOSED DUPLICATE QA Contact: Eran Kuris <ekuris>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: amuller, bhaley, chrisw, njohnston, psahoo, rchincho, skaplons
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---Flags: skaplons: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-24 18:24:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Robin Cernin 2018-12-01 06:18:53 UTC
Description of problem:

neutron-openvswitch-agent wont start and is hang in the reboot loop

/etc/neutron/plugins/ml2/ml2_conf.ini
firewall_driver = openvswitch 

/etc/neutron/plugins/ml2/openvswitch_agent.ini
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

2018-11-30 19:01:20.456 413650 WARNING stevedore.named [req-fce281e1-0c38-4cc4-bd7c-3082c2a246d9 - - - - -] Could not load neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
2018-11-30 19:01:25.667 413650 INFO oslo_rootwrap.client [req-fce281e1-0c38-4cc4-bd7c-3082c2a246d9 - - - - -] Spawned new rootwrap daemon process with pid=413692
2018-11-30 19:01:27.639 413650 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-eb5ddd06-2be6-48d0-a8bc-f5d5b05cb0c4 - - - - -] Agent initialized successfully, now running... 
2018-11-30 19:01:32.647 413650 CRITICAL neutron [-] Timeout: 5 seconds
2018-11-30 19:01:32.647 413650 ERROR neutron Traceback (most recent call last):
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2018-11-30 19:01:32.647 413650 ERROR neutron     sys.exit(main())
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     agent_main.main()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 51, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     mod.main()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     'neutron.plugins.ml2.drivers.openvswitch.agent.'
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 375, in run_apps
2018-11-30 19:01:32.647 413650 ERROR neutron     hub.joinall(services)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 97, in joinall
2018-11-30 19:01:32.647 413650 ERROR neutron     t.wait()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2018-11-30 19:01:32.647 413650 ERROR neutron     return self._exit_event.wait()
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2018-11-30 19:01:32.647 413650 ERROR neutron     current.throw(*self._exc)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2018-11-30 19:01:32.647 413650 ERROR neutron     result = function(*args, **kwargs)
2018-11-30 19:01:32.647 413650 ERROR neutron   File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 59, in _launch
2018-11-30 19:01:32.647 413650 ERROR neutron     raise e
2018-11-30 19:01:32.647 413650 ERROR neutron Timeout: 5 seconds
2018-11-30 19:01:32.647 413650 ERROR neutron

Version-Release number of selected component (if applicable):

OSP 10

openstack-neutron-openvswitch-9.4.1-5.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. systemctl start neutron-openvswitch-agent
2.
3.

Actual results:


Expected results:


Additional info:

Comment 5 Robin Cernin 2018-12-01 06:49:41 UTC
More detailed traceback attached by the customer:

Added additional debugging to /usr/lib/python2.7/site-packages/ryu/lib/hub.py:57

            except BaseException as e:
                LOG.error('hub: uncaught exception: %s',
                          traceback.format_exc())

Additional traceback generated by properly logging exception:

2018-12-01 01:39:14.955 984341 ERROR ryu.lib.hub [req-fdf8e6c0-dab5-4c22-88af-59b0d651e4e2 - - - - -] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 43, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2185, in main
    agent.daemon_loop()
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2101, in daemon_loop
    self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager
    pm.start()
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 57, in start
    self._monitor.start(block=True)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovsdb_monitor.py", line 117, in start
    while not self.is_active():
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 100, in is_active
    self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 159, in pid
    run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 297, in get_root_helper_child_pid
    pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 179, in find_child_pids
    log_fail_as_error=False)
  File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 128, in execute
    _stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1403, in _communicate
    stdout, stderr = self._communicate_with_select(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1504, in _communicate_with_select
    rlist, wlist, xlist = select.select(read_set, write_set, [])
  File "/usr/lib/python2.7/site-packages/eventlet/green/select.py", line 86, in select
    return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
Timeout: 5 seconds

2018-12-01 01:39:14.956 984341 CRITICAL neutron [-] Timeout: 5 seconds