Bug 1397017 - Parallel executions of AppManager.close()
Summary: Parallel executions of AppManager.close()
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-ryu
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z7
: 10.0 (Newton)
Assignee: RHOS Maint
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks: 1450223
TreeView+ depends on / blocked
 
Reported: 2016-11-21 12:17 UTC by Marian Krcmarik
Modified: 2022-08-16 13:52 UTC (History)
9 users (show)

Fixed In Version: python-ryu-4.9-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-22 13:25:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1589746 0 None None None 2016-11-21 12:17:46 UTC
Red Hat Issue Tracker OSP-4582 0 None None None 2022-08-16 13:52:35 UTC

Description Marian Krcmarik 2016-11-21 12:17:46 UTC
Description of problem:
If an AppManager.close call is started and all AppManager.services are stopped, AppManager.run_apps starts another close() call, resulting in a KeyError exception in close() (*1).  Prevent that using a semaphore.

The upstream patch:
https://github.com/osrg/ryu/commit/b0ab4f16028c452374c5a0f22bd970038194f142

I hit this problem when stopping neutron-openvswitch-agent service on Openstack computes/controllers in order to setup up Instance Ha (where the service is being stopped firstly and then put under pacemaker management for computes and left under systemd management for controllers).

The traceack on which openvswitch failed was following:
2016-11-20 07:30:59.464 102285 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: None
2016-11-20 07:30:59.465 102285 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: None
2016-11-20 07:30:59.465 102285 DEBUG neutron.agent.linux.async_process [-] Output received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: None _read_stdout /usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py:237
2016-11-20 07:30:59.466 102285 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 545, in close
    self.uninstantiate(app_name)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 528, in uninstantiate
    app = self.applications.pop(name)
KeyError: 'ofctl_service'

Version-Release number of selected component (if applicable):
python-ryu-4.3-2.el7ost.noarch.rpm  

How reproducible:
Sometimes

Steps to Reproduce:
1. Stop neutron-openvswitch-agent service

Actual results:
service fails, not sure about other consequences - maybe some binded ports remain active

Expected results:
Sane service stop

Additional info:

Comment 1 Mike Burns 2017-05-11 11:54:09 UTC
is this already fixed?  4.9-2 is already shipped?

Comment 2 Marian Krcmarik 2017-05-11 12:12:28 UTC
(In reply to Mike Burns from comment #1)
> is this already fixed?  4.9-2 is already shipped?

It does seem so in RHOSP11

Comment 3 Ihar Hrachyshka 2017-05-15 18:20:53 UTC
I believe this will be fixed for OSP10 with: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=534727

Comment 6 Lon Hohberger 2017-09-06 19:58:10 UTC
According to our records, this should be resolved by python-ryu-4.9-2.1.el7ost.  This build is available now.


Note You need to log in before you can comment on or make changes to this bug.