Bug 1685621 - Mistral's eventlet monkeypatching breaks git-python
Summary: Mistral's eventlet monkeypatching breaks git-python
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: 15.0 (Stein)
Assignee: Emilien Macchi
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-05 16:54 UTC by Damien Ciabrini
Modified: 2019-09-26 10:48 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-common-10.4.1-0.20190308154822.63ab018.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:20:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1818757 0 None None None 2019-03-06 00:40:10 UTC
OpenStack gerrit 641338 0 None MERGED Fix interaction between gitpython and eventlet 2020-11-07 11:42:37 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:20:47 UTC

Description Damien Ciabrini 2019-03-05 16:54:03 UTC
Description of problem:
When deploying an overcloud, the openstack overcloud commands uses mistral to signal progress on the deployment, and fails:

Enabling ssh admin - COMPLETE.
Waiting for messages on queue 'tripleo' with no timeout.
The action raised an exception [action_ex_id=23ec3e50-9a0f-448c-8f86-5f7553357fe3, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{'container': 'overcloud', 'container_config': 'overclou
d-config'}']
 module 'select' has no attribute 'poll'
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 29, in run 
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run 
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run 
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 945, in take_action
    verbosity=self.app_args.verbose_level)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 323, in config_download
    raise exceptions.DeploymentError("Overcloud configuration failed.")
tripleoclient.exceptions.DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.
Full error:
2019-03-02 03:43:20.127 1 WARNING mistral.executors.default_executor [req-0568807f-eb53-4a31-a05b-afcf0631330b 07085b77ed23481ab53cc174dfb53f08 ec0093095eae49b1a5e67d54894e2b0d - default default] The action raised an exception [action_ex_
id=23ec3e50-9a0f-448c-8f86-5f7553357fe3, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{'container': 'overcloud', 'container_config': 'overcloud-config'}']
 module 'select' has no attribute 'poll': AttributeError: module 'select' has no attribute 'poll'
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor Traceback (most recent call last):
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     result = action.run(action_ctx)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/tripleo_common/actions/config.py", line 87, in run
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     commit_message=message)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 492, in download_config
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     git_repo = self.initialize_git_repo(config_dir)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/tripleo_common/utils/config.py", line 140, in initialize_git_repo
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     repo = git.Repo.init(dirname)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/git/repo/base.py", line 764, in init
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     git.init(**kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/git/cmd.py", line 440, in <lambda>
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/git/cmd.py", line 834, in _call_process
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     return self.execute(make_call(), **_kwargs)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib/python3.6/site-packages/git/cmd.py", line 587, in execute
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     stdout_value, stderr_value = proc.communicate()
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     stdout, stderr = self._communicate(input, endtime, timeout)
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib64/python3.6/subprocess.py", line 1521, in _communicate
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     with _PopenSelector() as selector:
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor   File "/usr/lib64/python3.6/selectors.py", line 348, in __init__
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor     self._poll = select.poll()
2019-03-02 03:43:20.127 1 ERROR mistral.executors.default_executor AttributeError: module 'select' has no attribute 'poll'

Quoting Michele:
"""
This is (another) eventlet monkeypatching issue. Reproducer:
import git
import eventlet
from eventlet import timeout as ev_timeout
import sys

eventlet.monkey_patch(
    os=True,
    select=True,
    socket=True,
    thread=False if '--use-debugger' in sys.argv else True,
    time=True)
"""


 
Version-Release number of selected component (if applicable):
openstack-mistral-executor:20190226.1

How reproducible:
Always

Steps to Reproduce:
1. deploy an overcloud
2.
3.

Actual results:
mistral fails, deployment never proceeds

Expected results:
no failure

Additional info:

Comment 2 Adriano Petrich 2019-03-06 12:09:00 UTC
Submitted an upstream fix. The issue as discussed upstream is that gitpython subprocess is not being properly patched by mistral.

Comment 11 errata-xmlrpc 2019-09-21 11:20:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.