Bug 1467645 - Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not saf...
Status: ON_QA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-mistralclient (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: ga
: 12.0 (Pike)
Assigned To: RHOS Maint
nlevinki
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-04 08:35 EDT by Marius Cornea
Modified: 2017-09-05 20:35 EDT (History)
12 users (show)

See Also:
Fixed In Version: python-mistralclient-3.1.1-0.20170731092149.0f28372.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
undercloud upgrade (665.13 KB, text/plain)
2017-07-04 08:35 EDT, Marius Cornea
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1688518 None None None 2017-07-06 06:09 EDT
Launchpad 1693790 None None None 2017-07-04 08:36 EDT

  None (edit)
Description Marius Cornea 2017-07-04 08:35:51 EDT
Created attachment 1294197 [details]
undercloud upgrade

Description of problem:
Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

Version-Release number of selected component (if applicable):
[stack@undercloud-0 ~]$ rpm -qa | grep instack
instack-6.0.1-0.20170525020340.103d575.el7ost.noarch
instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 on rhel 7.4
2. Upgrade undercloud to OSP12

Actual results:
Upgrade fails with:

#############################################################################
Undercloud upgrade failed.

Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################


#############################################################################
Undercloud upgrade failed.

Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Expected results:
Upgrade succeeds.

Additional info:
[stack@undercloud-0 ~]$ cat undercloud.conf 
[DEFAULT]
local_ip = 192.168.0.1/25
network_gateway = 192.168.0.126
undercloud_public_vip = 192.168.0.2
undercloud_admin_vip = 192.168.0.3
local_interface = eth1
network_cidr = 192.168.0.0/25
masquerade_network = 192.168.0.0/25
dhcp_start = 192.168.0.10
dhcp_end = 192.168.0.30
inspection_iprange = 192.168.0.31,192.168.0.50
undercloud_debug = False
generate_service_certificate = True
[auth]

Attaching the complete output.
Comment 1 Marius Cornea 2017-07-04 08:44:54 EDT
Workaround: rerun openstack undercloud upgrade.
Comment 2 marios 2017-07-06 06:09:53 EDT
o/ mcornea - found this upstream bug https://bugs.launchpad.net/mistral/+bug/1688518 which I'll add to trackers (you may want to close the other launchpad bug as duplicate I think) 

There is some discussion there ^^ that a recent commit  in mistralclient may have fixed this https://github.com/openstack/python-mistralclient/commit/83b3d0d39cb8072682fac74f6a40877030e91c18 (well, not _that_ recent) so I am sceptical. On my environment with python2-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch the relevant code is like:

/usr/lib/python2.7/site-packages/mistralclient/api/v2/client.py :
 40 class Client(object):
 41     def __init__(self, auth_type='keystone', **kwargs):
 42         # We get the session at this point, as some instances of session
 43         # objects might have mutexes that can't be deep-copied.
 44         session = kwargs.pop('session', None)
 45         req = copy.deepcopy(kwargs)
 46         mistral_url = req.get('mistral_url')

and I do not hit this. Can we check the version of mistralclient you have as a start and take it from there. Really don't get why this doesn't reproduce on the second run though (same code is executed...)
Comment 3 Marius Cornea 2017-07-07 12:57:54 EDT
(In reply to marios from comment #2)
> and I do not hit this. Can we check the version of mistralclient you have as
> a start and take it from there. Really don't get why this doesn't reproduce
> on the second run though (same code is executed...)

Hey Marios o/ here is the mistral /var/log/yum.log snippet on an environment showing this issue:

Jul 07 05:28:44 Installed: puppet-mistral-10.3.1-1.el7ost.noarch
Jul 07 05:28:50 Installed: python-mistralclient-3.0.0-1.el7ost.noarch
Jul 07 05:28:51 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch
Jul 07 05:33:01 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch
Jul 07 05:33:02 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch
Jul 07 05:33:03 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch
Jul 07 05:33:04 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch
Jul 07 12:48:10 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch
Jul 07 12:48:14 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch
Jul 07 12:48:15 Updated: python-openstack-mistral-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:15 Updated: openstack-mistral-common-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-api-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-engine-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-executor-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:23 Updated: puppet-mistral-11.2.0-0.20170626052247.15e99c5.el7ost.noarch
Comment 4 Marius Cornea 2017-07-11 06:40:15 EDT
We can get it working by updating the python-mistralclient package before undercloud upgrade:

sudo systemctl stop 'openstack-*' 'neutron-*' httpd
sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient python-mistralclient
openstack undercloud upgrade &> undercloud_upgrade.log
Comment 5 marios 2017-07-11 06:46:12 EDT
(In reply to Marius Cornea from comment #4)
> We can get it working by updating the python-mistralclient package before
> undercloud upgrade:
> 
> sudo systemctl stop 'openstack-*' 'neutron-*' httpd
> sudo yum update -y instack-undercloud openstack-puppet-modules
> openstack-tripleo-common python-tripleoclient python-mistralclient
> openstack undercloud upgrade &> undercloud_upgrade.log

ACK thanks for the info Marius... we can carry it in the list of things we update before running the openstack undercloud upgrade for now I suppose. I am puzzled why the manual update works though since there _is_ a full package update before the puppet run so it should be using the latest mistralclient by then anyway I mean without you having to manually update it.
Comment 7 marios 2017-07-20 05:50:09 EDT
Note: there is a new 'pre-upgrade' hook for instack-undercloud  - being wired up here https://review.openstack.org/#/c/479082/ (the actual function is already landed in instack_undercloud/undercloud.py). That should put an end to the manual stop and update process.

Are we still hitting this consistently? Looks like this problem is present in python-mistralclient-3.0.0-1.el7ost.noarch but fixed in 3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch (possibly earlier, these are just the versions we're getting on this env).

So currently, manually yum update the mistralclient (to get python-mistralclient-3.1.0-0.20170425130122) before the undercloud upgrade is workaround.
Comment 8 Sofer Athlan-Guyot 2017-07-25 10:51:19 EDT
Moving to post as the code should be there.
Comment 9 Marius Cornea 2017-07-25 17:10:24 EDT
I wasn't able to reproduce the issue anylonger:

[root@undercloud-0 stack]# grep mistral /var/log/yum.log 
Jul 25 13:16:10 Installed: puppet-mistral-10.3.1-1.el7ost.noarch
Jul 25 13:16:17 Installed: python-mistralclient-3.0.0-1.el7ost.noarch
Jul 25 13:16:18 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch
Jul 25 13:20:21 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch
Jul 25 13:20:23 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch
Jul 25 13:20:25 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch
Jul 25 13:20:26 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch
Jul 25 16:57:13 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch
Jul 25 16:57:26 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch
Jul 25 16:57:28 Updated: python-openstack-mistral-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:28 Updated: openstack-mistral-common-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-engine-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-executor-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-api-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:32 Updated: puppet-mistral-11.2.0-0.20170714215857.ae0bd7c.el7ost.noarch
Comment 10 Thierry Vignaud 2017-08-21 06:09:53 EDT
Fixed in commit 83b3d0d39cb8072682fac74f6a40877030e91c18 according to https://bugs.launchpad.net/mistral/+bug/1688518

Note You need to log in before you can comment on or make changes to this bug.