Bug 1467645

Summary: Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: python-mistralclientAssignee: RHOS Maint <rhos-maint>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 12.0 (Pike)CC: augol, dbecker, jjoyce, jschluet, mandreou, mburns, morazi, nyechiel, rhel-osp-director-maint, sathlang, slinaber, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-mistralclient-3.1.1-0.20170731092149.0f28372.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:37:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
undercloud upgrade none

Description Marius Cornea 2017-07-04 12:35:51 UTC
Created attachment 1294197 [details]
undercloud upgrade

Description of problem:
Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

Version-Release number of selected component (if applicable):
[stack@undercloud-0 ~]$ rpm -qa | grep instack
instack-6.0.1-0.20170525020340.103d575.el7ost.noarch
instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 on rhel 7.4
2. Upgrade undercloud to OSP12

Actual results:
Upgrade fails with:

#############################################################################
Undercloud upgrade failed.

Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################


#############################################################################
Undercloud upgrade failed.

Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Expected results:
Upgrade succeeds.

Additional info:
[stack@undercloud-0 ~]$ cat undercloud.conf 
[DEFAULT]
local_ip = 192.168.0.1/25
network_gateway = 192.168.0.126
undercloud_public_vip = 192.168.0.2
undercloud_admin_vip = 192.168.0.3
local_interface = eth1
network_cidr = 192.168.0.0/25
masquerade_network = 192.168.0.0/25
dhcp_start = 192.168.0.10
dhcp_end = 192.168.0.30
inspection_iprange = 192.168.0.31,192.168.0.50
undercloud_debug = False
generate_service_certificate = True
[auth]

Attaching the complete output.

Comment 1 Marius Cornea 2017-07-04 12:44:54 UTC
Workaround: rerun openstack undercloud upgrade.

Comment 2 Marios Andreou 2017-07-06 10:09:53 UTC
o/ mcornea - found this upstream bug https://bugs.launchpad.net/mistral/+bug/1688518 which I'll add to trackers (you may want to close the other launchpad bug as duplicate I think) 

There is some discussion there ^^ that a recent commit  in mistralclient may have fixed this https://github.com/openstack/python-mistralclient/commit/83b3d0d39cb8072682fac74f6a40877030e91c18 (well, not _that_ recent) so I am sceptical. On my environment with python2-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch the relevant code is like:

/usr/lib/python2.7/site-packages/mistralclient/api/v2/client.py :
 40 class Client(object):
 41     def __init__(self, auth_type='keystone', **kwargs):
 42         # We get the session at this point, as some instances of session
 43         # objects might have mutexes that can't be deep-copied.
 44         session = kwargs.pop('session', None)
 45         req = copy.deepcopy(kwargs)
 46         mistral_url = req.get('mistral_url')

and I do not hit this. Can we check the version of mistralclient you have as a start and take it from there. Really don't get why this doesn't reproduce on the second run though (same code is executed...)

Comment 3 Marius Cornea 2017-07-07 16:57:54 UTC
(In reply to marios from comment #2)
> and I do not hit this. Can we check the version of mistralclient you have as
> a start and take it from there. Really don't get why this doesn't reproduce
> on the second run though (same code is executed...)

Hey Marios o/ here is the mistral /var/log/yum.log snippet on an environment showing this issue:

Jul 07 05:28:44 Installed: puppet-mistral-10.3.1-1.el7ost.noarch
Jul 07 05:28:50 Installed: python-mistralclient-3.0.0-1.el7ost.noarch
Jul 07 05:28:51 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch
Jul 07 05:33:01 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch
Jul 07 05:33:02 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch
Jul 07 05:33:03 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch
Jul 07 05:33:04 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch
Jul 07 12:48:10 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch
Jul 07 12:48:14 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch
Jul 07 12:48:15 Updated: python-openstack-mistral-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:15 Updated: openstack-mistral-common-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-api-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-engine-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:22 Updated: openstack-mistral-executor-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch
Jul 07 12:48:23 Updated: puppet-mistral-11.2.0-0.20170626052247.15e99c5.el7ost.noarch

Comment 4 Marius Cornea 2017-07-11 10:40:15 UTC
We can get it working by updating the python-mistralclient package before undercloud upgrade:

sudo systemctl stop 'openstack-*' 'neutron-*' httpd
sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient python-mistralclient
openstack undercloud upgrade &> undercloud_upgrade.log

Comment 5 Marios Andreou 2017-07-11 10:46:12 UTC
(In reply to Marius Cornea from comment #4)
> We can get it working by updating the python-mistralclient package before
> undercloud upgrade:
> 
> sudo systemctl stop 'openstack-*' 'neutron-*' httpd
> sudo yum update -y instack-undercloud openstack-puppet-modules
> openstack-tripleo-common python-tripleoclient python-mistralclient
> openstack undercloud upgrade &> undercloud_upgrade.log

ACK thanks for the info Marius... we can carry it in the list of things we update before running the openstack undercloud upgrade for now I suppose. I am puzzled why the manual update works though since there _is_ a full package update before the puppet run so it should be using the latest mistralclient by then anyway I mean without you having to manually update it.

Comment 7 Marios Andreou 2017-07-20 09:50:09 UTC
Note: there is a new 'pre-upgrade' hook for instack-undercloud  - being wired up here https://review.openstack.org/#/c/479082/ (the actual function is already landed in instack_undercloud/undercloud.py). That should put an end to the manual stop and update process.

Are we still hitting this consistently? Looks like this problem is present in python-mistralclient-3.0.0-1.el7ost.noarch but fixed in 3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch (possibly earlier, these are just the versions we're getting on this env).

So currently, manually yum update the mistralclient (to get python-mistralclient-3.1.0-0.20170425130122) before the undercloud upgrade is workaround.

Comment 8 Sofer Athlan-Guyot 2017-07-25 14:51:19 UTC
Moving to post as the code should be there.

Comment 9 Marius Cornea 2017-07-25 21:10:24 UTC
I wasn't able to reproduce the issue anylonger:

[root@undercloud-0 stack]# grep mistral /var/log/yum.log 
Jul 25 13:16:10 Installed: puppet-mistral-10.3.1-1.el7ost.noarch
Jul 25 13:16:17 Installed: python-mistralclient-3.0.0-1.el7ost.noarch
Jul 25 13:16:18 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch
Jul 25 13:20:21 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch
Jul 25 13:20:23 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch
Jul 25 13:20:25 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch
Jul 25 13:20:26 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch
Jul 25 16:57:13 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch
Jul 25 16:57:26 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch
Jul 25 16:57:28 Updated: python-openstack-mistral-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:28 Updated: openstack-mistral-common-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-engine-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-executor-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:29 Updated: openstack-mistral-api-5.0.0-0.20170718095321.61231ec.el7ost.noarch
Jul 25 16:57:32 Updated: puppet-mistral-11.2.0-0.20170714215857.ae0bd7c.el7ost.noarch

Comment 10 Thierry Vignaud 2017-08-21 10:09:53 UTC
Fixed in commit 83b3d0d39cb8072682fac74f6a40877030e91c18 according to https://bugs.launchpad.net/mistral/+bug/1688518

Comment 16 errata-xmlrpc 2017-12-13 21:37:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462