Hide Forgot
Created attachment 1294197 [details] undercloud upgrade Description of problem: Undercloud upgrade fails with: Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__() Version-Release number of selected component (if applicable): [stack@undercloud-0 ~]$ rpm -qa | grep instack instack-6.0.1-0.20170525020340.103d575.el7ost.noarch instack-undercloud-7.1.1-0.20170623182135.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 on rhel 7.4 2. Upgrade undercloud to OSP12 Actual results: Upgrade fails with: ############################################################################# Undercloud upgrade failed. Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__() See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# ############################################################################# Undercloud upgrade failed. Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__() See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Expected results: Upgrade succeeds. Additional info: [stack@undercloud-0 ~]$ cat undercloud.conf [DEFAULT] local_ip = 192.168.0.1/25 network_gateway = 192.168.0.126 undercloud_public_vip = 192.168.0.2 undercloud_admin_vip = 192.168.0.3 local_interface = eth1 network_cidr = 192.168.0.0/25 masquerade_network = 192.168.0.0/25 dhcp_start = 192.168.0.10 dhcp_end = 192.168.0.30 inspection_iprange = 192.168.0.31,192.168.0.50 undercloud_debug = False generate_service_certificate = True [auth] Attaching the complete output.
Workaround: rerun openstack undercloud upgrade.
o/ mcornea - found this upstream bug https://bugs.launchpad.net/mistral/+bug/1688518 which I'll add to trackers (you may want to close the other launchpad bug as duplicate I think) There is some discussion there ^^ that a recent commit in mistralclient may have fixed this https://github.com/openstack/python-mistralclient/commit/83b3d0d39cb8072682fac74f6a40877030e91c18 (well, not _that_ recent) so I am sceptical. On my environment with python2-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch the relevant code is like: /usr/lib/python2.7/site-packages/mistralclient/api/v2/client.py : 40 class Client(object): 41 def __init__(self, auth_type='keystone', **kwargs): 42 # We get the session at this point, as some instances of session 43 # objects might have mutexes that can't be deep-copied. 44 session = kwargs.pop('session', None) 45 req = copy.deepcopy(kwargs) 46 mistral_url = req.get('mistral_url') and I do not hit this. Can we check the version of mistralclient you have as a start and take it from there. Really don't get why this doesn't reproduce on the second run though (same code is executed...)
(In reply to marios from comment #2) > and I do not hit this. Can we check the version of mistralclient you have as > a start and take it from there. Really don't get why this doesn't reproduce > on the second run though (same code is executed...) Hey Marios o/ here is the mistral /var/log/yum.log snippet on an environment showing this issue: Jul 07 05:28:44 Installed: puppet-mistral-10.3.1-1.el7ost.noarch Jul 07 05:28:50 Installed: python-mistralclient-3.0.0-1.el7ost.noarch Jul 07 05:28:51 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch Jul 07 05:33:01 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch Jul 07 05:33:02 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch Jul 07 05:33:03 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch Jul 07 05:33:04 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch Jul 07 12:48:10 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch Jul 07 12:48:14 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch Jul 07 12:48:15 Updated: python-openstack-mistral-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch Jul 07 12:48:15 Updated: openstack-mistral-common-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch Jul 07 12:48:22 Updated: openstack-mistral-api-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch Jul 07 12:48:22 Updated: openstack-mistral-engine-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch Jul 07 12:48:22 Updated: openstack-mistral-executor-5.0.0-0.20170627151830.fe1a1e1.el7ost.noarch Jul 07 12:48:23 Updated: puppet-mistral-11.2.0-0.20170626052247.15e99c5.el7ost.noarch
We can get it working by updating the python-mistralclient package before undercloud upgrade: sudo systemctl stop 'openstack-*' 'neutron-*' httpd sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient python-mistralclient openstack undercloud upgrade &> undercloud_upgrade.log
(In reply to Marius Cornea from comment #4) > We can get it working by updating the python-mistralclient package before > undercloud upgrade: > > sudo systemctl stop 'openstack-*' 'neutron-*' httpd > sudo yum update -y instack-undercloud openstack-puppet-modules > openstack-tripleo-common python-tripleoclient python-mistralclient > openstack undercloud upgrade &> undercloud_upgrade.log ACK thanks for the info Marius... we can carry it in the list of things we update before running the openstack undercloud upgrade for now I suppose. I am puzzled why the manual update works though since there _is_ a full package update before the puppet run so it should be using the latest mistralclient by then anyway I mean without you having to manually update it.
Note: there is a new 'pre-upgrade' hook for instack-undercloud - being wired up here https://review.openstack.org/#/c/479082/ (the actual function is already landed in instack_undercloud/undercloud.py). That should put an end to the manual stop and update process. Are we still hitting this consistently? Looks like this problem is present in python-mistralclient-3.0.0-1.el7ost.noarch but fixed in 3.1.0-0.20170425130122.9e6ff99.el7.centos.noarch (possibly earlier, these are just the versions we're getting on this env). So currently, manually yum update the mistralclient (to get python-mistralclient-3.1.0-0.20170425130122) before the undercloud upgrade is workaround.
Moving to post as the code should be there.
I wasn't able to reproduce the issue anylonger: [root@undercloud-0 stack]# grep mistral /var/log/yum.log Jul 25 13:16:10 Installed: puppet-mistral-10.3.1-1.el7ost.noarch Jul 25 13:16:17 Installed: python-mistralclient-3.0.0-1.el7ost.noarch Jul 25 13:16:18 Installed: python-openstack-mistral-4.0.2-1.el7ost.noarch Jul 25 13:20:21 Installed: openstack-mistral-common-4.0.2-1.el7ost.noarch Jul 25 13:20:23 Installed: openstack-mistral-api-4.0.2-1.el7ost.noarch Jul 25 13:20:25 Installed: openstack-mistral-engine-4.0.2-1.el7ost.noarch Jul 25 13:20:26 Installed: openstack-mistral-executor-4.0.2-1.el7ost.noarch Jul 25 16:57:13 Installed: python-mistral-lib-0.2.0-0.20170529134325.bb1b87b.el7ost.noarch Jul 25 16:57:26 Updated: python-mistralclient-3.1.0-0.20170425130122.9e6ff99.el7ost.noarch Jul 25 16:57:28 Updated: python-openstack-mistral-5.0.0-0.20170718095321.61231ec.el7ost.noarch Jul 25 16:57:28 Updated: openstack-mistral-common-5.0.0-0.20170718095321.61231ec.el7ost.noarch Jul 25 16:57:29 Updated: openstack-mistral-engine-5.0.0-0.20170718095321.61231ec.el7ost.noarch Jul 25 16:57:29 Updated: openstack-mistral-executor-5.0.0-0.20170718095321.61231ec.el7ost.noarch Jul 25 16:57:29 Updated: openstack-mistral-api-5.0.0-0.20170718095321.61231ec.el7ost.noarch Jul 25 16:57:32 Updated: puppet-mistral-11.2.0-0.20170714215857.ae0bd7c.el7ost.noarch
Fixed in commit 83b3d0d39cb8072682fac74f6a40877030e91c18 according to https://bugs.launchpad.net/mistral/+bug/1688518
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462