1232997 – ironic client fails due to locked instance while deploying the osp-7 overcloud

Bug 1232997 - ironic client fails due to locked instance while deploying the osp-7 overcloud

Summary: ironic client fails due to locked instance while deploying the osp-7 overcloud

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-ironic
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	z5
Target Release:	7.0 (Kilo)
Assignee:	Lucas Alvares Gomes
QA Contact:	Toure Dunnon
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1273561
TreeView+	depends on / blocked

Reported:	2015-06-18 02:23 UTC by wes hayutin
Modified:	2020-05-14 02:18 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-10 10:55:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
logs from the undercloud (13.92 MB, application/octet-stream) 2015-06-18 02:26 UTC, wes hayutin	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1363766	0	unspecified	CLOSED	RHOSP Deployment dies at 90%	2021-02-22 00:41:40 UTC

Internal Links: 1363766

Description wes hayutin 2015-06-18 02:23:24 UTC

Description of problem:

02:17:42.484 failed: [undercloud] => {"changed": true, "cmd": "source /home/stack/stackrc; if [ -f \"/home/stack/deploy-overcloudrc\" ]; then\n source /home/stack/deploy-overcloudrc;\n fi; openstack overcloud deploy --plan-uuid 37b893ec-c78b-49e1-93be-5888ae6ef5fd --control-scale $CONTROLSCALE --compute-scale $COMPUTESCALE --ceph-storage-scale $CEPHSTORAGESCALE #Both swift and blockstorage are not supported downstream right now #--swift-storage-scale $SWIFTSTORAGESCALE #--block-storage-scale $BLOCKSTORAGESCALE;", "delta": "0:00:24.380112", "end": "2015-06-17 21:52:43.699863", "rc": 1, "start": "2015-06-17 21:52:19.319751", "warnings": []}
02:17:42.489 stderr: WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 1 of 6
02:17:42.489 WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 2 of 6
02:17:42.489 WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 3 of 6
02:17:42.489 WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 4 of 6
02:17:42.489 WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 5 of 6
02:17:42.489 WARNING: ironicclient.common.http Request returned failure status.
02:17:42.489 ERROR: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 6 of 6
02:17:42.499 ERROR: openstack Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409)
02:17:42.500 stdout: The following templates will be written:

Comment 2 wes hayutin 2015-06-18 02:26:26 UTC

Created attachment 1040221 [details]
logs from the undercloud

Comment 4 Tim Wilkinson 2015-09-23 13:30:01 UTC

This bug blocks both IO performance testing as well as hyper-converged storage evaluation of OSP7 on RHCS. It breaks discovery and prevents overcloud deployment. 

Additionally, the link in Comment 1 is no longer available.

Discovery fails with ... 
WARNING: ironicclient.common.http Error contacting Ironic server: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed. (HTTP 409). Attempt 15 of 61


In the ironic-api.log ...
  ...
2015-09-23 01:22:27.270 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists.
2015-09-23 01:22:29.304 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists.
2015-09-23 01:22:31.327 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists.
  ...
2015-09-23 01:23:28.406 17366 WARNING wsme.api [-] Client-side error: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed.
2015-09-23 01:23:30.435 17366 WARNING wsme.api [-] Client-side error: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed.
  ...

Comment 5 Tim Wilkinson 2015-09-23 13:32:00 UTC

# rpmq openstack
openstack-tempest-kilo-20150708.2.el7ost.noarch
openstack-nova-scheduler-2015.1.1-1.el7ost.noarch
openstack-nova-cert-2015.1.1-1.el7ost.noarch
redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-10.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.1-4.el7ost.noarch
openstack-nova-common-2015.1.1-1.el7ost.noarch
openstack-neutron-common-2015.1.1-5.el7ost.noarch
openstack-neutron-ml2-2015.1.1-5.el7ost.noarch
openstack-heat-engine-2015.1.1-4.el7ost.noarch
python-openstackclient-1.0.3-2.el7ost.noarch
openstack-selinux-0.6.37-1.el7ost.noarch
openstack-swift-container-2.3.0-1.el7ost.noarch
openstack-nova-api-2015.1.1-1.el7ost.noarch
openstack-tuskar-ui-0.4.0-3.el7ost.noarch
openstack-ceilometer-common-2015.1.1-1.el7ost.noarch
openstack-nova-compute-2015.1.1-1.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
python-django-openstack-auth-1.2.0-5.el7ost.noarch
openstack-dashboard-theme-2015.1.1-2.el7ost.noarch
openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-neutron-openvswitch-2015.1.1-5.el7ost.noarch
openstack-heat-api-2015.1.1-4.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-64.el7ost.noarch
openstack-ironic-conductor-2015.1.1-4.el7ost.noarch
openstack-nova-conductor-2015.1.1-1.el7ost.noarch
openstack-swift-account-2.3.0-1.el7ost.noarch
openstack-swift-proxy-2.3.0-1.el7ost.noarch
openstack-tuskar-0.4.18-4.el7ost.noarch
openstack-nova-console-2015.1.1-1.el7ost.noarch
openstack-ceilometer-notification-2015.1.1-1.el7ost.noarch
openstack-ceilometer-collector-2015.1.1-1.el7ost.noarch
openstack-dashboard-2015.1.1-2.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-heat-common-2015.1.1-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.1-4.el7ost.noarch
openstack-puppet-modules-2015.1.8-17.el7ost.noarch
openstack-ironic-common-2015.1.1-4.el7ost.noarch
openstack-ceilometer-api-2015.1.1-1.el7ost.noarch
openstack-ceilometer-alarm-2015.1.1-1.el7ost.noarch
openstack-ironic-api-2015.1.1-4.el7ost.noarch
openstack-keystone-2015.1.1-1.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-5.el7ost.noarch
openstack-ironic-discoverd-1.1.0-6.el7ost.noarch
openstack-ceilometer-central-2015.1.1-1.el7ost.noarch
openstack-swift-2.3.0-1.el7ost.noarch
openstack-nova-novncproxy-2015.1.1-1.el7ost.noarch
openstack-swift-object-2.3.0-1.el7ost.noarch
openstack-tripleo-common-0.0.1.dev6-2.git49b57eb.el7ost.noarch
openstack-glance-2015.1.1-1.el7ost.noarch
openstack-neutron-2015.1.1-5.el7ost.noarch

Comment 8 Dan Yocum 2016-04-04 16:04:37 UTC

I'm getting the same error:

2016-04-04 07:10:10.877 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581
2016-04-04 07:10:10.878 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296
2016-04-04 07:10:10.878 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603
2016-04-04 07:10:10.879 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin servic
e: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632
2016-04-04 07:10:10.893 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name
,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-Us
er-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenan
t _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672
2016-04-04 07:10:10.894 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581
2016-04-04 07:10:10.895 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296
2016-04-04 07:10:10.895 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603
2016-04-04 07:10:10.896 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin servic
e: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632
2016-04-04 07:10:10.908 1214 WARNING wsme.api [-] Client-side error: Node d094ef1e-e6c3-45f5-a948-550e24ec673c is locked by host director1.os1.phx2.redhat.com, please retry after the current operation is completed.
2016-04-04 07:10:12.668 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672
2016-04-04 07:10:12.668 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581
2016-04-04 07:10:12.669 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296
2016-04-04 07:10:12.670 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603
2016-04-04 07:10:12.670 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin service: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632
2016-04-04 07:10:12.915 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672
2016-04-04 07:10:12.916 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581
2016-04-04 07:10:12.917 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296
2016-04-04 07:10:12.918 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603
2016-04-04 07:10:12.918 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin service: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632
2016-04-04 07:10:12.928 1214 WARNING wsme.api [-] Client-side error: Node d094ef1e-e6c3-45f5-a948-550e24ec673c is locked by host director1.os1.phx2.redhat.com, please retry after the current operation is completed.


Swift-proxy is running:

[root@director1 ironic]# ps auxww | grep swift | grep proxy
swift     9208  0.0  0.4 332560 47804 ?        Ss   08:25   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     9369  0.0  0.4 338496 49812 ?        S    08:26   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     9370  0.0  0.4 338496 49812 ?        S    08:26   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     9372  0.0  0.4 338496 49744 ?        S    08:26   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift     9373  0.0  0.4 338496 49748 ?        S    08:26   0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

Comment 9 Thom Carlin 2016-08-04 15:26:47 UTC

I see the same in RHOSP 8 (QCI 1.0)

Comment 10 Dmitry Tantsur 2016-11-10 10:55:20 UTC

Hi folks! There were numerous fixes in this field, few of them made it into OSP 7 and 8, but most are not backportable. Please open new bugs if you see it again in newer versions.

Note You need to log in before you can comment on or make changes to this bug.