Bug 1232997
| Summary: | ironic client fails due to locked instance while deploying the osp-7 overcloud | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | wes hayutin <whayutin> | ||||
| Component: | openstack-ironic | Assignee: | Lucas Alvares Gomes <lmartins> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Toure Dunnon <tdunnon> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.0 (Kilo) | CC: | bugzilla, dhill, dtantsur, dyocum, hbrock, jmatthew, jtaleric, srevivo, tcarlin, twilkins, whayutin | ||||
| Target Milestone: | z5 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 7.0 (Kilo) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-11-10 10:55:20 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1273561 | ||||||
| Attachments: |
|
||||||
Created attachment 1040221 [details]
logs from the undercloud
This bug blocks both IO performance testing as well as hyper-converged storage evaluation of OSP7 on RHCS. It breaks discovery and prevents overcloud deployment. Additionally, the link in Comment 1 is no longer available. Discovery fails with ... WARNING: ironicclient.common.http Error contacting Ironic server: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed. (HTTP 409). Attempt 15 of 61 In the ironic-api.log ... ... 2015-09-23 01:22:27.270 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists. 2015-09-23 01:22:29.304 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists. 2015-09-23 01:22:31.327 17366 WARNING wsme.api [-] Client-side error: A port with MAC address d4:be:d9:f5:10:4f already exists. ... 2015-09-23 01:23:28.406 17366 WARNING wsme.api [-] Client-side error: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed. 2015-09-23 01:23:30.435 17366 WARNING wsme.api [-] Client-side error: Node 5b077041-cb60-400a-a83a-6d8f7f1b49e8 is locked by host localhost.localdomain, please retry after the current operation is completed. ... # rpmq openstack openstack-tempest-kilo-20150708.2.el7ost.noarch openstack-nova-scheduler-2015.1.1-1.el7ost.noarch openstack-nova-cert-2015.1.1-1.el7ost.noarch redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch openstack-tripleo-image-elements-0.9.6-10.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.1-4.el7ost.noarch openstack-nova-common-2015.1.1-1.el7ost.noarch openstack-neutron-common-2015.1.1-5.el7ost.noarch openstack-neutron-ml2-2015.1.1-5.el7ost.noarch openstack-heat-engine-2015.1.1-4.el7ost.noarch python-openstackclient-1.0.3-2.el7ost.noarch openstack-selinux-0.6.37-1.el7ost.noarch openstack-swift-container-2.3.0-1.el7ost.noarch openstack-nova-api-2015.1.1-1.el7ost.noarch openstack-tuskar-ui-0.4.0-3.el7ost.noarch openstack-ceilometer-common-2015.1.1-1.el7ost.noarch openstack-nova-compute-2015.1.1-1.el7ost.noarch openstack-heat-templates-0-0.6.20150605git.el7ost.noarch python-django-openstack-auth-1.2.0-5.el7ost.noarch openstack-dashboard-theme-2015.1.1-2.el7ost.noarch openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch openstack-neutron-openvswitch-2015.1.1-5.el7ost.noarch openstack-heat-api-2015.1.1-4.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-64.el7ost.noarch openstack-ironic-conductor-2015.1.1-4.el7ost.noarch openstack-nova-conductor-2015.1.1-1.el7ost.noarch openstack-swift-account-2.3.0-1.el7ost.noarch openstack-swift-proxy-2.3.0-1.el7ost.noarch openstack-tuskar-0.4.18-4.el7ost.noarch openstack-nova-console-2015.1.1-1.el7ost.noarch openstack-ceilometer-notification-2015.1.1-1.el7ost.noarch openstack-ceilometer-collector-2015.1.1-1.el7ost.noarch openstack-dashboard-2015.1.1-2.el7ost.noarch openstack-utils-2014.2-1.el7ost.noarch openstack-heat-common-2015.1.1-4.el7ost.noarch openstack-heat-api-cfn-2015.1.1-4.el7ost.noarch openstack-puppet-modules-2015.1.8-17.el7ost.noarch openstack-ironic-common-2015.1.1-4.el7ost.noarch openstack-ceilometer-api-2015.1.1-1.el7ost.noarch openstack-ceilometer-alarm-2015.1.1-1.el7ost.noarch openstack-ironic-api-2015.1.1-4.el7ost.noarch openstack-keystone-2015.1.1-1.el7ost.noarch openstack-swift-plugin-swift3-1.7-3.el7ost.noarch openstack-tripleo-puppet-elements-0.0.1-5.el7ost.noarch openstack-ironic-discoverd-1.1.0-6.el7ost.noarch openstack-ceilometer-central-2015.1.1-1.el7ost.noarch openstack-swift-2.3.0-1.el7ost.noarch openstack-nova-novncproxy-2015.1.1-1.el7ost.noarch openstack-swift-object-2.3.0-1.el7ost.noarch openstack-tripleo-common-0.0.1.dev6-2.git49b57eb.el7ost.noarch openstack-glance-2015.1.1-1.el7ost.noarch openstack-neutron-2015.1.1-5.el7ost.noarch I'm getting the same error: 2016-04-04 07:10:10.877 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581 2016-04-04 07:10:10.878 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296 2016-04-04 07:10:10.878 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603 2016-04-04 07:10:10.879 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin servic e: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632 2016-04-04 07:10:10.893 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name ,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-Us er-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenan t _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672 2016-04-04 07:10:10.894 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581 2016-04-04 07:10:10.895 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296 2016-04-04 07:10:10.895 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603 2016-04-04 07:10:10.896 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin servic e: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632 2016-04-04 07:10:10.908 1214 WARNING wsme.api [-] Client-side error: Node d094ef1e-e6c3-45f5-a948-550e24ec673c is locked by host director1.os1.phx2.redhat.com, please retry after the current operation is completed. 2016-04-04 07:10:12.668 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672 2016-04-04 07:10:12.668 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581 2016-04-04 07:10:12.669 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296 2016-04-04 07:10:12.670 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603 2016-04-04 07:10:12.670 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin service: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632 2016-04-04 07:10:12.915 1214 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672 2016-04-04 07:10:12.916 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581 2016-04-04 07:10:12.917 1214 DEBUG keystonemiddleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_cache.py:296 2016-04-04 07:10:12.918 1214 DEBUG keystonemiddleware.auth_token [-] Authenticating service token __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:603 2016-04-04 07:10:12.918 1214 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 075d4a5f8797455ea452a40377c0a2da, project_id 115774183ce149c79bdf1ce07e6aae50, roles _member_,admin service: user_id None, project_id None, roles None __call__ /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:632 2016-04-04 07:10:12.928 1214 WARNING wsme.api [-] Client-side error: Node d094ef1e-e6c3-45f5-a948-550e24ec673c is locked by host director1.os1.phx2.redhat.com, please retry after the current operation is completed. Swift-proxy is running: [root@director1 ironic]# ps auxww | grep swift | grep proxy swift 9208 0.0 0.4 332560 47804 ? Ss 08:25 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 9369 0.0 0.4 338496 49812 ? S 08:26 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 9370 0.0 0.4 338496 49812 ? S 08:26 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 9372 0.0 0.4 338496 49744 ? S 08:26 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf swift 9373 0.0 0.4 338496 49748 ? S 08:26 0:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf I see the same in RHOSP 8 (QCI 1.0) Hi folks! There were numerous fixes in this field, few of them made it into OSP 7 and 8, but most are not backportable. Please open new bugs if you see it again in newer versions. |
Description of problem: 02:17:42.484 failed: [undercloud] => {"changed": true, "cmd": "source /home/stack/stackrc; if [ -f \"/home/stack/deploy-overcloudrc\" ]; then\n source /home/stack/deploy-overcloudrc;\n fi; openstack overcloud deploy --plan-uuid 37b893ec-c78b-49e1-93be-5888ae6ef5fd --control-scale $CONTROLSCALE --compute-scale $COMPUTESCALE --ceph-storage-scale $CEPHSTORAGESCALE #Both swift and blockstorage are not supported downstream right now #--swift-storage-scale $SWIFTSTORAGESCALE #--block-storage-scale $BLOCKSTORAGESCALE;", "delta": "0:00:24.380112", "end": "2015-06-17 21:52:43.699863", "rc": 1, "start": "2015-06-17 21:52:19.319751", "warnings": []} 02:17:42.489 stderr: WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 1 of 6 02:17:42.489 WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 2 of 6 02:17:42.489 WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 3 of 6 02:17:42.489 WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 4 of 6 02:17:42.489 WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 WARNING: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 5 of 6 02:17:42.489 WARNING: ironicclient.common.http Request returned failure status. 02:17:42.489 ERROR: ironicclient.common.http Error contacting Ironic server: Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409). Attempt 6 of 6 02:17:42.499 ERROR: openstack Node 5f0847b4-c66f-47d2-aaf1-505920627493 is locked by host virtblade11.virt.lab.eng.bos.redhat.com, please retry after the current operation is completed. (HTTP 409) 02:17:42.500 stdout: The following templates will be written: