1285031 – Failed to create overcloud from scenario 2

Bug 1285031 - Failed to create overcloud from scenario 2

Summary: Failed to create overcloud from scenario 2

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	rhosp-director
Sub Component:
Version:	7.0 (Kilo)
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	10.0 (Newton)
Assignee:	Hugh Brock
QA Contact:	Shai Revivo
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-11-24 17:32 UTC by Julien Bonjean
Modified:	2016-10-11 03:11 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-10-11 03:11:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Julien Bonjean 2015-11-24 17:32:22 UTC

Description of problem:
I'm trying to setup a basic cloud (scenario 2) using the method described in the documentation https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Scenario_2_Using_the_CLI_to_Create_a_Basic_Overcloud.html.
Unfortunately, the creation hangs and eventually fails (timeout). For information, the machines are VMs (KVM).

# heat stack-list --show-nested
+--------------------------------------+-------------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+
| id                                   | stack_name                                                                    | stack_status    | creation_time        | parent                               |
+--------------------------------------+-------------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+
| 75bef810-36de-42b8-bc18-4eb5da79aa88 | overcloud                                                                     | CREATE_FAILED   | 2015-11-24T00:43:23Z | None                                 |
| 4575a10c-39d7-48d4-bb1e-c18babb36af4 | overcloud-VipConfig-tuvth2hxakcs                                              | CREATE_COMPLETE | 2015-11-24T00:43:24Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 291f67da-7df2-4c3b-875b-c6e1de1937aa | overcloud-Networks-vsic2ouasirc-TenantNetwork-a5e4jfrzu45u                    | CREATE_COMPLETE | 2015-11-24T00:43:25Z | d7a39444-38cc-4f74-a2ad-77555461c7f0 |
| 32cbad96-44f8-4ecf-8c9e-ad5d7df694a1 | overcloud-Networks-vsic2ouasirc-StorageMgmtNetwork-vscuis4oh2op               | CREATE_COMPLETE | 2015-11-24T00:43:25Z | d7a39444-38cc-4f74-a2ad-77555461c7f0 |
| d7a39444-38cc-4f74-a2ad-77555461c7f0 | overcloud-Networks-vsic2ouasirc                                               | CREATE_COMPLETE | 2015-11-24T00:43:25Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| d869259f-7358-4624-ba5b-7a048c670c50 | overcloud-Networks-vsic2ouasirc-InternalNetwork-i6kchwy2hfrb                  | CREATE_COMPLETE | 2015-11-24T00:43:25Z | d7a39444-38cc-4f74-a2ad-77555461c7f0 |
| 06c2f4ca-024a-4f81-9fb7-b299c90d9cd7 | overcloud-Networks-vsic2ouasirc-ExternalNetwork-4cfyq3if4ttm                  | CREATE_COMPLETE | 2015-11-24T00:43:26Z | d7a39444-38cc-4f74-a2ad-77555461c7f0 |
| 330e123c-c555-4fd3-8518-8d6473b62d11 | overcloud-Networks-vsic2ouasirc-StorageNetwork-lmhg3nzm24ew                   | CREATE_COMPLETE | 2015-11-24T00:43:26Z | d7a39444-38cc-4f74-a2ad-77555461c7f0 |
| 5ce2d7ea-29ad-4cc5-b2d5-25ae1528df67 | overcloud-ControlVirtualIP-kvmyjdx6gfrn                                       | CREATE_COMPLETE | 2015-11-24T00:43:29Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 8883307b-699c-4ba8-9c08-1fd25422e2a9 | overcloud-CephStorage-nyeela524wxu                                            | CREATE_COMPLETE | 2015-11-24T00:43:30Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 8c473f05-4135-4732-bebc-e941aa2dc1cd | overcloud-ObjectStorage-2orio4ump7se                                          | CREATE_COMPLETE | 2015-11-24T00:43:30Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 5e066789-79ae-459d-83bd-8c05b42def92 | overcloud-StorageVirtualIP-uj2knssjhphc                                       | CREATE_COMPLETE | 2015-11-24T00:43:31Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 68d3cdf1-1bbd-408b-b41b-a1d33b7ff90a | overcloud-RedisVirtualIP-jc4dti5gueag                                         | CREATE_COMPLETE | 2015-11-24T00:43:32Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| be1d4cd5-48c2-4779-9aca-d6da5649f140 | overcloud-PublicVirtualIP-cwoqtdvxie5h                                        | CREATE_COMPLETE | 2015-11-24T00:43:32Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| e61951b2-93d0-4a4b-a42c-a3c388da5e82 | overcloud-InternalApiVirtualIP-4vmpyu5grlfz                                   | CREATE_COMPLETE | 2015-11-24T00:43:32Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| f5bfa829-64f8-4e12-b044-ec4fe3c27be5 | overcloud-StorageMgmtVirtualIP-krsfahrgupoh                                   | CREATE_COMPLETE | 2015-11-24T00:43:33Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| d8241fc9-10b4-4866-850b-543af206d647 | overcloud-VipMap-rxuv3fj5wk5p                                                 | CREATE_COMPLETE | 2015-11-24T00:43:34Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 085070ba-8444-434e-9b11-6378200afcc9 | overcloud-BlockStorage-qz4xrffekvlf                                           | CREATE_COMPLETE | 2015-11-24T00:43:36Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 3efd39c3-5106-4228-adbe-892167898c6f | overcloud-Controller-bn4npbmdnhku                                             | CREATE_FAILED   | 2015-11-24T00:43:36Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 5bc24651-511a-4b88-8ca2-38fdef1ccd83 | overcloud-Compute-yfxtfjbtyn6h                                                | CREATE_FAILED   | 2015-11-24T00:43:37Z | 75bef810-36de-42b8-bc18-4eb5da79aa88 |
| 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6                              | CREATE_FAILED   | 2015-11-24T00:43:38Z | 3efd39c3-5106-4228-adbe-892167898c6f |
| 223fc513-e740-4515-b3c2-161aa5222f80 | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-UpdateConfig-jyhbb2ock3ss       | CREATE_COMPLETE | 2015-11-24T00:43:39Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| 353e66c5-a30d-4d8c-aa95-564e330c9e94 | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h                                 | CREATE_FAILED   | 2015-11-24T00:43:39Z | 5bc24651-511a-4b88-8ca2-38fdef1ccd83 |
| 886c6e7f-1322-4ca0-94f8-1bf1e4068f38 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-NodeUserData-fkxa7fkxqjfc    | CREATE_COMPLETE | 2015-11-24T00:43:39Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| d483ee2d-62c4-4046-9b49-4b683807405b | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-UpdateConfig-bmwmhrchgzbd    | CREATE_COMPLETE | 2015-11-24T00:43:39Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| c68fb3d4-11a1-49af-81d9-2c19cbec29dd | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-NodeUserData-hdki5bodp3l3       | CREATE_COMPLETE | 2015-11-24T00:43:40Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| 37e677ba-46be-40f8-b184-6182c6907a65 | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-StoragePort-636mwr3en5le        | CREATE_COMPLETE | 2015-11-24T00:45:50Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| 370e1c13-8e26-41dd-9ec2-f69f5ca8cdad | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-TenantPort-f2me2dz6py7c         | CREATE_COMPLETE | 2015-11-24T00:45:51Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| 0c58d9fd-0014-43ca-a1ab-bb223500af2c | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-InternalApiPort-tu7frpsrd27c    | CREATE_COMPLETE | 2015-11-24T00:45:52Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| da1b93a3-f9fc-4f31-8c56-5aac8f64a051 | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-NetIpMap-dm26bqsc4ty6           | CREATE_COMPLETE | 2015-11-24T00:45:54Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| cc0ea323-5d38-498d-bf9f-3d739e51b94f | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-StorageMgmtPort-7ftynfvqpgf2 | CREATE_COMPLETE | 2015-11-24T00:45:55Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| ef2907a7-96bb-4445-b38d-0a785a0afd3c | overcloud-Compute-yfxtfjbtyn6h-0-f7lfkrvgwj3h-NetworkConfig-y7zxyfqpxf4o      | CREATE_COMPLETE | 2015-11-24T00:45:55Z | 353e66c5-a30d-4d8c-aa95-564e330c9e94 |
| cc3bc18f-3b38-45ba-a88e-32955bbd3449 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-InternalApiPort-25llffph4dev | CREATE_COMPLETE | 2015-11-24T00:45:56Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| 5de6a189-0f8e-462d-9489-89652947998f | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-ExternalPort-2jbi6evddy4j    | CREATE_COMPLETE | 2015-11-24T00:45:57Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| f721c416-c474-4473-a489-8e303bdc1093 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-StoragePort-yq4rij5gn4y4     | CREATE_COMPLETE | 2015-11-24T00:45:57Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| 8560fd65-3160-4af5-a315-aed33f7a63d8 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-TenantPort-r3yt4ljpecjy      | CREATE_COMPLETE | 2015-11-24T00:45:59Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| ba817fa8-8af6-4c3b-918c-21bfaa1f0361 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-NetIpSubnetMap-fihedhdbpmgh  | CREATE_COMPLETE | 2015-11-24T00:46:04Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| 798df225-b64d-445d-b544-0e8bf30edc63 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-NetworkConfig-gd2l3kbanhyu   | CREATE_COMPLETE | 2015-11-24T00:46:05Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
| cde786a0-1036-4487-89dc-8d258327f0a5 | overcloud-Controller-bn4npbmdnhku-0-ofuzrgdxb2r6-NetIpMap-pjauvgxunjwf        | CREATE_COMPLETE | 2015-11-24T00:46:05Z | 8afbd100-09d8-4aa7-80f3-07bec4ef29a1 |
+--------------------------------------+-------------------------------------------------------------------------------+-----------------+----------------------+--------------------------------------+

Version-Release number of selected component (if applicable):
RHEL 7.2
RHOSP 7

How reproducible:
100%

Steps to Reproduce:
1. Create an overcloud following the documentation (scenario 2).

Actual results:
The overcloud creation fails with:
Stack failed with status: Create timed out
ERROR: openstack Heat Stack create failed.

Expected results:
Success.

Additional info:

The following error messages seem relevant:

/var/log/heat/heat-engine.log
2015-11-24 09:58:08.978 15892 WARNING heat.engine.clients.os.nova [req-2d6ef87b-d593-44c8-a840-aacf0f55962d - 17fc113025814bc7ac366cf5aea978ca-75bef810-36de-42b8-bc18-4eb5da7] Instance (9e790fc8-7bff-4750-b586-622016ef3cae) not found: Instance 9e790fc8-7bff-4750-b586-622016ef3cae could not be found. (HTTP 404) (Request-ID: req-8d8fc01c-9ae8-4c7c-b151-ffd1f9782ba0)
2015-11-24 09:58:26.009 15894 WARNING heat.engine.clients.os.nova [req-66700032-1406-45e1-ab20-d44fd6c11637 - 17fc113025814bc7ac366cf5aea978ca-75bef810-36de-42b8-bc18-4eb5da7] Instance (62789dd6-21f3-46f5-9709-8704fd6cf459) not found: Instance 62789dd6-21f3-46f5-9709-8704fd6cf459 could not be found. (HTTP 404) (Request-ID: req-b3238738-0ddc-485c-a8c5-be6082879660)

But the instances exist (in admin project):

# openstack server list
+--------------------------------------+------------------------+--------+--------------------+
| ID                                   | Name                   | Status | Networks           |
+--------------------------------------+------------------------+--------+--------------------+
| 9e790fc8-7bff-4750-b586-622016ef3cae | overcloud-compute-0    | ACTIVE | ctlplane=192.0.2.8 |
| 62789dd6-21f3-46f5-9709-8704fd6cf459 | overcloud-controller-0 | ACTIVE | ctlplane=192.0.2.7 |
+--------------------------------------+------------------------+--------+--------------------+

If I dig a little bit, the matching messages in nova-api logs are:

/var/log/nova/nova-api.log
---
2015-11-24 10:00:46.986 29448 DEBUG nova.compute.api [req-2282045e-2730-45c1-a23e-d7961eaeb768 7d1cffbae3fd4d048d74c66a0f47528f c9cc51c170264156aad549373d32a2e9 - - -] [instance: 9e790fc8-7bff-4750-b586-622016ef3cae] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:1911
2015-11-24 10:00:47.021 29448 INFO nova.api.openstack.wsgi [req-2282045e-2730-45c1-a23e-d7961eaeb768 7d1cffbae3fd4d048d74c66a0f47528f c9cc51c170264156aad549373d32a2e9 - - -] HTTP exception thrown: Instance 9e790fc8-7bff-4750-b586-622016ef3cae could not be found.
2015-11-24 10:00:47.021 29448 DEBUG nova.api.openstack.wsgi [req-2282045e-2730-45c1-a23e-d7961eaeb768 7d1cffbae3fd4d048d74c66a0f47528f c9cc51c170264156aad549373d32a2e9 - - -] Returning 404 to user: Instance 9e790fc8-7bff-4750-b586-622016ef3cae could not be found. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1166
2015-11-24 10:00:47.022 29448 INFO nova.osapi_compute.wsgi.server [req-2282045e-2730-45c1-a23e-d7961eaeb768 7d1cffbae3fd4d048d74c66a0f47528f c9cc51c170264156aad549373d32a2e9 - - -] 192.0.2.1 "GET /v2/c9cc51c170264156aad549373d32a2e9/servers/9e790fc8-7bff-4750-b586-622016ef3cae HTTP/1.1" status: 404 len: 330 time: 0.0373631
---
2015-11-24 10:01:03.791 29448 DEBUG nova.compute.api [req-4b254daa-6221-47f0-9cb8-1c5d02350466 64c3abdd81d24e6cb5a89348b90bc808 c9cc51c170264156aad549373d32a2e9 - - -] [instance: 62789dd6-21f3-46f5-9709-8704fd6cf459] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:1911
2015-11-24 10:01:03.825 29448 INFO nova.api.openstack.wsgi [req-4b254daa-6221-47f0-9cb8-1c5d02350466 64c3abdd81d24e6cb5a89348b90bc808 c9cc51c170264156aad549373d32a2e9 - - -] HTTP exception thrown: Instance 62789dd6-21f3-46f5-9709-8704fd6cf459 could not be found.
2015-11-24 10:01:03.826 29448 DEBUG nova.api.openstack.wsgi [req-4b254daa-6221-47f0-9cb8-1c5d02350466 64c3abdd81d24e6cb5a89348b90bc808 c9cc51c170264156aad549373d32a2e9 - - -] Returning 404 to user: Instance 62789dd6-21f3-46f5-9709-8704fd6cf459 could not be found. __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1166
2015-11-24 10:01:03.826 29448 INFO nova.osapi_compute.wsgi.server [req-4b254daa-6221-47f0-9cb8-1c5d02350466 64c3abdd81d24e6cb5a89348b90bc808 c9cc51c170264156aad549373d32a2e9 - - -] 192.0.2.1 "GET /v2/c9cc51c170264156aad549373d32a2e9/servers/62789dd6-21f3-46f5-9709-8704fd6cf459 HTTP/1.1" status: 404 len: 330 time: 0.0556030
---

So it's trying to retrieve the instance in project id c9cc51c170264156aad549373d32a2e9. I tried to find the corresponding project:

# openstack project list
+----------------------------------+----------+
| ID                               | Name     |
+----------------------------------+----------+
| 17fc113025814bc7ac366cf5aea978ca | admin    |
| 3f29da1ba1264558b5a7b886ef677bc9 | service  |
| bfd22542423546e9893ddcf6606c9bf1 | services |
+----------------------------------+----------+

And it's not here, meaning it must be the heat project, which I can confirm by directly querying the database:

# mysql -u root keystone -e 'select id,name,description from project'
+----------------------------------+------------------------------------------------------------------+-----------------------------------+
| id                               | name                                                             | description                       |
+----------------------------------+------------------------------------------------------------------+-----------------------------------+
| 17fc113025814bc7ac366cf5aea978ca | admin                                                            | admin tenant                      |
| 3f29da1ba1264558b5a7b886ef677bc9 | service                                                          | NULL                              |
| bfd22542423546e9893ddcf6606c9bf1 | services                                                         | Tenant for the openstack services |
| c9cc51c170264156aad549373d32a2e9 | 17fc113025814bc7ac366cf5aea978ca-75bef810-36de-42b8-bc18-4eb5da7 | Heat stack user project           |
+----------------------------------+------------------------------------------------------------------+-----------------------------------+

Comment 4 Mike Burns 2016-04-07 20:57:01 UTC

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 Jaromir Coufal 2016-10-11 03:11:30 UTC

This seems as an obsolete issue, can you please reconfirm if you are still experiencing the same issue and if yes re-open the bug? Thank you

Note You need to log in before you can comment on or make changes to this bug.