Bug 1236694

Summary: UPDATE_FAILED for ControlVirtualIP and Ceph-Storage
Product: Red Hat OpenStack Reporter: Ola Pavlenko <opavlenk>
Component: rhosp-directorAssignee: Lennart Regebro <lregebro>
Status: CLOSED ERRATA QA Contact: Ola Pavlenko <opavlenk>
Severity: urgent Docs Contact:
Priority: high    
Version: DirectorCC: akrivoka, calfonso, jslagle, mburns, ohochman, rhel-osp-director-maint, rrosa, sclewis
Target Milestone: gaKeywords: Triaged
Target Release: Director   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: python-rdomanager-oscplugin-0.0.8-16.el7ost.noarch Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 13:57:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ola Pavlenko 2015-06-29 18:21:16 UTC
Following https://bugzilla.redhat.com/show_bug.cgi?id=1234117 verification


Description of problem:
-------------------------------
Tried to update stack using UCLI. (tbh, there were nothing to update). After 10 minutes got UPDATE_FAILED for Ceph and Control VIP 

Version-Release number of selected component (if applicable):
--------------------------------------
rhos-release-0.62-1.noarch
python-rdomanager-oscplugin-0.0.8-13.el7ost.noarch
$ rpm -qa | grep heat
openstack-heat-engine-2015.1.0-4.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch
openstack-heat-api-2015.1.0-4.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.0-4.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-19.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch


How reproducible:
---------------------
100%

Steps to Reproduce:
1.Have overcloud successfully deployed with 1 controller, 1 compute, 1 ceph
2. update stack using:
export OVERCLOUD_PLAN_NAME=overcloud
openstack overcloud update stack overcloud

Actual results:
UPDATE_FAILED

Expected results:
-----------------------
UPDATE_SUCCESS

Additional info:
---------------------
[stack@instack ~]$ export OVERCLOUD_PLAN_NAME=overcloud
[stack@instack ~]$ openstack overcloud update stack overcloud 
stack overcloud status: WAITING
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| 541131cc-1559-4b43-b12d-b3ae33366bbc | overcloud  | UPDATE_IN_PROGRESS | 2015-06-29T17:20:58Z |
+--------------------------------------+------------+--------------------+----------------------+
-
-
-
-
-
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 541131cc-1559-4b43-b12d-b3ae33366bbc | overcloud  | UPDATE_FAILED | 2015-06-29T17:20:58Z |
+--------------------------------------+------------+---------------+----------------------+

[stack@instack ~]$ heat resource-list overcloud
+-----------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                     | physical_resource_id                          | resource_type                                     | resource_status | updated_time         |
+-----------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| BlockStorageAllNodesDeployment    | f9d94363-39e4-41fa-b416-762f6e677107          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| BlockStorageNodesPostDeployment   | e31b5749-6f75-4ac3-8fe7-7a53e36d8bb3          | OS::TripleO::BlockStoragePostDeployment           | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| CephClusterConfig                 | 2502bc59-c176-4c6e-8535-dcad7fc60db5          | OS::TripleO::CephClusterConfig::SoftwareConfig    | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| CephStorageAllNodesDeployment     | 5e33c964-69e4-45a9-ad3a-7c57fa84c8f6          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| CephStorageCephDeployment         | fb2b4cb1-09e9-4ac7-aae9-ecd1e197fe65          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| CephStorageNodesPostDeployment    | a975e90d-b25f-40b1-893b-cca5b7138420          | OS::TripleO::CephStoragePostDeployment            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| Cinder-Storage                    | 56dc957e-243b-4e15-b35c-bd281c08b692          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| Compute                           | 96b8eb18-6369-43d8-8ca6-d6b786624d9b          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ComputeAllNodesDeployment         | 2541836a-3793-4752-a4d4-54f2457dc8e8          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ComputeCephDeployment             | b5b8313a-cb8e-42d4-ba01-c4318fa84de6          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ComputeNodesPostDeployment        | 4575c68c-26db-439f-bc58-c2161a0426e6          | OS::TripleO::ComputePostDeployment                | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| Controller                        | 29e2bea7-6252-45b3-bab1-78062b39c131          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerAllNodesDeployment      | 7211d8b5-769e-492a-ae79-6bfc445b015a          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerBootstrapNodeConfig     | 9b1b7cd7-3e12-4cf8-9850-528755c91445          | OS::TripleO::BootstrapNode::SoftwareConfig        | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerBootstrapNodeDeployment | 734c38c8-469c-40f9-943c-56a8556c0b88          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerCephDeployment          | 21b1335d-a68e-4734-9918-092d86769f21          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerClusterConfig           | 4c2bef8e-6a8f-4cb1-ace6-b94444f5f0e5          | OS::Heat::StructuredConfig                        | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerClusterDeployment       | aa25fbdd-62a6-4d3c-8357-6708002960c1          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerIpListMap               | addbc50e-f78f-4a0d-9b53-6c44055ad946          | OS::TripleO::Network::Ports::NetIpListMap         | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerNodesPostDeployment     | c6984275-1595-429f-a16d-04ff42b3cee2          | OS::TripleO::ControllerPostDeployment             | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ControllerSwiftDeployment         | 8cd90c53-b2fa-43d3-a15f-8cc49a847222          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| HeatAuthEncryptionKey             | overcloud-HeatAuthEncryptionKey-k6rstqbu4m4m  | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| HorizonSecret                     | overcloud-HorizonSecret-ojxxmr4pheoi          | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| InternalApiVirtualIP              | 32236588-43da-4f35-8bed-697b9f5d745a          | OS::TripleO::Controller::Ports::InternalApiPort   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| MysqlClusterUniquePart            | overcloud-MysqlClusterUniquePart-4c3uiaann4x6 | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| MysqlRootPassword                 | overcloud-MysqlRootPassword-u2vas6hninmn      | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ObjectStorageAllNodesDeployment   | b33780be-d1f7-4b9b-ae53-65372af69afb          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ObjectStorageNodesPostDeployment  | 4eeb17e8-d5f3-4d8d-a5ff-1e5f7ebb722a          | OS::TripleO::ObjectStoragePostDeployment          | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| ObjectStorageSwiftDeployment      | 1ad898ce-6cc4-4558-968f-7dfa1d11c68f          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| PcsdPassword                      | overcloud-PcsdPassword-uddpaaozimkh           | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| PublicVirtualIP                   | 6d7e6de6-ae0d-4e54-817c-f0f881d00ac6          | OS::TripleO::Controller::Ports::ExternalPort      | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| RabbitCookie                      | overcloud-RabbitCookie-wp6srn4w3nws           | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| RedisVirtualIP                    | b93cfe58-cea4-4b37-9e0b-9fc042c254ed          | OS::TripleO::Controller::Ports::RedisVipPort      | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| StorageMgmtVirtualIP              | 1b08adb4-bc6d-4fae-89de-1e15eb182760          | OS::TripleO::Controller::Ports::StorageMgmtPort   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| StorageVirtualIP                  | d31e75b5-170a-4a41-b9ef-c827c92893b7          | OS::TripleO::Controller::Ports::StoragePort       | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| SwiftDevicesAndProxyConfig        | 9309becd-afc0-4989-9b98-5366fc82a94b          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| VipDeployment                     | 18ca82d2-4f8c-40cd-95f3-f82cc33c1a07          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| VipMap                            | 4398aa3a-96dc-4526-a863-d8bcc2832343          | OS::TripleO::Network::Ports::NetIpMap             | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| allNodesConfig                    | 53b6f0bd-156e-4dc1-96b4-99b43e74d523          | OS::TripleO::AllNodes::SoftwareConfig             | CREATE_COMPLETE | 2015-06-29T17:20:59Z |
| Ceph-Storage                      | cd7e7fa1-6cf8-4c46-923b-c9ac540b74a9          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2015-06-29T17:54:09Z |
| VipConfig                         | 6888431b-c500-4aa8-bb3c-8824230548ed          | OS::TripleO::VipConfig                            | UPDATE_COMPLETE | 2015-06-29T17:54:12Z |
| Swift-Storage                     | 5aec6fbc-1355-437a-a625-9f4f7f163a39          | OS::Heat::ResourceGroup                           | UPDATE_COMPLETE | 2015-06-29T17:54:13Z |
| Networks                          | 130222f7-8a81-45b6-9d91-f7792bb5442e          | OS::TripleO::Network                              | UPDATE_COMPLETE | 2015-06-29T17:54:15Z |
| ControlVirtualIP                  |                                               | OS::Neutron::Port                                 | CREATE_FAILED   | 2015-06-29T17:54:51Z |
+-----------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
[stack@instack ~]$ heat resource-show overcloud Ceph-Storage
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                              |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                  |
|                        |   "attributes": null,                                                                                                                              |
|                        |   "refs": null                                                                                                                                     |
|                        | }                                                                                                                                                  |
| description            |                                                                                                                                                    |
| links                  | http://192.0.2.1:8004/v1/94c97cfaa9304a1582cf18793b7135ea/stacks/overcloud/541131cc-1559-4b43-b12d-b3ae33366bbc/resources/Ceph-Storage (self)      |
|                        | http://192.0.2.1:8004/v1/94c97cfaa9304a1582cf18793b7135ea/stacks/overcloud/541131cc-1559-4b43-b12d-b3ae33366bbc (stack)                            |
|                        | http://192.0.2.1:8004/v1/94c97cfaa9304a1582cf18793b7135ea/stacks/overcloud-Ceph-Storage-xmx5et6b4ub6/cd7e7fa1-6cf8-4c46-923b-c9ac540b74a9 (nested) |
| logical_resource_id    | Ceph-Storage                                                                                                                                       |
| physical_resource_id   | cd7e7fa1-6cf8-4c46-923b-c9ac540b74a9                                                                                                               |
| required_by            | CephStorageAllNodesDeployment                                                                                                                      |
|                        | allNodesConfig                                                                                                                                     |
|                        | CephStorageNodesPostDeployment                                                                                                                     |
|                        | CephStorageCephDeployment                                                                                                                          |
| resource_name          | Ceph-Storage                                                                                                                                       |
| resource_status        | UPDATE_FAILED                                                                                                                                      |
| resource_status_reason | UPDATE aborted                                                                                                                                     |
| resource_type          | OS::Heat::ResourceGroup                                                                                                                            |
| updated_time           | 2015-06-29T17:54:09Z                                                                                                                               |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@instack ~]$ heat resource-show overcloud ControlVirtualIP
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                             |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                 |
|                        |   "status": null,                                                                                                                                 |
|                        |   "subnets": null,                                                                                                                                |
|                        |   "name": null,                                                                                                                                   |
|                        |   "allowed_address_pairs": null,                                                                                                                  |
|                        |   "show": null,                                                                                                                                   |
|                        |   "network_id": null,                                                                                                                             |
|                        |   "tenant_id": null,                                                                                                                              |
|                        |   "admin_state_up": null,                                                                                                                         |
|                        |   "device_owner": null,                                                                                                                           |
|                        |   "mac_address": null,                                                                                                                            |
|                        |   "fixed_ips": null,                                                                                                                              |
|                        |   "security_groups": null,                                                                                                                        |
|                        |   "device_id": null                                                                                                                               |
|                        | }                                                                                                                                                 |
| description            |                                                                                                                                                   |
| links                  | http://192.0.2.1:8004/v1/94c97cfaa9304a1582cf18793b7135ea/stacks/overcloud/541131cc-1559-4b43-b12d-b3ae33366bbc/resources/ControlVirtualIP (self) |
|                        | http://192.0.2.1:8004/v1/94c97cfaa9304a1582cf18793b7135ea/stacks/overcloud/541131cc-1559-4b43-b12d-b3ae33366bbc (stack)                           |
| logical_resource_id    | ControlVirtualIP                                                                                                                                  |
| physical_resource_id   |                                                                                                                                                   |
| required_by            | StorageMgmtVirtualIP                                                                                                                              |
|                        | PublicVirtualIP                                                                                                                                   |
|                        | InternalApiVirtualIP                                                                                                                              |
|                        | RedisVirtualIP                                                                                                                                    |
|                        | StorageVirtualIP                                                                                                                                  |
|                        | Cinder-Storage                                                                                                                                    |
|                        | VipDeployment                                                                                                                                     |
|                        | Controller                                                                                                                                        |
| resource_name          | ControlVirtualIP                                                                                                                                  |
| resource_status        | CREATE_FAILED                                                                                                                                     |
| resource_status_reason | BadRequest: Invalid input for network_id. Reason: '' is not a valid UUID.                                                                         |
| resource_type          | OS::Neutron::Port                                                                                                                                 |
| updated_time           | 2015-06-29T17:54:51Z                                                                                                                              |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@instack ~]$ echo $OVERCLOUD_PLAN_NAME
overcloud
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 541131cc-1559-4b43-b12d-b3ae33366bbc | overcloud  | UPDATE_FAILED | 2015-06-29T17:20:58Z |
+--------------------------------------+------------+---------------+----------------------+

[stack@instack ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
| a50beb09-2217-4b45-85b0-1e0ca319ef41 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| e40924f2-6430-43d2-b4c2-0ecbb56bf24e | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
| f0c03a2a-be75-4b6e-a1f0-51cb745ad15a | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.18 |
+--------------------------------------+-------------------------+--------+------------+-------------+---------------------+

Comment 3 James Slagle 2015-07-01 18:00:15 UTC
could be related to bz#1235334

Comment 4 Lennart Regebro 2015-07-02 11:02:12 UTC
Yeah, if this could be tested again, with python-rdomanager-oscplugin-0.0.8-16.el7ost.noarch or later, that would be great. The network_id should be picked up from the NeutronControlPlaneID parameter, which is being saved now.

https://review.gerrithub.io/#/c/237713/

I can successfully update the overcloud after that change.

Comment 6 Ola Pavlenko 2015-07-22 22:15:22 UTC
Verified 
python-rdomanager-oscplugin-0.0.8-41.el7ost.noarch

Comment 8 errata-xmlrpc 2015-08-05 13:57:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549