Bug 1303476

Summary: Update from 7.1 to 7.2 tries to (re-)create the ceph storage nodes and fails.
Product: Red Hat OpenStack Reporter: David Gurtner <dgurtner>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED DUPLICATE QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: hbrock, mburns, rhel-osp-director-maint, srevivo
Target Milestone: ---   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-01 00:26:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Gurtner 2016-02-01 00:16:32 UTC
Description of problem:
When doing an upgrade from 7.1 to 7.2 the upgrade fails. The reason is that the upgrade tries to (re-)create the Ceph storage nodes and then fails because it can't find suitable hosts:

[stack@undercloud ~]$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                    | Status | Task State | Power State | Networks             |
+--------------------------------------+-------------------------+--------+------------+-------------+----------------------+
| 11c75cd8-7155-4dca-b891-93fb33f6aa00 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=172.16.0.74 |
| f80f4d74-f51b-4ec2-a3af-2a60f4c2f630 | overcloud-cephstorage-0 | BUILD  | spawning   | NOSTATE     |                      |
| 19d24d3e-a25f-466b-8be6-867e008e2379 | overcloud-cephstorage-1 | ACTIVE | -          | Running     | ctlplane=172.16.0.73 |
| 47cc5d4b-87a5-4b8a-84a8-c748b006d3c7 | overcloud-cephstorage-1 | BUILD  | spawning   | NOSTATE     | ctlplane=172.16.0.81 |

2016-02-01 00:51:47.390 13916 INFO heat.engine.resource [-] UPDATE: TemplateResource "0" [4ca59819-bb44-43f0-9e3a-17357e373078] Stack "overcloud-CephStorage-4g7hzdtlii4c"
 [14b81fbf-5c91-40b0-9151-d7bfbabbe76b]
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource Traceback (most recent call last):
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 528, in _action_recorder
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     yield
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 796, in update
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     args=[after, tmpl_diff, prop_diff])
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 313, in wrapper
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     step = next(subtask)
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 572, in action_handler_task
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     while not check(handler_data):
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 410, in check_update_comp
lete
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     cookie=cookie)
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 340, in _check_status_com
plete
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource     action=action)
2016-02-01 00:51:47.390 13916 TRACE heat.engine.resource ResourceFailure: resources[0]: ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No v
alid host was found. There are not enough hosts available., Code: 500"


Version-Release number of selected component (if applicable):
python-rdomanager-oscplugin-0.0.10-22.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy 7.1 overcloud
2. Update OSP-d to 7.2
3. (Try to) Update overcloud to 7.2


Actual results:
[stack@undercloud ~]$ openstack overcloud update stack overcloud -i --templates /home/stack/my_templates -e  /home/stack/my_templates/overcloud-resource-registry-puppet.yaml -e /home/stack/my_templates/environments/storage-environment.yaml -e /home/stack/my_templates/environments/updates/update-from-vip.yaml -e /home/stack/my_templates/param-updates.yaml
starting package update on stack overcloud
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED


Expected results:
[stack@undercloud ~]$ openstack overcloud update stack overcloud -i --templates /home/stack/my_templates -e  /home/stack/my_templates/overcloud-resource-registry-puppet.yaml -e /home/stack/my_templates/environments/storage-environment.yaml -e /home/stack/my_templates/environments/updates/update-from-vip.yaml -e /home/stack/my_templates/param-updates.yaml
starting package update on stack overcloud
IN_PROGRESS
[...]
COMPLETE  
update finished with status COMPLETE


Additional info:
This is on a virtual lab, KVM based VMs with pxe_ssh driver.

Comment 2 David Gurtner 2016-02-01 00:26:02 UTC

*** This bug has been marked as a duplicate of bug 1303475 ***