Bug 1322659

Summary:	rhel-osp-director: Error: Could not find data item ceph::profile::params::fsid in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/ff7ba273-1fcc-4357-91ba-15ee76e29e8c.pp:902 on node overcloud-controller-0.localdomai
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	rhosp-director	Assignee:	Angus Thomas <athomas>
Status:	CLOSED NOTABUG	QA Contact:	Arik Chernetsky <achernet>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0 (Liberty)	CC:	dbecker, jstransk, mburns, mcornea, morazi, rhel-osp-director-maint
Target Milestone:	---
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-31 13:57:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Alexander Chuzhoy 2016-03-31 02:50:23 UTC

rhel-osp-director:  Error: Could not find data item ceph::profile::params::fsid in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/ff7ba273-1fcc-4357-91ba-15ee76e29e8c.pp:902 on node overcloud-controller-0.localdomai


Upgrade step with /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml fails with the above error.

Environment:
openstack-tripleo-heat-templates-0.8.14-2.el7ost.noarch
openstack-puppet-modules-7.0.17-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-2.el7ost.noarch
instack-undercloud-2.2.7-1.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud 7.3 (HA+ 1 compute + 1 swift node
2. Go trough the upgrade process, run "openstack overcloud deploy  --templates /usr/share/openstack-tripleo-heat-templates -e   /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml  -e  /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml  -e  /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml  -e  /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-converge.yaml"


Result:
2016-03-30 23:34:11 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:34:12 [NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:08 [1]: SIGNAL_IN_PROGRESS  Signal: deployment failed (1)
2016-03-30 23:35:08 [1]: CREATE_FAILED  Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2016-03-30 23:35:11 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:12 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:13 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:14 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:16 [ControllerDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:17 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:17 [1]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:19 [NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:20 [0]: SIGNAL_IN_PROGRESS  Signal: deployment failed (1)
2016-03-30 23:35:22 [0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2016-03-30 23:35:22 [2]: SIGNAL_IN_PROGRESS  Signal: deployment failed (1)
2016-03-30 23:35:23 [2]: CREATE_FAILED  Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2016-03-30 23:35:23 [0]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:24 [overcloud-ControllerNodesPostDeployment-kxk4ufgia3lr-ControllerOvercloudServicesDeployment_Step4-lj64kfmaltga]: UPDATE_FAILED  Error: resources[1]: Deployment to server failed: deploy_status_c
ode : Deployment exited with non-zero status code: 1
2016-03-30 23:35:25 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:25 [0]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:27 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:28 [0]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:29 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:30 [0]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:30 [ControllerDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:31 [ControllerDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:31 [0]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:32 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:33 [NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:33 [2]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:35:34 [NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-30 23:39:29 [ComputeNodesPostDeployment]: CREATE_FAILED  CREATE aborted
2016-03-30 23:39:29 [overcloud]: UPDATE_FAILED  resources.ControllerOvercloudServicesDeployment_Step4: resources.ControllerNodesPostDeployment.Error: resources[1]: Deployment to server failed: deploy_status_code:
Deployment exited with non-zero status code: 1
Stack overcloud UPDATE_FAILED
Heat Stack update failed.



running heat deployment-show reveals this:
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Keystone]): Keystone under Eventlet has been deprecated during the Kilo cycle. Support for deploying under eventlet will be dropped as of the M-release of OpenStack.
Warning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
Error: Could not find data item ceph::profile::params::fsid in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/ff7ba273-1fcc-4357-91ba-15ee76e29e8c.pp:902 on node overcloud-controller-0.localdomain
Error: Could not find data item ceph::profile::params::fsid in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/ff7ba273-1fcc-4357-91ba-15ee76e29e8c.pp:902 on node overcloud-controller-0.localdomain

Comment 2 Alexander Chuzhoy 2016-03-31 02:51:07 UTC

The issue seems to be intermittent.

Comment 4 Jiri Stransky 2016-03-31 13:57:06 UTC

The upgrading command included this environment file:

-e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml

... which by default enables Ceph backends for Cinder and glance. But from nova list it doesn't look like the overcloud has any ceph nodes, so the 7.3 overcloud probably wasn't deployed with Ceph.

[stack@instack ~]$ nova list
+--------------------------------------+---------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                      | Status | Task State | Power State | Networks            |
+--------------------------------------+---------------------------+--------+------------+-------------+---------------------+
| ca19d6d7-0c54-4e89-83f8-88547775b02d | overcloud-controller-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| bf2e8356-3a4c-46bb-a785-06dcd302d399 | overcloud-controller-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
| b03a6efe-4a86-4044-92ac-e67709e7222f | overcloud-controller-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| bc610f3f-9e20-4217-b800-dc59550266e4 | overcloud-novacompute-0   | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 9d57b18c-9c23-4c05-a9b1-286c952a8307 | overcloud-objectstorage-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.7  |
+--------------------------------------+---------------------------+--------+------------+-------------+---------------------+

I suspect the set of environment files passed to the upgrade command might be a bit different than what was passed into the 7.3 deployment command. When deploying/upgrading without Ceph nodes present, storage-environment.yaml shouldn't be used, at least not in its unmodified state.