Description of problem: Overcloud deployment fail during step 5 Overcloud deployment command: cd ; openstack overcloud deploy \ --debug \ --log-file ~/pilot/overcloud_deployment.log \ -t 1800 \ --stack overcloud \ --templates ~/pilot/templates/overcloud \ -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \ -e ~/pilot/templates/network-environment.yaml \ -e ~/pilot/templates/node-placement.yaml \ -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \ -e ~/pilot/templates/overcloud/environments/ceph-radosgw.yaml \ -e ~/pilot/templates/dell-environment.yaml \ -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml \ -e ~/pilot/templates/dell-cinder-backends.yaml \ --control-flavor baremetal \ --compute-flavor baremetal \ --ceph-storage-flavor baremetal \ --swift-storage-flavor baremetal \ --block-storage-flavor baremetal \ --neutron-public-interface bond1 \ --neutron-network-type vlan \ --neutron-disable-tunneling \ --libvirt-type kvm \ --os-auth-url http://192.168.120.202:5000/v2.0 \ --os-project-name admin \ --os-user-id admin \ --os-password 0147b920dc189f878a07a2289cefcfc12f45c35e \ --control-scale 3 \ --compute-scale 3 \ --ceph-storage-scale 3 \ --ntp-server 0.centos.pool.ntp.org \ --neutron-network-vlan-ranges physint:201:220,physext \ --neutron-bridge-mappings physint:br-tenant,physext:br-ex 2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.BlockStorageDeployment_Step5]: CREATE_COMPLETE state changed 2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step5]: CREATE_COMPLETE state changed 2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: CREATE_IN_PROGRESS state changed 2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: CREATE_IN_PROGRESS state changed 2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: CREATE_IN_PROGRESS state changed 2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: CREATE_IN_PROGRESS state changed 2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: CREATE_IN_PROGRESS state changed 2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: CREATE_IN_PROGRESS state changed 2017-01-19 19:33:26Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: SIGNAL_IN_PROGRESS Signal: deployment 8f55d6fa-414a-49a7-bb3e-5a065d53cc86 succeeded 2017-01-19 19:33:26Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: CREATE_COMPLETE state changed 2017-01-19 19:33:46Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.1]: SIGNAL_IN_PROGRESS Signal: deployment 405a9941-1902-4dab-a3b5-7dd686a3a77e succeeded 2017-01-19 19:33:46Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: SIGNAL_IN_PROGRESS Signal: deployment bd9daf26-2f0e-45ec-9a83-679af6c47022 succeeded 2017-01-19 19:33:47Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.1]: CREATE_COMPLETE state changed 2017-01-19 19:33:47Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: CREATE_COMPLETE state changed 2017-01-19 19:33:48Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: SIGNAL_IN_PROGRESS Signal: deployment 1f13b98c-47bc-4c78-b516-259fc88e17a1 failed (1) 2017-01-19 19:33:49Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.1]: SIGNAL_IN_PROGRESS Signal: deployment 0c1698d2-9a5e-48a9-b384-25eed8cde05d succeeded 2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.1]: CREATE_COMPLETE state changed 2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: SIGNAL_IN_PROGRESS Signal: deployment d9bb5602-ec52-479a-9fd2-47aaf062b613 succeeded 2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: CREATE_COMPLETE state changed 2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully 2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: SIGNAL_IN_PROGRESS Signal: deployment b1b7df18-81cc-472c-b14b-62a600b32c89 succeeded 2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE state changed 2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: CREATE_COMPLETE state changed 2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully 2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE state changed 2017-01-19 19:35:32Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.1]: SIGNAL_IN_PROGRESS Signal: deployment 1c9009c7-64d8-4913-976a-754c3a336c5e succeeded 2017-01-19 19:35:32Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.1]: CREATE_COMPLETE state changed 2017-01-19 19:37:18Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: SIGNAL_IN_PROGRESS Signal: deployment daf9ae1a-bd4d-4cf5-b7d7-72dc79aaac46 succeeded 2017-01-19 19:37:19Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: CREATE_COMPLETE state changed 2017-01-19 19:37:19Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2017-01-19 19:37:20Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-01-19 19:37:20Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-01-19 19:37:21Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-01-19 19:37:21Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 Stack overcloud CREATE_FAILED [2017-01-19 19:55:38,423] (heat-config) [INFO] exception: connect failed ^[[1;31mWarning: Scope(Class[Cinder::Api]): keystone_enabled is deprecated, use auth_strategy instead.^[[0m ^[[1;31mWarning: Scope(Class[Keystone]): Fernet token is recommended in Mitaka release. The default for token_provider will be changed to 'fernet' in O release.^[[0m ^[[1;31mWarning: Scope(Class[Glance::Api]): default_store not provided, it will be automatically set to glance.store.http.Store^[[0m ^[[1;31mWarning: Scope(Class[Heat]): keystone_user_domain_id is deprecated, use the name option instead.^[[0m ^[[1;31mWarning: Scope(Class[Heat]): keystone_project_domain_id is deprecated, use the name option instead.^[[0m ^[[1;31mWarning: Scope(Class[Neutron::Agents::L3]): parameter external_network_bridge is deprecated^[[0m ^[[1;31mWarning: Scope(Class[Neutron::Server::Notifications]): nova_url is deprecated and will be removed after Newton cycle.^[[0m ^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_user'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_password'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_tenant_name'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::auth_uri'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::auth_version'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::identity_uri'; class ::nova::api has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated^[[0m ^[[1;31mWarning: Scope(Class[Ceilometer]): Both $metering_secret and $telemetry_secret defined, using $telemetry_secret^[[0m ^[[1;31mWarning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 166 in file /etc/puppet/modules/gnocchi/manifests/api.pp^[[0m ^[[1;31mWarning: Scope(Class[Gnocchi::Api]): gnocchi:api::keystone_identity_uri is deprecated, use gnocchi::keystone::authtoken::auth_url instead^[[0m ^[[1;31mWarning: Scope(Class[Gnocchi::Api]): gnocchi::api::keystone_auth_uri is deprecated, use gnocchi::keystone::authtoken::auth_uri instead^[[0m ^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m ^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m ^[[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.^[[0m ^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m ^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m ^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m ^[[1;31mError: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org^[[0m ^[[1;31mError: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org^[[0m [2017-01-19 19:55:38,423] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/af951947-d8e4-4635-b6aa-0e5914b6c890.pp. [1] Version-Release number of selected component (if applicable): Puddle: 2017-01-17.1
Created attachment 1242602 [details] environment.json
Created attachment 1242603 [details] heat-debug.log.gz
I just redeployed the platform with a configuration that was working with the previous OSP10 puddle (ceph and eqx backends for Cinder are enabled). I face the same issue.
I tried to call the puppet script manually with all the content commented and then I reenable its content line be line. The error shows up when I uncomment: include ::tripleo::profile::base::keystone (...) Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[aodh] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[ceilometer] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[cinder] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[glance] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[gnocchi] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[heat] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[heat-cfn] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[neutron] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[nova] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user_role[admin@admin] with 'before' Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user_role[heat_stack_domain_admin::heat_stack@::heat_stack] with 'before' Error: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org Error: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org
I can avoid the problem if I comment these lines https://github.com/openstack/puppet-ceph/blob/3bb76e7364fd0e00418226e98535aa1463f0dfc0/manifests/rgw/keystone/auth.pp#L85-L89 in /usr/share/openstack-puppet/modules/ceph/manifests/rgw/keystone/auth.pp
(In reply to Gonéri Le Bouder from comment #3) > I just redeployed the platform with a configuration that was working with > the previous OSP10 puddle (ceph and eqx backends for Cinder are enabled). I > face the same issue. hi, are you suggesting that it is a particular combination of cinder/dell/rgw not working? does the issue reproduce with a standard overcloud with rgw?
At this point, I'm a bit confused. I managed to deploy the platform with the previous puddle and the very same configuration (ceph, dellsc, equallogic). I've no conclusion yet.
I believe it's a regression introduce here: http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/10.0-RHEL-7/2017-01-15.1/logs/changelog.log puppet-ceph: puppet-ceph-2.2.1-3.el7ost -> puppet-ceph-2.2.1-4.el7ost --------------------------------------------------------------------- Wed Jan 04 2017 Jon Schlueter <jschluet> 2.2.1-4 - Include and not redeclare apache mod_env (lp#1651987) - Update ceph::key to update keys - Unify parameter arguments between classes. <----------------------- - Use facts from puppet-openstack_spec_helper - Show team and repo badges on README - Switch puppet-ceph to rspec-puppet-facts - Fix puppet version for requirements in metadata This is the associated patch: https://github.com/openstack/puppet-ceph/commit/3bb76e7364fd0e00418226e98535aa1463f0dfc0 It removes the default value and introduce a new rgw_keystone_admin_project variable. This variable is not defined. I reassign the issue to puppet-ceph.
downgrading to puppet-ceph-2.2.1-3.el7ost solve the problem.
The problem is with the user parameter, not tenant ( https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/services/ceph-rgw.yaml#L73 ). Issue has been fixed upstream in master: https://review.openstack.org/#/c/405429/ and I pushed the backport for review: https://review.openstack.org/#/c/423526/ thanks Dimitri Savineau for the help.
(In reply to Gonéri Le Bouder from comment #10) > The problem is with the user parameter, not tenant ( > https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/ > puppet/services/ceph-rgw.yaml#L73 ). > > Issue has been fixed upstream in master: > https://review.openstack.org/#/c/405429/ > and I pushed the backport for review: > https://review.openstack.org/#/c/423526/ > > thanks Dimitri Savineau for the help. Goneri, from what I can tell the issue is in puppet-ceph and you pushed a workaround in THT for both master and newton branches, is it correct?
Yes except that the initial THT fix in master is from Emilien not me.
(In reply to Gonéri Le Bouder from comment #12) > Yes except that the initial THT fix in master is from Emilien not me. ack, thanks; let's merge the workaround first and see if we can fix the puppet-ceph too
Moving into POST state as the newton cherry-pick is merged now.
verified on openstack-tripleo-heat-templates-5.2.0-6.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1585