Bug 1414976 - Overcloud deployment fail (step5): Error: No set of title patterns matched the title "@service".
Summary: Overcloud deployment fail (step5): Error: No set of title patterns matched th...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: z3
: 10.0 (Newton)
Assignee: Giulio Fidente
QA Contact: Yogev Rabl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-19 21:14 UTC by Gonéri Le Bouder
Modified: 2017-06-28 14:44 UTC (History)
14 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.2.0-17.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-28 14:44:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
environment.json (47.23 KB, text/plain)
2017-01-19 21:27 UTC, Gonéri Le Bouder
no flags Details
heat-debug.log.gz (285.49 KB, application/x-gzip)
2017-01-19 21:28 UTC, Gonéri Le Bouder
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1642524 0 None None None 2017-01-23 08:59:27 UTC
OpenStack gerrit 423526 0 None MERGED ceph-rgw: add missing user parameter 2020-09-29 21:17:07 UTC
Red Hat Product Errata RHBA-2017:1585 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 director Bug Fix Advisory 2017-06-28 18:42:51 UTC

Description Gonéri Le Bouder 2017-01-19 21:14:50 UTC
Description of problem:

Overcloud deployment fail during step 5

Overcloud deployment command:
cd ; openstack overcloud deploy \
 --debug \
 --log-file ~/pilot/overcloud_deployment.log \
 -t 1800 \
 --stack overcloud \
 --templates ~/pilot/templates/overcloud \
 -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \
 -e ~/pilot/templates/network-environment.yaml \
 -e ~/pilot/templates/node-placement.yaml \
 -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \
 -e ~/pilot/templates/overcloud/environments/ceph-radosgw.yaml \
 -e ~/pilot/templates/dell-environment.yaml \
 -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml \
 -e ~/pilot/templates/dell-cinder-backends.yaml \
 --control-flavor baremetal \
 --compute-flavor baremetal \
 --ceph-storage-flavor baremetal \
 --swift-storage-flavor baremetal \
 --block-storage-flavor baremetal \
 --neutron-public-interface bond1 \
 --neutron-network-type vlan \
 --neutron-disable-tunneling \
 --libvirt-type kvm \
 --os-auth-url http://192.168.120.202:5000/v2.0 \
 --os-project-name admin \
 --os-user-id admin \
 --os-password 0147b920dc189f878a07a2289cefcfc12f45c35e \
 --control-scale 3 \
 --compute-scale 3 \
 --ceph-storage-scale 3 \
 --ntp-server 0.centos.pool.ntp.org \
 --neutron-network-vlan-ranges physint:201:220,physext \
 --neutron-bridge-mappings physint:br-tenant,physext:br-ex

2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.BlockStorageDeployment_Step5]: CREATE_COMPLETE  state changed
2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step5]: CREATE_COMPLETE  state changed
2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:32:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:32:58Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: CREATE_IN_PROGRESS  state changed
2017-01-19 19:33:26Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: SIGNAL_IN_PROGRESS  Signal: deployment 8f55d6fa-414a-49a7-bb3e-5a065d53cc86 succeeded
2017-01-19 19:33:26Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: CREATE_COMPLETE  state changed
2017-01-19 19:33:46Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.1]: SIGNAL_IN_PROGRESS  Signal: deployment 405a9941-1902-4dab-a3b5-7dd686a3a77e succeeded
2017-01-19 19:33:46Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: SIGNAL_IN_PROGRESS  Signal: deployment bd9daf26-2f0e-45ec-9a83-679af6c47022 succeeded
2017-01-19 19:33:47Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.1]: CREATE_COMPLETE  state changed
2017-01-19 19:33:47Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.0]: CREATE_COMPLETE  state changed
2017-01-19 19:33:48Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: SIGNAL_IN_PROGRESS  Signal: deployment 1f13b98c-47bc-4c78-b516-259fc88e17a1 failed (1)
2017-01-19 19:33:49Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.1]: SIGNAL_IN_PROGRESS  Signal: deployment 0c1698d2-9a5e-48a9-b384-25eed8cde05d succeeded
2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.1]: CREATE_COMPLETE  state changed
2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: SIGNAL_IN_PROGRESS  Signal: deployment d9bb5602-ec52-479a-9fd2-47aaf062b613 succeeded
2017-01-19 19:33:55Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: CREATE_COMPLETE  state changed
2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: SIGNAL_IN_PROGRESS  Signal: deployment b1b7df18-81cc-472c-b14b-62a600b32c89 succeeded
2017-01-19 19:33:56Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5]: CREATE_COMPLETE  state changed
2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5.2]: CREATE_COMPLETE  state changed
2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE  Stack CREATE completed successfully
2017-01-19 19:33:57Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE  state changed
2017-01-19 19:35:32Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.1]: SIGNAL_IN_PROGRESS  Signal: deployment 1c9009c7-64d8-4913-976a-754c3a336c5e succeeded
2017-01-19 19:35:32Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.1]: CREATE_COMPLETE  state changed
2017-01-19 19:37:18Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: SIGNAL_IN_PROGRESS  Signal: deployment daf9ae1a-bd4d-4cf5-b7d7-72dc79aaac46 succeeded
2017-01-19 19:37:19Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.2]: CREATE_COMPLETE  state changed
2017-01-19 19:37:19Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
2017-01-19 19:37:20Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_FAILED  Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2017-01-19 19:37:20Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2017-01-19 19:37:21Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
2017-01-19 19:37:21Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1

 Stack overcloud CREATE_FAILED




[2017-01-19 19:55:38,423] (heat-config) [INFO] exception: connect failed
^[[1;31mWarning: Scope(Class[Cinder::Api]): keystone_enabled is deprecated, use auth_strategy instead.^[[0m
^[[1;31mWarning: Scope(Class[Keystone]): Fernet token is recommended in Mitaka release. The default for token_provider will be changed to 'fernet' in O release.^[[0m
^[[1;31mWarning: Scope(Class[Glance::Api]): default_store not provided, it will be automatically set to glance.store.http.Store^[[0m
^[[1;31mWarning: Scope(Class[Heat]): keystone_user_domain_id is deprecated, use the name option instead.^[[0m
^[[1;31mWarning: Scope(Class[Heat]): keystone_project_domain_id is deprecated, use the name option instead.^[[0m
^[[1;31mWarning: Scope(Class[Neutron::Agents::L3]): parameter external_network_bridge is deprecated^[[0m
^[[1;31mWarning: Scope(Class[Neutron::Server::Notifications]): nova_url is deprecated and will be removed after Newton cycle.^[[0m
^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_user'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_password'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::admin_tenant_name'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::auth_uri'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::auth_version'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Keystone::Authtoken]): Could not look up qualified variable '::nova::api::identity_uri'; class ::nova::api has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated^[[0m
^[[1;31mWarning: Scope(Class[Ceilometer]): Both $metering_secret and $telemetry_secret defined, using $telemetry_secret^[[0m
^[[1;31mWarning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 166 in file /etc/puppet/modules/gnocchi/manifests/api.pp^[[0m
^[[1;31mWarning: Scope(Class[Gnocchi::Api]): gnocchi:api::keystone_identity_uri is deprecated, use gnocchi::keystone::authtoken::auth_url instead^[[0m
^[[1;31mWarning: Scope(Class[Gnocchi::Api]): gnocchi::api::keystone_auth_uri is deprecated, use gnocchi::keystone::authtoken::auth_uri instead^[[0m
^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m
^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m
^[[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.^[[0m
^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m
^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m
^[[1;31mWarning: Not collecting exported resources without storeconfigs^[[0m
^[[1;31mError: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org^[[0m
^[[1;31mError: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org^[[0m

[2017-01-19 19:55:38,423] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/af951947-d8e4-4635-b6aa-0e5914b6c890.pp. [1]


Version-Release number of selected component (if applicable):

Puddle: 2017-01-17.1

Comment 1 Gonéri Le Bouder 2017-01-19 21:27:18 UTC
Created attachment 1242602 [details]
environment.json

Comment 2 Gonéri Le Bouder 2017-01-19 21:28:24 UTC
Created attachment 1242603 [details]
heat-debug.log.gz

Comment 3 Gonéri Le Bouder 2017-01-20 13:49:16 UTC
I just redeployed the platform with a configuration that was working with the previous OSP10 puddle (ceph and eqx backends for Cinder are enabled). I face the same issue.

Comment 4 Gonéri Le Bouder 2017-01-20 15:30:47 UTC
I tried to call the puppet script manually with all the content commented and then I reenable its content line be line. The error shows up when I uncomment:

include ::tripleo::profile::base::keystone


(...)
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[aodh] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[ceilometer] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[cinder] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[glance] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[gnocchi] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[heat] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[heat-cfn] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[neutron] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user[nova] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user_role[admin@admin] with 'before'
Debug: Adding relationship from Anchor[keystone::service::end] to Keystone_user_role[heat_stack_domain_admin::heat_stack@::heat_stack] with 'before'
Error: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org
Error: No set of title patterns matched the title "@service". on node overcloud-controller-0.cwdg720xd01.org

Comment 5 Gonéri Le Bouder 2017-01-20 17:17:37 UTC
I can avoid the problem if I comment these lines
https://github.com/openstack/puppet-ceph/blob/3bb76e7364fd0e00418226e98535aa1463f0dfc0/manifests/rgw/keystone/auth.pp#L85-L89
in /usr/share/openstack-puppet/modules/ceph/manifests/rgw/keystone/auth.pp

Comment 6 Giulio Fidente 2017-01-20 17:34:43 UTC
(In reply to Gonéri Le Bouder from comment #3)
> I just redeployed the platform with a configuration that was working with
> the previous OSP10 puddle (ceph and eqx backends for Cinder are enabled). I
> face the same issue.

hi, are you suggesting that it is a particular combination of cinder/dell/rgw not working? does the issue reproduce with a standard overcloud with rgw?

Comment 7 Gonéri Le Bouder 2017-01-20 20:44:19 UTC
At this point, I'm a bit confused. I managed to deploy the platform with the previous puddle and the very same configuration (ceph, dellsc, equallogic). I've no conclusion yet.

Comment 8 Gonéri Le Bouder 2017-01-20 20:55:08 UTC
I believe it's a regression introduce here:

http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/10.0-RHEL-7/2017-01-15.1/logs/changelog.log
puppet-ceph: puppet-ceph-2.2.1-3.el7ost -> puppet-ceph-2.2.1-4.el7ost
---------------------------------------------------------------------
Wed Jan 04 2017 Jon Schlueter <jschluet> 2.2.1-4
- Include and not redeclare apache mod_env (lp#1651987)
- Update ceph::key to update keys
- Unify parameter arguments between classes. <-----------------------
- Use facts from puppet-openstack_spec_helper
- Show team and repo badges on README
- Switch puppet-ceph to rspec-puppet-facts
- Fix puppet version for requirements in metadata

This is the associated patch:
https://github.com/openstack/puppet-ceph/commit/3bb76e7364fd0e00418226e98535aa1463f0dfc0

It removes the default value and introduce a new rgw_keystone_admin_project variable. This variable is not defined. I reassign the issue to puppet-ceph.

Comment 9 Gonéri Le Bouder 2017-01-20 21:00:38 UTC
downgrading to puppet-ceph-2.2.1-3.el7ost solve the problem.

Comment 10 Gonéri Le Bouder 2017-01-20 21:39:47 UTC
The problem is with the user parameter, not tenant ( https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/services/ceph-rgw.yaml#L73 ).

Issue has been fixed upstream in master: https://review.openstack.org/#/c/405429/
and I pushed the backport for review: https://review.openstack.org/#/c/423526/

thanks Dimitri Savineau for the help.

Comment 11 Giulio Fidente 2017-01-23 09:03:37 UTC
(In reply to Gonéri Le Bouder from comment #10)
> The problem is with the user parameter, not tenant (
> https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/
> puppet/services/ceph-rgw.yaml#L73 ).
> 
> Issue has been fixed upstream in master:
> https://review.openstack.org/#/c/405429/
> and I pushed the backport for review:
> https://review.openstack.org/#/c/423526/
> 
> thanks Dimitri Savineau for the help.

Goneri, from what I can tell the issue is in puppet-ceph and you pushed a workaround in THT for both master and newton branches, is it correct?

Comment 12 Gonéri Le Bouder 2017-01-23 14:07:03 UTC
Yes except that the initial THT fix in master is from Emilien not me.

Comment 13 Giulio Fidente 2017-01-23 16:50:12 UTC
(In reply to Gonéri Le Bouder from comment #12)
> Yes except that the initial THT fix in master is from Emilien not me.

ack, thanks; let's merge the workaround first and see if we can fix the puppet-ceph too

Comment 14 Giulio Fidente 2017-01-25 10:47:54 UTC
Moving into POST state as the newton cherry-pick is merged now.

Comment 16 Yogev Rabl 2017-03-31 16:44:18 UTC
verified on openstack-tripleo-heat-templates-5.2.0-6.el7ost.noarch

Comment 19 errata-xmlrpc 2017-06-28 14:44:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1585


Note You need to log in before you can comment on or make changes to this bug.