| Summary: | stack UPDATE_FAILED while adding more computes (step 6) | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduard Barrera <ebarrera> |
| Component: | rhosp-director | Assignee: | Angus Thomas <athomas> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 (Kilo) | CC: | dbecker, jstransk, mburns, mcornea, morazi, pcaruana, plemenko, rhel-osp-director-maint |
| Target Milestone: | --- | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-03-18 21:26:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
It's probably that the first Puppet prefetch error due to non-ascii characters: Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: ERROR: openstack 'ascii' codec can't encode character u'\\xf3' in position 89: ordinal not in range(128)\n\"ID\",\"Name\",\"Description\",\"Enabled\"\n\"0161c65a362b446abbf896fac457f4d6\",\"DSN-ENG1\",\"Tenant reserved for migration from OST-PRE\",True\n\"038d38558df54e30ba72f8cf625515cd\",\"EndPointsTestJSON-446276246\",\"EndPointsTestJSON-446276246-desc\",True\n\"042cf907720145fea8ea1324394fd576\",\"demo\",\"Default project\",True\n\"05973815d133410192348bf3f4477c9e\",\"Test\",\"Test project for Admins\",True\n\"07ec661cbb6447759a1389bc69786c91\",\"AccountNegativeTest-825982852\",\"AccountNegativeTest-825982852-desc\",True\n\"08313c47cc5340ba8f03fee1e646e46c\",\"ContainerTest-619159989\",\"ContainerTest-619159989-desc\",True\n\"0b55ec4f0bf74e5e8933f55b284c8518\",\"ObjectTestACLs-1275316143\",\"ObjectTestACLs-1275316143-desc\",True\n\"11e5824e6e0749d2aa52412b87b8dae9\",\"ProjectsTestJSON-926624388\",\"ProjectsTestJSON-926624388-desc\",True ... causes that Puppet isn't able to list existing projects in Keystone, and it tries to create the 'admin' project again: Error: Execution of '/usr/bin/openstack project create --format shell admin --enable --description admin tenant' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-efaca269-2a83-4803-b2ea-006f91dce361) ... but the project already exists, so the command fails, and the Puppet run fails too. I think the root cause might be a general i18n issue with python-openstackclient, like described in bug 1218122. Until this is fixed (probably in python-openstackclient), it may be possible to work around the issue by not using non-ascii characters in OpenStack entity names (this particular issue is with Keystone projects, but it could pop up in other situations as well). |
Description of problem: It is not possible to upscalwe the cloud correctly, we are facing the folloing errors: # openstack overcloud deploy --templates ~/templates/my-overcloud --compute-scale 4 --compute-flavor compute --ntp-server 10.256.256.256 --neutron-network-type vxlan --neutron-tunnel-types vxlan -e ~/templates/global-environment-dsn.yaml -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/my-overcloud/environments/storage-environment.yaml -e /home/stack/templates/my-overcloud/enable-tls.yaml -e /home/stack/ssl/enable-tls-key.yaml -e /home/stack/templates/my-overcloud/inject-trust-anchor.yaml Deploying templates in the directory /home/stack/templates/my-overcloud Stack failed with status: resources.ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step6: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 ERROR: openstack Heat Stack update failed. ]$ heat resource-list -n 5 overcloud | grep FAILED | ControllerNodesPostDeployment | 9bc778fc-03f8-4071-9d94-4147590b94c4 | OS::TripleO::ControllerPostDeployment | UPDATE_FAILED | 2016-03-17T08:59:08Z | | | ControllerOvercloudServicesDeployment_Step6 | f6cb421c-de01-4b80-9055-4c820a50245c | OS::Heat::StructuredDeployments | UPDATE_FAILED | 2016-03-17T09:14:07Z | ControllerNodesPostDeployment | | 0 | 11b06f99-2075-4cbe-be94-bdc1c4af83a0 | OS::Heat::StructuredDeployment | UPDATE_FAILED | 2016-03-17T09:14:09Z | ControllerOvercloudServicesDeployment_Step6 | $ heat deployment-show 11b06f99-2075-4cbe-be94-bdc1c4af83a0 { "status": "FAILED", "server_id": "65a55e8e-31be-4023-b5bf-6bfc53a9f350", "config_id": "d825e229-0068-4036-a1e2-210dadab4970", "output_values": { "deploy_stdout": "\u001b[mNotice: Compiled catalog for ost-ah1-controller0.localdomain in environment production in 42.17 seconds\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Exec[galera-ready]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/File[/tmp/ceph-mon-keyring-ost-ah1-controller0]/ensure: defined content as '{md5}6e76a887817a550ce891521930b1010f'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[ceph-mon-mkfs-ost-ah1-controller0]/returns: ++ ceph-mon --id ost-ah1-controller0 --show-config-value mon_data\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[ceph-mon-mkfs-ost-ah1-controller0]/returns: + mon_data=/var/lib/ceph/mon/ceph-ost-ah1-controller0\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[ceph-mon-mkfs-ost-ah1-controller0]/returns: + '[' '!' -d /var/lib/ceph/mon/ceph-ost-ah1-controller0 ']'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[ceph-mon-mkfs-ost-ah1-controller0]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[ceph-mon-ceph.client.admin.keyring-ost-ah1-controller0]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Service[ceph-mon-ost-ah1-controller0]/ensure: ensure changed 'stopped' to 'running'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Profile::Mon/Ceph::Mon[ost-ah1-controller0]/Exec[rm-keyring-ost-ah1-controller0]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Dependency Keystone_tenant[admin] has failures: true\u001b[0m\n\u001b[mNotice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Keystone_tenant[admin] has failures: true\u001b[0m\n\u001b[mNotice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]: Dependency Keystone_tenant[services] has failures: true\u001b[0m\n\u001b[mNotice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]: Dependency Keystone_tenant[admin] has failures: true\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully\u001b[0m\n\u001b[mNotice: Pacemaker has reported quorum achieved\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/message: defined 'message' as 'Pacemaker has reported quorum achieved'\u001b[0m\n\u001b[mNotice: Finished catalog run in 167.19 seconds\u001b[0m\n", "deploy_stderr": "Device \"br_bond0\" does not exist.\nDevice \"br_ex\" does not exist.\nDevice \"br_int\" does not exist.\nDevice \"br_tun\" does not exist.\nDevice \"ovs_system\" does not exist.\n\u001b[1;31mWarning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.\u001b[0m\n\u001b[1;31mError: Could not prefetch keystone_tenant provider 'openstack': Execution of '/usr/bin/openstack project list --quiet --format csv --long' returned 1: ERROR: openstack 'ascii' codec can't encode character u'\\xf3' in position 89: ordinal not in range(128)\n\"ID\",\"Name\",\"Description\",\"Enabled\"\n\"0161c65a362b446abbf896fac457f4d6\",\"DSN-ENG1\",\"Tenant reserved for migration from OST-PRE\",True\n\"038d38558df54e30ba72f8cf625515cd\",\"EndPointsTestJSON-446276246\",\"EndPointsTestJSON-446276246-desc\",True\n\"042cf907720145fea8ea1324394fd576\",\"demo\",\"Default project\",True\n\"05973815d133410192348bf3f4477c9e\",\"Test\",\"Test project for Admins\",True\n\"07ec661cbb6447759a1389bc69786c91\",\"AccountNegativeTest-825982852\",\"AccountNegativeTest-825982852-desc\",True\n\"08313c47cc5340ba8f03fee1e646e46c\",\"ContainerTest-619159989\",\"ContainerTest-619159989-desc\",True\n\"0b55ec4f0bf74e5e8933f55b284c8518\",\"ObjectTestACLs-1275316143\",\"ObjectTestACLs-1275316143-desc\",True\n\"11e5824e6e0749d2aa52412b87b8dae9\",\"ProjectsTestJSON-926624388\",\"ProjectsTestJSON-926624388-desc\",True\u001b[0m\n\u001b[1;31mError: Execution of '/usr/bin/openstack project create --format shell admin --enable --description admin tenant' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-efaca269-2a83-4803-b2ea-006f91dce361)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Execution of '/usr/bin/openstack project create --format shell admin --enable --description admin tenant' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-efaca269-2a83-4803-b2ea-006f91dce361)\u001b[0m\n\u001b[1;31mError: Execution of '/usr/bin/openstack project create --format shell services --enable --description Tenant for the openstack services' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-1f87a68a-d9ef-4425-be5b-0334bf7475c0)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[services]/ensure: change from absent to present failed: Execution of '/usr/bin/openstack project create --format shell services --enable --description Tenant for the openstack services' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-1f87a68a-d9ef-4425-be5b-0334bf7475c0)\u001b[0m\n\u001b[1;31mWarning: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Skipping because of failed dependencies\u001b[0m\n\u001b[1;31mWarning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies\u001b[0m\n\u001b[1;31mWarning: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]: Skipping because of failed dependencies\u001b[0m\n", "deploy_status_code": 6 }, "creation_time": "2016-03-08T17:32:54Z", "updated_time": "2016-03-17T09:19:55Z", "input_values": {}, "action": "UPDATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 6", "id": "11b06f99-2075-4cbe-be94-bdc1c4af83a0" } please note: Error: Execution of '/usr/bin/openstack project create --format shell admin --enable --description admin tenant' returned 1: ERROR: openstack Conflict occurred attempting to store project - Duplicate Entry (HTTP 409) (Request-ID: req-efaca269-2a83-4803-b2ea-006f91dce361)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Execution of '/usr/bin/openstack project create --format shell admin --enable --description admin tenant' Other things in the logs: # /usr/bin/openstack project list --quiet --format csv --long' returned 1: ERROR: openstack 'ascii' codec can't encode character u'\\xf3' in position 89: ordinal not in range(12 Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. deploy an initial cloud 2. upscale the environment 3. Actual results: above errors Expected results: env upscaled successfuly Additional info: