Bug 1340453 - Heat stack on overcloud fails with: Resource CREATE failed: Error: Cannot get stack domain user token, no stack domain id configured, please fix your heat.conf
Summary: Heat stack on overcloud fails with: Resource CREATE failed: Error: Cannot get...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Gaël Chamoulaud
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-27 12:46 UTC by Marius Cornea
Modified: 2016-10-24 09:32 UTC (History)
20 users (show)

Fixed In Version: openstack-tripleo-heat-templates-2.0.0-17.el7ost
Doc Type: If docs needed, set a value
Doc Text:
In previous releases of Heat, domain resources were created before /etc/heat/heat.conf was configured on the overcloud. However, domain resources depended on settings from that file; as such, these resources were not created correctly, preventing users from creating heat stacks. Users had to manually restart the Pacemaker Heat Engine resource to work around the problem. This release corrects the sequence of steps for deploying the Heat service, thereby fixing the problem.
Clone Of:
Environment:
Last Closed: 2016-08-11 11:31:26 UTC
Target Upstream Version:


Attachments (Terms of Use)
heat.conf (1.75 KB, text/plain)
2016-05-27 12:46 UTC, Marius Cornea
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1599 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 director Release Candidate Advisory 2016-08-11 15:25:37 UTC
OpenStack gerrit 331652 'None' 'MERGED' 'Ensure the Heat Domain parameters before starting heat-engine pcmk service' 2019-11-13 01:45:27 UTC
OpenStack gerrit 342032 'None' 'MERGED' 'Ensure the Heat Domain parameters before starting heat-engine pcmk service' 2019-11-13 01:45:27 UTC
Launchpad 1599232 None None None 2016-07-19 13:14:36 UTC

Description Marius Cornea 2016-05-27 12:46:23 UTC
Created attachment 1162457 [details]
heat.conf

Description of problem:
Heat stack creation on overcloud fails with:

Heat stack on overcloud fails with: Resource CREATE failed: Error: Cannot get stack domain user token, no stack domain id configured, please fix your heat.conf 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-2.0.0-8.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud 
2. Launch a heat stack on overcloud

Actual results:
Heat stack creation fails.

Expected results:
Heat stack creation succeeds.

Additional info:

Attaching the heat.conf on one of the controllers. 

It looks that there is no heat_stack Keystone domain created:

[root@overcloud-controller-0 heat]# mysql -e 'use keystone;select * from domain;'
+--------------------------+--------------------------+---------+-------+
| id                       | name                     | enabled | extra |
+--------------------------+--------------------------+---------+-------+
| <<keystone.domain.root>> | <<keystone.domain.root>> |       0 | {}    |
+--------------------------+--------------------------+---------+-------+

Comment 2 Marius Cornea 2016-05-27 13:09:27 UTC
The heat_stack domain and user are set up:

[stack@undercloud ~]$ openstack --os-auth-url http://172.16.18.25:5000 --os-username admin --os-project-name admin --os-password AsxhfvUqmeTVVp33YTXCYPVjF --os-identity-api-version 3 domain list
+----------------------------------+------------+---------+--------------------+
| ID                               | Name       | Enabled | Description        |
+----------------------------------+------------+---------+--------------------+
| 8b38cc65098d405a8ffe6a2662ae634a | heat_stack | True    |                    |
| default                          | Default    | True    | The default domain |
+----------------------------------+------------+---------+--------------------+

[stack@undercloud ~]$ openstack --os-auth-url http://172.16.18.25:5000 --os-username admin --os-project-name admin --os-password AsxhfvUqmeTVVp33YTXCYPVjF --os-identity-api-version 3 user list
+----------------------------------+----------------------------------------+
| ID                               | Name                                   |
+----------------------------------+----------------------------------------+
| 4ae78026b6084014ad1d3429b1a48b2e | aodh                                   |
| 4d50cf7d684743afa9406efe0739b58c | neutron                                |
| 5e67934b18cc43ea8ebfbc85ea02f982 | glance                                 |
| 605e371c6f1f402c9f8baf558248eaed | sahara                                 |
| 61a30ca59daa45a3b7578f0849946649 | cinder                                 |
| 75876d2a3a6d432095870bbc43586210 | vm04-vm04-e7sxmlyfjdn4-wh-ch2vctxpcyho |
| 790ac57025104891ae2f7987d6faeff2 | admin                                  |
| 991bd3484ea14a5d9f499b7f2f1a7314 | heat                                   |
| adce358666c74531ae4b2a05ea72a194 | ceilometer                             |
| d760c4aed7324e3fa6d6d746d28d7046 | heat_stack_domain_admin                |
| dbf66c766b6a4133aac5a38fd942b7ed | gnocchi                                |
| e20607846ceb4f3c9b8573da5b7d6c84 | nova                                   |
| f084df27799a49efa6d6c2696275b6aa | cinderv2                               |
| f9e209747e6b4e7b9c18bb68eeb3a0f3 | swift                                  |
+----------------------------------+----------------------------------------+

Comment 3 Thomas Hervé 2016-05-27 14:45:22 UTC
Looking the logs, it seems that heat is started and the configuration is changed afterwards. Indeed, restarting heat-engine on all the controllers fix the issue.

It's most likely a problem in puppet-heat doing the keystone domain configuration too late, or not forcing a heat restart once done.

Comment 4 Marius Cornea 2016-06-02 09:36:57 UTC
OK, so it seems I was able to reproduce this on 8 as well. Indeed after running pcs resource restart openstack-heat-engine-clone on the overcloud controller I was able to create a stack.

Comment 5 Kevin Jones 2016-06-03 01:17:52 UTC
I reproduced this as well on OSP 8 deployment (w nuage integration) in a lab environment.

Running this as suggested on my controller also resolved the issue:
pcs resource restart openstack-heat-engine-clone

Comment 7 Mike Orazi 2016-06-27 20:30:10 UTC
We moved this out of post -> assigned as I bel8ieve reproducing has become an issue.  Gael or Marius, can you update with the present status please?

Comment 10 Emilien Macchi 2016-06-30 15:06:18 UTC
I have a question for Heat folks:

what needs to happen *before* starting Heat:

1) configure heat.conf (already done in puppet-heat, see[1]) ?
2) create Keystone domain, user and user role for heat_admin ?

[1] https://github.com/openstack/puppet-heat/blob/master/manifests/deps.pp#L17

In both case, it seems we have an orchestration issue.
Gaël is working on https://review.openstack.org/#/c/331652/ but I don't agree with this change, it's not the right way to fix the problem. We need to use puppet orchestration to solve it, that's why I need to know what needs to happen before starting Heat.

Comment 11 Thomas Hervé 2016-06-30 17:14:27 UTC
Heat needs stack_user_domain_id or stack_user_domain_name config options set in the heat.conf before starting. The domain itself doesn't need to exist until a stack is actually created, but the value needs to be there. Considering the exception that we got, it would seem it's not when heat-engine starts.

Comment 16 Jaromir Coufal 2016-07-13 22:00:21 UTC
Does it fail in all cases? If not, what are the cases, what is the severity?

Comment 17 Jaromir Coufal 2016-07-13 22:01:11 UTC
Do we have workaround?

Comment 18 Marius Cornea 2016-07-14 07:39:19 UTC
(In reply to Jaromir Coufal from comment #17)
> Do we have workaround?

Yes, the workaround is easy: run 'pcs resource restart openstack-heat-engine-clone' after deployment is done.

(In reply to Jaromir Coufal from comment #16)
> Does it fail in all cases? If not, what are the cases, what is the severity?

No, it doesn't fail in all cases, only with particular templates, i.e. from my testing stack that creates vm, network, volume, attaches volume, writes a file system and mount it inside and signals back once it's done.

Comment 19 Luigi Toscano 2016-07-21 13:07:55 UTC
Just a note to say that this blocks Sahara, which uses Heat as the engine to spawn Big Data clusters.

Comment 22 errata-xmlrpc 2016-08-11 11:31:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html


Note You need to log in before you can comment on or make changes to this bug.