Hide Forgot
Created attachment 1162457 [details] heat.conf Description of problem: Heat stack creation on overcloud fails with: Heat stack on overcloud fails with: Resource CREATE failed: Error: Cannot get stack domain user token, no stack domain id configured, please fix your heat.conf Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-2.0.0-8.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud 2. Launch a heat stack on overcloud Actual results: Heat stack creation fails. Expected results: Heat stack creation succeeds. Additional info: Attaching the heat.conf on one of the controllers. It looks that there is no heat_stack Keystone domain created: [root@overcloud-controller-0 heat]# mysql -e 'use keystone;select * from domain;' +--------------------------+--------------------------+---------+-------+ | id | name | enabled | extra | +--------------------------+--------------------------+---------+-------+ | <<keystone.domain.root>> | <<keystone.domain.root>> | 0 | {} | +--------------------------+--------------------------+---------+-------+
The heat_stack domain and user are set up: [stack@undercloud ~]$ openstack --os-auth-url http://172.16.18.25:5000 --os-username admin --os-project-name admin --os-password AsxhfvUqmeTVVp33YTXCYPVjF --os-identity-api-version 3 domain list +----------------------------------+------------+---------+--------------------+ | ID | Name | Enabled | Description | +----------------------------------+------------+---------+--------------------+ | 8b38cc65098d405a8ffe6a2662ae634a | heat_stack | True | | | default | Default | True | The default domain | +----------------------------------+------------+---------+--------------------+ [stack@undercloud ~]$ openstack --os-auth-url http://172.16.18.25:5000 --os-username admin --os-project-name admin --os-password AsxhfvUqmeTVVp33YTXCYPVjF --os-identity-api-version 3 user list +----------------------------------+----------------------------------------+ | ID | Name | +----------------------------------+----------------------------------------+ | 4ae78026b6084014ad1d3429b1a48b2e | aodh | | 4d50cf7d684743afa9406efe0739b58c | neutron | | 5e67934b18cc43ea8ebfbc85ea02f982 | glance | | 605e371c6f1f402c9f8baf558248eaed | sahara | | 61a30ca59daa45a3b7578f0849946649 | cinder | | 75876d2a3a6d432095870bbc43586210 | vm04-vm04-e7sxmlyfjdn4-wh-ch2vctxpcyho | | 790ac57025104891ae2f7987d6faeff2 | admin | | 991bd3484ea14a5d9f499b7f2f1a7314 | heat | | adce358666c74531ae4b2a05ea72a194 | ceilometer | | d760c4aed7324e3fa6d6d746d28d7046 | heat_stack_domain_admin | | dbf66c766b6a4133aac5a38fd942b7ed | gnocchi | | e20607846ceb4f3c9b8573da5b7d6c84 | nova | | f084df27799a49efa6d6c2696275b6aa | cinderv2 | | f9e209747e6b4e7b9c18bb68eeb3a0f3 | swift | +----------------------------------+----------------------------------------+
Looking the logs, it seems that heat is started and the configuration is changed afterwards. Indeed, restarting heat-engine on all the controllers fix the issue. It's most likely a problem in puppet-heat doing the keystone domain configuration too late, or not forcing a heat restart once done.
OK, so it seems I was able to reproduce this on 8 as well. Indeed after running pcs resource restart openstack-heat-engine-clone on the overcloud controller I was able to create a stack.
I reproduced this as well on OSP 8 deployment (w nuage integration) in a lab environment. Running this as suggested on my controller also resolved the issue: pcs resource restart openstack-heat-engine-clone
We moved this out of post -> assigned as I bel8ieve reproducing has become an issue. Gael or Marius, can you update with the present status please?
I have a question for Heat folks: what needs to happen *before* starting Heat: 1) configure heat.conf (already done in puppet-heat, see[1]) ? 2) create Keystone domain, user and user role for heat_admin ? [1] https://github.com/openstack/puppet-heat/blob/master/manifests/deps.pp#L17 In both case, it seems we have an orchestration issue. Gaël is working on https://review.openstack.org/#/c/331652/ but I don't agree with this change, it's not the right way to fix the problem. We need to use puppet orchestration to solve it, that's why I need to know what needs to happen before starting Heat.
Heat needs stack_user_domain_id or stack_user_domain_name config options set in the heat.conf before starting. The domain itself doesn't need to exist until a stack is actually created, but the value needs to be there. Considering the exception that we got, it would seem it's not when heat-engine starts.
Does it fail in all cases? If not, what are the cases, what is the severity?
Do we have workaround?
(In reply to Jaromir Coufal from comment #17) > Do we have workaround? Yes, the workaround is easy: run 'pcs resource restart openstack-heat-engine-clone' after deployment is done. (In reply to Jaromir Coufal from comment #16) > Does it fail in all cases? If not, what are the cases, what is the severity? No, it doesn't fail in all cases, only with particular templates, i.e. from my testing stack that creates vm, network, volume, attaches volume, writes a file system and mount it inside and signals back once it's done.
Just a note to say that this blocks Sahara, which uses Heat as the engine to spawn Big Data clusters.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1599.html