Bug 1388474

steve, can you triage this one? i suppose it might be related to heat policy somehow

The non random diff part of the heat config, comparing the old(working) with (new) not working version:

Common subdirectories: old/controller-0/etc/heat/environment.d and new/controller-0/etc/heat/environment.d
diff -u old/controller-0/etc/heat/heat.conf new/controller-0/etc/heat/heat.conf
--- old/controller-0/etc/heat/heat.conf	2016-10-19 17:43:49.000000000 +0200
+++ new/controller-0/etc/heat/heat.conf	2016-10-25 10:00:17.000000000 +0200
@@ -193,16 +193,13 @@
 # Keystone domain name which contains heat template-defined users. If
 # `stack_user_domain_id` option is set, this option is ignored. (string value)
 #stack_user_domain_name = <None>
-stack_user_domain_name = heat_stack
 
 # Keystone username, a user with roles sufficient to manage users and projects
 # in the stack_user_domain. (string value)
 #stack_domain_admin = <None>
-stack_domain_admin = heat_stack_domain_admin
 
 # Keystone password for stack_domain_admin user. (string value)
 #stack_domain_admin_password = <None>
-stack_domain_admin_password = GZQHKXp4Kus9suAx2xYGPHDtj
 
 # Maximum raw byte size of any template. (integer value)
 #max_template_size = 524288

Attila, this might already be fixed upstream. Can you confirm that your tripleo-heat-templates has the following changes?

https://review.openstack.org/#/c/387288/
https://review.openstack.org/#/c/387289/

I think either these changes will fix the problem, or caused the problem.

https://review.openstack.org/#/c/387288/1/manifests/profile/base/keystone.pp was in puppet-tripleo-5.3.0-1.el7ost.noarch .

https://review.openstack.org/#/c/387289/ is not in
openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost.noarch

OK, I suggest retesting when https://review.openstack.org/#/c/387289/ is available in a puddle build.

openstack-tripleo-heat-templates                                 -5.0.0-1.1.el7ost has the 2th change, and still failing

/etc/heat/heat.conf:

[DEFAULT]
heat_metadata_server_url = http://127.0.0.1:8000
heat_waitcondition_server_url = http://127.0.0.1:8000/v1/waitcondition
heat_watch_server_url =http://127.0.0.1:8003
num_engine_workers = 0
auth_encryption_key = XNK0jWPZcVwyOcprSQLjalNhneqffmC0
debug = True
log_dir = /var/log/heat
rpc_backend = rabbit
[auth_password]
[clients]
[clients_aodh]
[clients_barbican]
[clients_ceilometer]
[clients_cinder]
[clients_designate]
[clients_glance]
[clients_heat]
[clients_keystone]
auth_uri =http://192.168.0.15:35357
[clients_magnum]
[clients_manila]
[clients_mistral]
[clients_monasca]
[clients_neutron]
[clients_nova]
[clients_sahara]
[clients_senlin]
[clients_swift]
[clients_trove]
[clients_zaqar]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://heat:9xKnzxjKNh4y9fKKUdDym3NKH.0.16/heat
max_retries = -1
db_max_retries = -1
[ec2authtoken]
auth_uri = http://10.0.0.16:5000/v2.0/ec2tokens
[eventlet_opts]
[heat_api]
bind_host = 10.0.0.10
workers = 0
[heat_api_cfn]
bind_host = 10.0.0.10
workers = 0
[heat_api_cloudwatch]
bind_host = 10.0.0.10
workers = 0
[matchmaker_redis]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
driver =messaging
[oslo_messaging_rabbit]
rabbit_hosts = 10.0.0.23
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = Bb23X4tM8uJERZMwDe6jVYPvB
heartbeat_timeout_threshold = 60
[oslo_messaging_zmq]
[oslo_middleware]
enable_proxy_headers_parsing = True
[oslo_policy]
policy_file = /etc/heat/policy.json
[paste_deploy]
[profiler]
[revision]
[ssl]
[trustee]
auth_url = http://192.168.0.15:35357
project_domain_name = Default
username = heat
user_domain_name = Default
password = 9xKnzxjKNh4y9fKKUdDym3NKH
auth_plugin=password
[volumes]
[keystone_authtoken]
password=9xKnzxjKNh4y9fKKUdDym3NKH
project_name=service
project_domain_name=Default
auth_url=http://192.168.0.15:35357
username=heat
auth_uri=http://10.0.0.16:5000/v2.0
auth_type=password
user_domain_name=Default

openstack --os-identity-api-version 3 domain list
+----------------------------------+-------------------+---------+--------------------+
| ID                               | Name              | Enabled | Description        |
+----------------------------------+-------------------+---------+--------------------+
| d93a131125714434839803aaac91d30b | <SERVICE DEFAULT> | True    |                    |
| default                          | Default           | True    | The default domain |
+----------------------------------+-------------------+---------+--------------------+

openstack --os-identity-api-version 3 user list --domain Default
+----------------------------------+------------+
| ID                               | Name       |
+----------------------------------+------------+
| ca9dae39524c49c980bc6b4df575cbdb | admin      |
| 4f85409148584adb8cb75e8af98f15a3 | neutron    |
| c90dbbac41c54f1bbb0ad60e0e5d15c7 | heat       |
| 06a7be6314c04c0b8efab24ba92e62f2 | gnocchi    |
| 729de663581344e7a5019d02600c583a | aodh       |
| 7840a28097234c7680b43d3140f64026 | nova       |
| b1d329a361854bfa9d6a8c6e8d0b41c1 | glance     |
| 7e7652da59094229af9bfff012fc0614 | ceilometer |
| c06da5359f9a45c690f5242a25b099a3 | cinder     |
| 481f7a9badb84f9dab6c685b7550e7ea | heat-cfn   |
| dab1c30bb75b437ab3c892109d212eb4 | swift      |
| 712ba1fdf79a465280a73d2344be6e48 | testuser   |
+----------------------------------+------------+

Ok so I've reproduced and the problem appears to be puppet is incorrectly ensuring the domain does *not* exist:

    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]/ensure: removed^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]/ensure: removed^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_user[<SERVICE DEFAULT>::<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[<SERVICE DEFAULT>::<SERVICE DEFAULT>@::<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]/ensure: removed^[[0m

Currently trying to figure out why this is happening

I tested https://review.openstack.org/#/c/392288 and it appears to fix this

[stack@undercloud-0 ~]$ openstack stack show test1
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                    |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+
| id                    | ebfaa0f8-5125-4e97-a2bf-01d3efdeb220                                                                                     |
| stack_name            | test1                                                                                                                    |
| description           | Template which creates only a new user                                                                                   |
|                       |                                                                                                                          |
| creation_time         | 2016-11-10T17:18:57Z                                                                                                     |
| updated_time          | None                                                                                                                     |
| stack_status          | CREATE_COMPLETE                                                                                                          |
| stack_status_reason   | Stack CREATE completed successfully                                                                                      |
| parameters            | AWS::Region: ap-southeast-1                                                                                              |
|                       | AWS::StackId: arn:openstack:heat::fb267f9c3d11474abb150483e0583c95:stacks/test1/ebfaa0f8-5125-4e97-a2bf-01d3efdeb220     |
|                       | AWS::StackName: test1                                                                                                    |
|                       |                                                                                                                          |
| outputs               | []                                                                                                                       |
|                       |                                                                                                                          |
| links                 | - href: https://172.16.18.32:13004/v1/fb267f9c3d11474abb150483e0583c95/stacks/test1/ebfaa0f8-5125-4e97-a2bf-01d3efdeb220 |
|                       |   rel: self                                                                                                              |
|                       |                                                                                                                          |
| parent                | None                                                                                                                     |
| disable_rollback      | True                                                                                                                     |
| deletion_time         | None                                                                                                                     |
| stack_user_project_id | b0ba2225c275442d8a6c3e171d599d9d                                                                                         |
| capabilities          | []                                                                                                                       |
| notification_topics   | []                                                                                                                       |
| stack_owner           | None                                                                                                                     |
| timeout_mins          | None                                                                                                                     |
| tags                  | null                                                                                                                     |
|                       | ...                                                                                                                      |
|                       |                                                                                                                          |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html