Bug 1388474 - Overcloud heat fails to create an IAM user
Summary: Overcloud heat fails to create an IAM user
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Steven Hardy
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-25 12:51 UTC by Attila Fazekas
Modified: 2016-12-14 16:25 UTC (History)
15 users (show)

Fixed In Version: puppet-tripleo-5.3.0-7.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:25:06 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 393000 None None None 2016-11-03 12:45:15 UTC

Description Attila Fazekas 2016-10-25 12:51:56 UTC
Description of problem:

Normally all user is able to create AWS::IAM::User ,
but in current setup it faild.


Version-Release number of selected component (if applicable):
python-keystoneclient-3.5.0-1.el7ost.noarch
openstack-tripleo-common-5.3.0-1.el7ost.noarch
python-keystoneauth1-2.12.1-1.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost.noarch
puppet-keystone-9.4.0-1.el7ost.noarch
puppet-heat-9.4.1-1.el7ost.noarch
openstack-heat-engine-7.0.0-3.el7ost.noarch
python-keystone-10.0.0-2.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch
openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch
puppet-tripleo-5.3.0-1.el7ost.noarch
openstack-heat-api-cfn-7.0.0-3.el7ost.noarch
openstack-keystone-10.0.0-2.el7ost.noarch
openstack-heat-templates-0.0.1-0.20161011152629.40a4ed0.el7ost.noarch
python-heatclient-1.5.0-1.el7ost.noarch
python-tripleoclient-5.3.0-1.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
python-keystonemiddleware-4.9.0-1.el7ost.noarch
openstack-heat-api-7.0.0-3.el7ost.noarch
openstack-heat-common-7.0.0-3.el7ost.noarch
openstack-tripleo-validations-5.1.1-0.20161003173643.3652f12.el7ost.noarch
python-heat-agent-0.0.1-0.20161011152629.40a4ed0.el7ost.noarch
openstack-tripleo-ui-1.0.4-1.el7ost.noarch


How reproducible:
always

Steps to Reproduce:
$ source <admin_creads>
openstack user create testuser --password $OS_PASSWORD
openstack project create testproject
openstack user role add --project testproject --user testuser --role swiftoperator

cat >/test.hot <<EOF
HeatTemplateFormatVersion: '2012-12-12'
Description: |
  Template which creates only a new user
Resources:
  CfnUser:
    Type: AWS::IAM::User
EOF



Actual results:
$ OS_USERNAME=testuser OS_TENANT_NAME=testproject heat stack-create --template-file test.hot test1
WARNING (shell) "heat stack-create" is deprecated, please use "openstack stack create" instead
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+----------------------+--------------+
| id                                   | stack_name | stack_status  | creation_time        | updated_time |
+--------------------------------------+------------+---------------+----------------------+--------------+
| b0fec8bc-e424-45f1-ae96-1fa7a7e5bdbf | test1      | CREATE_FAILED | 2016-10-25T12:39:15Z | None         |
+--------------------------------------+------------+---------------+----------------------+--------------+
[afazekas@seal46 infrared]$ OS_USERNAME=testuser OS_TENANT_NAME=testproject heat stack-show test1
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+------------------------------------------------------------------------------------------------------------------------------+
| Property              | Value                                                                                                                        |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------+
| capabilities          | []                                                                                                                           |
| creation_time         | 2016-10-25T12:39:15Z                                                                                                         |
| deletion_time         | None                                                                                                                         |
| description           | Template which creates only a new user                                                                                       |
| disable_rollback      | True                                                                                                                         |
| id                    | b0fec8bc-e424-45f1-ae96-1fa7a7e5bdbf                                                                                         |
| links                 | http://10.0.0.109:8004/v1/72f68abcd82b4009b4baf926f37a8537/stacks/test1/b0fec8bc-e424-45f1-ae96-1fa7a7e5bdbf (self)          |
| notification_topics   | []                                                                                                                           |
| outputs               | []                                                                                                                           |
| parameters            | {                                                                                                                            |
|                       |   "AWS::StackId": "arn:openstack:heat::72f68abcd82b4009b4baf926f37a8537:stacks/test1/b0fec8bc-e424-45f1-ae96-1fa7a7e5bdbf",  |
|                       |   "AWS::Region": "ap-southeast-1",                                                                                           |
|                       |   "AWS::StackName": "test1"                                                                                                  |
|                       | }                                                                                                                            |
| parent                | None                                                                                                                         |
| stack_name            | test1                                                                                                                        |
| stack_owner           | None                                                                                                                         |
| stack_status          | CREATE_FAILED                                                                                                                |
| stack_status_reason   | Resource CREATE failed: AuthorizationFailure:                                                                                |
|                       | resources.CfnUser: Authorization failed.                                                                                     |
| stack_user_project_id | 72f68abcd82b4009b4baf926f37a8537                                                                                             |
| tags                  | null                                                                                                                         |
| template_description  | Template which creates only a new user                                                                                       |
| timeout_mins          | None                                                                                                                         |
| updated_time          | None                                                                                                                         |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------+



Additional info:
tempest.api.orchestration.stacks.test_templates.TemplateAWSTestJSON fails with:

Traceback (most recent call last):
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/root/tempest-dir/tempest/test.py", line 241, in setUpClass
    six.reraise(etype, value, trace)
  File "/root/tempest-dir/tempest/test.py", line 234, in setUpClass
    cls.resource_setup()
  File "/root/tempest-dir/tempest/api/orchestration/stacks/test_templates.py", line 34, in resource_setup
    'CREATE_COMPLETE')
  File "/root/tempest-dir/tempest/services/orchestration/json/orchestration_client.py", line 174, in wait_for_stack_status
    stack_status_reason=body['stack_status_reason'])
tempest.exceptions.StackBuildErrorException: Stack tempest-heat-871894728/9ade44c5-d7d4-4f12-a4b6-f1629f6eb538 is in CREATE_FAILED status due to 'Resource CREATE failed: Forbidden: resources.CfnUser: You are not authorized to perform the requested action: identity:list_roles (HTTP 403) (Request-ID: req-e67a481a-22da-49ec-bbb9-d0e99e0f0682)'

Comment 1 James Slagle 2016-10-25 13:20:51 UTC
steve, can you triage this one? i suppose it might be related to heat policy somehow

Comment 4 Attila Fazekas 2016-10-31 12:01:40 UTC
The non random diff part of the heat config, comparing the old(working) with (new) not working version:

Common subdirectories: old/controller-0/etc/heat/environment.d and new/controller-0/etc/heat/environment.d
diff -u old/controller-0/etc/heat/heat.conf new/controller-0/etc/heat/heat.conf
--- old/controller-0/etc/heat/heat.conf	2016-10-19 17:43:49.000000000 +0200
+++ new/controller-0/etc/heat/heat.conf	2016-10-25 10:00:17.000000000 +0200
@@ -193,16 +193,13 @@
 # Keystone domain name which contains heat template-defined users. If
 # `stack_user_domain_id` option is set, this option is ignored. (string value)
 #stack_user_domain_name = <None>
-stack_user_domain_name = heat_stack
 
 # Keystone username, a user with roles sufficient to manage users and projects
 # in the stack_user_domain. (string value)
 #stack_domain_admin = <None>
-stack_domain_admin = heat_stack_domain_admin
 
 # Keystone password for stack_domain_admin user. (string value)
 #stack_domain_admin_password = <None>
-stack_domain_admin_password = GZQHKXp4Kus9suAx2xYGPHDtj
 
 # Maximum raw byte size of any template. (integer value)
 #max_template_size = 524288

Comment 5 Steve Baker 2016-11-01 02:48:38 UTC
Attila, this might already be fixed upstream. Can you confirm that your tripleo-heat-templates has the following changes?

https://review.openstack.org/#/c/387288/
https://review.openstack.org/#/c/387289/

I think either these changes will fix the problem, or caused the problem.

Comment 6 Attila Fazekas 2016-11-01 07:28:16 UTC
https://review.openstack.org/#/c/387288/1/manifests/profile/base/keystone.pp was in puppet-tripleo-5.3.0-1.el7ost.noarch .

https://review.openstack.org/#/c/387289/ is not in
openstack-tripleo-heat-templates-5.0.0-0.6.0rc3.el7ost.noarch

Comment 7 Steve Baker 2016-11-01 21:31:52 UTC
OK, I suggest retesting when https://review.openstack.org/#/c/387289/ is available in a puddle build.

Comment 8 Attila Fazekas 2016-11-02 08:50:10 UTC
openstack-tripleo-heat-templates                                 -5.0.0-1.1.el7ost has the 2th change, and still failing

Comment 9 Marius Cornea 2016-11-02 11:35:49 UTC
/etc/heat/heat.conf:

[DEFAULT]
heat_metadata_server_url = http://127.0.0.1:8000
heat_waitcondition_server_url = http://127.0.0.1:8000/v1/waitcondition
heat_watch_server_url =http://127.0.0.1:8003
num_engine_workers = 0
auth_encryption_key = XNK0jWPZcVwyOcprSQLjalNhneqffmC0
debug = True
log_dir = /var/log/heat
rpc_backend = rabbit
[auth_password]
[clients]
[clients_aodh]
[clients_barbican]
[clients_ceilometer]
[clients_cinder]
[clients_designate]
[clients_glance]
[clients_heat]
[clients_keystone]
auth_uri =http://192.168.0.15:35357
[clients_magnum]
[clients_manila]
[clients_mistral]
[clients_monasca]
[clients_neutron]
[clients_nova]
[clients_sahara]
[clients_senlin]
[clients_swift]
[clients_trove]
[clients_zaqar]
[cors]
[cors.subdomain]
[database]
connection = mysql+pymysql://heat:9xKnzxjKNh4y9fKKUdDym3NKH@10.0.0.16/heat
max_retries = -1
db_max_retries = -1
[ec2authtoken]
auth_uri = http://10.0.0.16:5000/v2.0/ec2tokens
[eventlet_opts]
[heat_api]
bind_host = 10.0.0.10
workers = 0
[heat_api_cfn]
bind_host = 10.0.0.10
workers = 0
[heat_api_cloudwatch]
bind_host = 10.0.0.10
workers = 0
[matchmaker_redis]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
driver =messaging
[oslo_messaging_rabbit]
rabbit_hosts = 10.0.0.23
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = Bb23X4tM8uJERZMwDe6jVYPvB
heartbeat_timeout_threshold = 60
[oslo_messaging_zmq]
[oslo_middleware]
enable_proxy_headers_parsing = True
[oslo_policy]
policy_file = /etc/heat/policy.json
[paste_deploy]
[profiler]
[revision]
[ssl]
[trustee]
auth_url = http://192.168.0.15:35357
project_domain_name = Default
username = heat
user_domain_name = Default
password = 9xKnzxjKNh4y9fKKUdDym3NKH
auth_plugin=password
[volumes]
[keystone_authtoken]
password=9xKnzxjKNh4y9fKKUdDym3NKH
project_name=service
project_domain_name=Default
auth_url=http://192.168.0.15:35357
username=heat
auth_uri=http://10.0.0.16:5000/v2.0
auth_type=password
user_domain_name=Default

openstack --os-identity-api-version 3 domain list
+----------------------------------+-------------------+---------+--------------------+
| ID                               | Name              | Enabled | Description        |
+----------------------------------+-------------------+---------+--------------------+
| d93a131125714434839803aaac91d30b | <SERVICE DEFAULT> | True    |                    |
| default                          | Default           | True    | The default domain |
+----------------------------------+-------------------+---------+--------------------+

openstack --os-identity-api-version 3 user list --domain Default
+----------------------------------+------------+
| ID                               | Name       |
+----------------------------------+------------+
| ca9dae39524c49c980bc6b4df575cbdb | admin      |
| 4f85409148584adb8cb75e8af98f15a3 | neutron    |
| c90dbbac41c54f1bbb0ad60e0e5d15c7 | heat       |
| 06a7be6314c04c0b8efab24ba92e62f2 | gnocchi    |
| 729de663581344e7a5019d02600c583a | aodh       |
| 7840a28097234c7680b43d3140f64026 | nova       |
| b1d329a361854bfa9d6a8c6e8d0b41c1 | glance     |
| 7e7652da59094229af9bfff012fc0614 | ceilometer |
| c06da5359f9a45c690f5242a25b099a3 | cinder     |
| 481f7a9badb84f9dab6c685b7550e7ea | heat-cfn   |
| dab1c30bb75b437ab3c892109d212eb4 | swift      |
| 712ba1fdf79a465280a73d2344be6e48 | testuser   |
+----------------------------------+------------+

Comment 10 Steven Hardy 2016-11-02 13:39:51 UTC
Ok so I've reproduced and the problem appears to be puppet is incorrectly ensuring the domain does *not* exist:

    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]/ensure: removed^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]/ensure: removed^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_user[<SERVICE DEFAULT>::<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[<SERVICE DEFAULT>::<SERVICE DEFAULT>@::<SERVICE DEFAULT>]/ensure: created^[[0m
    ^[[mNotice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]/ensure: removed^[[0m

Currently trying to figure out why this is happening

Comment 11 Steven Hardy 2016-11-02 15:09:05 UTC
I tested https://review.openstack.org/#/c/392288 and it appears to fix this

Comment 15 Marius Cornea 2016-11-10 17:19:52 UTC
[stack@undercloud-0 ~]$ openstack stack show test1
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                    |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+
| id                    | ebfaa0f8-5125-4e97-a2bf-01d3efdeb220                                                                                     |
| stack_name            | test1                                                                                                                    |
| description           | Template which creates only a new user                                                                                   |
|                       |                                                                                                                          |
| creation_time         | 2016-11-10T17:18:57Z                                                                                                     |
| updated_time          | None                                                                                                                     |
| stack_status          | CREATE_COMPLETE                                                                                                          |
| stack_status_reason   | Stack CREATE completed successfully                                                                                      |
| parameters            | AWS::Region: ap-southeast-1                                                                                              |
|                       | AWS::StackId: arn:openstack:heat::fb267f9c3d11474abb150483e0583c95:stacks/test1/ebfaa0f8-5125-4e97-a2bf-01d3efdeb220     |
|                       | AWS::StackName: test1                                                                                                    |
|                       |                                                                                                                          |
| outputs               | []                                                                                                                       |
|                       |                                                                                                                          |
| links                 | - href: https://172.16.18.32:13004/v1/fb267f9c3d11474abb150483e0583c95/stacks/test1/ebfaa0f8-5125-4e97-a2bf-01d3efdeb220 |
|                       |   rel: self                                                                                                              |
|                       |                                                                                                                          |
| parent                | None                                                                                                                     |
| disable_rollback      | True                                                                                                                     |
| deletion_time         | None                                                                                                                     |
| stack_user_project_id | b0ba2225c275442d8a6c3e171d599d9d                                                                                         |
| capabilities          | []                                                                                                                       |
| notification_topics   | []                                                                                                                       |
| stack_owner           | None                                                                                                                     |
| timeout_mins          | None                                                                                                                     |
| tags                  | null                                                                                                                     |
|                       | ...                                                                                                                      |
|                       |                                                                                                                          |
+-----------------------+--------------------------------------------------------------------------------------------------------------------------+

Comment 17 errata-xmlrpc 2016-12-14 16:25:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.