Bug 1129760

Summary: F20 juno install fails w/ duplicate user errors
Product: [Community] RDO Reporter: wes hayutin <whayutin>
Component: openstack-puppet-modulesAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED CURRENTRELEASE QA Contact: Ami Jeain <ajeain>
Severity: urgent Docs Contact:
Priority: medium    
Version: unspecifiedCC: aortega, apevec, apevec, lbezdick, lhh, mmagr, rmeggins, yeylon
Target Milestone: ---   
Target Release: Juno   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-puppet-modules-2014.2-0.3.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-28 23:07:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1134328    
Bug Blocks:    
Attachments:
Description Flags
log of keystone.pp.log, see errors
none
attempt to clean up users prior to rerun
none
another attempt to clean up prior to rerun none

Description wes hayutin 2014-08-13 15:28:12 UTC 
Created attachment 926484 [details]
log of keystone.pp.log, see errors

Description of problem:

Example of one of the errors.

Error: /Stage[main]/Neutron::Keystone::Auth/Keystone_user[neutron]/ensure: change from absent to present failed: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass redhat --tenant_id 13bc386edf8d48ea8eb37d4c1847ff9e' returned 1: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)

Steps to recreate:
1. provision a f20 machine, yum -y update
2. setup juno stage repo: http://team.virt.bos.redhat.com/openstack/openstack-juno/fedora-20/
3. packstack --allinone


Attempted to drop the mysql db and removing the users and rerunning the install after an initial failure.  It looks like the users are installed once and then attempted once more in the keystone.pp.  I need to investigate a bit more on the puppet side of things.

See the three log files for details.
Comment 1 wes hayutin 2014-08-13 15:28:48 UTC 
Created attachment 926485 [details]
attempt to clean up users prior to rerun
Comment 2 wes hayutin 2014-08-13 15:29:23 UTC 
Created attachment 926486 [details]
another attempt to clean up prior to rerun
Comment 3 wes hayutin 2014-08-13 15:30:10 UTC 
[fedora@westest-junof20 ~]$ rpm -qa | grep packstack
openstack-packstack-2014.1.1-0.26.dev1220.fc21.noarch
openstack-packstack-puppet-2014.1.1-0.26.dev1220.fc21.noarch

[fedora@westest-junof20 ~]$ rpm -qa | grep puppet
openstack-puppet-modules-2014.1-19.3.fc22.noarch
puppet-3.4.3-3.fc20.noarch
openstack-packstack-puppet-2014.1.1-0.26.dev1220.fc21.noarch
Comment 4 Alan Pevec 2014-08-14 22:24:44 UTC 
FWIW same error on rhel7+epel7, puppet-3.6.2-2.el7.noarch same packstack and o-p-m version

BTW puppet actions are supposed to be idempotent so puppet-keystone should be fixed to handle this situation, whatever the actual root cause might be
Comment 5 Martin Magr 2014-08-18 14:44:34 UTC 
From puppet-keystone point of view it is doing precisely what is supposed to do, eg.:
1. check if user exists
2. if does not exist, create it

Below is part of Puppet debug log enhanced with additional debug messages: 
"""
Debug: Prefetching keystone resources for keystone_user
>>>list_keystone, user, 4, []
* running keystone --os-endpoint http://127.0.0.1:35357/v2.0/ ["user-list", []]
Debug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-list'
* output:

>>>>>>[]
>>>list_keystone, tenant, 3, []
* running keystone --os-endpoint http://127.0.0.1:35357/v2.0/ ["tenant-list", []]
Debug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-list'
* output:
+----------------------------------+----------+---------+
|                id                |   name   | enabled |
+----------------------------------+----------+---------+
| e1edfc2c32d144208dc9e26aae294553 |  admin   |   True  |
| 5205b29537b44d2d87708dd6142e318b | services |   True  |
+----------------------------------+----------+---------+
>>>>>>["e1edfc2c32d144208dc9e26aae294553", "admin", "True"]
>>>>>>["5205b29537b44d2d87708dd6142e318b", "services", "True"]
>>>>>>[["e1edfc2c32d144208dc9e26aae294553", "admin", "True"], ["5205b29537b44d2d87708dd6142e318b", "services", "True"]]
* running keystone --os-endpoint http://127.0.0.1:35357/v2.0/ [["user-create", "--name", "neutron", "--enabled", "True", ["--email", "neutron@localhost", "--pass", "1b22eb2b8ff14126", "--tenant_id", "5205b29537b44d2d87708dd6142e318b"]]]
Debug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass 1b22eb2b8ff14126 --tenant_id 5205b29537b44d2d87708dd6142e318b'
!!!Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass 1b22eb2b8ff14126 --tenant_id 5205b29537b44d2d87708dd6142e318b' returned 1: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
Error: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass 1b22eb2b8ff14126 --tenant_id 5205b29537b44d2d87708dd6142e318b' returned 1: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)

Error: /Stage[main]/Neutron::Keystone::Auth/Keystone_user[neutron]/ensure: change from absent to present failed: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass 1b22eb2b8ff14126 --tenant_id 5205b29537b44d2d87708dd6142e318b' returned 1: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
"""

When I run user-list after installation fails, all users are in DB:
[root@localhost modules]# keystone --os-endpoint http://127.0.0.1:35357/v2.0/ --os-token 1029522011d54996ad57763be39e1918 user-list
+----------------------------------+------------+---------+----------------------+
|                id                |    name    | enabled |        email         |
+----------------------------------+------------+---------+----------------------+
| 9a82dec5da3a4a1bbc5bfcd4b96a2c1b |   admin    |   True  |    test     |
| 4148c2f1b1cd4c80881d42de3c7351a2 | ceilometer |   True  | ceilometer@localhost |
| 2ee7aeacd5404ce69dc3b6f84d0d761f |   cinder   |   True  |   cinder@localhost   |
| c747df68edb843009dd46ecbec50ff2b |   glance   |   True  |   glance@localhost   |
| d7b1d47a10c2486c9ea8c196ca65c9d6 |  neutron   |   True  |  neutron@localhost   |
| 7ad512f330284db19d14c44118dfd9ce |    nova    |   True  |    nova@localhost    |
| a2c8fc2d10634e888d9e7654303bc63a |   swift    |   True  |   swift@localhost    |
+----------------------------------+------------+---------+----------------------+

So it seems like users are created parallel twice. I'm quite confused because same packstack and OPM works for Icehouse packages. Will have to further investigate to find cause of duplicity...
Comment 6 Martin Magr 2014-08-21 13:06:29 UTC 
Setting correct priority.
Comment 7 wes hayutin 2014-08-22 14:32:03 UTC 
Any updates on a resolution?
Comment 8 Lukas Bezdicka 2014-08-22 15:07:44 UTC 
Workaround that gets you a bit further is to remove _member_ role creation from puppet keystone /usr/share/openstack-puppet/modules/keystone/manifests/roles/admin.pp. The issue there is that it gets now created with random id instead of id specified in member_role_id config option. This role is than checked when adding users and recreated automatically with correct id which fails because it exists already. I don't know the fix yet and so far this got me to broken glance and nova.
Comment 9 Lukas Bezdicka 2014-08-25 07:30:37 UTC 
Calls from puppet:
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ role-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ role-create --name _member_'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name neutron --type network --description Neutron Networking Service'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name ceilometer --type metering --description Openstack Metering Service'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-create --name admin --enabled True --description admin tenant'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ role-create --name ResellerAdmin'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name nova_ec2 --type ec2 --description EC2 Service'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name cinderv2 --type volumev2 --description Cinder Service v2'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ role-create --name SwiftOperator'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ endpoint-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ service-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ endpoint-create --service-id ed7aec9a4d0442a1abe48ad7a94292ea --publicurl http://192.168.122.133:8777 --internalurl http://192.168.122.133:8777 --adminurl http://192.168.122.133:8777 --region RegionOne'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-create --name services --enabled True --description Tenant for the openstack services'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-list'[0m
[0;36mDebug: Executing '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass c974265af37e42a0 --tenant_id e12fcdff59b7476198a722466fb68033'[0m
[1;31mError: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost --pass c974265af37e42a0 --tenant_id e12fcdff59b7476198a722466fb68033' returned 1: Conflict occurred attempting to store role - Duplicate Entry (HTTP 409)
[1;31mError: /Stage[main]/Neutron::Keystone::Auth/Keystone_user[neutron]/ensure: change from absent to present failed: Execution of '/usr/bin/keystone --os-endpoint http://127.0.0.1:35357/v2.0/ user-create --name neutron --enabled True --email neutron@localhost 

What happens in mysql:
140821  7:22:48	  501 Connect	keystone_admin.122.133 as anonymous on keystone
		  501 Query	set autocommit=0
		  501 Query	SELECT DATABASE()
		  501 Query	SELECT @@tx_isolation
		  501 Query	SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1
		  501 Query	SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1
		  501 Query	rollback
		  501 Query	SHOW VARIABLES LIKE 'sql_mode'
		  501 Query	SET SESSION sql_mode = 'TRADITIONAL'
		  501 Query	select 1
		  501 Query	SHOW VARIABLES LIKE 'sql_mode'
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT role.id AS role_id, role.name AS role_name, role.extra AS role_extra 
FROM role
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO role (id, name, extra) VALUES ('08111af684c74d289cc0eacc03f418e6', '_member_', '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service
		  501 Query	rollback
140821  7:22:49	  501 Query	select 1
		  501 Query	INSERT INTO service (id, type, enabled, extra) VALUES ('fec7d3835c784d6585f0e72ddff72a31', 'network', 1, '{\"name\": \"neutron\", \"description\": \"Neutron Networking Service\"}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO service (id, type, enabled, extra) VALUES ('326fa9e372dd4833868aef8b6b46d38e', 'metering', 1, '{\"name\": \"ceilometer\", \"description\": \"Openstack Metering Service\"}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT domain.id AS domain_id, domain.name AS domain_name, domain.enabled AS domain_enabled, domain.extra AS domain_extra 
FROM domain 
WHERE domain.id = 'default'
		  501 Query	SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra 
FROM project 
WHERE project.domain_id = 'default'
		  501 Query	commit
		  501 Query	rollback
140821  7:22:50	  501 Query	select 1
		  501 Query	INSERT INTO project (id, name, domain_id, description, enabled, extra) VALUES ('3adfa10908ab4819990698cbce2f6bd0', 'admin', 'default', 'admin tenant', 1, '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO role (id, name, extra) VALUES ('d05b059e63bc4868a200b2595a8c9059', 'ResellerAdmin', '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO service (id, type, enabled, extra) VALUES ('4d50c57fe407427b91ab5a9765851ecb', 'ec2', 1, '{\"name\": \"nova_ec2\", \"description\": \"EC2 Service\"}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO service (id, type, enabled, extra) VALUES ('6fa8f7b2343f4e9ea9c1d92a24b0177f', 'volumev2', 1, '{\"name\": \"cinderv2\", \"description\": \"Cinder Service v2\"}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO role (id, name, extra) VALUES ('f047b28e90c34db1a09da20d7d04d71f', 'SwiftOperator', '{}')
		  501 Query	commit
		  501 Query	rollback
140821  7:22:51	  501 Query	select 1
		  501 Query	SELECT endpoint.id AS endpoint_id, endpoint.legacy_endpoint_id AS endpoint_legacy_endpoint_id, endpoint.interface AS endpoint_interface, endpoint.region AS endpoint_region, endpoint.service_id AS endpoint_service_id, endpoint.url AS endpoint_url, endpoint.enabled AS endpoint_enabled, endpoint.extra AS endpoint_extra 
FROM endpoint
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service 
WHERE service.id = '326fa9e372dd4833868aef8b6b46d38e'
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service 
WHERE service.id = '326fa9e372dd4833868aef8b6b46d38e'
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO endpoint (id, legacy_endpoint_id, interface, region, service_id, url, enabled, extra) VALUES ('922ced8b5569454fa519757db5c484fc', 'e5321ebd7aa84f3ea1143a4626b95fb0', 'admin', 'RegionOne', '326fa9e372dd4833868aef8b6b46d38e', 'http://192.168.122.133:8777', 1, '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service 
WHERE service.id = '326fa9e372dd4833868aef8b6b46d38e'
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO endpoint (id, legacy_endpoint_id, interface, region, service_id, url, enabled, extra) VALUES ('ccfeaca0f3544b429c96c516918d8096', 'e5321ebd7aa84f3ea1143a4626b95fb0', 'internal', 'RegionOne', '326fa9e372dd4833868aef8b6b46d38e', 'http://192.168.122.133:8777', 1, '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT service.id AS service_id, service.type AS service_type, service.enabled AS service_enabled, service.extra AS service_extra 
FROM service 
WHERE service.id = '326fa9e372dd4833868aef8b6b46d38e'
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO endpoint (id, legacy_endpoint_id, interface, region, service_id, url, enabled, extra) VALUES ('627b91ca8954430bb4173ed955dbdf2e', 'e5321ebd7aa84f3ea1143a4626b95fb0', 'public', 'RegionOne', '326fa9e372dd4833868aef8b6b46d38e', 'http://192.168.122.133:8777', 1, '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO project (id, name, domain_id, description, enabled, extra) VALUES ('93b2181100ef46dbb4a270576a74c317', 'services', 'default', 'Tenant for the openstack services', 1, '{}')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id 
FROM user 
WHERE user.domain_id = 'default'
		  501 Query	rollback
140821  7:22:52	  501 Query	select 1
		  501 Query	SELECT domain.id AS domain_id, domain.name AS domain_name, domain.enabled AS domain_enabled, domain.extra AS domain_extra 
FROM domain 
WHERE domain.id = 'default'
		  501 Query	SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra 
FROM project 
WHERE project.domain_id = 'default'
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra 
FROM project 
WHERE project.id = '93b2181100ef46dbb4a270576a74c317'
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT domain.id AS domain_id, domain.name AS domain_name, domain.enabled AS domain_enabled, domain.extra AS domain_extra 
FROM domain 
WHERE domain.id = 'default'
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO user (id, name, domain_id, password, enabled, extra, default_project_id) VALUES ('20ea72a83dd24a40bb1414d187fa1c68', 'neutron', 'default', '$6$rounds=40000$Lwj327XLLA1eXabn$ESFtYQ2Fj7CIDaek6HkXYV89AjculJ4FYptSAJkbRwysQMgNwDbkBimnJfEAV0XH3gAiMgALwqnmJ129oQMuc.', 1, '{\"email\": \"neutron@localhost\"}', '93b2181100ef46dbb4a270576a74c317')
		  501 Query	commit
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra 
FROM project 
WHERE project.id = '93b2181100ef46dbb4a270576a74c317'
		  501 Query	SELECT role.id AS role_id, role.name AS role_name, role.extra AS role_extra 
FROM role 
WHERE role.id = '9fe2ff9ee4384b1894a90878d3e92bab'
		  501 Query	rollback
		  501 Query	rollback
		  501 Query	select 1
		  501 Query	INSERT INTO role (id, name, extra) VALUES ('9fe2ff9ee4384b1894a90878d3e92bab', '_member_', '{}')
		  501 Query	rollback
		  501 Query	rollback
		  501 Query	select 1
Comment 10 Lukas Bezdicka 2014-08-26 12:47:18 UTC 
The https://review.openstack.org/#/c/116856/ should allow us to switch to master branches of puppet-{keystone,nova,neutron,...} modules.
Comment 11 Alan Pevec 2014-09-01 13:54:37 UTC 
https://review.openstack.org/118155 required patch for puppet-neutron in openstack-puppet-modules package
Comment 12 Lukas Bezdicka 2014-09-12 08:10:02 UTC 
New o-p-m package should contain all the fixes needed.