Description of problem: OpenStack Keystone API v3 supports domains as a high level container for projects. CloudForms should support this new feature, starting with a new "Domain" field in the openstack provider configuration in the GUI, and modifying all the methods related to openstack to login with the domain+user+password combination and the keystone V3 endpoint. Version-Release number of selected component (if applicable): 3.2 BETA How reproducible: Try to add openstack as provider when keystone V3 api is enabled and domains are being used (i.e.- the cloudforms user for openstack belongs to a non default domain) Steps to Reproduce: 1. OpenStack installation with keystone v3 api 2. OpenStack cloudform user belonging to a non default domain 3. Try to fill the connection details in CF (domain + user + password) Actual results: Domain field is missing Expected results: Domain field is enabled Additional info: Fog ruby gem now supports keystone domains https://github.com/fog/fog/issues/3084 Upstream forum post http://talk.manageiq.org/t/support-for-keystone-v3-domains/663 Keystone domains explained http://www.madorn.com/keystone-v3-api.html
Support of keystone v3 is here https://github.com/ManageIQ/manageiq/pull/3850 But only with 'cloud_admin' domain, for multi domain support, we will need to do some architectural changes. Since one user cannot do operations across multiple domains.
https://github.com/ManageIQ/manageiq/pull/8177
https://github.com/ManageIQ/manageiq/pull/8180
From my testing it appears, that with the default keystone policy.json, the user added to ManageIQ should be an admin of the domain, so he can list projects inside. Could we add this to the Docs for Keytone V3? The OpenStack CLI query for adding Admin rights to Domain looks like: openstack --os-identity-api-version 3 --os-auth-url http://localhost:5000/v3 --os-username cloud_admin --os-password <password> --os-domain-name admin_domain --os-user-domain-name admin_domain role add --domain Default --user admin admin example shows adding admin rights to user 'admin' on the domain 'Default'. User required for this operation is the admin of the all domains.
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=059185a1f2f085eda22c0296b6a3d46983ce6ea7 commit 059185a1f2f085eda22c0296b6a3d46983ce6ea7 Author: Ladislav Smola <lsmola> AuthorDate: Fri Apr 22 10:29:49 2016 +0200 Commit: Ladislav Smola <lsmola> CommitDate: Fri Apr 22 13:15:03 2016 +0200 OpenStack make Keystone V3 domain ID configurable OpenStack make Keystone V3 domain ID configurable Fixes GH issue: https://github.com/ManageIQ/manageiq/issues/7342 Fixes BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1228542 Conflicts: app/assets/javascripts/controllers/ems_common/ems_common_form_controller.js app/controllers/mixins/ems_common_angular.rb gems/pending/openstack/openstack_handle/identity_delegate.rb .../ems_common/ems_common_form_controller.js | 5 ++- app/controllers/ems_cloud_controller.rb | 48 ++++++++++++++-------- app/helpers/ems_cloud_helper/textual_summary.rb | 8 +++- .../manageiq/providers/openstack/manager_mixin.rb | 2 + .../views/ems_common/angular/_form.html.haml | 11 +++++ gems/pending/openstack/openstack_handle/handle.rb | 10 +++-- .../openstack_handle/identity_delegate.rb | 4 +- 7 files changed, 64 insertions(+), 24 deletions(-)
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/b1bc248698d578183a5a2b9abf82bac550bd3ec6 commit b1bc248698d578183a5a2b9abf82bac550bd3ec6 Author: Ladislav Smola <lsmola> AuthorDate: Fri Apr 22 10:29:49 2016 +0200 Commit: Ladislav Smola <lsmola> CommitDate: Fri Apr 22 12:45:24 2016 +0200 OpenStack make Keystone V3 domain ID configurable OpenStack make Keystone V3 domain ID configurable Fixes GH issue: https://github.com/ManageIQ/manageiq/issues/7342 Fixes BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1228542 .../ems_common/ems_common_form_controller.js | 2 + app/controllers/mixins/ems_common_angular.rb | 50 ++++++++++++++-------- app/helpers/ems_cloud_helper/textual_summary.rb | 8 +++- .../manageiq/providers/openstack/manager_mixin.rb | 2 + .../views/ems_common/angular/_form.html.haml | 11 +++++ gems/pending/openstack/openstack_handle/handle.rb | 10 +++-- .../openstack_handle/identity_delegate.rb | 4 +- 7 files changed, 63 insertions(+), 24 deletions(-)
https://github.com/ManageIQ/manageiq/pull/8473
I created a domain "dom1" and a user "dom1admin". I gave the user the admin role on the domain and checked that this user can successfully list the projects in there. I also created a project, and registered new service endpoints for identityv3 (and as per comment #6 in bz 1329635 I also deleted the endpoints for v2). The bug fails, I can't register the cloud provider using the credentials of this user. I get an error: "Credential validation was not successful: Login failed due to a bad username or password."
Currently we also require these roles on each project: admin, heat_stack_owner, SwiftOperator, there is a RFE to ignore projects, where we do not have a role.
Udi, as commented in https://bugzilla.redhat.com/show_bug.cgi?id=1329635, comment #9, we were able to test this with mcornea. 2 more bugs were identified: https://bugzilla.redhat.com/show_bug.cgi?id=1341550 and https://bugzilla.redhat.com/show_bug.cgi?id=1340874
Some testing steps that I ran, tested on 5.6.0.10-rc2.1-nightly.20160613100711_27d575b with 1 ctrl + 1 compute overcloud, osp8. I wasn't able to validate the credentials when registering the cloud provider in Cloudforms Step 1: This script will create the admin domain and cloud admin user then create dom1 domain with adm1 user as admin for dom1, password 'password'. It then adjusts the overcloud controller policy.json file and restart keystone service. Run it from undercloud as the stack user: https://gist.github.com/mcornea/68fa6b75dfc6c7f870c8a2d936752f9a Step 2: Create a credentials file for adm1 user: cat adm1_rc export OS_USERNAME=adm1 export OS_USER_DOMAIN_NAME=dom1 export OS_PASSWORD=password export OS_AUTH_URL=http://172.16.18.25:5000/v3 Step 3: Source the adm1_rc file and create a project; note the ID_DOM1 resulted from step 1 source adm1_rc openstack --os-identity-api-version 3 --os-default-domain $ID_DOM1 --os-domain-name dom1 project create --domain $ID_DOM1 --description "Project 01" project01 +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | Project 01 | | domain_id | 68a3465e207644e88123bc84ebc354c7 | | enabled | True | | id | 1957ff7fcdb74f568e122b06735ed976 | | is_domain | False | | name | project01 | | parent_id | None | +-------------+----------------------------------+ Step 4: Verify project01 was created: source adm1_rc openstack --os-identity-api-version 3 --os-default-domain $ID_DOM1 --os-domain-name dom1 project list --domain $ID_DOM1 +----------------------------------+-----------+ | ID | Name | +----------------------------------+-----------+ | 1957ff7fcdb74f568e122b06735ed976 | project01 | +----------------------------------+-----------+ At this point I'm trying to register the cloud provider in Cloudforms but credentials validation fails. Log on Cloudforms side: INFO -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#with_provider_connection) Connecting through ManageIQ::Providers::Openstack::CloudManager: [overcloud] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#verify_api_credentials) Error Class=MiqException::ServiceNotAvailable, Message=MiqException::ServiceNotAvailable WARN -- : MIQ(ManageIQ::Providers::Openstack::CloudManager#authentication_check_no_validation) type: ["default"] for [] [overcloud] Validation failed: invalid, Login failed due to a bad username or password. ERROR -- : MIQ(ems_cloud_controller-create): Credential validation was not successful: Login failed due to a bad username or password. Log on Keystone side: 2016-06-14 13:28:21.096 26939 DEBUG keystone.middleware.core [req-96688895-301c-4165-8baa-ed477c4ef013 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2016-06-14 13:28:21.100 26939 INFO keystone.common.wsgi [req-96688895-301c-4165-8baa-ed477c4ef013 - - - - -] POST http://172.16.18.25:5000/v3/auth/tokens 2016-06-14 13:28:21.240 26939 INFO eventlet.wsgi.server [req-96688895-301c-4165-8baa-ed477c4ef013 - - - - -] 10.0.0.12 - - [14/Jun/2016 13:28:21] "POST /v3/auth/tokens HTTP/1.1" 201 3585 0.144329 2016-06-14 13:28:21.267 26938 DEBUG keystone.middleware.core [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'd47da9fd281945378cf47b5ee8ae74dd', 'roles': [u'admin'], 'trustee_id': None, 'domain_name': u'dom1', 'trustor_id': None, 'token': <KeystoneToken (audit_id=b47qioFwQv6GJh42dfDuVg, audit_chain_id=b47qioFwQv6GJh42dfDuVg) at 0x6f5a500>, 'consumer_id': None, 'domain_id': u'68a3465e207644e88123bc84ebc354c7', 'trust_id': None} process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:314 2016-06-14 13:28:21.270 26938 INFO keystone.common.wsgi [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] GET http://172.16.18.25:5000/v3/users/d47da9fd281945378cf47b5ee8ae74dd/projects 2016-06-14 13:28:21.271 26938 DEBUG keystone.common.controller [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] RBAC: Authorizing identity:list_user_projects(user_id=d47da9fd281945378cf47b5ee8ae74dd) _build_policy_check_credentials /usr/lib/python2.7/site-packages/keystone/common/controller.py:62 2016-06-14 13:28:21.271 26938 DEBUG keystone.common.controller [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/site-packages/keystone/common/controller.py:67 2016-06-14 13:28:21.272 26938 DEBUG keystone.common.controller [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] RBAC: Adding query filter params () wrapper /usr/lib/python2.7/site-packages/keystone/common/controller.py:194 2016-06-14 13:28:21.272 26938 DEBUG keystone.policy.backends.rules [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] enforce identity:list_user_projects: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'd47da9fd281945378cf47b5ee8ae74dd', 'roles': [u'admin'], 'trustee_id': None, 'domain_name': u'dom1', 'trustor_id': None, 'token': <KeystoneToken (audit_id=b47qioFwQv6GJh42dfDuVg, audit_chain_id=b47qioFwQv6GJh42dfDuVg) at 0x6f5a500>, 'consumer_id': None, 'domain_id': u'68a3465e207644e88123bc84ebc354c7', 'trust_id': None} enforce /usr/lib/python2.7/site-packages/keystone/policy/backends/rules.py:76 2016-06-14 13:28:21.273 26938 DEBUG keystone.common.controller [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] RBAC: Authorization granted wrapper /usr/lib/python2.7/site-packages/keystone/common/controller.py:204 2016-06-14 13:28:21.286 26938 INFO eventlet.wsgi.server [req-8fb41907-ef6b-4a7e-af94-6c6c018a6dd6 - - - - -] 10.0.0.12 - - [14/Jun/2016 13:28:21] "GET /v3/users/d47da9fd281945378cf47b5ee8ae74dd/projects HTTP/1.1" 200 341 0.035778 2016-06-14 13:28:21.291 26940 DEBUG keystone.middleware.core [req-1ca0559b-8be6-427e-9771-070f8d89b6d8 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 2016-06-14 13:28:21.348 26940 INFO keystone.common.wsgi [req-1ca0559b-8be6-427e-9771-070f8d89b6d8 - - - - -] POST http://172.16.18.25:5000/v3/auth/tokens 2016-06-14 13:28:21.422 26940 INFO eventlet.wsgi.server [req-1ca0559b-8be6-427e-9771-070f8d89b6d8 - - - - -] 10.0.0.12 - - [14/Jun/2016 13:28:21] "POST /v3/auth/tokens HTTP/1.1" 201 3620 0.131509
clearing the needinfo
Update: I managed to get the credentials validation work after adding the Keystone v3 endpoints. So let's add a Step 0: source overcloudrc keystone service-create --name keystonev3 --type identityv3 --description "Keystone Identity Service v3" keystone endpoint-create --region regionOne --service keystonev3 --publicurl "http://172.16.18.25:5000/v3" --adminurl "http://192.168.0.17:35357/v3" --internalurl "http://10.0.0.10:5000/v3" but refresh still fails with: MiqException::ServiceNotAvailable
Going further for refresh, I had to manually assign the admin role to the domain admin for the project it has created: openstack --os-identity-api-version 3 --os-auth-url http://172.16.18.25:5000/v3 --os-username cloud_admin --os-password password --os-domain-name admin_domain --os-user-domain-name admin_domain role add --project project01 --user adm1 admin Now refresh fails due to the Cinder quota issue described in BZ#21341550: [----] E, [2016-06-14T11:26:57.307599 #2931:b57994] ERROR -- : [Excon::Errors::InternalServerError]: Expected(200) <=> Actual(500 InternalServerError) excon.error.response :body => "{\"computeFault\": {\"message\": \"The server has either erred or is incapable of performing the requested operation.\", \"code\": 500}}" :cookies => [ ] :headers => { "Content-Length" => "128" "Content-Type" => "application/json; charset=UTF-8" "Date" => "Tue, 14 Jun 2016 15:26:57 GMT" "X-Compute-Request-Id" => "req-53869fe5-8d8d-49c3-98f4-d96a59c3ab09" "X-Openstack-Request-Id" => "req-53869fe5-8d8d-49c3-98f4-d96a59c3ab09" } :host => "172.16.18.25" :local_address => "172.16.17.148" :local_port => 35321 :path => "/v2/ddb3e479d7094fa793059c1423b77291//os-quota-sets/ddb3e479d7094fa793059c1423b77291" :port => 8776 :reason_phrase => "Internal Server Error" :remote_ip => "172.16.18.25" :status => 500 :status_line => "HTTP/1.1 500 Internal Server Error\r\n"
FWIW, to workaround the Cinder quota issue I had to edit on the controller node, /etc/cinder/cinder.conf, keystone_authtoken section: auth_uri = http://10.0.0.11:5000/v2.0 to auth_uri = http://10.0.0.11:5000/v3
The dependent bugs were finished, this should be ready.
Created attachment 1217722 [details] last 2000 lines of evm.log
Failed to create V3 domain Credential validation was not successful: Unexpected response returned from system: MiqExce ption::ServiceNotAvailable Domain details below: curl -H "x-auth-token:7c62a57efc494f4e8a1b7f0f0daee5ab" http://10.0.0.107:5000/v3/users?domain_id=a7a7789958f54266b3bc5b1082006a41 |python -mjson.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 429 100 429 0 0 9761 0 --:--:-- --:--:-- --:--:-- 9976 { "links": { "next": null, "previous": null, "self": "http://10.0.0.107:5000/v3/users?domain_id=a7a7789958f54266b3bc5b1082006a41" }, "users": [ { "description": "Administrator of domain dom1", "domain_id": "a7a7789958f54266b3bc5b1082006a41", "enabled": true, "id": "c78f46d850ea486b98e091e1433ebd51", "links": { "self": "http://10.0.0.107:5000/v3/users/c78f46d850ea486b98e091e1433ebd51" }, "name": "adm1", "password_expires_at": null } ] }
Ronnie this BZ was mainly for refresh, which seems to work, the error: [----] E, [2016-11-06T07:27:50.799299 #2738:91b144] ERROR -- : MIQ(ManageIQ::Providers::Openstack::NetworkManager#event_monitor_available?) Exception trying to find openstack event monitor for uc_ci Network Manager(192.0.2.1). MiqException::ServiceNotAvailable is for event monitor. Please do fill another BZs for event monitor and metrics collection. Also we need a OpenStack version stated in the BZ.
Created attachment 1218931 [details] evm log 5000 lines
@ladas for a proper dV3 domain creation Credential validation was not successful: Unexpected response returned from system: MiqException::ServiceNotAvailable
Your OS setup is missing Ceilometer?
@ Ladas I did not configure the Ceilometer/Telemetry in my setup. Does it make any difference adding new cloud provider with keystone V3?
Yes the error seems to be caused by missing Ceilometer, seems like it's needed when you validate with Ceilometer eventing. It doesn't seem to be related to keystone v3 at all.
Verified on 5.7.0.11 RC1
Doc text revised.
I still have this issue, using the latest release. Can someone help how to fix it? And I do have celiometer running as well. But is there any specific configuration for it? With v2, I 'm able to authorize but the instances and images are not getting listed.